diff --git a/PHASE0_PLAN.md b/PHASE0_PLAN.md index a2cf228..0907fff 100644 --- a/PHASE0_PLAN.md +++ b/PHASE0_PLAN.md @@ -77,7 +77,7 @@ Every Phase 0 component maps to a source-RFP requirement or a NIST control. Tabl | **0.4** | RBAC seed loaded | ✅ **Complete 2026-09-30.** 8 roles seeded from `seed/roles.py`. Permissions matrix at `seed/permissions_matrix.json` — role-first grants across 30 entities × 8 actions. `python -m ypms.seed.run` wired into docker-compose. Loader validates every cell against declared actions/entities and refuses unknown roles at seed time. 53 tests, 92.20% coverage; end-to-end `can()` decisions verified per role including admin bypass. | | **0.5** | Registrant vertical slice | ✅ **Complete 2026-09-30.** `Registrant` model + schemas + router (`src/ypms/youth/`) + migration `0003_registrants`. SSN AES-256-GCM encrypted at the app layer (`security/column_crypto.py`), write-only on the model, serializer emits only `XXX-XX-1234`. `pii_read` audit on GET, list, and history; every state change emits an audit row with ciphertext redacted via new `__audit_redact__` framework hook. Portal-only roles restricted to own row (Phase 0 minimum; PARENT_USER blocked until Guardian). 98 tests, 94.83% coverage. Coverage doc at `docs/coverage/registrant.md`. | | **0.6** | 501-stub routers for remaining entities | ✅ **Complete 2026-09-30.** 26 stub entities registered in `src/ypms/stubs.py`, each with a router from `framework/stub_router.py` exposing one route per matrix action (8). Every route runs the real RBAC check and writes an audit row before returning 501. `tests/test_stubs.py` sweeps entity × role × route against the matrix JSON (501 + one audit row when granted; 403 + none when not). Registrant restore moved to the matrix `Restore` action. 138 tests, 95.05% coverage. | -| **0.7** | Retention scaffold | `retention_policy` attribute on every entity model (even 501-stub entities have a model). Nightly job in `src/ypms/framework/retention.py` runs, logs "0 records to purge" — no-op but wired. | +| **0.7** | Retention scaffold | ✅ **Complete 2026-10-01.** Every mapped table resolves a `RetentionPolicy` (entity models declare `retention_policy`; vendored framework tables mapped by name). PIA §9 interim periods encoded in `src/ypms/retention_classes.py`, all unapproved pending `YPMS-POAM-054`; classes without a PIA basis report as `unset`. Report-only job in `src/ypms/framework/retention.py` (`python -m ypms.framework.retention`) logs "0 records to purge" and writes a `retention_run` audit row. Stub entity models + migration `0004_stub_entity_tables`. Migration-vs-model drift test added. 156 tests, 95.91% coverage. | | **0.8** | CI pipeline | GitHub Actions workflow with lint (ruff), types (mypy/pyright — match OWC's pick), tests, bandit, pip-audit, gitleaks, SBOM. Green build on `main`. | | **0.9** | Docs + acceptance | `README.md` with runnable quickstart; `CONTRIBUTING.md` with hard rules; `docs/nist-mapping.md` skeleton; this file marked complete. | diff --git a/PHASE0_STATUS.md b/PHASE0_STATUS.md index b15b2a3..5991143 100644 --- a/PHASE0_STATUS.md +++ b/PHASE0_STATUS.md @@ -1,6 +1,6 @@ # Phase 0 — Handoff status -**Last updated:** 2026-09-30 +**Last updated:** 2026-10-01 **Phase:** 0 — Foundations (in flight) **Owner:** Chris Tonjes, DOES CIO (System Owner / ISSO / AO) **Companion docs:** [`PHASE0_PLAN.md`](PHASE0_PLAN.md) (scope + milestones), [`ARCHITECTURE.md`](ARCHITECTURE.md) (design), [`README.md`](README.md) (runnable quickstart) @@ -9,7 +9,7 @@ ## TL;DR -Phase 0 milestones **0.1 → 0.6** are shipped. **0.7 (retention scaffold)** is next; **0.8–0.9** follow. All ATO artifacts (SSP, control mapping, POA&M, PIA) are drafted at v0.1–v0.3 and awaiting review signatures. The repo builds, tests are green (**138 passing, 95.05% coverage**), and Docker Compose boots. `main` is pushed to Gitea. +Phase 0 milestones **0.1 → 0.7** are shipped. **0.8 (CI pipeline)** is next, then **0.9 (docs + acceptance)**. All ATO artifacts (SSP, control mapping, POA&M, PIA) are drafted at v0.1–v0.3 and awaiting review signatures. The repo builds, tests are green (**156 passing, 95.91% coverage**), and Docker Compose boots. `main` is pushed to Gitea. --- @@ -17,12 +17,12 @@ Phase 0 milestones **0.1 → 0.6** are shipped. **0.7 (retention scaffold)** is | Dimension | Status | |---|---| -| Phase | 0 (Foundations), milestones 0.1–0.6 shipped | +| Phase | 0 (Foundations), milestones 0.1–0.7 shipped | | Git branch | `main`, tracking `origin/main` (Gitea `https://gitea.doesworks.net/chris.tonjes/YPMS.git`) | -| Tests | **138 passing**, **95.05% coverage** (floor 80% enforced in CI) | -| Lint / types | `ruff` clean, `mypy` clean (39 files, strict) | +| Tests | **156 passing**, **95.91% coverage** (floor 80% enforced in CI) | +| Lint / types | `ruff` clean, `mypy` clean (43 files, strict) | | Runtime | FastAPI + Postgres on Docker Compose; boots to `/healthz` + `/readyz`; `/auth/*` and `/registrants/*` live; 26 stub entities return 501 behind RBAC + audit | -| Migrations | 3 revisions (`0001_baseline`, `0002_framework_tables`, `0003_registrants`); 0003 rendered offline, live Postgres run pending | +| Migrations | 4 revisions (`0001`–`0004_stub_entity_tables`); all apply cleanly to SQLite in the drift test; live Postgres run pending (no Docker on the dev workstation) | | Seed | 8 roles + permissions matrix + bootstrap admin (`chris` / `DOES_SYSTEM_ADMIN`) | | ATO docs | SSP v0.3, control mapping v0.2, POA&M v0.1, PIA v0.1 — all drafts, awaiting signatures | @@ -98,21 +98,33 @@ Guardrails baked in: `OYP_STAFF` can't see FERPA/CFSA (that's `ELIGIBILITY_EXAMI - Registrant restore now requires the matrix's `Restore` action (was `Edit`). - Test infra: `roles` / `make_user` / `api` fixtures live in `tests/conftest.py`; `api` uses one app per session. +### 0.7 — Retention scaffold ✅ + +- `src/ypms/framework/retention.py` — `RetentionPolicy` (data class, disposal, `period_days`, anchor column, cadence, `approved`, authority) with a derived status: `retain` / `unset` / `interim` / `approved`. `run_retention()` + `python -m ypms.framework.retention`. +- **Report-only in Phase 0.** Interim-period rows are counted as `interim_eligible`; approved-period rows as `to_purge`; nothing is ever modified. Each run logs `"N records to purge"` and writes one `retention_run` audit row (actor null). +- `src/ypms/retention_classes.py` — PIA §9 classes at interim periods, **all `approved=False`** until `YPMS-POAM-054` signs. Unset today: consent (derived from related record, Phase 1), host records, operational logs, accounts (`-071`). Signing off = flip `approved=True` and cite the schedule in `authority`; `test_nothing_is_approved_before_records_officer_signoff` must be updated in the same commit. +- `src/ypms/stub_models.py` + `migrations/versions/0004_stub_entity_tables.py` — id + mixins for the 26 stub entities. Domain phases move each class into its module; the table name stays. +- `src/ypms/models.py` — the one import-all list (Alembic env, retention job, test fixtures). +- `tests/framework/test_retention.py::test_migrations_match_models` — upgrades a scratch SQLite DB to head and diffs against `Base.metadata`. Verified it fails on an unmigrated column. + --- ## What's next -### Immediate next milestone: 0.7 — Retention scaffold +### Immediate next milestone: 0.8 — CI pipeline -`retention_policy` on every entity model, and a nightly job in `src/ypms/framework/retention.py` that runs and logs "0 records to purge". The exit criterion requires a model for every stub entity, so 0.7 also adds id + mixins models for the 26 entities in `src/ypms/stubs.py` (plus a migration). Retention periods are blocked on `YPMS-POAM-054` (records schedule); use a placeholder policy class per PIA §9 and leave the durations unset. +`.github/workflows/ci.yml` already runs ruff, mypy, pytest (80% floor), bandit, pip-audit, gitleaks. 0.8 is getting it green on the actual CI host and adding SBOM (CycloneDX; `YPMS-POAM-042`). **Open question first:** the repo lives on Gitea, so GitHub Actions won't run as-is — Gitea Actions (largely workflow-compatible) vs. Azure DevOps (ARCH §9.3 says match OWC). Confirm what OWC uses before building. -### After 0.7 +### After 0.8 -- **0.8** — CI pipeline hardening — SAST/SCA/SBOM configured in `.github/workflows/ci.yml` already; 0.8 is the discipline of green builds and any needed suppressions. - **0.9** — Docs + acceptance (README expanded, `docs/nist-mapping.md` skeleton, "Phase 0 done" acceptance checklist) Then Phase 1 (applications + identity + parental consent) starts. +### Schema follow-up (surfaced by 0.7 drift test) + +Migration `0002_framework_tables` creates named unique constraints (`uq_roles_code`, `uq_users_username`, `uq_users_email`) **and** unique indexes on the same columns; the models declare only the indexes. Harmless (double enforcement on Postgres) but real drift. 0002 is immutable — options are a `0005` that drops the three constraints, or leave as-is. Allow-listed in `KNOWN_0002_DRIFT` until decided. + ### Permission-matrix review items (surfaced by 0.6) The stub sweep runs every role against every action, so these are now visible. **Not changed** — they're policy calls for the Phase 0 review: @@ -189,7 +201,8 @@ git -C /Users/christonjes/cc/ypms config user.email | `youth/` (models, router, schemas) | 99–100% | | `security/column_crypto.py` | 100% | | `framework/stub_router.py`, `stubs.py` | 100% | -| Overall | **95.05%** | +| `framework/retention.py`, `retention_classes.py`, `stub_models.py` | 99–100% | +| Overall | **95.91%** | `framework/router.py` is still excluded from coverage — Registrant uses a hand-built router, so `make_router` gets its first real callers in 0.6. Coverage config in `pyproject.toml`. @@ -273,6 +286,10 @@ The GitHub Actions workflow at `.github/workflows/ci.yml` runs the same commands 13. **Adding an entity to `permissions_matrix.json` fails `test_every_matrix_entity_is_real_or_stubbed`** until it's either stubbed in `stubs.py` or has a real router (then add it to `REAL_ENTITIES` in that test). +14. **Every mapped table must resolve a retention policy** — including test-only models (`tests/framework/_fixtures.py::Thing` declares one). A new model without `retention_policy` fails `test_every_mapped_table_has_a_policy` and makes the job raise `LookupError`. + +15. **New model modules go in `src/ypms/models.py`.** Alembic, the retention job, and the test DB all read that list. + --- ## File map @@ -309,13 +326,13 @@ Then the ATO artifacts as needed: --- -## Signals to watch for as we approach milestone 0.7 +## Signals to watch for as we approach milestone 0.8 -- Stub models should be id + mixins only (PHASE0_PLAN §4). Resist adding real fields — that's domain-phase work with its own migration. -- Don't invent retention durations. `YPMS-POAM-054` owns them; the scaffold should make "no policy set" an explicit, logged state rather than a silent default. -- FosterIndicator and EducationRecord have a distinct two-year annual purge (ARCH §6.4) — model it as its own policy class now so the job structure doesn't need reshaping later. +- Don't weaken a gate to get green (no blanket bandit skips, no `--cov-fail-under` drop). Suppressions are per-finding with a reason. +- The drift test runs Alembic inside pytest; CI needs the repo root as cwd so `alembic.ini` resolves. +- Retention execution (actual disposal) is Phase 5 work, alongside the OSSE/CFSA ingest pipelines. Don't add it in passing. - Any entity that will hold PII in Phase 1+ (Guardian, TaxDocument, Attachment) should get the Registrant treatment then — `pii_read` on reads, `__audit_redact__` on encrypted columns. --- -**End of handoff v0.3.** +**End of handoff v0.4.** diff --git a/POAM.md b/POAM.md index ed5d917..628271b 100644 --- a/POAM.md +++ b/POAM.md @@ -178,6 +178,7 @@ Closures retained for traceability. | — | Phase 0 milestone 0.4 shipped (partial closure of YPMS-POAM-040) | 2026-09-30 | Data-driven RBAC seed: 8 roles + permissions matrix (30 entities × 8 actions) at `src/ypms/seed/`. Bootstrap admin auto-seeded (`chris` / DOES_SYSTEM_ADMIN). Loader idempotent + validates cell references. `python -m ypms.seed.run` chained into docker-compose. 53 tests passing, 92.20% coverage. | | — | Phase 0 milestone 0.5 shipped (partial closure of YPMS-POAM-040) | 2026-09-30 | Registrant vertical slice: model, schemas, router, migration 0003. SSN app-layer encrypted (AES-256-GCM, column-bound AAD) and masked in the serializer (C.5.7.50 / SC-28). `pii_read` audit on every read path (C.5.6.9 #5 / AU-14). Audit + version-history snapshots redact ciphertext via `__audit_redact__`. Portal-only roles limited to own row. 98 tests passing, 94.83% coverage. Evidence trace at `docs/coverage/registrant.md`. | | — | Phase 0 milestone 0.6 shipped (partial closure of YPMS-POAM-040) | 2026-09-30 | 501-stub routers for 26 entities; every route enforces RBAC and writes an audit row before returning 501 (source RFP C.5.7.27; NIST AC-3, AU-12). Matrix-driven test sweep covers every entity × role × action. Permission-matrix review items recorded in PHASE0_STATUS.md. 138 tests passing, 95.05% coverage. | +| — | Phase 0 milestone 0.7 shipped (partial closure of YPMS-POAM-040) | 2026-10-01 | Retention scaffold (NIST AU-11, SI-12; source RFP C.5.7.42): every table carries a policy; PIA §9 interim periods encoded but unapproved; report-only nightly job logs eligible counts and audits each run. Implementation is ready to consume the signed schedule from YPMS-POAM-054 (flip `approved`, cite authority). 156 tests passing, 95.91% coverage. | ---