Phase 0 milestone 0.2: vendor OWC framework + YPMS pii_audit

Brings in a working generic-CRUD framework plus its dependencies from OWC
(../owc-system/src/owc/, snapshot 2026-04-20). Every vendored module carries
a header comment naming the source; the OWC → ypms rename is the only
mechanical change.

Vendored modules:

- framework/base.py           TimestampMixin, SoftDeleteMixin, VersionedMixin
- framework/pagination.py     Page[T] with to_dict()
- framework/version_history.py  Snapshot table
- framework/crud.py           CRUDService with audit emission
- framework/bulk.py           BulkOps (BulkResult / BulkUpdateItem)
- framework/router.py         make_router() factory (kept without
                              `from __future__ import annotations` for
                              FastAPI type introspection — see the module
                              docstring)
- audit/models.py             AuditLog
- audit/service.py            record()
- security/models.py          Permission
- security/rbac.py            can() + require_permission() dependency
                              (DOES_SYSTEM_ADMIN as the super-role)
- users/models.py             User, Role, UserRole (YPMS adds
                              mfa_verified and assurance_level; drops
                              OWC-specific claimant_id)
- auth/deps.py                get_current_user() placeholder — real JWT
                              verify lands in 0.3

YPMS-specific additions on top of the vendored framework:

- framework/pii_audit.py      emit_pii_read() — NIST AU-14 / source RFP
                              C.5.6.9 #5. OWC doesn't need this because
                              its case-management surface is staff-only;
                              YPMS must audit reads of minors' PII, SSN,
                              banking, FERPA-covered OSSE records, and
                              CFSA foster/ward flags.

Bugfix vs. upstream OWC:

- framework/crud.py           Pagination `next_cursor` was previously the
                              peek-ahead row's id, which combined with
                              `where id > cursor` on the next page
                              silently dropped one row per boundary.
                              Caught by test_list_pagination_forwards_cursor.
                              Fixed here; noted in the module header so
                              re-vendoring OWC surfaces it upstream.

Migrations:

- 0002_framework_tables.py    Baseline for users, roles, user_roles,
                              permissions, audit_log, version_history.
                              Immutable once merged per CONTRIBUTING.md.
- migrations/env.py           Imports the new model modules so
                              autogenerate compares against them.

Tests (26 passing, 93.10% coverage, floor 80%):

- tests/framework/test_crud.py         CRUDService lifecycle + snapshots
                                       + audit emission (7 tests)
- tests/framework/test_bulk.py         BulkOps happy paths + failure
                                       reporting (4 tests)
- tests/framework/test_pagination.py   Page.to_dict serializer contract
                                       (2 tests)
- tests/framework/test_pii_audit.py    emit_pii_read row shape,
                                       anonymous case (2 tests)
- tests/security/test_rbac.py          can() + admin bypass + role union
                                       (4 tests)
- tests/test_readyz.py                 /readyz smoke (1 test)
- tests/test_logging.py                sanitize_for_log contract (3 tests)

Green locally: ruff clean, mypy clean (with a targeted override on
framework/router.py and framework/crud.py for TypeVar limitations that
OWC also accepts).

Advances YPMS-POAM-040. Prepares milestone 0.3 (JWT auth) and 0.4 (RBAC
seed) to plug into an already-live framework.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Chris TonjesandClaude Opus 4.7 committed 2026-09-30 16:55:56 -04:00
1 parent 09d862bf4a
commit ba756586aa
31 files changed
+1493 -11

No files matched your search

+166
View File
@@ -0,0 +1,166 @@
"""Generic CRUD service. One instance per entity model.
Handles list/get/create/update/soft-delete/restore + version snapshot + audit
emission. Bulk variants live in `ypms.framework.bulk`.
Vendored from OWC `framework/crud.py` (snapshot 2026-04-20). Changes from
upstream:
- `owc` → `ypms` import renames.
- **Pagination bugfix**: `next_cursor` is now the id of the last **returned**
row (not the peek-ahead row). Upstream returned the (limit+1)th row's id,
which combined with `where id > cursor` on the next call silently skipped
that row. Caught by `tests/framework/test_crud.py::test_list_pagination_forwards_cursor`.
Flag to OWC when re-vendoring.
"""
from __future__ import annotations
from datetime import UTC, datetime
from typing import Any, Generic, TypeVar
from pydantic import BaseModel
from sqlalchemy import select
from sqlalchemy.exc import NoResultFound
from sqlalchemy.orm import Session
from ypms.audit import service as audit
from ypms.framework.pagination import Page
from ypms.framework.version_history import VersionHistory
from ypms.users.models import User
ModelT = TypeVar("ModelT")
CreateT = TypeVar("CreateT", bound=BaseModel)
UpdateT = TypeVar("UpdateT", bound=BaseModel)
def _to_dict(obj: Any) -> dict[str, Any]:
if obj is None:
return {}
out: dict[str, Any] = {}
for c in obj.__table__.columns:
v = getattr(obj, c.name, None)
if isinstance(v, datetime):
v = v.isoformat()
out[c.name] = v
return out
class CRUDService(Generic[ModelT, CreateT, UpdateT]): # noqa: UP046 — TypeVars reused by bulk.py
def __init__(self, model: type[ModelT], entity: str):
self.model = model
self.entity = entity
# ------------- read -------------
def get(self, db: Session, id: int, *, include_deleted: bool = False) -> ModelT:
obj = db.get(self.model, id)
if obj is None:
raise NoResultFound(f"{self.entity} {id} not found")
if not include_deleted and getattr(obj, "deleted_at", None) is not None:
raise NoResultFound(f"{self.entity} {id} not found")
return obj
def list(
self,
db: Session,
*,
cursor: int | None = None,
limit: int = 50,
include_deleted: bool = False,
newest_first: bool = False,
) -> Page[ModelT]:
if newest_first:
stmt = select(self.model).order_by(self.model.id.desc()).limit(limit + 1) # type: ignore[attr-defined]
if cursor is not None:
stmt = stmt.where(self.model.id < cursor) # type: ignore[attr-defined]
else:
stmt = select(self.model).order_by(self.model.id.asc()).limit(limit + 1) # type: ignore[attr-defined]
if cursor is not None:
stmt = stmt.where(self.model.id > cursor) # type: ignore[attr-defined]
if not include_deleted and hasattr(self.model, "deleted_at"):
stmt = stmt.where(self.model.deleted_at.is_(None)) # type: ignore[attr-defined]
rows = db.execute(stmt).scalars().all()
# Fetched limit+1 to peek ahead; if we got more than `limit`, there is
# another page. next_cursor is the id of the LAST RETURNED row so the
# next call's `where id > cursor` starts strictly after it (no skip).
items = list(rows[:limit])
next_cursor = items[-1].id if len(rows) > limit else None # type: ignore[attr-defined]
return Page(items=items, next_cursor=next_cursor)
# ------------- write -------------
def create(self, db: Session, payload: CreateT, *, actor: User | None = None) -> ModelT:
data = payload.model_dump(exclude_unset=True)
obj = self.model(**data) # type: ignore[call-arg]
db.add(obj)
db.flush()
self._snapshot(db, obj, actor)
audit.record(
db, actor=actor, action="create", entity=self.entity, entity_id=obj.id,
after=_to_dict(obj),
)
return obj
def update(self, db: Session, id: int, payload: UpdateT, *, actor: User | None = None) -> ModelT:
obj = self.get(db, id)
before = _to_dict(obj)
for k, v in payload.model_dump(exclude_unset=True).items():
setattr(obj, k, v)
if hasattr(obj, "version"):
obj.version = (obj.version or 1) + 1
db.flush()
self._snapshot(db, obj, actor)
audit.record(
db, actor=actor, action="update", entity=self.entity, entity_id=obj.id,
before=before, after=_to_dict(obj),
)
return obj
def soft_delete(self, db: Session, id: int, *, actor: User | None = None) -> ModelT:
obj = self.get(db, id)
if not hasattr(obj, "deleted_at"):
raise TypeError(f"{self.entity} does not support soft delete")
before = _to_dict(obj)
obj.deleted_at = datetime.now(UTC)
db.flush()
audit.record(
db, actor=actor, action="delete", entity=self.entity, entity_id=obj.id,
before=before, after=_to_dict(obj),
)
return obj
def restore(self, db: Session, id: int, *, actor: User | None = None) -> ModelT:
obj = self.get(db, id, include_deleted=True)
before = _to_dict(obj)
obj.deleted_at = None
db.flush()
audit.record(
db, actor=actor, action="restore", entity=self.entity, entity_id=obj.id,
before=before, after=_to_dict(obj),
)
return obj
# ------------- internals -------------
def _snapshot(self, db: Session, obj: ModelT, actor: User | None) -> None:
if not hasattr(obj, "version"):
return
db.add(
VersionHistory(
entity=self.entity,
entity_id=obj.id, # type: ignore[attr-defined]
version=obj.version, # type: ignore[attr-defined]
actor_id=actor.id if actor else None,
snapshot=_to_dict(obj),
)
)
db.flush()
def history(self, db: Session, id: int) -> list[VersionHistory]:
stmt = (
select(VersionHistory)
.where(VersionHistory.entity == self.entity, VersionHistory.entity_id == id)
.order_by(VersionHistory.version.asc())
)
return list(db.execute(stmt).scalars().all())