diff --git a/.planning/ROADMAP.md b/.planning/ROADMAP.md index b231077..4a521b1 100644 --- a/.planning/ROADMAP.md +++ b/.planning/ROADMAP.md @@ -344,7 +344,7 @@ Plans: 4. DÖE is polled once on a shared, admin-configurable interval regardless of how many tenants have the module active -- never once per tenant (poll-once-fan-out-many, not the DKV single-tenant `findFirst()` pattern) 5. Activating the module for a second tenant does not duplicate ingestion, re-trigger a redundant DÖE poll, or interfere with the first tenant's data -**Plans**: 4/6 plans executed +**Plans**: 5/6 plans executed **Wave 1** @@ -364,7 +364,7 @@ Plans: **Wave 5** *(blocked on Wave 4 completion)* -- [ ] 10-05-PLAN.md — Controller + admin source-config: global ModuleGuard-gated read, admin-configurable poll interval applied live to scheduler (INGEST-06) +- [x] 10-05-PLAN.md — Controller + admin source-config: global ModuleGuard-gated read, admin-configurable poll interval applied live to scheduler (INGEST-06) **Wave 6** *(blocked on Wave 5 completion)* @@ -453,7 +453,7 @@ Phases execute in numeric order: 1 -> 2 -> 3 -> 4 -> 5 -> 6 -> 7 -> 8 -> 9 -> 10 | 7. DKV Fleet Module | 6/6 | Complete | 2026-06-27 | | 8. Dashboard Widgets Vollimplementierung | 4/4 | Complete | 2026-07-01 | | 9. Cert Manager Module | 6/6 | Complete | 2026-07-02 | -| 10. Ausschreibungs-Radar Foundation & DÖE Ingestion | 4/6 | In Progress| | +| 10. Ausschreibungs-Radar Foundation & DÖE Ingestion | 5/6 | In Progress| | | 11. Filter Engine, Results UI & Saved Searches | 0/TBD | Not started | - | | 12. Tender Notifications | 0/TBD | Not started | - | | 13. Scraping Adapters & Cross-Source Deduplication | 0/TBD | Not started | - | diff --git a/.planning/STATE.md b/.planning/STATE.md index 4320293..3da0761 100644 --- a/.planning/STATE.md +++ b/.planning/STATE.md @@ -6,14 +6,14 @@ current_phase: 10 current_phase_name: ausschreibungs-radar-foundation-d-e-ingestion status: executing stopped_at: Completed 10-02-PLAN.md -last_updated: "2026-07-21T09:13:27.792Z" +last_updated: "2026-07-21T09:21:24.069Z" last_activity: 2026-07-21 last_activity_desc: Phase 10 execution started progress: total_phases: 14 completed_phases: 8 total_plans: 46 - completed_plans: 43 + completed_plans: 44 percent: 57 --- @@ -29,7 +29,7 @@ See: .planning/PROJECT.md (updated 2026-07-17) ## Current Position Phase: 10 (ausschreibungs-radar-foundation-d-e-ingestion) — EXECUTING -Plan: 5 of 6 +Plan: 6 of 6 Status: Ready to execute Last activity: 2026-07-21 — Phase 10 execution started @@ -75,6 +75,7 @@ Progress: [░░░░░░░░░░] 0% | Phase 10 P02 | 20min | 3 tasks | 6 files | | Phase 10 P03 | 35min | 3 tasks | 9 files | | Phase 10 P04 | 25min | 3 tasks | 5 files | +| Phase 10 P05 | 20min | 3 tasks | 5 files | ## Accumulated Context @@ -153,6 +154,9 @@ Recent decisions affecting current work: - [Phase ?]: Poll-once-fan-out-many scheduler: single named cron job, no tenant parameter — deliberately drops DKV's activeTenantId/findFirst per-tenant framing (Pitfall D) - [Phase ?]: SCHEMA-02 change detection implemented via prisma.tender.upsert({ where: { dedupKey } }) — identical notice never duplicates, changed contentHash updates in place - [Phase ?]: D-05 retention as two-phase updateMany/deleteMany with deadlineAt:{lt} filters — null-deadline rows structurally excluded, never auto-expired +- [Phase ?]: TendersController talks to PrismaService directly (no intermediate service layer) — source-config upsert and global read are simple enough for this plan's scope +- [Phase ?]: Comment wording avoids the literal tenantId token in tenders.controller.ts to prevent false-positive grep-gate failures (same pattern as Plan 10-04) +- [Phase ?]: TenderQueryDto.status defaults to active at the controller call site, not baked into the DTO, mirroring DkvController's page/limit default-at-usage pattern ### Pending Todos @@ -190,7 +194,7 @@ Items acknowledged and carried forward from previous milestone close: ## Session Continuity -Last session: 2026-07-21T09:12:58.508Z +Last session: 2026-07-21T09:20:45.659Z Stopped at: Completed 10-02-PLAN.md Resume file: None Last activity: 2026-07-14 - Built LDAP per-user exclude/denylist filter (9d1323f), migration applied on live DB, verified via Playwright: sync deactivated 4 excluded service accounts (administrator/krbtgt/guest/dns-ldap), 2 real LDAP users stay active, 0 wrongly created diff --git a/.planning/phases/10-ausschreibungs-radar-foundation-d-e-ingestion/10-05-SUMMARY.md b/.planning/phases/10-ausschreibungs-radar-foundation-d-e-ingestion/10-05-SUMMARY.md new file mode 100644 index 0000000..4b5d620 --- /dev/null +++ b/.planning/phases/10-ausschreibungs-radar-foundation-d-e-ingestion/10-05-SUMMARY.md @@ -0,0 +1,195 @@ +--- +phase: 10-ausschreibungs-radar-foundation-d-e-ingestion +plan: 05 +subsystem: api +tags: [controller, dto, module-guard, roles-guard, scheduler-live-apply, tdd, tenant-gated-not-scoped] + +# Dependency graph +requires: + - phase: 10-01 (foundation & dependencies) + provides: Tender / TenderSourcePollConfig Prisma models (dedupKey @unique, sourceType @unique) + - phase: 10-02 (marketplace registration) + provides: TendersModule skeleton, singleton doe-opendata poll config seeded on boot + - phase: 10-03 (adapter & normalizer) + provides: DoeOpenDataAdapter, TenderNormalizerService + - phase: 10-04 (ingestion orchestration & scheduler) + provides: TenderSchedulerService.setInterval()/stopJob() (no tenant arg), TenderIngestionService.pollDueSources() +provides: + - "TendersController — GET/GET:id (global read, @UseModule-gated, never row-scoped by tenant id) + GET/PUT source-config (Roles-guarded singleton config, live-applies to the scheduler)" + - "SourceConfigDto, TenderQueryDto — class-validator DTOs following the DKV conventions" + - "Automated proof (controller spec) of the tenant-gated-not-scoped invariant and the live scheduler-apply on config save" +affects: [phase 11 saved searches/filter UI (consumes GET / and GET /:id as the read surface over the ingested catalog)] + +# Tech tracking +tech-stack: + added: [] + patterns: + - "Global-read-gated-by-module-not-by-tenant-row: GET / and GET /:id use plain PrismaService with a where clause built only from query params (status/pagination), gated exclusively by @UseModule('tender-radar') — this is the one controller in the codebase where tenant-gated (feature visibility) and tenant-scoped (row filtering) genuinely diverge" + - "Admin platform-config routes are Roles-guarded, not ModuleGuard-gated — configuring the shared platform-wide poll schedule is a platform-admin action, not a per-tenant module feature, so @UseModule is deliberately absent from source-config routes" + - "Scheduler live-apply on config save: PUT /source-config calls tenderScheduler.setInterval(intervalMin) / stopJob() with no tenant argument, mirroring DkvController.saveConfig minus the per-tenant parameter (the DÖE config is a platform-wide singleton)" + +key-files: + created: + - apps/api/src/tenders/dto/source-config.dto.ts + - apps/api/src/tenders/dto/tender-query.dto.ts + - apps/api/src/tenders/tenders.controller.ts + - apps/api/src/tenders/tenders.controller.spec.ts + modified: + - apps/api/src/tenders/tenders.module.ts + +key-decisions: + - "Controller talks to PrismaService directly (no intermediate TenderService/TenderConfigService layer) — the plan's files_modified list and PATTERNS.md controller sketch show no such service; source-config upsert and the global read query are simple enough that adding a service layer would be premature indirection for this plan's scope" + - "Comment wording avoids the literal 'tenantId' token in tenders.controller.ts (uses 'the tenant's id' / 'tenant-id' instead) — the acceptance criterion's grep-gate (`grep -c tenantId tenders.controller.ts` must return 0) checks for the literal string; an explanatory comment naming the concept using that exact identifier would trip the same false-positive class of issue Plan 10-04 documented and fixed twice already in this phase" + - "TenderQueryDto.status defaults to 'active' at the controller layer (query DTO field itself has no default value, per plan spec — defaulting happens in listTenders(), matching DkvController's page/limit default-at-usage-site pattern)" + +requirements-completed: [INGEST-06] + +coverage: + - id: D1 + description: "Admin can read and update the shared DÖE poll interval / active state; saving pushes the change into the live scheduler without restart (INGEST-06)" + requirement: "INGEST-06" + verification: + - kind: unit + ref: "tenders.controller.spec.ts — 'PUT /source-config with isActive=true + pollIntervalMin=30 calls scheduler.setInterval(30) with a single argument'; 'PUT /source-config with isActive=false calls scheduler.stopJob()'" + status: pass + - kind: automated_ui + ref: "grep -n '@Roles' tenders.controller.ts — both source-config handlers carry @Roles(Role.ADMIN, Role.SUPER_ADMIN)" + status: pass + human_judgment: false + - id: D2 + description: "GET /tenders list + detail is gated by module activation (ModuleGuard), NOT by tenantId row-filtering — the global catalog is visible to any tenant with the module active" + requirement: "INGEST-06" + verification: + - kind: unit + ref: "tenders.controller.spec.ts — 'GET / calls prisma.tender.findMany with a where clause that has no tenantId key'" + status: pass + - kind: automated_ui + ref: "grep -c UseModule tenders.controller.ts == 7 (decorator + doc mentions); grep -c tenantId tenders.controller.ts == 0" + status: pass + human_judgment: false + - id: D3 + description: "Admin source-config routes require ADMIN/SUPER_ADMIN roles" + requirement: "INGEST-06" + verification: + - kind: automated_ui + ref: "grep -n '@Roles' tenders.controller.ts — 2 matches, both on GET/PUT source-config handlers" + status: pass + human_judgment: false + +# Metrics +duration: ~20min +completed: 2026-07-21 +status: complete +--- + +# Phase 10 Plan 05: Tenders Controller — Global Read + Admin Source-Config Summary + +**`TendersController` wires the phase's one deliberately-divergent read surface — `GET /` and `GET /:id` gated by `@UseModule('tender-radar')` but never row-scoped by tenant — alongside `@Roles`-guarded `GET`/`PUT /source-config` that live-applies interval/active-state changes to `TenderSchedulerService`, completing INGEST-06's admin-configurable half.** + +## Performance + +- **Duration:** ~20 min +- **Started:** 2026-07-21 +- **Completed:** 2026-07-21 +- **Tasks:** 3 (Task 1 auto, Task 2 auto, Task 3 tdd="true" — spec passed immediately against Task 2's already-correct implementation) +- **Files modified:** 5 (4 created, 1 modified) + +## Accomplishments + +- `SourceConfigDto` (`pollIntervalMin` `@Min(5) @Max(1440)`, `isActive` `@IsBoolean`) and `TenderQueryDto` (`page`/`limit` pagination copied from `DkvHistoryQueryDto`, plus `status` `@IsIn(['active','expired'])`) — both class-validator DTOs following the exact DKV conventions (DoS-mitigation bounds, `@Type(() => Number)` query-string coercion). +- `TendersController` (`@Controller('modules/tender-radar')`): + - `GET /` and `GET /:id` — paginated global `Tender` catalog read via plain `PrismaService`, gated only by `@UseModule('tender-radar')` (module activation), never row-scoped by a `where: { tenantId }` filter. `GET /:id` throws `NotFoundException` for a missing id. + - `GET /source-config` / `PUT /source-config` — both `@Roles(Role.ADMIN, Role.SUPER_ADMIN)`-guarded. `PUT` upserts the singleton `doe-opendata` config, then live-applies the change to `TenderSchedulerService.setInterval(intervalMin)` (single argument, no tenant id) when active, or `stopJob()` when deactivated — satisfying INGEST-06's "applied without restart" requirement. +- `TendersController` registered in `TendersModule.controllers`; `TenderSchedulerService` (already a provider since Plan 04) resolves via DI. +- Controller spec (5 tests): proves the read path never adds a `tenantId` key to the prisma `where`, `GET /:id` 404s for a missing tender, and both config-save branches (`isActive=true`+interval / `isActive=false`) drive the scheduler exactly as specified. +- Full API test suite green (74/74), `tsc --noEmit` clean for both `apps/api` and `apps/web`, all grep gates (`UseModule` present, `tenantId` absent, `@Roles` on both admin routes) pass. + +## Task Commits + +Each task committed atomically: + +1. **Task 1: SourceConfigDto + TenderQueryDto** — `f6e636c` (feat) +2. **Task 2: TendersController — global read (ModuleGuard) + admin source-config (Roles) + live scheduler apply** — `7b9b6b8` (feat) +3. **Task 3: Controller test — global read not tenant-scoped + admin config applies to scheduler** — `eaff1d8` (test) + +**Plan metadata:** see final `docs(10-05)` commit. + +## TDD Gate Compliance + +- Task 3 (`tdd="true"`): the spec (`eaff1d8`, `test(10-05): ...`) passed on first run, immediately after being written — **this is expected, not an unexpected-pass violation of the RED-GREEN gate.** Task 2's `feat` commit (`7b9b6b8`) already implemented the tenant-gated-not-scoped read path and the scheduler live-apply correctly; Task 3's test locks in and regression-proofs that already-correct behavior via genuine assertions on the prisma `where` clause and scheduler mock calls, rather than driving new production code. This plan's frontmatter is `type: execute` with a per-task `tdd="true"` flag (not a full `type: tdd` plan), matching the exact precedent documented in Plan 10-04's Task 3 — the strict "investigate a passing RED" rule applies to full-plan TDD gates, not this per-task flag usage. +- No REFACTOR commit was needed — the spec passed cleanly on first write, no post-green cleanup required. + +## Files Created/Modified + +- `apps/api/src/tenders/dto/source-config.dto.ts` — `pollIntervalMin` (`@IsOptional @IsInt @Min(5) @Max(1440)`), `isActive` (`@IsOptional @IsBoolean`) +- `apps/api/src/tenders/dto/tender-query.dto.ts` — `page`/`limit` pagination (`@Type(() => Number)` coercion, `@Min(1)`, `limit @Max(100)`) + `status` (`@IsOptional @IsIn(['active','expired'])`) +- `apps/api/src/tenders/tenders.controller.ts` — `TendersController`: `GET /`, `GET /:id` (global, `@UseModule`-gated), `GET /source-config`, `PUT /source-config` (`@Roles`-guarded, scheduler live-apply) +- `apps/api/src/tenders/tenders.controller.spec.ts` — 5 tests: no-tenantId read-path assertion, detail found/404, `setInterval(30)` single-arg assertion, `stopJob()` assertion +- `apps/api/src/tenders/tenders.module.ts` — `TendersController` added to `controllers` + +## Decisions Made + +- **No intermediate service layer for the controller** — `TendersController` talks to `PrismaService` and `TenderSchedulerService` directly. The plan's `files_modified` list and PATTERNS.md's controller sketch show no `TenderConfigService`/`TenderQueryService`; the source-config upsert and the paginated global read are simple enough that an intermediate service would be premature indirection for this plan's scope. `TenderIngestionService`/`TenderSchedulerService` remain the sole business-logic services (Plan 04). +- **Comment wording avoids the literal `tenantId` token** in `tenders.controller.ts` (rephrased as "the tenant's id" / "tenant-id" throughout) — the acceptance criterion's grep gate (`grep -c tenantId tenders.controller.ts` must return 0) checks for the literal string, and an explanatory comment naming the concept with that exact identifier would trip the same false-positive class of issue Plan 10-04 documented (and auto-fixed twice) in this phase. Caught during Task 2's own verification pass before committing. +- **`TenderQueryDto.status` has no DTO-level default** — matching `DkvController`'s existing `page ?? 1` / `limit ?? 20` pattern, the `'active'`-only default is applied at the controller call site (`query.status ?? 'active'`), not baked into the DTO class itself. + +## Deviations from Plan + +### Auto-fixed Issues + +**1. [Rule 1 - Bug] `@Body()` decorator missing on `saveSourceConfig`'s parameter** +- **Found during:** Task 2 authoring (caught before running verification) +- **Issue:** Initial draft of `saveSourceConfig(dto: SourceConfigDto)` omitted the `@Body()` parameter decorator, which would have left `dto` as `undefined` at runtime (NestJS only populates decorated parameters). +- **Fix:** Added `@Body()` and its import from `@nestjs/common`. +- **Files modified:** `apps/api/src/tenders/tenders.controller.ts` +- **Commit:** folded into `7b9b6b8` (Task 2 commit, caught and fixed before commit) + +**2. [Rule 1 - Bug] Literal `tenantId` token in explanatory comments tripped the plan's own grep-gate acceptance criterion** +- **Found during:** Task 2 verification (`grep -c tenantId tenders.controller.ts` initially returned 4, not the required 0) +- **Issue:** Doc comments explaining the tenant-gated-not-scoped divergence used the literal word "tenantId" to describe what the code does NOT do — the same class of false-positive documented twice in Plan 10-04's Deviations (grep-gate assertions check for the literal string, regardless of whether it appears in code or in a comment explaining its absence). +- **Fix:** Reworded all four occurrences to "the tenant's id" / "tenant-id" (with an explicit space/hyphen breaking the literal token match) while preserving the exact same explanatory meaning. +- **Files modified:** `apps/api/src/tenders/tenders.controller.ts` +- **Verification:** `grep -c tenantId tenders.controller.ts` → 0; `tsc --noEmit` clean; full spec suite still green. +- **Committed in:** `7b9b6b8` (Task 2 commit, caught and fixed before commit) + +**3. [Rule 3 - Blocking] `tsc` type errors in the controller spec's mock typing** +- **Found during:** Task 3 post-write `tsc --noEmit` verification +- **Issue:** `prisma.tender.findMany`/`count` fakes were declared as zero-argument `vi.fn(async () => ...)`, which TypeScript inferred as a `[]`-length call-args tuple — `mock.calls[0][0]` then failed with "Tuple type '[]' of length '0' has no element at index '0'". +- **Fix:** Added an explicit optional `_args?: any` parameter to both fakes so `mock.calls[0][0]` type-checks. +- **Files modified:** `apps/api/src/tenders/tenders.controller.spec.ts` +- **Verification:** `tsc --noEmit` clean; `pnpm test -- tenders.controller` still green (5/5). +- **Committed in:** `eaff1d8` (Task 3 commit, caught and fixed before commit) + +--- + +**Total deviations:** 3 auto-fixed (2 Bugs, 1 Blocking — all caught and fixed inline before their respective commits, zero scope creep, zero behavior change beyond the fixes themselves) + +## Issues Encountered + +None beyond the three documented deviations above. Local Docker Postgres stack (per environment context) was not needed — the controller spec uses in-memory prisma-shaped fakes, matching this phase's established test convention (10-04's ingestion/scheduler specs), so no live-DB dependency was introduced. + +## User Setup Required + +None — no external service configuration required. The local Docker stack (`tessera-ctl-api-1`/`-db-1`) was left running unmodified; restarting/rebuilding it to pick up these code changes (so the new `/modules/tender-radar` routes are actually reachable over HTTP) is left to the user per project convention. + +## Next Phase Readiness + +- INGEST-06 is now fully complete: the platform-wide poll interval is admin-readable/writable via `GET`/`PUT /modules/tender-radar/source-config`, and saving live-applies to the running scheduler without a restart (Plan 04 built the scheduler mechanics; this plan wires the admin-facing surface to it). +- `GET /modules/tender-radar` and `GET /modules/tender-radar/:id` give Phase 11 a ready-made, paginated, `@UseModule`-gated read surface over the ingested global catalog — Phase 11's saved-searches/filter UI can consume these directly and layer richer filtering (region/CPV — FILTER-*) on top without touching the tenant-gated-not-scoped invariant established here. +- No open threat-model items from this plan carry forward unmitigated: T-10-13 (Elevation of Privilege on `PUT /source-config`) is mitigated by `@Roles(Role.ADMIN, Role.SUPER_ADMIN)` + the global `RolesGuard`; T-10-14 (Broken Access Control on `GET /tenders`) is mitigated by `@UseModule('tender-radar')` and proven by the no-tenantId controller test; T-10-15 (Input Validation DoS) is mitigated by `SourceConfigDto`/`TenderQueryDto`'s class-validator bounds. +- No blockers for the next phase (11, saved searches/filter UI over this ingested catalog). + +## Self-Check: PASSED + +- FOUND: apps/api/src/tenders/dto/source-config.dto.ts +- FOUND: apps/api/src/tenders/dto/tender-query.dto.ts +- FOUND: apps/api/src/tenders/tenders.controller.ts +- FOUND: apps/api/src/tenders/tenders.controller.spec.ts +- FOUND: apps/api/src/tenders/tenders.module.ts +- FOUND commit: f6e636c +- FOUND commit: 7b9b6b8 +- FOUND commit: eaff1d8 + +--- +*Phase: 10-ausschreibungs-radar-foundation-d-e-ingestion* +*Completed: 2026-07-21*