docs(10-04): complete ingestion orchestration & multi-tenant-safe scheduler plan
This commit is contained in:
@@ -0,0 +1,207 @@
|
||||
---
|
||||
phase: 10-ausschreibungs-radar-foundation-d-e-ingestion
|
||||
plan: 04
|
||||
subsystem: api
|
||||
tags: [scheduler, ingestion, multi-tenant-safety, cron, scheduler-registry, tdd, poll-once-fan-out-many]
|
||||
|
||||
# Dependency graph
|
||||
requires:
|
||||
- phase: 10-01 (foundation & dependencies)
|
||||
provides: Tender / TenderSourcePollConfig Prisma models (dedupKey @unique, sourceType @unique)
|
||||
- phase: 10-02 (marketplace registration)
|
||||
provides: TendersModule skeleton, singleton doe-opendata poll config seeded on boot
|
||||
- phase: 10-03 (adapter & normalizer)
|
||||
provides: DoeOpenDataAdapter.fetchTenders(dayCursor), TenderNormalizerService.normalize()
|
||||
provides:
|
||||
- TenderIngestionService — pollDueSources() (day-cursor gate, SCHEMA-02 upsert change-detection, catch-up loop) + pruneExpiredTenders() (D-05 retention)
|
||||
- TenderSchedulerService — single global cron job 'tender-doe-poll' (poll-once-fan-out-many, INGEST-06)
|
||||
- Automated proof of the phase's headline acceptance criterion (Success Criteria 4 & 5): 2nd-tenant activation triggers zero additional DÖE calls/cron jobs/Tender rows
|
||||
affects: [10-05 controller/DTOs (consumes setInterval()/stopJob() on admin config save), phase 11 saved searches/filter UI (consumes the ingested Tender delta)]
|
||||
|
||||
# Tech tracking
|
||||
tech-stack:
|
||||
added: []
|
||||
patterns:
|
||||
- "Poll-once-fan-out-many scheduler: single named cron job, no tenant parameter anywhere in the class — the DKV findFirst()/activeTenantId single-tenant framing is explicitly NOT reused (Pitfall D)"
|
||||
- "Day-cursor gate lives in the ingestion service, not the scheduler — cron-tick frequency (D-04, admin-configurable) is decoupled from actual-fetch frequency (day-granularity, Pitfall A)"
|
||||
- "Singleton platform-wide config loaded via findUnique on a fixed slug (sourceType: 'doe-opendata'), never findFirst — self-documenting against future copy-paste into a per-tenant source"
|
||||
- "prisma.tender.upsert({ where: { dedupKey } }) as the SCHEMA-02 change-detection seam — identical notice never duplicates, changed contentHash updates in place"
|
||||
- "D-05 retention as two-phase updateMany/deleteMany with deadlineAt:{lt:...} filters — null-deadline rows are structurally excluded (SQL NULL comparisons are always false), not just by convention"
|
||||
- "In-memory prisma-shaped fake (Maps) for service-level tests — matches this repo's established convention (ldap.service.spec.ts) rather than a live DB connection"
|
||||
|
||||
key-files:
|
||||
created:
|
||||
- apps/api/src/tenders/tender-ingestion.service.ts
|
||||
- apps/api/src/tenders/tender-ingestion.service.spec.ts
|
||||
- apps/api/src/tenders/tender-scheduler.service.ts
|
||||
- apps/api/src/tenders/tender-scheduler.service.spec.ts
|
||||
modified:
|
||||
- apps/api/src/tenders/tenders.module.ts
|
||||
|
||||
key-decisions:
|
||||
- "Comment wording avoiding literal 'forTenant'/'activeTenantId'/'findFirst' tokens in the two new service files — both grep-gate assertions (task-level and spec-level) check for the literal string, and an explanatory comment mentioning the anti-pattern by name would itself trip the assertion (same class of issue documented as a Rule 1 fix in Plan 10-01)"
|
||||
- "TenderIngestionService.politeDelayMs is a protected, test-overridable field (not a hardcoded sleep) so the catch-up-loop unit test doesn't sleep for real while production still gets the ~1.5s politeness delay between successive day-fetches"
|
||||
- "Task 3's two-tenant test drives the REAL (unmocked) ModuleRegistryService against a fake prisma, not a stand-in activation function — this exercises the genuine activateForTenant() call path, making the 'zero additional calls/jobs/rows' assertion a true integration proof rather than a tautology"
|
||||
- "pruneExpiredTenders() is called once per successful pollDueSources() tick (i.e., only when at least one day was actually fetched) — not on the day-cursor no-op branch, so a no-op tick stays a true no-op with zero DB writes beyond the read"
|
||||
|
||||
requirements-completed: [SCHEMA-02, INGEST-06]
|
||||
|
||||
coverage:
|
||||
- id: D1
|
||||
description: "A changed DÖE notice (same dedupKey, new contentHash) updates the existing Tender row instead of creating a duplicate (SCHEMA-02)"
|
||||
requirement: "SCHEMA-02"
|
||||
verification:
|
||||
- kind: unit
|
||||
ref: "tender-ingestion.service.spec.ts — 'updates the existing row in place when the same dedupKey reappears with a changed contentHash' — asserts store size stays 1, contentHash/deadlineAt reflect the new version"
|
||||
status: pass
|
||||
- kind: unit
|
||||
ref: "tender-ingestion.service.spec.ts — 'inserts a fresh notice, and does NOT duplicate when the identical notice ... reappears' — asserts store size stays 1 across 2 catch-up-day fetches of the identical record"
|
||||
status: pass
|
||||
human_judgment: false
|
||||
- id: D2
|
||||
description: "DÖE is polled once on a shared global schedule regardless of tenant count — poll-once-fan-out-many, never per-tenant (INGEST-06)"
|
||||
requirement: "INGEST-06"
|
||||
verification:
|
||||
- kind: integration
|
||||
ref: "tender-scheduler.service.spec.ts — 2nd-tenant activation asserts addCronJob still called exactly once, pollDueSources never called by activation, tender.count() stays 0"
|
||||
status: pass
|
||||
- kind: automated_ui
|
||||
ref: "grep -c activeTenantId src/tenders/tender-scheduler.service.ts == 0; grep -c findFirst == 0; grep -c findUnique >= 1"
|
||||
status: pass
|
||||
human_judgment: false
|
||||
- id: D3
|
||||
description: "The poll tick is day-cursor gated: no upstream fetch when dayCursor >= today Europe/Berlin (D-01 from-now, no historical backfill)"
|
||||
verification:
|
||||
- kind: unit
|
||||
ref: "tender-ingestion.service.spec.ts — 'makes NO adapter call and returns when the day-cursor is not strictly before Berlin-today' + 'does nothing when ... config is not active'"
|
||||
status: pass
|
||||
human_judgment: false
|
||||
- id: D4
|
||||
description: "Tenders past their deadline are marked expired and pruned after 90 days; deadline-less rows are never auto-expired (D-05)"
|
||||
verification:
|
||||
- kind: unit
|
||||
ref: "tender-ingestion.service.spec.ts — pruneExpiredTenders test: active+past-deadline -> expired, expired+91d-old -> deleted, expired+30d-old -> retained, null-deadline -> untouched, active+future-deadline -> untouched"
|
||||
status: pass
|
||||
human_judgment: false
|
||||
- id: D5
|
||||
description: "Service uses the plain PrismaService — no tenant RLS extension on Tender/TenderSourcePollConfig (D-03, T-10-09)"
|
||||
verification:
|
||||
- kind: automated_ui
|
||||
ref: "tender-ingestion.service.spec.ts source-inspection test: grep-equivalent expect(source).not.toMatch(/forTenant/) on tender-ingestion.service.ts"
|
||||
status: pass
|
||||
human_judgment: false
|
||||
|
||||
# Metrics
|
||||
duration: ~25min
|
||||
completed: 2026-07-21
|
||||
status: complete
|
||||
---
|
||||
|
||||
# Phase 10 Plan 04: Ingestion Orchestration & Multi-Tenant-Safe Scheduler Summary
|
||||
|
||||
**`TenderIngestionService` (day-cursor gate, SCHEMA-02 upsert-based change detection, D-05 retention) and `TenderSchedulerService` (single global cron job, zero tenant dimension) built test-first, with an automated two-tenant integration test proving the phase's headline acceptance criterion: activating the module for a 2nd tenant triggers zero additional DÖE polls, cron jobs, or Tender rows.**
|
||||
|
||||
## Performance
|
||||
|
||||
- **Duration:** ~25 min
|
||||
- **Started:** 2026-07-21
|
||||
- **Completed:** 2026-07-21
|
||||
- **Tasks:** 3 (Task 1 TDD RED->GREEN, Task 2 auto, Task 3 TDD spec-only — passed immediately against Task 2's already-correct implementation)
|
||||
- **Files modified:** 5 (4 created, 1 modified)
|
||||
|
||||
## Accomplishments
|
||||
|
||||
- `TenderIngestionService.pollDueSources()`: loads the singleton `doe-opendata` config via `findUnique` on the fixed slug, applies the day-cursor gate (`nextDayToFetch()`, no-op when nothing new — Pitfall A), catches up across missed days with a politeness delay between fetches, and upserts each normalized notice by `dedupKey` — the SCHEMA-02 change-detection seam (identical notice never duplicates; changed `contentHash` updates the row in place).
|
||||
- `pruneExpiredTenders()` (D-05): marks past-deadline active rows `expired`, deletes `expired` rows older than 90 days, and structurally never touches `deadlineAt IS NULL` rows.
|
||||
- `TenderSchedulerService`: reuses `DkvSchedulerService`'s `CronJob`/`SchedulerRegistry` mechanics verbatim, but drops the per-tenant `activeTenantId` framing entirely — exactly one cron job (`tender-doe-poll`) for the whole platform, `setInterval()` takes no tenant argument.
|
||||
- Two-tenant safety integration test (`tender-scheduler.service.spec.ts`) drives the real `ModuleRegistryService.activateForTenant()` against a fake prisma and asserts, after a 2nd tenant activates the module: zero additional `addCronJob` calls, zero additional `pollDueSources()` invocations, zero additional `Tender` rows — the absence of tenant-scaled behavior, not the presence of correct per-tenant iteration.
|
||||
- Full API test suite green (69/69), `tsc --noEmit` clean, all grep gates (`activeTenantId`, `forTenant`, `findFirst`) return 0 in both new service files.
|
||||
|
||||
## Task Commits
|
||||
|
||||
Each task committed atomically, RED before GREEN where TDD applies:
|
||||
|
||||
1. **Task 1a: Failing spec — day-cursor gate, SCHEMA-02, D-05 (RED)** — `f9e52ab` (test)
|
||||
2. **Task 1b: TenderIngestionService implementation (GREEN)** — `6fb734b` (feat)
|
||||
3. **Task 2: TenderSchedulerService — single global cron (poll-once-fan-out-many)** — `0ba7410` (feat)
|
||||
4. **Task 3: Two-tenant safety integration test (Success Criteria 4 & 5)** — `444c68b` (test)
|
||||
|
||||
**Plan metadata:** see final `docs(10-04)` commit.
|
||||
|
||||
## TDD Gate Compliance
|
||||
|
||||
- Task 1 (`tdd="true"`): genuine RED->GREEN. `f9e52ab` (`test(10-04): ...`) failed at module-resolution time (`tender-ingestion.service.ts` did not exist yet), confirmed via `pnpm exec vitest run -- tender-ingestion` before any implementation existed. `6fb734b` (`feat(10-04): ...`) brought all 7 tests to green.
|
||||
- Task 3 (`tdd="true"`): the spec (`444c68b`, `test(10-04): ...`) passed on first run, immediately after being written — **this is expected, not an unexpected-pass violation of the RED-GREEN gate.** Task 2's `feat` commit already implemented the poll-once-fan-out-many invariant correctly (no tenant dimension exists anywhere in `TenderSchedulerService` to begin with); Task 3's test locks in and regression-proofs that already-correct architecture via a genuine two-tenant activation exercise, rather than driving new production code. The plan's own task sequencing (implement scheduler in Task 2, then prove the property in Task 3) anticipates this — it is not a full-plan `type: tdd` gate (this plan's frontmatter is `type: execute` with per-task `tdd="true"` flags), so the strict "investigate a passing RED" rule does not apply at the plan level here.
|
||||
- No REFACTOR commit was needed for either task — both implementations passed cleanly, no post-green cleanup required.
|
||||
|
||||
## Files Created/Modified
|
||||
|
||||
- `apps/api/src/tenders/tender-ingestion.service.ts` — `pollDueSources()` (day-cursor gate, catch-up loop, upsert change-detection) + `pruneExpiredTenders()` (D-05); exports `nextDayToFetch()` for direct testing
|
||||
- `apps/api/src/tenders/tender-ingestion.service.spec.ts` — 7 tests: 2 day-cursor gate, 2 SCHEMA-02 change-detection, 1 catch-up cursor advance, 1 D-05 retention (5 sub-assertions), 1 no-forTenant source check
|
||||
- `apps/api/src/tenders/tender-scheduler.service.ts` — single global cron job (`tender-doe-poll`), `setInterval()`/`stopJob()`, `onModuleInit()` via `findUnique` on fixed slug
|
||||
- `apps/api/src/tenders/tender-scheduler.service.spec.ts` — 3 tests: two-tenant safety integration proof, single-job/no-tenant-param check, source-level anti-pattern grep checks
|
||||
- `apps/api/src/tenders/tenders.module.ts` — `TenderIngestionService` + `TenderSchedulerService` added to `providers`
|
||||
|
||||
## Decisions Made
|
||||
|
||||
- **Comment wording avoids the literal `forTenant`/`activeTenantId`/`findFirst` tokens** in both new service files — the grep-gate assertions (task-level bash grep and spec-level source-inspection tests) check for the literal string, and an explanatory comment naming the anti-pattern directly would itself trip the assertion. This is the same class of issue Plan 10-01 documented as a Rule 1 fix; caught and fixed inline during this plan (see Deviations).
|
||||
- **`politeDelayMs` is a protected, test-overridable instance field**, not a hardcoded `setTimeout` call — production gets the ~1.5s politeness delay between successive catch-up day-fetches (RESEARCH Open Question 1), while the catch-up-loop unit test zeroes it out (`(service as any).politeDelayMs = 0`) to run instantly.
|
||||
- **Task 3's test drives the real, unmocked `ModuleRegistryService`** against a fake prisma rather than stubbing `activateForTenant()` directly — this makes the "zero additional calls/jobs/rows on 2nd activation" assertion a genuine integration proof of the actual activation call path, not a tautological check against a hand-written stand-in.
|
||||
- **`pruneExpiredTenders()` runs once per successful tick** (only when the day-cursor loop actually processed at least one day), not on the no-op branch — keeps a true no-op tick free of any DB write beyond the initial config read.
|
||||
- **In-memory prisma-shaped fakes (Maps)**, not a live DB connection, for both new spec files — matches this repo's established test convention (`ldap.service.spec.ts`) rather than introducing a new DB-integration-test pattern; the local Docker Postgres stack was available (per environment context) but no existing vitest precedent uses it, so the repo-consistent approach was chosen.
|
||||
|
||||
## Deviations from Plan
|
||||
|
||||
### Auto-fixed Issues
|
||||
|
||||
**1. [Rule 1 - Bug] Explanatory comment in `tender-ingestion.service.ts` contained the literal `forTenant` token**
|
||||
- **Found during:** Task 1 GREEN verification (first `pnpm exec vitest run -- tender-ingestion` after implementation)
|
||||
- **Issue:** The class-level doc comment explained the multi-tenant-safety rationale using the phrase "Never wraps ... in `forTenant()`" — this literal substring made the spec's `expect(source).not.toMatch(/forTenant/)` assertion fail, a false positive against the actual D-03 invariant (the code itself never calls `forTenant()`; only the comment mentioned it by name).
|
||||
- **Fix:** Reworded the comment to describe the same rationale without the literal token ("uses the plain, non-tenant-scoped PrismaService... Never wraps ... queries in the tenant RLS extension").
|
||||
- **Files modified:** `apps/api/src/tenders/tender-ingestion.service.ts`
|
||||
- **Verification:** `grep -c forTenant` → 0; `tsc --noEmit` exit 0; full spec re-run green.
|
||||
- **Committed in:** `6fb734b` (Task 1 GREEN commit, folded in before commit since caught during the same verification pass)
|
||||
|
||||
**2. [Rule 1 - Bug] Explanatory comments in `tender-scheduler.service.ts` contained the literal `activeTenantId` and `findFirst` tokens**
|
||||
- **Found during:** Task 2 verification (`grep -c activeTenantId` returned 1, not the required 0)
|
||||
- **Issue:** The class-level doc comment listed the anti-patterns being deliberately avoided using their literal names ("No `activeTenantId` field", "never `findFirst()`") — both tripped their respective grep-gate assertions the same way as Deviation 1, and would additionally have broken Task 3's planned source-inspection test if left uncorrected.
|
||||
- **Fix:** Reworded both mentions to describe the same design intent without the literal tokens ("No per-tenant 'active tenant id' instance field at all", "a fixed-slug lookup, not an unfiltered/ordered 'first match' query").
|
||||
- **Files modified:** `apps/api/src/tenders/tender-scheduler.service.ts`
|
||||
- **Verification:** `grep -c activeTenantId` → 0; `grep -c findFirst` → 0; `grep -c findUnique` → 2; `tsc --noEmit` exit 0.
|
||||
- **Committed in:** `0ba7410` (Task 2 commit, caught and fixed before commit)
|
||||
|
||||
---
|
||||
|
||||
**Total deviations:** 2 auto-fixed (2 Bugs, both comment-wording false-positives against grep-gate assertions — no production logic was ever incorrect)
|
||||
**Impact on plan:** Zero scope creep, zero behavior change. Both fixes are pure documentation-wording corrections needed to satisfy the plan's own literal-string acceptance criteria.
|
||||
|
||||
## Issues Encountered
|
||||
|
||||
None beyond the two documented deviations above. The local Docker Postgres stack (per the environment context) was not needed for this plan's tests — both new spec files use in-memory prisma-shaped fakes, matching this repo's established test convention, so no live-DB dependency was introduced.
|
||||
|
||||
## User Setup Required
|
||||
|
||||
None — no external service configuration required. The local Docker stack (`tessera-ctl-api-1`/`-db-1`) was left running unmodified; restarting/rebuilding it to pick up these code changes (so the scheduler actually starts polling against the live `oeffentlichevergabe.de` API) is left to the user per project convention.
|
||||
|
||||
## Next Phase Readiness
|
||||
|
||||
- `TenderIngestionService` and `TenderSchedulerService` are both implemented, tested, and registered in `TendersModule.providers` — the full ingestion pipeline (fetch -> normalize -> upsert -> schedule) is wired end-to-end for the first time this phase.
|
||||
- Plan 05 (controller/DTOs) can now wire `POST /tenders/source-config` to call `tenderScheduler.setInterval()`/`stopJob()` after an admin updates `pollIntervalMin`/`isActive` — both methods already exist with the exact signature the `PATTERNS.md` controller sketch expects.
|
||||
- No open threat-model items from this plan carry forward unmitigated: T-10-09 (RLS-exempt global-table access) and T-10-10 (per-tenant scheduler scaling) are both mitigated and proven by automated tests; T-10-11 (URL tampering via the day-cursor) and T-10-12 (catch-up-loop DoS against DÖE) are both structurally addressed (internally-computed cursor, politeness delay) with no admin-facing input path introduced.
|
||||
- No blockers for Plan 10-05.
|
||||
|
||||
## Self-Check: PASSED
|
||||
|
||||
- FOUND: apps/api/src/tenders/tender-ingestion.service.ts
|
||||
- FOUND: apps/api/src/tenders/tender-ingestion.service.spec.ts
|
||||
- FOUND: apps/api/src/tenders/tender-scheduler.service.ts
|
||||
- FOUND: apps/api/src/tenders/tender-scheduler.service.spec.ts
|
||||
- FOUND: apps/api/src/tenders/tenders.module.ts
|
||||
- FOUND commit: f9e52ab
|
||||
- FOUND commit: 6fb734b
|
||||
- FOUND commit: 0ba7410
|
||||
- FOUND commit: 444c68b
|
||||
|
||||
---
|
||||
*Phase: 10-ausschreibungs-radar-foundation-d-e-ingestion*
|
||||
*Completed: 2026-07-21*
|
||||
Reference in New Issue
Block a user