docs(10-04): complete ingestion orchestration & multi-tenant-safe scheduler plan
This commit is contained in:
@@ -15,13 +15,13 @@
|
|||||||
- [ ] **INGEST-03**: Das System importiert Ausschreibungen vom cosinex-Vergabemarktplatz (DTVP) über einen Adapter.
|
- [ ] **INGEST-03**: Das System importiert Ausschreibungen vom cosinex-Vergabemarktplatz (DTVP) über einen Adapter.
|
||||||
- [ ] **INGEST-04**: Das System importiert Ausschreibungen aus RSS-Feeds (subreport-elvis, service.bund.de).
|
- [ ] **INGEST-04**: Das System importiert Ausschreibungen aus RSS-Feeds (subreport-elvis, service.bund.de).
|
||||||
- [ ] **INGEST-05**: Das System liest Portal-Benachrichtigungs-E-Mails aus einem konfigurierten Postfach ein (nutzt bestehende DKV-Inbox-Infrastruktur) und extrahiert daraus Ausschreibungen.
|
- [ ] **INGEST-05**: Das System liest Portal-Benachrichtigungs-E-Mails aus einem konfigurierten Postfach ein (nutzt bestehende DKV-Inbox-Infrastruktur) und extrahiert daraus Ausschreibungen.
|
||||||
- [ ] **INGEST-06**: Jede Quelle wird einmal zentral pro Zeitplan abgefragt (poll-once-fan-out-many), Intervall pro Quelle im Admin-Bereich konfigurierbar; das Ergebnis wird an alle passenden Mandanten-Suchprofile verteilt.
|
- [x] **INGEST-06**: Jede Quelle wird einmal zentral pro Zeitplan abgefragt (poll-once-fan-out-many), Intervall pro Quelle im Admin-Bereich konfigurierbar; das Ergebnis wird an alle passenden Mandanten-Suchprofile verteilt.
|
||||||
- [ ] **INGEST-07**: vergabe24 und aumass sind als harte Denylist hinterlegt und können nicht als automatische Scraping-Quelle registriert werden (AGB-Verbot).
|
- [ ] **INGEST-07**: vergabe24 und aumass sind als harte Denylist hinterlegt und können nicht als automatische Scraping-Quelle registriert werden (AGB-Verbot).
|
||||||
|
|
||||||
### SCHEMA — Normalisierung & Deduplizierung
|
### SCHEMA — Normalisierung & Deduplizierung
|
||||||
|
|
||||||
- [x] **SCHEMA-01**: Alle Quellen werden in ein einheitliches, OCDS-orientiertes Ausschreibungs-Schema normalisiert (Titel, Auftraggeber, CPV-Codes, Region/PLZ, Frist, geschätzter Wert, Verfahrensart, Quell-URL, Rohdaten). Ausschreibungsdaten sind plattform-global, nicht mandantengebunden.
|
- [x] **SCHEMA-01**: Alle Quellen werden in ein einheitliches, OCDS-orientiertes Ausschreibungs-Schema normalisiert (Titel, Auftraggeber, CPV-Codes, Region/PLZ, Frist, geschätzter Wert, Verfahrensart, Quell-URL, Rohdaten). Ausschreibungsdaten sind plattform-global, nicht mandantengebunden.
|
||||||
- [ ] **SCHEMA-02**: Das System erkennt Änderungen an bereits importierten Ausschreibungen (Fristverlängerung, Aufhebung) über einen Content-Hash und aktualisiert den Datensatz.
|
- [x] **SCHEMA-02**: Das System erkennt Änderungen an bereits importierten Ausschreibungen (Fristverlängerung, Aufhebung) über einen Content-Hash und aktualisiert den Datensatz.
|
||||||
- [ ] **SCHEMA-03**: Dieselbe Ausschreibung aus mehreren Quellen wird zu einem Eintrag mit mehreren Quell-Links dedupliziert (Schlüssel: OCID → Quelle:NoticeId → Fuzzy-Fingerprint aus Auftraggeber+Titel+CPV+Frist+Wert). Dedup greift erst ab der zweiten aktiven Quelle.
|
- [ ] **SCHEMA-03**: Dieselbe Ausschreibung aus mehreren Quellen wird zu einem Eintrag mit mehreren Quell-Links dedupliziert (Schlüssel: OCID → Quelle:NoticeId → Fuzzy-Fingerprint aus Auftraggeber+Titel+CPV+Frist+Wert). Dedup greift erst ab der zweiten aktiven Quelle.
|
||||||
|
|
||||||
### FILTER — Suche & Profile
|
### FILTER — Suche & Profile
|
||||||
@@ -78,9 +78,9 @@
|
|||||||
|-------------|-------|--------|
|
|-------------|-------|--------|
|
||||||
| CONFIG-01 | Phase 10 | Complete |
|
| CONFIG-01 | Phase 10 | Complete |
|
||||||
| INGEST-01 | Phase 10 | Complete |
|
| INGEST-01 | Phase 10 | Complete |
|
||||||
| INGEST-06 | Phase 10 | Pending |
|
| INGEST-06 | Phase 10 | Complete |
|
||||||
| SCHEMA-01 | Phase 10 | Complete |
|
| SCHEMA-01 | Phase 10 | Complete |
|
||||||
| SCHEMA-02 | Phase 10 | Pending |
|
| SCHEMA-02 | Phase 10 | Complete |
|
||||||
| FILTER-01 | Phase 11 | Pending |
|
| FILTER-01 | Phase 11 | Pending |
|
||||||
| FILTER-02 | Phase 11 | Pending |
|
| FILTER-02 | Phase 11 | Pending |
|
||||||
| FILTER-03 | Phase 11 | Pending |
|
| FILTER-03 | Phase 11 | Pending |
|
||||||
|
|||||||
@@ -344,7 +344,7 @@ Plans:
|
|||||||
4. DÖE is polled once on a shared, admin-configurable interval regardless of how many tenants have the module active -- never once per tenant (poll-once-fan-out-many, not the DKV single-tenant `findFirst()` pattern)
|
4. DÖE is polled once on a shared, admin-configurable interval regardless of how many tenants have the module active -- never once per tenant (poll-once-fan-out-many, not the DKV single-tenant `findFirst()` pattern)
|
||||||
5. Activating the module for a second tenant does not duplicate ingestion, re-trigger a redundant DÖE poll, or interfere with the first tenant's data
|
5. Activating the module for a second tenant does not duplicate ingestion, re-trigger a redundant DÖE poll, or interfere with the first tenant's data
|
||||||
|
|
||||||
**Plans**: 3/6 plans executed
|
**Plans**: 4/6 plans executed
|
||||||
|
|
||||||
**Wave 1**
|
**Wave 1**
|
||||||
|
|
||||||
@@ -360,7 +360,7 @@ Plans:
|
|||||||
|
|
||||||
**Wave 4** *(blocked on Wave 3 completion)*
|
**Wave 4** *(blocked on Wave 3 completion)*
|
||||||
|
|
||||||
- [ ] 10-04-PLAN.md — Ingestion + shared scheduler: day-cursor gate, upsert change-detection, single global cron (poll-once-fan-out-many), 90-day retention, two-tenant safety test (SCHEMA-02, INGEST-06)
|
- [x] 10-04-PLAN.md — Ingestion + shared scheduler: day-cursor gate, upsert change-detection, single global cron (poll-once-fan-out-many), 90-day retention, two-tenant safety test (SCHEMA-02, INGEST-06)
|
||||||
|
|
||||||
**Wave 5** *(blocked on Wave 4 completion)*
|
**Wave 5** *(blocked on Wave 4 completion)*
|
||||||
|
|
||||||
@@ -453,7 +453,7 @@ Phases execute in numeric order: 1 -> 2 -> 3 -> 4 -> 5 -> 6 -> 7 -> 8 -> 9 -> 10
|
|||||||
| 7. DKV Fleet Module | 6/6 | Complete | 2026-06-27 |
|
| 7. DKV Fleet Module | 6/6 | Complete | 2026-06-27 |
|
||||||
| 8. Dashboard Widgets Vollimplementierung | 4/4 | Complete | 2026-07-01 |
|
| 8. Dashboard Widgets Vollimplementierung | 4/4 | Complete | 2026-07-01 |
|
||||||
| 9. Cert Manager Module | 6/6 | Complete | 2026-07-02 |
|
| 9. Cert Manager Module | 6/6 | Complete | 2026-07-02 |
|
||||||
| 10. Ausschreibungs-Radar Foundation & DÖE Ingestion | 3/6 | In Progress| |
|
| 10. Ausschreibungs-Radar Foundation & DÖE Ingestion | 4/6 | In Progress| |
|
||||||
| 11. Filter Engine, Results UI & Saved Searches | 0/TBD | Not started | - |
|
| 11. Filter Engine, Results UI & Saved Searches | 0/TBD | Not started | - |
|
||||||
| 12. Tender Notifications | 0/TBD | Not started | - |
|
| 12. Tender Notifications | 0/TBD | Not started | - |
|
||||||
| 13. Scraping Adapters & Cross-Source Deduplication | 0/TBD | Not started | - |
|
| 13. Scraping Adapters & Cross-Source Deduplication | 0/TBD | Not started | - |
|
||||||
|
|||||||
+8
-4
@@ -6,14 +6,14 @@ current_phase: 10
|
|||||||
current_phase_name: ausschreibungs-radar-foundation-d-e-ingestion
|
current_phase_name: ausschreibungs-radar-foundation-d-e-ingestion
|
||||||
status: executing
|
status: executing
|
||||||
stopped_at: Completed 10-02-PLAN.md
|
stopped_at: Completed 10-02-PLAN.md
|
||||||
last_updated: "2026-07-21T09:01:03.926Z"
|
last_updated: "2026-07-21T09:13:27.792Z"
|
||||||
last_activity: 2026-07-21
|
last_activity: 2026-07-21
|
||||||
last_activity_desc: Phase 10 execution started
|
last_activity_desc: Phase 10 execution started
|
||||||
progress:
|
progress:
|
||||||
total_phases: 14
|
total_phases: 14
|
||||||
completed_phases: 8
|
completed_phases: 8
|
||||||
total_plans: 46
|
total_plans: 46
|
||||||
completed_plans: 42
|
completed_plans: 43
|
||||||
percent: 57
|
percent: 57
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -29,7 +29,7 @@ See: .planning/PROJECT.md (updated 2026-07-17)
|
|||||||
## Current Position
|
## Current Position
|
||||||
|
|
||||||
Phase: 10 (ausschreibungs-radar-foundation-d-e-ingestion) — EXECUTING
|
Phase: 10 (ausschreibungs-radar-foundation-d-e-ingestion) — EXECUTING
|
||||||
Plan: 4 of 6
|
Plan: 5 of 6
|
||||||
Status: Ready to execute
|
Status: Ready to execute
|
||||||
Last activity: 2026-07-21 — Phase 10 execution started
|
Last activity: 2026-07-21 — Phase 10 execution started
|
||||||
|
|
||||||
@@ -74,6 +74,7 @@ Progress: [░░░░░░░░░░] 0%
|
|||||||
| Phase 10 P01 | 15min | 3 tasks | 4 files |
|
| Phase 10 P01 | 15min | 3 tasks | 4 files |
|
||||||
| Phase 10 P02 | 20min | 3 tasks | 6 files |
|
| Phase 10 P02 | 20min | 3 tasks | 6 files |
|
||||||
| Phase 10 P03 | 35min | 3 tasks | 9 files |
|
| Phase 10 P03 | 35min | 3 tasks | 9 files |
|
||||||
|
| Phase 10 P04 | 25min | 3 tasks | 5 files |
|
||||||
|
|
||||||
## Accumulated Context
|
## Accumulated Context
|
||||||
|
|
||||||
@@ -149,6 +150,9 @@ Recent decisions affecting current work:
|
|||||||
- [Phase 10-03]: sourceNoticeId = OCDS release.id (stable, no version suffix), not the zip entry filename
|
- [Phase 10-03]: sourceNoticeId = OCDS release.id (stable, no version suffix), not the zip entry filename
|
||||||
- [Phase 10-03]: eForms-DE XML primary for deadlineAt/estimatedValue/procedureType; OCDS primary for ocid/buyerName/title/cpvCodes/region/plz
|
- [Phase 10-03]: eForms-DE XML primary for deadlineAt/estimatedValue/procedureType; OCDS primary for ocid/buyerName/title/cpvCodes/region/plz
|
||||||
- [Phase 10-03]: bundesland left null this plan — NUTS-to-Bundesland mapping deferred to Phase 11 filter UI
|
- [Phase 10-03]: bundesland left null this plan — NUTS-to-Bundesland mapping deferred to Phase 11 filter UI
|
||||||
|
- [Phase ?]: Poll-once-fan-out-many scheduler: single named cron job, no tenant parameter — deliberately drops DKV's activeTenantId/findFirst per-tenant framing (Pitfall D)
|
||||||
|
- [Phase ?]: SCHEMA-02 change detection implemented via prisma.tender.upsert({ where: { dedupKey } }) — identical notice never duplicates, changed contentHash updates in place
|
||||||
|
- [Phase ?]: D-05 retention as two-phase updateMany/deleteMany with deadlineAt:{lt} filters — null-deadline rows structurally excluded, never auto-expired
|
||||||
|
|
||||||
### Pending Todos
|
### Pending Todos
|
||||||
|
|
||||||
@@ -186,7 +190,7 @@ Items acknowledged and carried forward from previous milestone close:
|
|||||||
|
|
||||||
## Session Continuity
|
## Session Continuity
|
||||||
|
|
||||||
Last session: 2026-07-21T09:00:23.857Z
|
Last session: 2026-07-21T09:12:58.508Z
|
||||||
Stopped at: Completed 10-02-PLAN.md
|
Stopped at: Completed 10-02-PLAN.md
|
||||||
Resume file: None
|
Resume file: None
|
||||||
Last activity: 2026-07-14 - Built LDAP per-user exclude/denylist filter (9d1323f), migration applied on live DB, verified via Playwright: sync deactivated 4 excluded service accounts (administrator/krbtgt/guest/dns-ldap), 2 real LDAP users stay active, 0 wrongly created
|
Last activity: 2026-07-14 - Built LDAP per-user exclude/denylist filter (9d1323f), migration applied on live DB, verified via Playwright: sync deactivated 4 excluded service accounts (administrator/krbtgt/guest/dns-ldap), 2 real LDAP users stay active, 0 wrongly created
|
||||||
|
|||||||
@@ -0,0 +1,207 @@
|
|||||||
|
---
|
||||||
|
phase: 10-ausschreibungs-radar-foundation-d-e-ingestion
|
||||||
|
plan: 04
|
||||||
|
subsystem: api
|
||||||
|
tags: [scheduler, ingestion, multi-tenant-safety, cron, scheduler-registry, tdd, poll-once-fan-out-many]
|
||||||
|
|
||||||
|
# Dependency graph
|
||||||
|
requires:
|
||||||
|
- phase: 10-01 (foundation & dependencies)
|
||||||
|
provides: Tender / TenderSourcePollConfig Prisma models (dedupKey @unique, sourceType @unique)
|
||||||
|
- phase: 10-02 (marketplace registration)
|
||||||
|
provides: TendersModule skeleton, singleton doe-opendata poll config seeded on boot
|
||||||
|
- phase: 10-03 (adapter & normalizer)
|
||||||
|
provides: DoeOpenDataAdapter.fetchTenders(dayCursor), TenderNormalizerService.normalize()
|
||||||
|
provides:
|
||||||
|
- TenderIngestionService — pollDueSources() (day-cursor gate, SCHEMA-02 upsert change-detection, catch-up loop) + pruneExpiredTenders() (D-05 retention)
|
||||||
|
- TenderSchedulerService — single global cron job 'tender-doe-poll' (poll-once-fan-out-many, INGEST-06)
|
||||||
|
- Automated proof of the phase's headline acceptance criterion (Success Criteria 4 & 5): 2nd-tenant activation triggers zero additional DÖE calls/cron jobs/Tender rows
|
||||||
|
affects: [10-05 controller/DTOs (consumes setInterval()/stopJob() on admin config save), phase 11 saved searches/filter UI (consumes the ingested Tender delta)]
|
||||||
|
|
||||||
|
# Tech tracking
|
||||||
|
tech-stack:
|
||||||
|
added: []
|
||||||
|
patterns:
|
||||||
|
- "Poll-once-fan-out-many scheduler: single named cron job, no tenant parameter anywhere in the class — the DKV findFirst()/activeTenantId single-tenant framing is explicitly NOT reused (Pitfall D)"
|
||||||
|
- "Day-cursor gate lives in the ingestion service, not the scheduler — cron-tick frequency (D-04, admin-configurable) is decoupled from actual-fetch frequency (day-granularity, Pitfall A)"
|
||||||
|
- "Singleton platform-wide config loaded via findUnique on a fixed slug (sourceType: 'doe-opendata'), never findFirst — self-documenting against future copy-paste into a per-tenant source"
|
||||||
|
- "prisma.tender.upsert({ where: { dedupKey } }) as the SCHEMA-02 change-detection seam — identical notice never duplicates, changed contentHash updates in place"
|
||||||
|
- "D-05 retention as two-phase updateMany/deleteMany with deadlineAt:{lt:...} filters — null-deadline rows are structurally excluded (SQL NULL comparisons are always false), not just by convention"
|
||||||
|
- "In-memory prisma-shaped fake (Maps) for service-level tests — matches this repo's established convention (ldap.service.spec.ts) rather than a live DB connection"
|
||||||
|
|
||||||
|
key-files:
|
||||||
|
created:
|
||||||
|
- apps/api/src/tenders/tender-ingestion.service.ts
|
||||||
|
- apps/api/src/tenders/tender-ingestion.service.spec.ts
|
||||||
|
- apps/api/src/tenders/tender-scheduler.service.ts
|
||||||
|
- apps/api/src/tenders/tender-scheduler.service.spec.ts
|
||||||
|
modified:
|
||||||
|
- apps/api/src/tenders/tenders.module.ts
|
||||||
|
|
||||||
|
key-decisions:
|
||||||
|
- "Comment wording avoiding literal 'forTenant'/'activeTenantId'/'findFirst' tokens in the two new service files — both grep-gate assertions (task-level and spec-level) check for the literal string, and an explanatory comment mentioning the anti-pattern by name would itself trip the assertion (same class of issue documented as a Rule 1 fix in Plan 10-01)"
|
||||||
|
- "TenderIngestionService.politeDelayMs is a protected, test-overridable field (not a hardcoded sleep) so the catch-up-loop unit test doesn't sleep for real while production still gets the ~1.5s politeness delay between successive day-fetches"
|
||||||
|
- "Task 3's two-tenant test drives the REAL (unmocked) ModuleRegistryService against a fake prisma, not a stand-in activation function — this exercises the genuine activateForTenant() call path, making the 'zero additional calls/jobs/rows' assertion a true integration proof rather than a tautology"
|
||||||
|
- "pruneExpiredTenders() is called once per successful pollDueSources() tick (i.e., only when at least one day was actually fetched) — not on the day-cursor no-op branch, so a no-op tick stays a true no-op with zero DB writes beyond the read"
|
||||||
|
|
||||||
|
requirements-completed: [SCHEMA-02, INGEST-06]
|
||||||
|
|
||||||
|
coverage:
|
||||||
|
- id: D1
|
||||||
|
description: "A changed DÖE notice (same dedupKey, new contentHash) updates the existing Tender row instead of creating a duplicate (SCHEMA-02)"
|
||||||
|
requirement: "SCHEMA-02"
|
||||||
|
verification:
|
||||||
|
- kind: unit
|
||||||
|
ref: "tender-ingestion.service.spec.ts — 'updates the existing row in place when the same dedupKey reappears with a changed contentHash' — asserts store size stays 1, contentHash/deadlineAt reflect the new version"
|
||||||
|
status: pass
|
||||||
|
- kind: unit
|
||||||
|
ref: "tender-ingestion.service.spec.ts — 'inserts a fresh notice, and does NOT duplicate when the identical notice ... reappears' — asserts store size stays 1 across 2 catch-up-day fetches of the identical record"
|
||||||
|
status: pass
|
||||||
|
human_judgment: false
|
||||||
|
- id: D2
|
||||||
|
description: "DÖE is polled once on a shared global schedule regardless of tenant count — poll-once-fan-out-many, never per-tenant (INGEST-06)"
|
||||||
|
requirement: "INGEST-06"
|
||||||
|
verification:
|
||||||
|
- kind: integration
|
||||||
|
ref: "tender-scheduler.service.spec.ts — 2nd-tenant activation asserts addCronJob still called exactly once, pollDueSources never called by activation, tender.count() stays 0"
|
||||||
|
status: pass
|
||||||
|
- kind: automated_ui
|
||||||
|
ref: "grep -c activeTenantId src/tenders/tender-scheduler.service.ts == 0; grep -c findFirst == 0; grep -c findUnique >= 1"
|
||||||
|
status: pass
|
||||||
|
human_judgment: false
|
||||||
|
- id: D3
|
||||||
|
description: "The poll tick is day-cursor gated: no upstream fetch when dayCursor >= today Europe/Berlin (D-01 from-now, no historical backfill)"
|
||||||
|
verification:
|
||||||
|
- kind: unit
|
||||||
|
ref: "tender-ingestion.service.spec.ts — 'makes NO adapter call and returns when the day-cursor is not strictly before Berlin-today' + 'does nothing when ... config is not active'"
|
||||||
|
status: pass
|
||||||
|
human_judgment: false
|
||||||
|
- id: D4
|
||||||
|
description: "Tenders past their deadline are marked expired and pruned after 90 days; deadline-less rows are never auto-expired (D-05)"
|
||||||
|
verification:
|
||||||
|
- kind: unit
|
||||||
|
ref: "tender-ingestion.service.spec.ts — pruneExpiredTenders test: active+past-deadline -> expired, expired+91d-old -> deleted, expired+30d-old -> retained, null-deadline -> untouched, active+future-deadline -> untouched"
|
||||||
|
status: pass
|
||||||
|
human_judgment: false
|
||||||
|
- id: D5
|
||||||
|
description: "Service uses the plain PrismaService — no tenant RLS extension on Tender/TenderSourcePollConfig (D-03, T-10-09)"
|
||||||
|
verification:
|
||||||
|
- kind: automated_ui
|
||||||
|
ref: "tender-ingestion.service.spec.ts source-inspection test: grep-equivalent expect(source).not.toMatch(/forTenant/) on tender-ingestion.service.ts"
|
||||||
|
status: pass
|
||||||
|
human_judgment: false
|
||||||
|
|
||||||
|
# Metrics
|
||||||
|
duration: ~25min
|
||||||
|
completed: 2026-07-21
|
||||||
|
status: complete
|
||||||
|
---
|
||||||
|
|
||||||
|
# Phase 10 Plan 04: Ingestion Orchestration & Multi-Tenant-Safe Scheduler Summary
|
||||||
|
|
||||||
|
**`TenderIngestionService` (day-cursor gate, SCHEMA-02 upsert-based change detection, D-05 retention) and `TenderSchedulerService` (single global cron job, zero tenant dimension) built test-first, with an automated two-tenant integration test proving the phase's headline acceptance criterion: activating the module for a 2nd tenant triggers zero additional DÖE polls, cron jobs, or Tender rows.**
|
||||||
|
|
||||||
|
## Performance
|
||||||
|
|
||||||
|
- **Duration:** ~25 min
|
||||||
|
- **Started:** 2026-07-21
|
||||||
|
- **Completed:** 2026-07-21
|
||||||
|
- **Tasks:** 3 (Task 1 TDD RED->GREEN, Task 2 auto, Task 3 TDD spec-only — passed immediately against Task 2's already-correct implementation)
|
||||||
|
- **Files modified:** 5 (4 created, 1 modified)
|
||||||
|
|
||||||
|
## Accomplishments
|
||||||
|
|
||||||
|
- `TenderIngestionService.pollDueSources()`: loads the singleton `doe-opendata` config via `findUnique` on the fixed slug, applies the day-cursor gate (`nextDayToFetch()`, no-op when nothing new — Pitfall A), catches up across missed days with a politeness delay between fetches, and upserts each normalized notice by `dedupKey` — the SCHEMA-02 change-detection seam (identical notice never duplicates; changed `contentHash` updates the row in place).
|
||||||
|
- `pruneExpiredTenders()` (D-05): marks past-deadline active rows `expired`, deletes `expired` rows older than 90 days, and structurally never touches `deadlineAt IS NULL` rows.
|
||||||
|
- `TenderSchedulerService`: reuses `DkvSchedulerService`'s `CronJob`/`SchedulerRegistry` mechanics verbatim, but drops the per-tenant `activeTenantId` framing entirely — exactly one cron job (`tender-doe-poll`) for the whole platform, `setInterval()` takes no tenant argument.
|
||||||
|
- Two-tenant safety integration test (`tender-scheduler.service.spec.ts`) drives the real `ModuleRegistryService.activateForTenant()` against a fake prisma and asserts, after a 2nd tenant activates the module: zero additional `addCronJob` calls, zero additional `pollDueSources()` invocations, zero additional `Tender` rows — the absence of tenant-scaled behavior, not the presence of correct per-tenant iteration.
|
||||||
|
- Full API test suite green (69/69), `tsc --noEmit` clean, all grep gates (`activeTenantId`, `forTenant`, `findFirst`) return 0 in both new service files.
|
||||||
|
|
||||||
|
## Task Commits
|
||||||
|
|
||||||
|
Each task committed atomically, RED before GREEN where TDD applies:
|
||||||
|
|
||||||
|
1. **Task 1a: Failing spec — day-cursor gate, SCHEMA-02, D-05 (RED)** — `f9e52ab` (test)
|
||||||
|
2. **Task 1b: TenderIngestionService implementation (GREEN)** — `6fb734b` (feat)
|
||||||
|
3. **Task 2: TenderSchedulerService — single global cron (poll-once-fan-out-many)** — `0ba7410` (feat)
|
||||||
|
4. **Task 3: Two-tenant safety integration test (Success Criteria 4 & 5)** — `444c68b` (test)
|
||||||
|
|
||||||
|
**Plan metadata:** see final `docs(10-04)` commit.
|
||||||
|
|
||||||
|
## TDD Gate Compliance
|
||||||
|
|
||||||
|
- Task 1 (`tdd="true"`): genuine RED->GREEN. `f9e52ab` (`test(10-04): ...`) failed at module-resolution time (`tender-ingestion.service.ts` did not exist yet), confirmed via `pnpm exec vitest run -- tender-ingestion` before any implementation existed. `6fb734b` (`feat(10-04): ...`) brought all 7 tests to green.
|
||||||
|
- Task 3 (`tdd="true"`): the spec (`444c68b`, `test(10-04): ...`) passed on first run, immediately after being written — **this is expected, not an unexpected-pass violation of the RED-GREEN gate.** Task 2's `feat` commit already implemented the poll-once-fan-out-many invariant correctly (no tenant dimension exists anywhere in `TenderSchedulerService` to begin with); Task 3's test locks in and regression-proofs that already-correct architecture via a genuine two-tenant activation exercise, rather than driving new production code. The plan's own task sequencing (implement scheduler in Task 2, then prove the property in Task 3) anticipates this — it is not a full-plan `type: tdd` gate (this plan's frontmatter is `type: execute` with per-task `tdd="true"` flags), so the strict "investigate a passing RED" rule does not apply at the plan level here.
|
||||||
|
- No REFACTOR commit was needed for either task — both implementations passed cleanly, no post-green cleanup required.
|
||||||
|
|
||||||
|
## Files Created/Modified
|
||||||
|
|
||||||
|
- `apps/api/src/tenders/tender-ingestion.service.ts` — `pollDueSources()` (day-cursor gate, catch-up loop, upsert change-detection) + `pruneExpiredTenders()` (D-05); exports `nextDayToFetch()` for direct testing
|
||||||
|
- `apps/api/src/tenders/tender-ingestion.service.spec.ts` — 7 tests: 2 day-cursor gate, 2 SCHEMA-02 change-detection, 1 catch-up cursor advance, 1 D-05 retention (5 sub-assertions), 1 no-forTenant source check
|
||||||
|
- `apps/api/src/tenders/tender-scheduler.service.ts` — single global cron job (`tender-doe-poll`), `setInterval()`/`stopJob()`, `onModuleInit()` via `findUnique` on fixed slug
|
||||||
|
- `apps/api/src/tenders/tender-scheduler.service.spec.ts` — 3 tests: two-tenant safety integration proof, single-job/no-tenant-param check, source-level anti-pattern grep checks
|
||||||
|
- `apps/api/src/tenders/tenders.module.ts` — `TenderIngestionService` + `TenderSchedulerService` added to `providers`
|
||||||
|
|
||||||
|
## Decisions Made
|
||||||
|
|
||||||
|
- **Comment wording avoids the literal `forTenant`/`activeTenantId`/`findFirst` tokens** in both new service files — the grep-gate assertions (task-level bash grep and spec-level source-inspection tests) check for the literal string, and an explanatory comment naming the anti-pattern directly would itself trip the assertion. This is the same class of issue Plan 10-01 documented as a Rule 1 fix; caught and fixed inline during this plan (see Deviations).
|
||||||
|
- **`politeDelayMs` is a protected, test-overridable instance field**, not a hardcoded `setTimeout` call — production gets the ~1.5s politeness delay between successive catch-up day-fetches (RESEARCH Open Question 1), while the catch-up-loop unit test zeroes it out (`(service as any).politeDelayMs = 0`) to run instantly.
|
||||||
|
- **Task 3's test drives the real, unmocked `ModuleRegistryService`** against a fake prisma rather than stubbing `activateForTenant()` directly — this makes the "zero additional calls/jobs/rows on 2nd activation" assertion a genuine integration proof of the actual activation call path, not a tautological check against a hand-written stand-in.
|
||||||
|
- **`pruneExpiredTenders()` runs once per successful tick** (only when the day-cursor loop actually processed at least one day), not on the no-op branch — keeps a true no-op tick free of any DB write beyond the initial config read.
|
||||||
|
- **In-memory prisma-shaped fakes (Maps)**, not a live DB connection, for both new spec files — matches this repo's established test convention (`ldap.service.spec.ts`) rather than introducing a new DB-integration-test pattern; the local Docker Postgres stack was available (per environment context) but no existing vitest precedent uses it, so the repo-consistent approach was chosen.
|
||||||
|
|
||||||
|
## Deviations from Plan
|
||||||
|
|
||||||
|
### Auto-fixed Issues
|
||||||
|
|
||||||
|
**1. [Rule 1 - Bug] Explanatory comment in `tender-ingestion.service.ts` contained the literal `forTenant` token**
|
||||||
|
- **Found during:** Task 1 GREEN verification (first `pnpm exec vitest run -- tender-ingestion` after implementation)
|
||||||
|
- **Issue:** The class-level doc comment explained the multi-tenant-safety rationale using the phrase "Never wraps ... in `forTenant()`" — this literal substring made the spec's `expect(source).not.toMatch(/forTenant/)` assertion fail, a false positive against the actual D-03 invariant (the code itself never calls `forTenant()`; only the comment mentioned it by name).
|
||||||
|
- **Fix:** Reworded the comment to describe the same rationale without the literal token ("uses the plain, non-tenant-scoped PrismaService... Never wraps ... queries in the tenant RLS extension").
|
||||||
|
- **Files modified:** `apps/api/src/tenders/tender-ingestion.service.ts`
|
||||||
|
- **Verification:** `grep -c forTenant` → 0; `tsc --noEmit` exit 0; full spec re-run green.
|
||||||
|
- **Committed in:** `6fb734b` (Task 1 GREEN commit, folded in before commit since caught during the same verification pass)
|
||||||
|
|
||||||
|
**2. [Rule 1 - Bug] Explanatory comments in `tender-scheduler.service.ts` contained the literal `activeTenantId` and `findFirst` tokens**
|
||||||
|
- **Found during:** Task 2 verification (`grep -c activeTenantId` returned 1, not the required 0)
|
||||||
|
- **Issue:** The class-level doc comment listed the anti-patterns being deliberately avoided using their literal names ("No `activeTenantId` field", "never `findFirst()`") — both tripped their respective grep-gate assertions the same way as Deviation 1, and would additionally have broken Task 3's planned source-inspection test if left uncorrected.
|
||||||
|
- **Fix:** Reworded both mentions to describe the same design intent without the literal tokens ("No per-tenant 'active tenant id' instance field at all", "a fixed-slug lookup, not an unfiltered/ordered 'first match' query").
|
||||||
|
- **Files modified:** `apps/api/src/tenders/tender-scheduler.service.ts`
|
||||||
|
- **Verification:** `grep -c activeTenantId` → 0; `grep -c findFirst` → 0; `grep -c findUnique` → 2; `tsc --noEmit` exit 0.
|
||||||
|
- **Committed in:** `0ba7410` (Task 2 commit, caught and fixed before commit)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
**Total deviations:** 2 auto-fixed (2 Bugs, both comment-wording false-positives against grep-gate assertions — no production logic was ever incorrect)
|
||||||
|
**Impact on plan:** Zero scope creep, zero behavior change. Both fixes are pure documentation-wording corrections needed to satisfy the plan's own literal-string acceptance criteria.
|
||||||
|
|
||||||
|
## Issues Encountered
|
||||||
|
|
||||||
|
None beyond the two documented deviations above. The local Docker Postgres stack (per the environment context) was not needed for this plan's tests — both new spec files use in-memory prisma-shaped fakes, matching this repo's established test convention, so no live-DB dependency was introduced.
|
||||||
|
|
||||||
|
## User Setup Required
|
||||||
|
|
||||||
|
None — no external service configuration required. The local Docker stack (`tessera-ctl-api-1`/`-db-1`) was left running unmodified; restarting/rebuilding it to pick up these code changes (so the scheduler actually starts polling against the live `oeffentlichevergabe.de` API) is left to the user per project convention.
|
||||||
|
|
||||||
|
## Next Phase Readiness
|
||||||
|
|
||||||
|
- `TenderIngestionService` and `TenderSchedulerService` are both implemented, tested, and registered in `TendersModule.providers` — the full ingestion pipeline (fetch -> normalize -> upsert -> schedule) is wired end-to-end for the first time this phase.
|
||||||
|
- Plan 05 (controller/DTOs) can now wire `POST /tenders/source-config` to call `tenderScheduler.setInterval()`/`stopJob()` after an admin updates `pollIntervalMin`/`isActive` — both methods already exist with the exact signature the `PATTERNS.md` controller sketch expects.
|
||||||
|
- No open threat-model items from this plan carry forward unmitigated: T-10-09 (RLS-exempt global-table access) and T-10-10 (per-tenant scheduler scaling) are both mitigated and proven by automated tests; T-10-11 (URL tampering via the day-cursor) and T-10-12 (catch-up-loop DoS against DÖE) are both structurally addressed (internally-computed cursor, politeness delay) with no admin-facing input path introduced.
|
||||||
|
- No blockers for Plan 10-05.
|
||||||
|
|
||||||
|
## Self-Check: PASSED
|
||||||
|
|
||||||
|
- FOUND: apps/api/src/tenders/tender-ingestion.service.ts
|
||||||
|
- FOUND: apps/api/src/tenders/tender-ingestion.service.spec.ts
|
||||||
|
- FOUND: apps/api/src/tenders/tender-scheduler.service.ts
|
||||||
|
- FOUND: apps/api/src/tenders/tender-scheduler.service.spec.ts
|
||||||
|
- FOUND: apps/api/src/tenders/tenders.module.ts
|
||||||
|
- FOUND commit: f9e52ab
|
||||||
|
- FOUND commit: 6fb734b
|
||||||
|
- FOUND commit: 0ba7410
|
||||||
|
- FOUND commit: 444c68b
|
||||||
|
|
||||||
|
---
|
||||||
|
*Phase: 10-ausschreibungs-radar-foundation-d-e-ingestion*
|
||||||
|
*Completed: 2026-07-21*
|
||||||
Reference in New Issue
Block a user