From 761077814ffa3723f700c3a074d9875ed1d9da8e Mon Sep 17 00:00:00 2001 From: Schalli Date: Tue, 21 Jul 2026 11:01:50 +0200 Subject: [PATCH] =?UTF-8?q?docs(10-03):=20complete=20D=C3=96E=20source=20a?= =?UTF-8?q?dapter=20&=20normalizer=20plan?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .planning/REQUIREMENTS.md | 4 +- .planning/ROADMAP.md | 6 +- .planning/STATE.md | 13 +- .../10-03-SUMMARY.md | 182 ++++++++++++++++++ 4 files changed, 196 insertions(+), 9 deletions(-) create mode 100644 .planning/phases/10-ausschreibungs-radar-foundation-d-e-ingestion/10-03-SUMMARY.md diff --git a/.planning/REQUIREMENTS.md b/.planning/REQUIREMENTS.md index 61c8682..b5e8eaf 100644 --- a/.planning/REQUIREMENTS.md +++ b/.planning/REQUIREMENTS.md @@ -10,7 +10,7 @@ ### INGEST — Datenquellen -- [ ] **INGEST-01**: Das System ruft Ausschreibungen zeitgesteuert über die DÖE OpenData-API (oeffentlichevergabe.de, eForms/OCDS, auth-frei) ab. +- [x] **INGEST-01**: Das System ruft Ausschreibungen zeitgesteuert über die DÖE OpenData-API (oeffentlichevergabe.de, eForms/OCDS, auth-frei) ab. - [ ] **INGEST-02**: Das System importiert Ausschreibungen von Administration-Intelligence-NetServer-Portalen (lhs-vpbw, tender24, vergabe.landbw) über einen konfigurierbaren Adapter. - [ ] **INGEST-03**: Das System importiert Ausschreibungen vom cosinex-Vergabemarktplatz (DTVP) über einen Adapter. - [ ] **INGEST-04**: Das System importiert Ausschreibungen aus RSS-Feeds (subreport-elvis, service.bund.de). @@ -77,7 +77,7 @@ | Requirement | Phase | Status | |-------------|-------|--------| | CONFIG-01 | Phase 10 | Complete | -| INGEST-01 | Phase 10 | Pending | +| INGEST-01 | Phase 10 | Complete | | INGEST-06 | Phase 10 | Pending | | SCHEMA-01 | Phase 10 | Complete | | SCHEMA-02 | Phase 10 | Pending | diff --git a/.planning/ROADMAP.md b/.planning/ROADMAP.md index 5813872..131f9c1 100644 --- a/.planning/ROADMAP.md +++ b/.planning/ROADMAP.md @@ -344,7 +344,7 @@ Plans: 4. DÖE is polled once on a shared, admin-configurable interval regardless of how many tenants have the module active -- never once per tenant (poll-once-fan-out-many, not the DKV single-tenant `findFirst()` pattern) 5. Activating the module for a second tenant does not duplicate ingestion, re-trigger a redundant DÖE poll, or interfere with the first tenant's data -**Plans**: 2/6 plans executed +**Plans**: 3/6 plans executed **Wave 1** @@ -356,7 +356,7 @@ Plans: **Wave 3** *(blocked on Wave 2 completion)* -- [ ] 10-03-PLAN.md — DÖE adapter + normalizer (TDD): fetch/extract/parse day-export ZIP, D-02 open-tender filter, eForms-primary normalize with dedupKey + contentHash (INGEST-01, SCHEMA-01) +- [x] 10-03-PLAN.md — DÖE adapter + normalizer (TDD): fetch/extract/parse day-export ZIP, D-02 open-tender filter, eForms-primary normalize with dedupKey + contentHash (INGEST-01, SCHEMA-01) **Wave 4** *(blocked on Wave 3 completion)* @@ -453,7 +453,7 @@ Phases execute in numeric order: 1 -> 2 -> 3 -> 4 -> 5 -> 6 -> 7 -> 8 -> 9 -> 10 | 7. DKV Fleet Module | 6/6 | Complete | 2026-06-27 | | 8. Dashboard Widgets Vollimplementierung | 4/4 | Complete | 2026-07-01 | | 9. Cert Manager Module | 6/6 | Complete | 2026-07-02 | -| 10. Ausschreibungs-Radar Foundation & DÖE Ingestion | 2/6 | In Progress| | +| 10. Ausschreibungs-Radar Foundation & DÖE Ingestion | 3/6 | In Progress| | | 11. Filter Engine, Results UI & Saved Searches | 0/TBD | Not started | - | | 12. Tender Notifications | 0/TBD | Not started | - | | 13. Scraping Adapters & Cross-Source Deduplication | 0/TBD | Not started | - | diff --git a/.planning/STATE.md b/.planning/STATE.md index 648c2fa..701119c 100644 --- a/.planning/STATE.md +++ b/.planning/STATE.md @@ -6,14 +6,14 @@ current_phase: 10 current_phase_name: ausschreibungs-radar-foundation-d-e-ingestion status: executing stopped_at: Completed 10-02-PLAN.md -last_updated: "2026-07-21T08:46:07.001Z" +last_updated: "2026-07-21T09:01:03.926Z" last_activity: 2026-07-21 last_activity_desc: Phase 10 execution started progress: total_phases: 14 completed_phases: 8 total_plans: 46 - completed_plans: 41 + completed_plans: 42 percent: 57 --- @@ -29,7 +29,7 @@ See: .planning/PROJECT.md (updated 2026-07-17) ## Current Position Phase: 10 (ausschreibungs-radar-foundation-d-e-ingestion) — EXECUTING -Plan: 3 of 6 +Plan: 4 of 6 Status: Ready to execute Last activity: 2026-07-21 — Phase 10 execution started @@ -73,6 +73,7 @@ Progress: [░░░░░░░░░░] 0% | Phase 09 P06 | 7 | 3 tasks | 7 files | | Phase 10 P01 | 15min | 3 tasks | 4 files | | Phase 10 P02 | 20min | 3 tasks | 6 files | +| Phase 10 P03 | 35min | 3 tasks | 9 files | ## Accumulated Context @@ -144,6 +145,10 @@ Recent decisions affecting current work: - [Phase 10-02]: category: 'procurement' fuer Ausschreibungs-Radar im Marketplace gewaehlt (freies kebab-case, keine Enum-Beschraenkung) - [Phase 10-02]: Singleton doe-opendata Poll-Config wird direkt in TendersModule.onModuleInit() upserted, isActive:true per Default (D-04) - [Phase 10-02]: Platzhalter-Seite tender-radar/page.tsx nutzt hartkodierten deutschen Text statt next-intl (volle i18n ist CONFIG-03, Phase 14) +- [Phase 10-03]: Real DÖE fixtures live-captured (pubDay=2026-07-19), not synthetic — 8 notices spanning all D-02 tag classes +- [Phase 10-03]: sourceNoticeId = OCDS release.id (stable, no version suffix), not the zip entry filename +- [Phase 10-03]: eForms-DE XML primary for deadlineAt/estimatedValue/procedureType; OCDS primary for ocid/buyerName/title/cpvCodes/region/plz +- [Phase 10-03]: bundesland left null this plan — NUTS-to-Bundesland mapping deferred to Phase 11 filter UI ### Pending Todos @@ -181,7 +186,7 @@ Items acknowledged and carried forward from previous milestone close: ## Session Continuity -Last session: 2026-07-21T08:46:06.990Z +Last session: 2026-07-21T09:00:23.857Z Stopped at: Completed 10-02-PLAN.md Resume file: None Last activity: 2026-07-14 - Built LDAP per-user exclude/denylist filter (9d1323f), migration applied on live DB, verified via Playwright: sync deactivated 4 excluded service accounts (administrator/krbtgt/guest/dns-ldap), 2 real LDAP users stay active, 0 wrongly created diff --git a/.planning/phases/10-ausschreibungs-radar-foundation-d-e-ingestion/10-03-SUMMARY.md b/.planning/phases/10-ausschreibungs-radar-foundation-d-e-ingestion/10-03-SUMMARY.md new file mode 100644 index 0000000..08fffa2 --- /dev/null +++ b/.planning/phases/10-ausschreibungs-radar-foundation-d-e-ingestion/10-03-SUMMARY.md @@ -0,0 +1,182 @@ +--- +phase: 10-ausschreibungs-radar-foundation-d-e-ingestion +plan: 03 +subsystem: api +tags: [doe-opendata, adm-zip, fast-xml-parser, eforms, ocds, tdd, normalizer, ingestion] + +# Dependency graph +requires: + - phase: 10-01 (foundation & dependencies) + provides: Tender/TenderSourcePollConfig Prisma models, fast-xml-parser/adm-zip/csv-parse installed + - phase: 10-02 (marketplace registration) + provides: TendersModule skeleton with empty providers array, ready for adapter/normalizer wiring +provides: + - DoeOpenDataAdapter — fetches, extracts, and D-02-filters the DÖE day-batch export into RawTenderRecord[] (INGEST-01) + - TenderNormalizerService — maps RawTenderRecord to Tender fields, dedupKey, contentHash (SCHEMA-01) + - Real, trimmed DÖE fixture ZIPs (doe-eforms-sample.zip / doe-ocds-sample.zip) for future ingestion-service tests + - RawTenderRecord/NormalizedTenderFields/SourceType shared types + TenderSourceAdapter interface (day-cursor signature) +affects: [10-04 ingestion/scheduler (consumes fetchTenders + normalize), 10-05 controller/DTOs, phase 11 saved searches/filter UI] + +# Tech tracking +tech-stack: + added: [] + patterns: + - "Day-cursor adapter contract (fetchTenders(dayCursor: string)) — not since:Date — per DÖE's daily-batch-only API shape" + - "eForms-DE XML as PRIMARY source for deadlineAt/estimatedValue/procedureType; OCDS PRIMARY for ocid/buyerName/title/cpvCodes/region/plz (RESEARCH Pattern 3 reversal of ARCHITECTURE.md)" + - "Pre-extraction decompression-bomb ceiling check (sum entry.header.size before any getData() call) — adm-zip getEntries() reads only the central directory" + - "Positive tag-presence D-02 filter (tag.includes('tender')) — missing/other tags conservatively excluded, never default-open" + - "Pure normalizer service (no I/O), module-level extraction helpers, single normalize() entry point — mirrors DkvParserService shape" + +key-files: + created: + - apps/api/src/tenders/__fixtures__/doe-eforms-sample.zip + - apps/api/src/tenders/__fixtures__/doe-ocds-sample.zip + - apps/api/src/tenders/tender.types.ts + - apps/api/src/tenders/adapters/tender-source-adapter.interface.ts + - apps/api/src/tenders/adapters/doe-opendata.adapter.ts + - apps/api/src/tenders/adapters/doe-opendata.adapter.spec.ts + - apps/api/src/tenders/tender-normalizer.service.ts + - apps/api/src/tenders/tender-normalizer.service.spec.ts + modified: + - apps/api/src/tenders/tenders.module.ts + +key-decisions: + - "Fixtures captured LIVE from oeffentlichevergabe.de (pubDay=2026-07-19, not synthetic) — 8 real notices trimmed from a 594-notice day, spanning all four D-02 tag classes (tender/award/planning/untagged-with-awards) plus one directly cross-checked eForms-primary-deadline pair, per this repo's real-fixture precedent (DKV PDF parser)" + - "sourceNoticeId = OCDS release.id (stable, no version suffix), NOT the zip entry filename (which carries a -NN version suffix) — matches the dedupKey fallback tier's intent of a stable per-notice identifier" + - "Ceiling check runs per-archive, sequentially (eforms.zip checked+extracted before ocds.zip is even fetched) — a bomb in the first archive short-circuits before a second network call is made" + - "bundesland (Bundesland name) intentionally left null — NUTS-code-to-Bundesland-name mapping is deferred to Phase 11's filter UI, out of this phase's ingestion-core scope" + - "XMLParser configured with removeNSPrefix:true — eForms-DE's cac:/cbc:/efac: namespace prefixes are stripped so normalizer code addresses tags by local name only (ContractNotice.ProcurementProjectLot[0].TenderingProcess.TenderSubmissionDeadlinePeriod.EndDate)" + +requirements-completed: [INGEST-01, SCHEMA-01] + +coverage: + - id: D1 + description: "DoeOpenDataAdapter fetches a day's DÖE export ZIP, extracts it, and parses eForms-DE XML (primary) + OCDS JSON (ocid) into RawTenderRecord[] (INGEST-01)" + requirement: "INGEST-01" + verification: + - kind: unit + ref: "apps/api/src/tenders/adapters/doe-opendata.adapter.spec.ts — 6/6 green: sourceType, parse+pair into RawTenderRecord[], D-02 filter, 400-no-op, zip-bomb ceiling, no-axios" + status: pass + human_judgment: false + - id: D2 + description: "Only open tenders survive the D-02 filter: tag=['tender'] included; award/planning/untagged-with-awards excluded, exact retained count proven against real fixture" + requirement: "INGEST-01" + verification: + - kind: unit + ref: "doe-opendata.adapter.spec.ts — fetchTenders() on the 8-notice real fixture (4 tender / 2 award / 1 planning / 1 untagged-with-awards) returns exactly 4 records; explicit not.toContain assertions for each excluded notice id" + status: pass + human_judgment: false + - id: D3 + description: "TenderNormalizer maps a real eForms+OCDS notice pair into Tender fields with nullable deadline/value, a stable dedupKey (ocid -> sourcePortal:noticeId), and a contentHash (SCHEMA-01)" + requirement: "SCHEMA-01" + verification: + - kind: unit + ref: "tender-normalizer.service.spec.ts — 6/6 green: eForms-only deadline recovery (real cross-checked pair, OCDS tenderPeriod absent), missing-deadline/value -> null, dedupKey=ocid, dedupKey fallback, contentHash stability, contentHash changes on deadline mutation" + status: pass + human_judgment: false + - id: D4 + description: "Zip-slip / decompression-bomb safety present in the extract path (T-10-07)" + verification: + - kind: unit + ref: "doe-opendata.adapter.spec.ts — 60MB-declared synthetic archive (highly compressible, tiny on disk) rejected before any entry.getData() call; entries never written to disk (zip-slip structurally absent, documented in code)" + status: pass + human_judgment: false + +# Metrics +duration: ~35min +completed: 2026-07-21 +status: complete +--- + +# Phase 10 Plan 03: DÖE Source Adapter & Normalizer Summary + +**`DoeOpenDataAdapter` (native fetch + adm-zip + fast-xml-parser, D-02-filtered) and `TenderNormalizerService` (eForms-primary field mapping, dedupKey, contentHash) built and proven test-first against real DÖE fixture ZIPs captured live this session — the structurally hardest slice of INGEST-01/SCHEMA-01.** + +## Performance + +- **Duration:** ~35 min +- **Started:** 2026-07-21 +- **Completed:** 2026-07-21 +- **Tasks:** 3 (all auto, TDD RED->GREEN->GREEN, no checkpoints) +- **Files modified:** 9 (6 created API source/spec, 2 created fixtures, 1 modified module) + +## Accomplishments + +- Captured 2 real, trimmed DÖE day-export fixture ZIPs (`doe-eforms-sample.zip` / `doe-ocds-sample.zip`) live from `oeffentlichevergabe.de` (pubDay=2026-07-19), 8 notices spanning all four D-02 tag classes plus a directly cross-checked eForms-only-deadline pair — not synthetic data, per the DKV PDF-parser real-fixture precedent. +- `tender.types.ts` (`RawTenderRecord`/`NormalizedTenderFields`/`SourceType`) and `TenderSourceAdapter` interface (day-cursor `fetchTenders(dayCursor: string)`, correcting ARCHITECTURE.md's `since?: Date` sketch per RESEARCH Pattern 1) defined. +- `DoeOpenDataAdapter`: native fetch + AbortController 15s timeout, HTTP-400-as-no-op, adm-zip extraction with a pre-extraction decompression-bomb ceiling guard (T-10-07), and a positive-match D-02 open-tender filter (`tag.includes('tender')`) — proven against the real fixture's exact 4/8 retained count. +- `TenderNormalizerService`: pure `normalize()` mapping eForms-DE XML as PRIMARY for deadline/value/procedureType and OCDS as PRIMARY for ocid/buyer/title/CPV — proven on the real cross-checked notice pair where OCDS `tender.tenderPeriod` is entirely absent but the eForms XML carries a structured `TenderSubmissionDeadlinePeriod/EndDate`. +- Both services registered in `TendersModule.providers`; full API test suite green (59/59), `tsc --noEmit` clean. + +## Task Commits + +Each task committed atomically, RED before GREEN: + +1. **Task 1: Real fixtures + shared types + adapter interface + failing specs (RED)** — `f88e2a8` (test) +2. **Task 2: DoeOpenDataAdapter — fetch + adm-zip extract + parse + D-02 filter (GREEN)** — `3764feb` (feat) +3. **Task 3: TenderNormalizerService — fields + dedupKey + contentHash (GREEN)** — `31607df` (feat) + +**Plan metadata:** see final `docs(10-03)` commit. + +## TDD Gate Compliance + +- RED gate: `f88e2a8` (`test(10-03): ...`) — both spec files failed at module-resolution time (adapter/normalizer not yet implemented), confirmed via `pnpm exec vitest run -- doe-opendata.adapter tender-normalizer` before any implementation existed. +- GREEN gate (adapter): `3764feb` (`feat(10-03): ...`) — all 6 `doe-opendata.adapter.spec.ts` tests pass. +- GREEN gate (normalizer): `31607df` (`feat(10-03): ...`) — all 6 `tender-normalizer.service.spec.ts` tests pass. +- No REFACTOR commit was needed — both implementations passed cleanly on first GREEN attempt, no post-green cleanup required. + +## Files Created/Modified + +- `apps/api/src/tenders/__fixtures__/doe-eforms-sample.zip` / `doe-ocds-sample.zip` — 8 real notices (4 tender-tagged incl. one directly cross-checked eForms-only-deadline pair, 2 award-tagged, 1 planning-tagged, 1 untagged-with-populated-awards), trimmed from a real 594-notice day (2026-07-19) +- `apps/api/src/tenders/tender.types.ts` — `SourceType`, `RawTenderRecord`, `NormalizedTenderFields` +- `apps/api/src/tenders/adapters/tender-source-adapter.interface.ts` — `TenderSourceAdapter` (day-cursor signature) +- `apps/api/src/tenders/adapters/doe-opendata.adapter.ts` — fetch/extract/parse/D-02-filter, exports `isOpenTenderNotice()` for reuse +- `apps/api/src/tenders/adapters/doe-opendata.adapter.spec.ts` — 6 tests +- `apps/api/src/tenders/tender-normalizer.service.ts` — pure `normalize()` + module-level extraction helpers +- `apps/api/src/tenders/tender-normalizer.service.spec.ts` — 6 tests +- `apps/api/src/tenders/tenders.module.ts` — `DoeOpenDataAdapter` + `TenderNormalizerService` added to `providers` + +## Decisions Made + +- **Real fixtures, live-captured this session:** downloaded the actual `eforms.zip`/`ocds.zip` for pubDay=2026-07-19 (594 real notices), classified all by OCDS `releases[0].tag`, and trimmed to 8 representative notices covering every D-02 tag class plus a directly verified eForms-primary-deadline cross-check pair — matches this repo's stated preference for real fixtures over synthetic ones (DKV PDF parser precedent), no fallback-to-synthetic path was needed since the DÖE host was reachable. +- **`sourceNoticeId` = OCDS `release.id`, not the zip entry basename:** the entry filename carries a version suffix (e.g. `-01`, `-03`) while `release.id` is the stable per-notice identifier without it — using the filename would make the `sourcePortal:sourceNoticeId` dedupKey fallback tier version-sensitive, defeating its purpose. +- **Ceiling check is sequential per-archive:** `eforms.zip` is fetched, extracted, and ceiling-checked before `ocds.zip` is even requested — a bomb in the first archive short-circuits the whole call with zero extra network I/O, and keeps the zip-bomb test's mock trivial (only `eforms.zip`'s response needs to be the oversized archive). +- **`bundesland` left `null` for this plan:** NUTS-region-code (e.g. `DEA41`) to human Bundesland-name mapping belongs to Phase 11's filter UI, not this phase's ingestion core — documented inline in `tender.types.ts` and the normalizer. +- **`removeNSPrefix: true` on the `XMLParser`:** eForms-DE XML uses `cac:`/`cbc:`/`efac:`/`efbc:` namespace prefixes throughout; stripping them lets the normalizer address tags by local name (`ProcurementProjectLot[0].TenderingProcess.TenderSubmissionDeadlinePeriod.EndDate`) without namespace-aware traversal, at negligible collision risk for the specific fields this phase reads. + +## Deviations from Plan + +None — plan executed exactly as written. The plan's `must_haves.key_links` anticipated a `checkpoint:human-verify` gate before `pnpm add adm-zip` (package-legitimacy protocol); that gate was already satisfied in Plan 10-01 (adm-zip was installed and user-approved there), so no new checkpoint was needed in this plan. + +## Issues Encountered + +None. The DÖE host (`oeffentlichevergabe.de`) was reachable from this execution environment, so the real-fixture path (not the documented XML-reconstruction fallback) was used throughout. + +## User Setup Required + +None — no external service configuration required. Local Docker stack was left running unmodified; a live DÖE ingestion run (Plan 04's scheduler) is a separate concern from this plan's pure fetch/parse/normalize units, which were verified entirely via the committed fixtures, not the live DB/stack. + +## Next Phase Readiness + +- Both pure units (`DoeOpenDataAdapter.fetchTenders()` and `TenderNormalizerService.normalize()`) are implemented, tested, and registered in `TendersModule.providers` — ready for Plan 04 (`TenderIngestionService`) to compose them: `pollDueSources()` will call `fetchTenders(nextDay)` then `normalize()` each record, then `prisma.tender.upsert({ where: { dedupKey } })`. +- The day-cursor gate (`nextDayToFetch()`, RESEARCH.md's "Day-cursor gate" snippet) and the singleton `TenderSourcePollConfig` row (seeded in Plan 02) are the remaining pieces Plan 04 wires together — no blockers. +- No open threat-model items from this plan carry forward: T-10-06 (SSRF) mitigated by the hardcoded DÖE host constant, T-10-07 (decompression bomb) mitigated and tested, T-10-08 (untrusted text fields) is satisfied by storing raw values with no HTML emission anywhere in the normalizer. + +## Self-Check: PASSED + +- FOUND: apps/api/src/tenders/__fixtures__/doe-eforms-sample.zip +- FOUND: apps/api/src/tenders/__fixtures__/doe-ocds-sample.zip +- FOUND: apps/api/src/tenders/tender.types.ts +- FOUND: apps/api/src/tenders/adapters/tender-source-adapter.interface.ts +- FOUND: apps/api/src/tenders/adapters/doe-opendata.adapter.ts +- FOUND: apps/api/src/tenders/adapters/doe-opendata.adapter.spec.ts +- FOUND: apps/api/src/tenders/tender-normalizer.service.ts +- FOUND: apps/api/src/tenders/tender-normalizer.service.spec.ts +- FOUND: apps/api/src/tenders/tenders.module.ts +- FOUND commit: f88e2a8 +- FOUND commit: 3764feb +- FOUND commit: 31607df + +--- +*Phase: 10-ausschreibungs-radar-foundation-d-e-ingestion* +*Completed: 2026-07-21*