docs(10-03): complete DÖE source adapter & normalizer plan

.planning/REQUIREMENTS.md

@@ -10,7 +10,7 @@
 
 ### INGEST — Datenquellen
 
-- [ ] **INGEST-01**: Das System ruft Ausschreibungen zeitgesteuert über die DÖE OpenData-API (oeffentlichevergabe.de, eForms/OCDS, auth-frei) ab.
+- [x] **INGEST-01**: Das System ruft Ausschreibungen zeitgesteuert über die DÖE OpenData-API (oeffentlichevergabe.de, eForms/OCDS, auth-frei) ab.
 - [ ] **INGEST-02**: Das System importiert Ausschreibungen von Administration-Intelligence-NetServer-Portalen (lhs-vpbw, tender24, vergabe.landbw) über einen konfigurierbaren Adapter.
 - [ ] **INGEST-03**: Das System importiert Ausschreibungen vom cosinex-Vergabemarktplatz (DTVP) über einen Adapter.
 - [ ] **INGEST-04**: Das System importiert Ausschreibungen aus RSS-Feeds (subreport-elvis, service.bund.de).
@@ -77,7 +77,7 @@
 | Requirement | Phase | Status |
 |-------------|-------|--------|
 | CONFIG-01 | Phase 10 | Complete |
-| INGEST-01 | Phase 10 | Pending |
+| INGEST-01 | Phase 10 | Complete |
 | INGEST-06 | Phase 10 | Pending |
 | SCHEMA-01 | Phase 10 | Complete |
 | SCHEMA-02 | Phase 10 | Pending |

.planning/ROADMAP.md

@@ -344,7 +344,7 @@ Plans:
   4. DÖE is polled once on a shared, admin-configurable interval regardless of how many tenants have the module active -- never once per tenant (poll-once-fan-out-many, not the DKV single-tenant `findFirst()` pattern)
   5. Activating the module for a second tenant does not duplicate ingestion, re-trigger a redundant DÖE poll, or interfere with the first tenant's data
 
-**Plans**: 2/6 plans executed
+**Plans**: 3/6 plans executed
 
 **Wave 1**
 
@@ -356,7 +356,7 @@ Plans:
 
 **Wave 3** *(blocked on Wave 2 completion)*
 
-- [ ] 10-03-PLAN.md — DÖE adapter + normalizer (TDD): fetch/extract/parse day-export ZIP, D-02 open-tender filter, eForms-primary normalize with dedupKey + contentHash (INGEST-01, SCHEMA-01)
+- [x] 10-03-PLAN.md — DÖE adapter + normalizer (TDD): fetch/extract/parse day-export ZIP, D-02 open-tender filter, eForms-primary normalize with dedupKey + contentHash (INGEST-01, SCHEMA-01)
 
 **Wave 4** *(blocked on Wave 3 completion)*
 
@@ -453,7 +453,7 @@ Phases execute in numeric order: 1 -> 2 -> 3 -> 4 -> 5 -> 6 -> 7 -> 8 -> 9 -> 10
 | 7. DKV Fleet Module | 6/6 | Complete   | 2026-06-27 |
 | 8. Dashboard Widgets Vollimplementierung | 4/4 | Complete   | 2026-07-01 |
 | 9. Cert Manager Module | 6/6 | Complete   | 2026-07-02 |
-| 10. Ausschreibungs-Radar Foundation & DÖE Ingestion | 2/6 | In Progress|  |
+| 10. Ausschreibungs-Radar Foundation & DÖE Ingestion | 3/6 | In Progress|  |
 | 11. Filter Engine, Results UI & Saved Searches | 0/TBD | Not started | - |
 | 12. Tender Notifications | 0/TBD | Not started | - |
 | 13. Scraping Adapters & Cross-Source Deduplication | 0/TBD | Not started | - |

.planning/STATE.md

@@ -6,14 +6,14 @@ current_phase: 10
 current_phase_name: ausschreibungs-radar-foundation-d-e-ingestion
 status: executing
 stopped_at: Completed 10-02-PLAN.md
-last_updated: "2026-07-21T08:46:07.001Z"
+last_updated: "2026-07-21T09:01:03.926Z"
 last_activity: 2026-07-21
 last_activity_desc: Phase 10 execution started
 progress:
   total_phases: 14
   completed_phases: 8
   total_plans: 46
-  completed_plans: 41
+  completed_plans: 42
   percent: 57
 ---
 
@@ -29,7 +29,7 @@ See: .planning/PROJECT.md (updated 2026-07-17)
 ## Current Position
 
 Phase: 10 (ausschreibungs-radar-foundation-d-e-ingestion) — EXECUTING
-Plan: 3 of 6
+Plan: 4 of 6
 Status: Ready to execute
 Last activity: 2026-07-21 — Phase 10 execution started
 
@@ -73,6 +73,7 @@ Progress: [░░░░░░░░░░] 0%
 | Phase 09 P06 | 7 | 3 tasks | 7 files |
 | Phase 10 P01 | 15min | 3 tasks | 4 files |
 | Phase 10 P02 | 20min | 3 tasks | 6 files |
+| Phase 10 P03 | 35min | 3 tasks | 9 files |
 
 ## Accumulated Context
 
@@ -144,6 +145,10 @@ Recent decisions affecting current work:
 - [Phase 10-02]: category: 'procurement' fuer Ausschreibungs-Radar im Marketplace gewaehlt (freies kebab-case, keine Enum-Beschraenkung)
 - [Phase 10-02]: Singleton doe-opendata Poll-Config wird direkt in TendersModule.onModuleInit() upserted, isActive:true per Default (D-04)
 - [Phase 10-02]: Platzhalter-Seite tender-radar/page.tsx nutzt hartkodierten deutschen Text statt next-intl (volle i18n ist CONFIG-03, Phase 14)
+- [Phase 10-03]: Real DÖE fixtures live-captured (pubDay=2026-07-19), not synthetic — 8 notices spanning all D-02 tag classes
+- [Phase 10-03]: sourceNoticeId = OCDS release.id (stable, no version suffix), not the zip entry filename
+- [Phase 10-03]: eForms-DE XML primary for deadlineAt/estimatedValue/procedureType; OCDS primary for ocid/buyerName/title/cpvCodes/region/plz
+- [Phase 10-03]: bundesland left null this plan — NUTS-to-Bundesland mapping deferred to Phase 11 filter UI
 
 ### Pending Todos
 
@@ -181,7 +186,7 @@ Items acknowledged and carried forward from previous milestone close:
 
 ## Session Continuity
 
-Last session: 2026-07-21T08:46:06.990Z
+Last session: 2026-07-21T09:00:23.857Z
 Stopped at: Completed 10-02-PLAN.md
 Resume file: None
 Last activity: 2026-07-14 - Built LDAP per-user exclude/denylist filter (9d1323f), migration applied on live DB, verified via Playwright: sync deactivated 4 excluded service accounts (administrator/krbtgt/guest/dns-ldap), 2 real LDAP users stay active, 0 wrongly created

.planning/phases/10-ausschreibungs-radar-foundation-d-e-ingestion/10-03-SUMMARY.md

@@ -0,0 +1,182 @@
+---
+phase: 10-ausschreibungs-radar-foundation-d-e-ingestion
+plan: 03
+subsystem: api
+tags: [doe-opendata, adm-zip, fast-xml-parser, eforms, ocds, tdd, normalizer, ingestion]
+
+# Dependency graph
+requires:
+  - phase: 10-01 (foundation & dependencies)
+    provides: Tender/TenderSourcePollConfig Prisma models, fast-xml-parser/adm-zip/csv-parse installed
+  - phase: 10-02 (marketplace registration)
+    provides: TendersModule skeleton with empty providers array, ready for adapter/normalizer wiring
+provides:
+  - DoeOpenDataAdapter — fetches, extracts, and D-02-filters the DÖE day-batch export into RawTenderRecord[] (INGEST-01)
+  - TenderNormalizerService — maps RawTenderRecord to Tender fields, dedupKey, contentHash (SCHEMA-01)
+  - Real, trimmed DÖE fixture ZIPs (doe-eforms-sample.zip / doe-ocds-sample.zip) for future ingestion-service tests
+  - RawTenderRecord/NormalizedTenderFields/SourceType shared types + TenderSourceAdapter interface (day-cursor signature)
+affects: [10-04 ingestion/scheduler (consumes fetchTenders + normalize), 10-05 controller/DTOs, phase 11 saved searches/filter UI]
+
+# Tech tracking
+tech-stack:
+  added: []
+  patterns:
+    - "Day-cursor adapter contract (fetchTenders(dayCursor: string)) — not since:Date — per DÖE's daily-batch-only API shape"
+    - "eForms-DE XML as PRIMARY source for deadlineAt/estimatedValue/procedureType; OCDS PRIMARY for ocid/buyerName/title/cpvCodes/region/plz (RESEARCH Pattern 3 reversal of ARCHITECTURE.md)"
+    - "Pre-extraction decompression-bomb ceiling check (sum entry.header.size before any getData() call) — adm-zip getEntries() reads only the central directory"
+    - "Positive tag-presence D-02 filter (tag.includes('tender')) — missing/other tags conservatively excluded, never default-open"
+    - "Pure normalizer service (no I/O), module-level extraction helpers, single normalize() entry point — mirrors DkvParserService shape"
+
+key-files:
+  created:
+    - apps/api/src/tenders/__fixtures__/doe-eforms-sample.zip
+    - apps/api/src/tenders/__fixtures__/doe-ocds-sample.zip
+    - apps/api/src/tenders/tender.types.ts
+    - apps/api/src/tenders/adapters/tender-source-adapter.interface.ts
+    - apps/api/src/tenders/adapters/doe-opendata.adapter.ts
+    - apps/api/src/tenders/adapters/doe-opendata.adapter.spec.ts
+    - apps/api/src/tenders/tender-normalizer.service.ts
+    - apps/api/src/tenders/tender-normalizer.service.spec.ts
+  modified:
+    - apps/api/src/tenders/tenders.module.ts
+
+key-decisions:
+  - "Fixtures captured LIVE from oeffentlichevergabe.de (pubDay=2026-07-19, not synthetic) — 8 real notices trimmed from a 594-notice day, spanning all four D-02 tag classes (tender/award/planning/untagged-with-awards) plus one directly cross-checked eForms-primary-deadline pair, per this repo's real-fixture precedent (DKV PDF parser)"
+  - "sourceNoticeId = OCDS release.id (stable, no version suffix), NOT the zip entry filename (which carries a -NN version suffix) — matches the dedupKey fallback tier's intent of a stable per-notice identifier"
+  - "Ceiling check runs per-archive, sequentially (eforms.zip checked+extracted before ocds.zip is even fetched) — a bomb in the first archive short-circuits before a second network call is made"
+  - "bundesland (Bundesland name) intentionally left null — NUTS-code-to-Bundesland-name mapping is deferred to Phase 11's filter UI, out of this phase's ingestion-core scope"
+  - "XMLParser configured with removeNSPrefix:true — eForms-DE's cac:/cbc:/efac: namespace prefixes are stripped so normalizer code addresses tags by local name only (ContractNotice.ProcurementProjectLot[0].TenderingProcess.TenderSubmissionDeadlinePeriod.EndDate)"
+
+requirements-completed: [INGEST-01, SCHEMA-01]
+
+coverage:
+  - id: D1
+    description: "DoeOpenDataAdapter fetches a day's DÖE export ZIP, extracts it, and parses eForms-DE XML (primary) + OCDS JSON (ocid) into RawTenderRecord[] (INGEST-01)"
+    requirement: "INGEST-01"
+    verification:
+      - kind: unit
+        ref: "apps/api/src/tenders/adapters/doe-opendata.adapter.spec.ts — 6/6 green: sourceType, parse+pair into RawTenderRecord[], D-02 filter, 400-no-op, zip-bomb ceiling, no-axios"
+        status: pass
+    human_judgment: false
+  - id: D2
+    description: "Only open tenders survive the D-02 filter: tag=['tender'] included; award/planning/untagged-with-awards excluded, exact retained count proven against real fixture"
+    requirement: "INGEST-01"
+    verification:
+      - kind: unit
+        ref: "doe-opendata.adapter.spec.ts — fetchTenders() on the 8-notice real fixture (4 tender / 2 award / 1 planning / 1 untagged-with-awards) returns exactly 4 records; explicit not.toContain assertions for each excluded notice id"
+        status: pass
+    human_judgment: false
+  - id: D3
+    description: "TenderNormalizer maps a real eForms+OCDS notice pair into Tender fields with nullable deadline/value, a stable dedupKey (ocid -> sourcePortal:noticeId), and a contentHash (SCHEMA-01)"
+    requirement: "SCHEMA-01"
+    verification:
+      - kind: unit
+        ref: "tender-normalizer.service.spec.ts — 6/6 green: eForms-only deadline recovery (real cross-checked pair, OCDS tenderPeriod absent), missing-deadline/value -> null, dedupKey=ocid, dedupKey fallback, contentHash stability, contentHash changes on deadline mutation"
+        status: pass
+    human_judgment: false
+  - id: D4
+    description: "Zip-slip / decompression-bomb safety present in the extract path (T-10-07)"
+    verification:
+      - kind: unit
+        ref: "doe-opendata.adapter.spec.ts — 60MB-declared synthetic archive (highly compressible, tiny on disk) rejected before any entry.getData() call; entries never written to disk (zip-slip structurally absent, documented in code)"
+        status: pass
+    human_judgment: false
+
+# Metrics
+duration: ~35min
+completed: 2026-07-21
+status: complete
+---
+
+# Phase 10 Plan 03: DÖE Source Adapter & Normalizer Summary
+
+**`DoeOpenDataAdapter` (native fetch + adm-zip + fast-xml-parser, D-02-filtered) and `TenderNormalizerService` (eForms-primary field mapping, dedupKey, contentHash) built and proven test-first against real DÖE fixture ZIPs captured live this session — the structurally hardest slice of INGEST-01/SCHEMA-01.**
+
+## Performance
+
+- **Duration:** ~35 min
+- **Started:** 2026-07-21
+- **Completed:** 2026-07-21
+- **Tasks:** 3 (all auto, TDD RED->GREEN->GREEN, no checkpoints)
+- **Files modified:** 9 (6 created API source/spec, 2 created fixtures, 1 modified module)
+
+## Accomplishments
+
+- Captured 2 real, trimmed DÖE day-export fixture ZIPs (`doe-eforms-sample.zip` / `doe-ocds-sample.zip`) live from `oeffentlichevergabe.de` (pubDay=2026-07-19), 8 notices spanning all four D-02 tag classes plus a directly cross-checked eForms-only-deadline pair — not synthetic data, per the DKV PDF-parser real-fixture precedent.
+- `tender.types.ts` (`RawTenderRecord`/`NormalizedTenderFields`/`SourceType`) and `TenderSourceAdapter` interface (day-cursor `fetchTenders(dayCursor: string)`, correcting ARCHITECTURE.md's `since?: Date` sketch per RESEARCH Pattern 1) defined.
+- `DoeOpenDataAdapter`: native fetch + AbortController 15s timeout, HTTP-400-as-no-op, adm-zip extraction with a pre-extraction decompression-bomb ceiling guard (T-10-07), and a positive-match D-02 open-tender filter (`tag.includes('tender')`) — proven against the real fixture's exact 4/8 retained count.
+- `TenderNormalizerService`: pure `normalize()` mapping eForms-DE XML as PRIMARY for deadline/value/procedureType and OCDS as PRIMARY for ocid/buyer/title/CPV — proven on the real cross-checked notice pair where OCDS `tender.tenderPeriod` is entirely absent but the eForms XML carries a structured `TenderSubmissionDeadlinePeriod/EndDate`.
+- Both services registered in `TendersModule.providers`; full API test suite green (59/59), `tsc --noEmit` clean.
+
+## Task Commits
+
+Each task committed atomically, RED before GREEN:
+
+1. **Task 1: Real fixtures + shared types + adapter interface + failing specs (RED)** — `f88e2a8` (test)
+2. **Task 2: DoeOpenDataAdapter — fetch + adm-zip extract + parse + D-02 filter (GREEN)** — `3764feb` (feat)
+3. **Task 3: TenderNormalizerService — fields + dedupKey + contentHash (GREEN)** — `31607df` (feat)
+
+**Plan metadata:** see final `docs(10-03)` commit.
+
+## TDD Gate Compliance
+
+- RED gate: `f88e2a8` (`test(10-03): ...`) — both spec files failed at module-resolution time (adapter/normalizer not yet implemented), confirmed via `pnpm exec vitest run -- doe-opendata.adapter tender-normalizer` before any implementation existed.
+- GREEN gate (adapter): `3764feb` (`feat(10-03): ...`) — all 6 `doe-opendata.adapter.spec.ts` tests pass.
+- GREEN gate (normalizer): `31607df` (`feat(10-03): ...`) — all 6 `tender-normalizer.service.spec.ts` tests pass.
+- No REFACTOR commit was needed — both implementations passed cleanly on first GREEN attempt, no post-green cleanup required.
+
+## Files Created/Modified
+
+- `apps/api/src/tenders/__fixtures__/doe-eforms-sample.zip` / `doe-ocds-sample.zip` — 8 real notices (4 tender-tagged incl. one directly cross-checked eForms-only-deadline pair, 2 award-tagged, 1 planning-tagged, 1 untagged-with-populated-awards), trimmed from a real 594-notice day (2026-07-19)
+- `apps/api/src/tenders/tender.types.ts` — `SourceType`, `RawTenderRecord`, `NormalizedTenderFields`
+- `apps/api/src/tenders/adapters/tender-source-adapter.interface.ts` — `TenderSourceAdapter` (day-cursor signature)
+- `apps/api/src/tenders/adapters/doe-opendata.adapter.ts` — fetch/extract/parse/D-02-filter, exports `isOpenTenderNotice()` for reuse
+- `apps/api/src/tenders/adapters/doe-opendata.adapter.spec.ts` — 6 tests
+- `apps/api/src/tenders/tender-normalizer.service.ts` — pure `normalize()` + module-level extraction helpers
+- `apps/api/src/tenders/tender-normalizer.service.spec.ts` — 6 tests
+- `apps/api/src/tenders/tenders.module.ts` — `DoeOpenDataAdapter` + `TenderNormalizerService` added to `providers`
+
+## Decisions Made
+
+- **Real fixtures, live-captured this session:** downloaded the actual `eforms.zip`/`ocds.zip` for pubDay=2026-07-19 (594 real notices), classified all by OCDS `releases[0].tag`, and trimmed to 8 representative notices covering every D-02 tag class plus a directly verified eForms-primary-deadline cross-check pair — matches this repo's stated preference for real fixtures over synthetic ones (DKV PDF parser precedent), no fallback-to-synthetic path was needed since the DÖE host was reachable.
+- **`sourceNoticeId` = OCDS `release.id`, not the zip entry basename:** the entry filename carries a version suffix (e.g. `-01`, `-03`) while `release.id` is the stable per-notice identifier without it — using the filename would make the `sourcePortal:sourceNoticeId` dedupKey fallback tier version-sensitive, defeating its purpose.
+- **Ceiling check is sequential per-archive:** `eforms.zip` is fetched, extracted, and ceiling-checked before `ocds.zip` is even requested — a bomb in the first archive short-circuits the whole call with zero extra network I/O, and keeps the zip-bomb test's mock trivial (only `eforms.zip`'s response needs to be the oversized archive).
+- **`bundesland` left `null` for this plan:** NUTS-region-code (e.g. `DEA41`) to human Bundesland-name mapping belongs to Phase 11's filter UI, not this phase's ingestion core — documented inline in `tender.types.ts` and the normalizer.
+- **`removeNSPrefix: true` on the `XMLParser`:** eForms-DE XML uses `cac:`/`cbc:`/`efac:`/`efbc:` namespace prefixes throughout; stripping them lets the normalizer address tags by local name (`ProcurementProjectLot[0].TenderingProcess.TenderSubmissionDeadlinePeriod.EndDate`) without namespace-aware traversal, at negligible collision risk for the specific fields this phase reads.
+
+## Deviations from Plan
+
+None — plan executed exactly as written. The plan's `must_haves.key_links` anticipated a `checkpoint:human-verify` gate before `pnpm add adm-zip` (package-legitimacy protocol); that gate was already satisfied in Plan 10-01 (adm-zip was installed and user-approved there), so no new checkpoint was needed in this plan.
+
+## Issues Encountered
+
+None. The DÖE host (`oeffentlichevergabe.de`) was reachable from this execution environment, so the real-fixture path (not the documented XML-reconstruction fallback) was used throughout.
+
+## User Setup Required
+
+None — no external service configuration required. Local Docker stack was left running unmodified; a live DÖE ingestion run (Plan 04's scheduler) is a separate concern from this plan's pure fetch/parse/normalize units, which were verified entirely via the committed fixtures, not the live DB/stack.
+
+## Next Phase Readiness
+
+- Both pure units (`DoeOpenDataAdapter.fetchTenders()` and `TenderNormalizerService.normalize()`) are implemented, tested, and registered in `TendersModule.providers` — ready for Plan 04 (`TenderIngestionService`) to compose them: `pollDueSources()` will call `fetchTenders(nextDay)` then `normalize()` each record, then `prisma.tender.upsert({ where: { dedupKey } })`.
+- The day-cursor gate (`nextDayToFetch()`, RESEARCH.md's "Day-cursor gate" snippet) and the singleton `TenderSourcePollConfig` row (seeded in Plan 02) are the remaining pieces Plan 04 wires together — no blockers.
+- No open threat-model items from this plan carry forward: T-10-06 (SSRF) mitigated by the hardcoded DÖE host constant, T-10-07 (decompression bomb) mitigated and tested, T-10-08 (untrusted text fields) is satisfied by storing raw values with no HTML emission anywhere in the normalizer.
+
+## Self-Check: PASSED
+
+- FOUND: apps/api/src/tenders/__fixtures__/doe-eforms-sample.zip
+- FOUND: apps/api/src/tenders/__fixtures__/doe-ocds-sample.zip
+- FOUND: apps/api/src/tenders/tender.types.ts
+- FOUND: apps/api/src/tenders/adapters/tender-source-adapter.interface.ts
+- FOUND: apps/api/src/tenders/adapters/doe-opendata.adapter.ts
+- FOUND: apps/api/src/tenders/adapters/doe-opendata.adapter.spec.ts
+- FOUND: apps/api/src/tenders/tender-normalizer.service.ts
+- FOUND: apps/api/src/tenders/tender-normalizer.service.spec.ts
+- FOUND: apps/api/src/tenders/tenders.module.ts
+- FOUND commit: f88e2a8
+- FOUND commit: 3764feb
+- FOUND commit: 31607df
+
+---
+*Phase: 10-ausschreibungs-radar-foundation-d-e-ingestion*
+*Completed: 2026-07-21*