---
phase: 10-ausschreibungs-radar-foundation-d-e-ingestion
plan: 03
subsystem: api
tags: [doe-opendata, adm-zip, fast-xml-parser, eforms, ocds, tdd, normalizer, ingestion]

# Dependency graph
requires:
  - phase: 10-01 (foundation & dependencies)
    provides: Tender/TenderSourcePollConfig Prisma models, fast-xml-parser/adm-zip/csv-parse installed
  - phase: 10-02 (marketplace registration)
    provides: TendersModule skeleton with empty providers array, ready for adapter/normalizer wiring
provides:
  - DoeOpenDataAdapter — fetches, extracts, and D-02-filters the DÖE day-batch export into RawTenderRecord[] (INGEST-01)
  - TenderNormalizerService — maps RawTenderRecord to Tender fields, dedupKey, contentHash (SCHEMA-01)
  - Real, trimmed DÖE fixture ZIPs (doe-eforms-sample.zip / doe-ocds-sample.zip) for future ingestion-service tests
  - RawTenderRecord/NormalizedTenderFields/SourceType shared types + TenderSourceAdapter interface (day-cursor signature)
affects: [10-04 ingestion/scheduler (consumes fetchTenders + normalize), 10-05 controller/DTOs, phase 11 saved searches/filter UI]

# Tech tracking
tech-stack:
  added: []
  patterns:
    - "Day-cursor adapter contract (fetchTenders(dayCursor: string)) — not since:Date — per DÖE's daily-batch-only API shape"
    - "eForms-DE XML as PRIMARY source for deadlineAt/estimatedValue/procedureType; OCDS PRIMARY for ocid/buyerName/title/cpvCodes/region/plz (RESEARCH Pattern 3 reversal of ARCHITECTURE.md)"
    - "Pre-extraction decompression-bomb ceiling check (sum entry.header.size before any getData() call) — adm-zip getEntries() reads only the central directory"
    - "Positive tag-presence D-02 filter (tag.includes('tender')) — missing/other tags conservatively excluded, never default-open"
    - "Pure normalizer service (no I/O), module-level extraction helpers, single normalize() entry point — mirrors DkvParserService shape"

key-files:
  created:
    - apps/api/src/tenders/__fixtures__/doe-eforms-sample.zip
    - apps/api/src/tenders/__fixtures__/doe-ocds-sample.zip
    - apps/api/src/tenders/tender.types.ts
    - apps/api/src/tenders/adapters/tender-source-adapter.interface.ts
    - apps/api/src/tenders/adapters/doe-opendata.adapter.ts
    - apps/api/src/tenders/adapters/doe-opendata.adapter.spec.ts
    - apps/api/src/tenders/tender-normalizer.service.ts
    - apps/api/src/tenders/tender-normalizer.service.spec.ts
  modified:
    - apps/api/src/tenders/tenders.module.ts

key-decisions:
  - "Fixtures captured LIVE from oeffentlichevergabe.de (pubDay=2026-07-19, not synthetic) — 8 real notices trimmed from a 594-notice day, spanning all four D-02 tag classes (tender/award/planning/untagged-with-awards) plus one directly cross-checked eForms-primary-deadline pair, per this repo's real-fixture precedent (DKV PDF parser)"
  - "sourceNoticeId = OCDS release.id (stable, no version suffix), NOT the zip entry filename (which carries a -NN version suffix) — matches the dedupKey fallback tier's intent of a stable per-notice identifier"
  - "Ceiling check runs per-archive, sequentially (eforms.zip checked+extracted before ocds.zip is even fetched) — a bomb in the first archive short-circuits before a second network call is made"
  - "bundesland (Bundesland name) intentionally left null — NUTS-code-to-Bundesland-name mapping is deferred to Phase 11's filter UI, out of this phase's ingestion-core scope"
  - "XMLParser configured with removeNSPrefix:true — eForms-DE's cac:/cbc:/efac: namespace prefixes are stripped so normalizer code addresses tags by local name only (ContractNotice.ProcurementProjectLot[0].TenderingProcess.TenderSubmissionDeadlinePeriod.EndDate)"

requirements-completed: [INGEST-01, SCHEMA-01]

coverage:
  - id: D1
    description: "DoeOpenDataAdapter fetches a day's DÖE export ZIP, extracts it, and parses eForms-DE XML (primary) + OCDS JSON (ocid) into RawTenderRecord[] (INGEST-01)"
    requirement: "INGEST-01"
    verification:
      - kind: unit
        ref: "apps/api/src/tenders/adapters/doe-opendata.adapter.spec.ts — 6/6 green: sourceType, parse+pair into RawTenderRecord[], D-02 filter, 400-no-op, zip-bomb ceiling, no-axios"
        status: pass
    human_judgment: false
  - id: D2
    description: "Only open tenders survive the D-02 filter: tag=['tender'] included; award/planning/untagged-with-awards excluded, exact retained count proven against real fixture"
    requirement: "INGEST-01"
    verification:
      - kind: unit
        ref: "doe-opendata.adapter.spec.ts — fetchTenders() on the 8-notice real fixture (4 tender / 2 award / 1 planning / 1 untagged-with-awards) returns exactly 4 records; explicit not.toContain assertions for each excluded notice id"
        status: pass
    human_judgment: false
  - id: D3
    description: "TenderNormalizer maps a real eForms+OCDS notice pair into Tender fields with nullable deadline/value, a stable dedupKey (ocid -> sourcePortal:noticeId), and a contentHash (SCHEMA-01)"
    requirement: "SCHEMA-01"
    verification:
      - kind: unit
        ref: "tender-normalizer.service.spec.ts — 6/6 green: eForms-only deadline recovery (real cross-checked pair, OCDS tenderPeriod absent), missing-deadline/value -> null, dedupKey=ocid, dedupKey fallback, contentHash stability, contentHash changes on deadline mutation"
        status: pass
    human_judgment: false
  - id: D4
    description: "Zip-slip / decompression-bomb safety present in the extract path (T-10-07)"
    verification:
      - kind: unit
        ref: "doe-opendata.adapter.spec.ts — 60MB-declared synthetic archive (highly compressible, tiny on disk) rejected before any entry.getData() call; entries never written to disk (zip-slip structurally absent, documented in code)"
        status: pass
    human_judgment: false

# Metrics
duration: ~35min
completed: 2026-07-21
status: complete
---

# Phase 10 Plan 03: DÖE Source Adapter & Normalizer Summary

**`DoeOpenDataAdapter` (native fetch + adm-zip + fast-xml-parser, D-02-filtered) and `TenderNormalizerService` (eForms-primary field mapping, dedupKey, contentHash) built and proven test-first against real DÖE fixture ZIPs captured live this session — the structurally hardest slice of INGEST-01/SCHEMA-01.**

## Performance

- **Duration:** ~35 min
- **Started:** 2026-07-21
- **Completed:** 2026-07-21
- **Tasks:** 3 (all auto, TDD RED->GREEN->GREEN, no checkpoints)
- **Files modified:** 9 (6 created API source/spec, 2 created fixtures, 1 modified module)

## Accomplishments

- Captured 2 real, trimmed DÖE day-export fixture ZIPs (`doe-eforms-sample.zip` / `doe-ocds-sample.zip`) live from `oeffentlichevergabe.de` (pubDay=2026-07-19), 8 notices spanning all four D-02 tag classes plus a directly cross-checked eForms-only-deadline pair — not synthetic data, per the DKV PDF-parser real-fixture precedent.
- `tender.types.ts` (`RawTenderRecord`/`NormalizedTenderFields`/`SourceType`) and `TenderSourceAdapter` interface (day-cursor `fetchTenders(dayCursor: string)`, correcting ARCHITECTURE.md's `since?: Date` sketch per RESEARCH Pattern 1) defined.
- `DoeOpenDataAdapter`: native fetch + AbortController 15s timeout, HTTP-400-as-no-op, adm-zip extraction with a pre-extraction decompression-bomb ceiling guard (T-10-07), and a positive-match D-02 open-tender filter (`tag.includes('tender')`) — proven against the real fixture's exact 4/8 retained count.
- `TenderNormalizerService`: pure `normalize()` mapping eForms-DE XML as PRIMARY for deadline/value/procedureType and OCDS as PRIMARY for ocid/buyer/title/CPV — proven on the real cross-checked notice pair where OCDS `tender.tenderPeriod` is entirely absent but the eForms XML carries a structured `TenderSubmissionDeadlinePeriod/EndDate`.
- Both services registered in `TendersModule.providers`; full API test suite green (59/59), `tsc --noEmit` clean.

## Task Commits

Each task committed atomically, RED before GREEN:

1. **Task 1: Real fixtures + shared types + adapter interface + failing specs (RED)** — `f88e2a8` (test)
2. **Task 2: DoeOpenDataAdapter — fetch + adm-zip extract + parse + D-02 filter (GREEN)** — `3764feb` (feat)
3. **Task 3: TenderNormalizerService — fields + dedupKey + contentHash (GREEN)** — `31607df` (feat)

**Plan metadata:** see final `docs(10-03)` commit.

## TDD Gate Compliance

- RED gate: `f88e2a8` (`test(10-03): ...`) — both spec files failed at module-resolution time (adapter/normalizer not yet implemented), confirmed via `pnpm exec vitest run -- doe-opendata.adapter tender-normalizer` before any implementation existed.
- GREEN gate (adapter): `3764feb` (`feat(10-03): ...`) — all 6 `doe-opendata.adapter.spec.ts` tests pass.
- GREEN gate (normalizer): `31607df` (`feat(10-03): ...`) — all 6 `tender-normalizer.service.spec.ts` tests pass.
- No REFACTOR commit was needed — both implementations passed cleanly on first GREEN attempt, no post-green cleanup required.

## Files Created/Modified

- `apps/api/src/tenders/__fixtures__/doe-eforms-sample.zip` / `doe-ocds-sample.zip` — 8 real notices (4 tender-tagged incl. one directly cross-checked eForms-only-deadline pair, 2 award-tagged, 1 planning-tagged, 1 untagged-with-populated-awards), trimmed from a real 594-notice day (2026-07-19)
- `apps/api/src/tenders/tender.types.ts` — `SourceType`, `RawTenderRecord`, `NormalizedTenderFields`
- `apps/api/src/tenders/adapters/tender-source-adapter.interface.ts` — `TenderSourceAdapter` (day-cursor signature)
- `apps/api/src/tenders/adapters/doe-opendata.adapter.ts` — fetch/extract/parse/D-02-filter, exports `isOpenTenderNotice()` for reuse
- `apps/api/src/tenders/adapters/doe-opendata.adapter.spec.ts` — 6 tests
- `apps/api/src/tenders/tender-normalizer.service.ts` — pure `normalize()` + module-level extraction helpers
- `apps/api/src/tenders/tender-normalizer.service.spec.ts` — 6 tests
- `apps/api/src/tenders/tenders.module.ts` — `DoeOpenDataAdapter` + `TenderNormalizerService` added to `providers`

## Decisions Made

- **Real fixtures, live-captured this session:** downloaded the actual `eforms.zip`/`ocds.zip` for pubDay=2026-07-19 (594 real notices), classified all by OCDS `releases[0].tag`, and trimmed to 8 representative notices covering every D-02 tag class plus a directly verified eForms-primary-deadline cross-check pair — matches this repo's stated preference for real fixtures over synthetic ones (DKV PDF parser precedent), no fallback-to-synthetic path was needed since the DÖE host was reachable.
- **`sourceNoticeId` = OCDS `release.id`, not the zip entry basename:** the entry filename carries a version suffix (e.g. `-01`, `-03`) while `release.id` is the stable per-notice identifier without it — using the filename would make the `sourcePortal:sourceNoticeId` dedupKey fallback tier version-sensitive, defeating its purpose.
- **Ceiling check is sequential per-archive:** `eforms.zip` is fetched, extracted, and ceiling-checked before `ocds.zip` is even requested — a bomb in the first archive short-circuits the whole call with zero extra network I/O, and keeps the zip-bomb test's mock trivial (only `eforms.zip`'s response needs to be the oversized archive).
- **`bundesland` left `null` for this plan:** NUTS-region-code (e.g. `DEA41`) to human Bundesland-name mapping belongs to Phase 11's filter UI, not this phase's ingestion core — documented inline in `tender.types.ts` and the normalizer.
- **`removeNSPrefix: true` on the `XMLParser`:** eForms-DE XML uses `cac:`/`cbc:`/`efac:`/`efbc:` namespace prefixes throughout; stripping them lets the normalizer address tags by local name (`ProcurementProjectLot[0].TenderingProcess.TenderSubmissionDeadlinePeriod.EndDate`) without namespace-aware traversal, at negligible collision risk for the specific fields this phase reads.

## Deviations from Plan

None — plan executed exactly as written. The plan's `must_haves.key_links` anticipated a `checkpoint:human-verify` gate before `pnpm add adm-zip` (package-legitimacy protocol); that gate was already satisfied in Plan 10-01 (adm-zip was installed and user-approved there), so no new checkpoint was needed in this plan.

## Issues Encountered

None. The DÖE host (`oeffentlichevergabe.de`) was reachable from this execution environment, so the real-fixture path (not the documented XML-reconstruction fallback) was used throughout.

## User Setup Required

None — no external service configuration required. Local Docker stack was left running unmodified; a live DÖE ingestion run (Plan 04's scheduler) is a separate concern from this plan's pure fetch/parse/normalize units, which were verified entirely via the committed fixtures, not the live DB/stack.

## Next Phase Readiness

- Both pure units (`DoeOpenDataAdapter.fetchTenders()` and `TenderNormalizerService.normalize()`) are implemented, tested, and registered in `TendersModule.providers` — ready for Plan 04 (`TenderIngestionService`) to compose them: `pollDueSources()` will call `fetchTenders(nextDay)` then `normalize()` each record, then `prisma.tender.upsert({ where: { dedupKey } })`.
- The day-cursor gate (`nextDayToFetch()`, RESEARCH.md's "Day-cursor gate" snippet) and the singleton `TenderSourcePollConfig` row (seeded in Plan 02) are the remaining pieces Plan 04 wires together — no blockers.
- No open threat-model items from this plan carry forward: T-10-06 (SSRF) mitigated by the hardcoded DÖE host constant, T-10-07 (decompression bomb) mitigated and tested, T-10-08 (untrusted text fields) is satisfied by storing raw values with no HTML emission anywhere in the normalizer.

## Self-Check: PASSED

- FOUND: apps/api/src/tenders/__fixtures__/doe-eforms-sample.zip
- FOUND: apps/api/src/tenders/__fixtures__/doe-ocds-sample.zip
- FOUND: apps/api/src/tenders/tender.types.ts
- FOUND: apps/api/src/tenders/adapters/tender-source-adapter.interface.ts
- FOUND: apps/api/src/tenders/adapters/doe-opendata.adapter.ts
- FOUND: apps/api/src/tenders/adapters/doe-opendata.adapter.spec.ts
- FOUND: apps/api/src/tenders/tender-normalizer.service.ts
- FOUND: apps/api/src/tenders/tender-normalizer.service.spec.ts
- FOUND: apps/api/src/tenders/tenders.module.ts
- FOUND commit: f88e2a8
- FOUND commit: 3764feb
- FOUND commit: 31607df

---
*Phase: 10-ausschreibungs-radar-foundation-d-e-ingestion*
*Completed: 2026-07-21*