tessera-ctl/.planning/phases/10-ausschreibungs-radar-foundation-d-e-ingestion/10-03-SUMMARY.md

phase, plan, subsystem, tags, requires, provides, affects, tech-stack, key-files, key-decisions, requirements-completed, coverage, duration, completed, status

phase	plan	subsystem	tags	requires	provides	affects	tech-stack	key-files	key-decisions	requirements-completed	coverage	duration	completed	status
10-ausschreibungs-radar-foundation-d-e-ingestion	03	api	doe-opendata adm-zip fast-xml-parser eforms ocds tdd normalizer ingestion	phase provides 10-01 (foundation & dependencies) Tender/TenderSourcePollConfig Prisma models, fast-xml-parser/adm-zip/csv-parse installed phase provides 10-02 (marketplace registration) TendersModule skeleton with empty providers array, ready for adapter/normalizer wiring	DoeOpenDataAdapter — fetches, extracts, and D-02-filters the DÖE day-batch export into RawTenderRecord[] (INGEST-01) TenderNormalizerService — maps RawTenderRecord to Tender fields, dedupKey, contentHash (SCHEMA-01) Real, trimmed DÖE fixture ZIPs (doe-eforms-sample.zip / doe-ocds-sample.zip) for future ingestion-service tests RawTenderRecord/NormalizedTenderFields/SourceType shared types + TenderSourceAdapter interface (day-cursor signature)	10-04 ingestion/scheduler (consumes fetchTenders + normalize) 10-05 controller/DTOs phase 11 saved searches/filter UI	added patterns Day-cursor adapter contract (fetchTenders(dayCursor: string)) — not since:Date — per DÖE's daily-batch-only API shape eForms-DE XML as PRIMARY source for deadlineAt/estimatedValue/procedureType; OCDS PRIMARY for ocid/buyerName/title/cpvCodes/region/plz (RESEARCH Pattern 3 reversal of ARCHITECTURE.md) Pre-extraction decompression-bomb ceiling check (sum entry.header.size before any getData() call) — adm-zip getEntries() reads only the central directory Positive tag-presence D-02 filter (tag.includes('tender')) — missing/other tags conservatively excluded, never default-open Pure normalizer service (no I/O), module-level extraction helpers, single normalize() entry point — mirrors DkvParserService shape	created modified apps/api/src/tenders/__fixtures__/doe-eforms-sample.zip apps/api/src/tenders/__fixtures__/doe-ocds-sample.zip apps/api/src/tenders/tender.types.ts apps/api/src/tenders/adapters/tender-source-adapter.interface.ts apps/api/src/tenders/adapters/doe-opendata.adapter.ts apps/api/src/tenders/adapters/doe-opendata.adapter.spec.ts apps/api/src/tenders/tender-normalizer.service.ts apps/api/src/tenders/tender-normalizer.service.spec.ts apps/api/src/tenders/tenders.module.ts	Fixtures captured LIVE from oeffentlichevergabe.de (pubDay=2026-07-19, not synthetic) — 8 real notices trimmed from a 594-notice day, spanning all four D-02 tag classes (tender/award/planning/untagged-with-awards) plus one directly cross-checked eForms-primary-deadline pair, per this repo's real-fixture precedent (DKV PDF parser) sourceNoticeId = OCDS release.id (stable, no version suffix), NOT the zip entry filename (which carries a -NN version suffix) — matches the dedupKey fallback tier's intent of a stable per-notice identifier Ceiling check runs per-archive, sequentially (eforms.zip checked+extracted before ocds.zip is even fetched) — a bomb in the first archive short-circuits before a second network call is made bundesland (Bundesland name) intentionally left null — NUTS-code-to-Bundesland-name mapping is deferred to Phase 11's filter UI, out of this phase's ingestion-core scope XMLParser configured with removeNSPrefix:true — eForms-DE's cac:/cbc:/efac: namespace prefixes are stripped so normalizer code addresses tags by local name only (ContractNotice.ProcurementProjectLot[0].TenderingProcess.TenderSubmissionDeadlinePeriod.EndDate)	INGEST-01 SCHEMA-01	id description requirement verification human_judgment D1 DoeOpenDataAdapter fetches a day's DÖE export ZIP, extracts it, and parses eForms-DE XML (primary) + OCDS JSON (ocid) into RawTenderRecord[] (INGEST-01) INGEST-01 kind ref status unit apps/api/src/tenders/adapters/doe-opendata.adapter.spec.ts — 6/6 green: sourceType, parse+pair into RawTenderRecord[], D-02 filter, 400-no-op, zip-bomb ceiling, no-axios pass false id description requirement verification human_judgment D2 Only open tenders survive the D-02 filter: tag=['tender'] included; award/planning/untagged-with-awards excluded, exact retained count proven against real fixture INGEST-01 kind ref status unit doe-opendata.adapter.spec.ts — fetchTenders() on the 8-notice real fixture (4 tender / 2 award / 1 planning / 1 untagged-with-awards) returns exactly 4 records; explicit not.toContain assertions for each excluded notice id pass false id description requirement verification human_judgment D3 TenderNormalizer maps a real eForms+OCDS notice pair into Tender fields with nullable deadline/value, a stable dedupKey (ocid -> sourcePortal:noticeId), and a contentHash (SCHEMA-01) SCHEMA-01 kind ref status unit tender-normalizer.service.spec.ts — 6/6 green: eForms-only deadline recovery (real cross-checked pair, OCDS tenderPeriod absent), missing-deadline/value -> null, dedupKey=ocid, dedupKey fallback, contentHash stability, contentHash changes on deadline mutation pass false id description verification human_judgment D4 Zip-slip / decompression-bomb safety present in the extract path (T-10-07) kind ref status unit doe-opendata.adapter.spec.ts — 60MB-declared synthetic archive (highly compressible, tiny on disk) rejected before any entry.getData() call; entries never written to disk (zip-slip structurally absent, documented in code) pass false	~35min	2026-07-21	complete

Phase 10 Plan 03: DÖE Source Adapter & Normalizer Summary

DoeOpenDataAdapter (native fetch + adm-zip + fast-xml-parser, D-02-filtered) and TenderNormalizerService (eForms-primary field mapping, dedupKey, contentHash) built and proven test-first against real DÖE fixture ZIPs captured live this session — the structurally hardest slice of INGEST-01/SCHEMA-01.

Performance

• Duration: ~35 min
• Started: 2026-07-21
• Completed: 2026-07-21
• Tasks: 3 (all auto, TDD RED->GREEN->GREEN, no checkpoints)
• Files modified: 9 (6 created API source/spec, 2 created fixtures, 1 modified module)

Accomplishments

• Captured 2 real, trimmed DÖE day-export fixture ZIPs (doe-eforms-sample.zip / doe-ocds-sample.zip) live from oeffentlichevergabe.de (pubDay=2026-07-19), 8 notices spanning all four D-02 tag classes plus a directly cross-checked eForms-only-deadline pair — not synthetic data, per the DKV PDF-parser real-fixture precedent.
• tender.types.ts (RawTenderRecord/NormalizedTenderFields/SourceType) and TenderSourceAdapter interface (day-cursor fetchTenders(dayCursor: string), correcting ARCHITECTURE.md's since?: Date sketch per RESEARCH Pattern 1) defined.
• DoeOpenDataAdapter: native fetch + AbortController 15s timeout, HTTP-400-as-no-op, adm-zip extraction with a pre-extraction decompression-bomb ceiling guard (T-10-07), and a positive-match D-02 open-tender filter (tag.includes('tender')) — proven against the real fixture's exact 4/8 retained count.
• TenderNormalizerService: pure normalize() mapping eForms-DE XML as PRIMARY for deadline/value/procedureType and OCDS as PRIMARY for ocid/buyer/title/CPV — proven on the real cross-checked notice pair where OCDS tender.tenderPeriod is entirely absent but the eForms XML carries a structured TenderSubmissionDeadlinePeriod/EndDate.
• Both services registered in TendersModule.providers; full API test suite green (59/59), tsc --noEmit clean.

Task Commits

Each task committed atomically, RED before GREEN:

1. Task 1: Real fixtures + shared types + adapter interface + failing specs (RED) — f88e2a8 (test)
2. Task 2: DoeOpenDataAdapter — fetch + adm-zip extract + parse + D-02 filter (GREEN) — 3764feb (feat)
3. Task 3: TenderNormalizerService — fields + dedupKey + contentHash (GREEN) — 31607df (feat)

Plan metadata: see final docs(10-03) commit.

TDD Gate Compliance

• RED gate: f88e2a8 (test(10-03): ...) — both spec files failed at module-resolution time (adapter/normalizer not yet implemented), confirmed via pnpm exec vitest run -- doe-opendata.adapter tender-normalizer before any implementation existed.
• GREEN gate (adapter): 3764feb (feat(10-03): ...) — all 6 doe-opendata.adapter.spec.ts tests pass.
• GREEN gate (normalizer): 31607df (feat(10-03): ...) — all 6 tender-normalizer.service.spec.ts tests pass.
• No REFACTOR commit was needed — both implementations passed cleanly on first GREEN attempt, no post-green cleanup required.

Files Created/Modified

• apps/api/src/tenders/__fixtures__/doe-eforms-sample.zip / doe-ocds-sample.zip — 8 real notices (4 tender-tagged incl. one directly cross-checked eForms-only-deadline pair, 2 award-tagged, 1 planning-tagged, 1 untagged-with-populated-awards), trimmed from a real 594-notice day (2026-07-19)
• apps/api/src/tenders/tender.types.ts — SourceType, RawTenderRecord, NormalizedTenderFields
• apps/api/src/tenders/adapters/tender-source-adapter.interface.ts — TenderSourceAdapter (day-cursor signature)
• apps/api/src/tenders/adapters/doe-opendata.adapter.ts — fetch/extract/parse/D-02-filter, exports isOpenTenderNotice() for reuse
• apps/api/src/tenders/adapters/doe-opendata.adapter.spec.ts — 6 tests
• apps/api/src/tenders/tender-normalizer.service.ts — pure normalize() + module-level extraction helpers
• apps/api/src/tenders/tender-normalizer.service.spec.ts — 6 tests
• apps/api/src/tenders/tenders.module.ts — DoeOpenDataAdapter + TenderNormalizerService added to providers

Decisions Made

• Real fixtures, live-captured this session: downloaded the actual eforms.zip/ocds.zip for pubDay=2026-07-19 (594 real notices), classified all by OCDS releases[0].tag, and trimmed to 8 representative notices covering every D-02 tag class plus a directly verified eForms-primary-deadline cross-check pair — matches this repo's stated preference for real fixtures over synthetic ones (DKV PDF parser precedent), no fallback-to-synthetic path was needed since the DÖE host was reachable.
• sourceNoticeId = OCDS release.id, not the zip entry basename: the entry filename carries a version suffix (e.g. -01, -03) while release.id is the stable per-notice identifier without it — using the filename would make the sourcePortal:sourceNoticeId dedupKey fallback tier version-sensitive, defeating its purpose.
• Ceiling check is sequential per-archive: eforms.zip is fetched, extracted, and ceiling-checked before ocds.zip is even requested — a bomb in the first archive short-circuits the whole call with zero extra network I/O, and keeps the zip-bomb test's mock trivial (only eforms.zip's response needs to be the oversized archive).
• bundesland left null for this plan: NUTS-region-code (e.g. DEA41) to human Bundesland-name mapping belongs to Phase 11's filter UI, not this phase's ingestion core — documented inline in tender.types.ts and the normalizer.
• removeNSPrefix: true on the XMLParser: eForms-DE XML uses cac:/cbc:/efac:/efbc: namespace prefixes throughout; stripping them lets the normalizer address tags by local name (ProcurementProjectLot[0].TenderingProcess.TenderSubmissionDeadlinePeriod.EndDate) without namespace-aware traversal, at negligible collision risk for the specific fields this phase reads.

Deviations from Plan

None — plan executed exactly as written. The plan's must_haves.key_links anticipated a checkpoint:human-verify gate before pnpm add adm-zip (package-legitimacy protocol); that gate was already satisfied in Plan 10-01 (adm-zip was installed and user-approved there), so no new checkpoint was needed in this plan.

Issues Encountered

None. The DÖE host (oeffentlichevergabe.de) was reachable from this execution environment, so the real-fixture path (not the documented XML-reconstruction fallback) was used throughout.

User Setup Required

None — no external service configuration required. Local Docker stack was left running unmodified; a live DÖE ingestion run (Plan 04's scheduler) is a separate concern from this plan's pure fetch/parse/normalize units, which were verified entirely via the committed fixtures, not the live DB/stack.

Next Phase Readiness

• Both pure units (DoeOpenDataAdapter.fetchTenders() and TenderNormalizerService.normalize()) are implemented, tested, and registered in TendersModule.providers — ready for Plan 04 (TenderIngestionService) to compose them: pollDueSources() will call fetchTenders(nextDay) then normalize() each record, then prisma.tender.upsert({ where: { dedupKey } }).
• The day-cursor gate (nextDayToFetch(), RESEARCH.md's "Day-cursor gate" snippet) and the singleton TenderSourcePollConfig row (seeded in Plan 02) are the remaining pieces Plan 04 wires together — no blockers.
• No open threat-model items from this plan carry forward: T-10-06 (SSRF) mitigated by the hardcoded DÖE host constant, T-10-07 (decompression bomb) mitigated and tested, T-10-08 (untrusted text fields) is satisfied by storing raw values with no HTML emission anywhere in the normalizer.

Self-Check: PASSED

• FOUND: apps/api/src/tenders/fixtures/doe-eforms-sample.zip
• FOUND: apps/api/src/tenders/fixtures/doe-ocds-sample.zip
• FOUND: apps/api/src/tenders/tender.types.ts
• FOUND: apps/api/src/tenders/adapters/tender-source-adapter.interface.ts
• FOUND: apps/api/src/tenders/adapters/doe-opendata.adapter.ts
• FOUND: apps/api/src/tenders/adapters/doe-opendata.adapter.spec.ts
• FOUND: apps/api/src/tenders/tender-normalizer.service.ts
• FOUND: apps/api/src/tenders/tender-normalizer.service.spec.ts
• FOUND: apps/api/src/tenders/tenders.module.ts
• FOUND commit: f88e2a8
• FOUND commit: 3764feb
• FOUND commit: 31607df

---

Phase: 10-ausschreibungs-radar-foundation-d-e-ingestion Completed: 2026-07-21