186 lines
14 KiB
Markdown
186 lines
14 KiB
Markdown
---
|
|
phase: 10-ausschreibungs-radar-foundation-d-e-ingestion
|
|
plan: 03
|
|
type: tdd
|
|
wave: 3
|
|
depends_on: ["10-01", "10-02"]
|
|
files_modified:
|
|
- apps/api/src/tenders/__fixtures__/doe-eforms-sample.zip
|
|
- apps/api/src/tenders/__fixtures__/doe-ocds-sample.zip
|
|
- apps/api/src/tenders/tender.types.ts
|
|
- apps/api/src/tenders/adapters/tender-source-adapter.interface.ts
|
|
- apps/api/src/tenders/adapters/doe-opendata.adapter.ts
|
|
- apps/api/src/tenders/adapters/doe-opendata.adapter.spec.ts
|
|
- apps/api/src/tenders/tender-normalizer.service.ts
|
|
- apps/api/src/tenders/tender-normalizer.service.spec.ts
|
|
- apps/api/src/tenders/tenders.module.ts
|
|
autonomous: true
|
|
requirements: [INGEST-01, SCHEMA-01]
|
|
must_haves:
|
|
truths:
|
|
- "DoeOpenDataAdapter fetches a day's DÖE export ZIP, extracts it, and parses eForms-DE XML (primary) + OCDS JSON (ocid) into RawTenderRecord[] (INGEST-01)"
|
|
- "Only open tenders survive the D-02 filter: tag=['tender'] included; award/planning/untagged-with-awards excluded"
|
|
- "TenderNormalizer maps a real eForms+OCDS notice pair into Tender fields with nullable deadline/value, a stable dedupKey (ocid → sourcePortal:noticeId), and a contentHash (SCHEMA-01)"
|
|
artifacts:
|
|
- apps/api/src/tenders/adapters/doe-opendata.adapter.ts
|
|
- apps/api/src/tenders/tender-normalizer.service.ts
|
|
- apps/api/src/tenders/__fixtures__/doe-eforms-sample.zip
|
|
key_links:
|
|
- "Adapter uses native fetch + AbortController timeout (icon-discovery idiom); HTTP 400 = expected no-op (today/future pubDay), 5xx/network throws"
|
|
- "dedupKey (ocid, else sourcePortal:sourceNoticeId) is the @unique upsert target consumed by Plan 04 ingestion"
|
|
---
|
|
|
|
<objective>
|
|
Build the DÖE source adapter and the normalizer — the parse+map core of INGEST-01/SCHEMA-01 — test-first against REAL captured fixture ZIPs (not synthetic data), per this repo's real-fixture precedent (DKV PDF parser).
|
|
|
|
## Phase Goal (user story)
|
|
**As a** Tessera-Administrator, **I want to** dass echte DÖE-Ausschreibungen aus dem Tages-Export korrekt gelesen und in ein einheitliches Schema normalisiert werden, **so that** nur offene, bietbare Vergaben (D-02) mit stabilen Dedup-Schluesseln im globalen Katalog landen.
|
|
|
|
Purpose: This is the structurally hardest part of the phase — correctly interpreting the DÖE day-batch ZIP shape, choosing eForms-DE XML as the primary structured-field source (deadline/value that OCDS drops), and applying the tag-presence D-02 filter. Both units are pure/deterministic → ideal for TDD.
|
|
Output: `DoeOpenDataAdapter`, `TenderNormalizerService`, the adapter interface, shared types, and committed real fixtures.
|
|
</objective>
|
|
|
|
<execution_context>
|
|
@$HOME/.claude/gsd-core/workflows/execute-plan.md
|
|
@$HOME/.claude/gsd-core/templates/summary.md
|
|
</execution_context>
|
|
|
|
<context>
|
|
@.planning/PROJECT.md
|
|
@.planning/ROADMAP.md
|
|
@.planning/phases/10-ausschreibungs-radar-foundation-d-e-ingestion/10-RESEARCH.md
|
|
@.planning/phases/10-ausschreibungs-radar-foundation-d-e-ingestion/10-PATTERNS.md
|
|
@apps/api/src/favorites/icon-discovery.service.ts
|
|
@apps/api/src/dkv/dkv-parser.service.ts
|
|
</context>
|
|
|
|
<tasks>
|
|
|
|
<task type="auto">
|
|
<name>Task 1: Wave 0 — capture real DÖE fixtures + shared types + adapter interface + failing specs</name>
|
|
<read_first>
|
|
- .planning/phases/10-ausschreibungs-radar-foundation-d-e-ingestion/10-RESEARCH.md (DÖE API Live Findings: endpoint, pubDay boundary rules, tag distribution, eForms-vs-OCDS field cross-check)
|
|
- apps/api/src/dkv/dkv.types.ts (types file conventions)
|
|
- .planning/research/ARCHITECTURE.md (Pattern 1: RawTenderRecord / TenderSourceAdapter interface sketch)
|
|
</read_first>
|
|
<files>apps/api/src/tenders/__fixtures__/doe-eforms-sample.zip, apps/api/src/tenders/__fixtures__/doe-ocds-sample.zip, apps/api/src/tenders/tender.types.ts, apps/api/src/tenders/adapters/tender-source-adapter.interface.ts, apps/api/src/tenders/adapters/doe-opendata.adapter.spec.ts, apps/api/src/tenders/tender-normalizer.service.spec.ts</files>
|
|
<action>
|
|
Capture REAL fixtures: compute a pubDay at least 2 calendar days in the past (Europe/Berlin), then download `https://oeffentlichevergabe.de/api/notice-exports?pubDay={day}&format=eforms.zip` and `...&format=ocds.zip`. Trim each archive to ~5-10 representative notices spanning the tag classes RESEARCH observed (`tender`, `award`, `planning`, and at least one untagged-with-awards) and save the trimmed ZIPs as `doe-eforms-sample.zip` / `doe-ocds-sample.zip` under `__fixtures__/`. Keep them small (do not commit the full 105-file archive). If the host is unreachable from the execution environment, construct minimal fixtures faithfully reproducing the real element shapes documented in RESEARCH (eForms `TenderSubmissionDeadlinePeriod/EndDate`, OCDS `releases[].tag`, `ocid` `ocds-mnwr74-...`) and note the substitution in the summary.
|
|
|
|
Define `tender.types.ts`: `SourceType` (`'doe-opendata'` for this phase), `RawTenderRecord` (sourceType, sourcePortal, sourceNoticeId, ocid?, sourceUrl, fetchedAt, eformsPayload/ocdsPayload or a unified payload, publishedAt), and `NormalizedTenderFields` (the Tender column subset the normalizer emits, plus dedupKey + contentHash).
|
|
|
|
Define `tender-source-adapter.interface.ts`: `TenderSourceAdapter` with `readonly sourceType: SourceType` and `fetchTenders(dayCursor: string): Promise<RawTenderRecord[]>` (day-cursor string `YYYY-MM-DD`, NOT a `since: Date` — RESEARCH Pattern 1 corrects ARCHITECTURE.md's signature).
|
|
|
|
Write FAILING specs first (RED): `doe-opendata.adapter.spec.ts` loads the fixture ZIPs and asserts (a) N raw records parsed, (b) the D-02 filter keeps only `tag=['tender']` and drops award/planning/untagged-with-awards (assert exact filtered count), (c) HTTP 400 fetch path returns `[]` (no-op). `tender-normalizer.service.spec.ts` asserts a known eForms+OCDS notice pair maps to the expected Tender fields including a non-null deadline recovered from eForms XML where OCDS is null, nullable value handled, and dedupKey = the notice's `ocid`.
|
|
</action>
|
|
<verify>
|
|
<automated>cd apps/api && test -f src/tenders/__fixtures__/doe-eforms-sample.zip && pnpm test -- doe-opendata.adapter tender-normalizer 2>&1 | tail -20</automated>
|
|
</verify>
|
|
<acceptance_criteria>
|
|
- Fixture ZIPs exist under `__fixtures__/`.
|
|
- `tender.types.ts` and `tender-source-adapter.interface.ts` compile.
|
|
- Both spec files exist and FAIL for the right reason (implementation not yet written) — RED state confirmed.
|
|
</acceptance_criteria>
|
|
<done>Real fixtures captured, types + interface defined, failing specs committed (test(...) commit).</done>
|
|
</task>
|
|
|
|
<task type="auto" tdd="true">
|
|
<name>Task 2: DoeOpenDataAdapter — fetch + adm-zip extract + parse + D-02 filter (GREEN)</name>
|
|
<read_first>
|
|
- apps/api/src/favorites/icon-discovery.service.ts (native fetch + AbortController timeout idiom, lines ~254-287)
|
|
- .planning/phases/10-ausschreibungs-radar-foundation-d-e-ingestion/10-PATTERNS.md (doe-opendata.adapter section: fetchDoeDay shape, 400=no-op)
|
|
- .planning/phases/10-ausschreibungs-radar-foundation-d-e-ingestion/10-RESEARCH.md (Security Domain: zip-bomb ceiling; D-02 isOpenTenderNotice)
|
|
- apps/api/src/tenders/adapters/doe-opendata.adapter.spec.ts (the RED spec from Task 1)
|
|
</read_first>
|
|
<behavior>
|
|
- Test: fetching a valid past pubDay returns RawTenderRecord[] parsed from the fixture ZIP.
|
|
- Test: a pubDay yielding HTTP 400 (today/future) returns [] without throwing.
|
|
- Test: D-02 filter keeps tag=['tender'], excludes award/planning/untagged — assert exact retained count for the fixture.
|
|
- Test: total uncompressed ZIP size over the ceiling (e.g. 50MB) is rejected before extraction (zip-bomb guard).
|
|
</behavior>
|
|
<files>apps/api/src/tenders/adapters/doe-opendata.adapter.ts, apps/api/src/tenders/tenders.module.ts</files>
|
|
<action>
|
|
Implement `DoeOpenDataAdapter` (Injectable) with `sourceType = 'doe-opendata'` and `fetchTenders(dayCursor)`:
|
|
- Fetch `eforms.zip` and `ocds.zip` for the given `dayCursor` via native `fetch` wrapped in an `AbortController` + 15s `setTimeout` (icon-discovery idiom — no axios). Treat HTTP 400 as an expected no-op signal (return `[]`); any other `!res.ok` throws.
|
|
- Extract with `adm-zip`: `new AdmZip(buffer).getEntries()`. BEFORE reading entry buffers, sum `entry.header.size` across entries and reject if it exceeds a sane ceiling (~50MB) — decompression-bomb defence-in-depth (RESEARCH Security Domain). Zip-slip is not a write risk here (in-memory only, never writing entries to disk) — do not write extracted entries to the filesystem.
|
|
- Parse each eForms-DE XML entry with `fast-xml-parser` (primary structured fields); parse the paired OCDS JSON entries with `JSON.parse` for `ocid` and buyer/party names. Pair eForms↔OCDS by notice id.
|
|
- Apply the D-02 open-tender filter on the OCDS release tag: keep when `release.tag` includes `'tender'`; exclude award, planning, and untagged-with-awards notices (positive tag-presence match per D-02 / RESEARCH Pattern 2 — never "absence of award = open"). Ingest is whole-Germany with NO region/CPV pre-filtering (D-03).
|
|
- Return `RawTenderRecord[]`.
|
|
Register `DoeOpenDataAdapter` in `TendersModule.providers`.
|
|
</action>
|
|
<verify>
|
|
<automated>cd apps/api && pnpm test -- doe-opendata.adapter 2>&1 | tail -15</automated>
|
|
</verify>
|
|
<acceptance_criteria>
|
|
- All `doe-opendata.adapter.spec.ts` tests pass (GREEN).
|
|
- D-02 retained-count assertion matches the fixture's `tender`-tagged subset exactly.
|
|
- Zip-bomb ceiling test passes (oversized archive rejected pre-extraction).
|
|
- Uses native fetch (assert no axios import).
|
|
</acceptance_criteria>
|
|
<done>Adapter parses fixtures into filtered RawTenderRecord[]; spec green; feat(...) commit.</done>
|
|
</task>
|
|
|
|
<task type="auto" tdd="true">
|
|
<name>Task 3: TenderNormalizerService — fields + dedupKey + contentHash (GREEN)</name>
|
|
<read_first>
|
|
- apps/api/src/dkv/dkv-parser.service.ts (transform-service shape: one public entry point + private helpers, no I/O)
|
|
- .planning/phases/10-ausschreibungs-radar-foundation-d-e-ingestion/10-PATTERNS.md (tender-normalizer.service section)
|
|
- .planning/phases/10-ausschreibungs-radar-foundation-d-e-ingestion/10-RESEARCH.md (Pattern 3 eForms-primary; Pattern 4 nullable deadline/value; dedup key priority)
|
|
- apps/api/src/tenders/tender-normalizer.service.spec.ts (the RED spec from Task 1)
|
|
</read_first>
|
|
<behavior>
|
|
- Test: eForms `TenderSubmissionDeadlinePeriod/EndDate` populates `deadlineAt` even when OCDS deadline is null.
|
|
- Test: missing deadline/value normalize to null (not thrown, not zero).
|
|
- Test: dedupKey = ocid when present; falls back to `${sourcePortal}:${sourceNoticeId}` when ocid absent.
|
|
- Test: contentHash is a stable sha256 over title+deadline+value+status; changing the deadline changes the hash.
|
|
</behavior>
|
|
<files>apps/api/src/tenders/tender-normalizer.service.ts, apps/api/src/tenders/tenders.module.ts</files>
|
|
<action>
|
|
Implement `TenderNormalizerService` (Injectable, pure — no I/O) with `normalize(raw: RawTenderRecord): NormalizedTenderFields`. Map eForms-DE XML as the PRIMARY source for `deadlineAt`, `estimatedValue`, `procedureType` (Pattern 3 — reverses ARCHITECTURE.md's OCDS-first guidance based on live cross-check); use OCDS for `ocid`, `buyerName`, party names. Extract `cpvCodes`, `region`/`plz`/`bundesland`, `title`, `sourceUrl`, `publishedAt`. Keep `deadlineAt`/`estimatedValue` nullable (Pattern 4 — null is the common case for Unterschwelle). Compute `dedupKey` priority-ordered: `ocid` → `${sourcePortal}:${sourceNoticeId}` (fuzzy fingerprint is Phase 13, out of scope here). Compute `contentHash = sha256(title + deadlineAt + estimatedValue + status)` (SCHEMA-02 hook). Set initial `status = 'active'`. Register the service in `TendersModule.providers`.
|
|
</action>
|
|
<verify>
|
|
<automated>cd apps/api && pnpm test -- tender-normalizer 2>&1 | tail -15</automated>
|
|
</verify>
|
|
<acceptance_criteria>
|
|
- All `tender-normalizer.service.spec.ts` tests pass (GREEN).
|
|
- Deadline recovered from eForms XML where OCDS is null (the key SCHEMA-01 assertion).
|
|
- dedupKey falls back correctly when ocid absent; contentHash changes when deadline changes.
|
|
</acceptance_criteria>
|
|
<done>Normalizer maps notice pairs to Tender fields deterministically; spec green; feat(...) commit.</done>
|
|
</task>
|
|
|
|
</tasks>
|
|
|
|
<threat_model>
|
|
## Trust Boundaries
|
|
|
|
| Boundary | Description |
|
|
|----------|-------------|
|
|
| DÖE host → API process | Outbound HTTP to a fixed external government host; untrusted archive + XML/JSON content enters the Node process |
|
|
| ZIP archive → extraction | Compressed third-party payload decompressed in-memory |
|
|
|
|
## STRIDE Threat Register
|
|
|
|
| Threat ID | Category | Component | Severity | Disposition | Mitigation Plan |
|
|
|-----------|----------|-----------|----------|-------------|-----------------|
|
|
| T-10-06 | Server-Side Request Forgery | `DoeOpenDataAdapter` outbound fetch | high | mitigate | URL host is a hardcoded constant (`oeffentlichevergabe.de`); only the internally-computed `pubDay` cursor is interpolated (never user input). Native fetch + AbortController 15s timeout bounds the request (icon-discovery idiom). No admin-supplied URL flows into the fetch |
|
|
| T-10-07 | Denial of Service | `adm-zip` extraction | high | mitigate | Sum `entry.header.size` and reject archives over a ~50MB uncompressed ceiling BEFORE reading entry buffers (decompression-bomb guard). Entries are read in-memory only — never written to disk, so zip-slip path traversal is structurally absent |
|
|
| T-10-08 | Tampering / Injection | parsed eForms/OCDS text fields | medium | mitigate | All ingested text (buyer names, titles, descriptions) is external untrusted input; store raw, do NOT assume "safe" because it came from a government API. Escaping happens at render time (Phase 11 UI) — the normalizer must not emit HTML |
|
|
</threat_model>
|
|
|
|
<verification>
|
|
- `pnpm --filter @tessera/api test -- doe-opendata.adapter tender-normalizer` all green.
|
|
- D-02 filter proven by exact retained-count assertion on real fixtures.
|
|
- eForms-primary deadline recovery proven (deadline present where OCDS is null).
|
|
- No axios import; zip-bomb ceiling enforced.
|
|
</verification>
|
|
|
|
<success_criteria>
|
|
- INGEST-01: DÖE day-export ZIP is fetched, extracted, and parsed into filtered open-tender raw records.
|
|
- SCHEMA-01: raw records normalize into the unified Tender field shape with stable dedupKey + contentHash.
|
|
</success_criteria>
|
|
|
|
<output>
|
|
Create `.planning/phases/10-ausschreibungs-radar-foundation-d-e-ingestion/10-03-SUMMARY.md` when done.
|
|
</output>
|