From 3a51d9bfedf035ad221ee2f74be1d833dcf3e6e1 Mon Sep 17 00:00:00 2001 From: Schalli Date: Tue, 21 Jul 2026 09:46:20 +0200 Subject: [PATCH] =?UTF-8?q?docs(10):=20research=20D=C3=96E=20OpenData=20AP?= =?UTF-8?q?I=20live=20verification=20and=20phase=20domain?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../10-RESEARCH.md | 418 ++++++++++++++++++ 1 file changed, 418 insertions(+) create mode 100644 .planning/phases/10-ausschreibungs-radar-foundation-d-e-ingestion/10-RESEARCH.md diff --git a/.planning/phases/10-ausschreibungs-radar-foundation-d-e-ingestion/10-RESEARCH.md b/.planning/phases/10-ausschreibungs-radar-foundation-d-e-ingestion/10-RESEARCH.md new file mode 100644 index 0000000..b8db88a --- /dev/null +++ b/.planning/phases/10-ausschreibungs-radar-foundation-d-e-ingestion/10-RESEARCH.md @@ -0,0 +1,418 @@ +# Phase 10: Ausschreibungs-Radar Foundation & DÖE Ingestion - Research + +**Researched:** 2026-07-21 +**Domain:** German public-procurement (DÖE OpenData) live API ingestion, OCDS/eForms-DE normalization, multi-tenant poll-once-fan-out-many scheduler, module-registry integration on the existing Tessera NestJS/Prisma platform +**Confidence:** HIGH (DÖE API mechanics — verified live against the production API this session, not just the JS-rendered Swagger UI); HIGH (platform integration — verified against existing `dkv/`/`module-registry/` codebase); MEDIUM (OCDS field completeness — verified against real live export, one day's sample; genuinely variable by buyer/portal) + + +## User Constraints (from CONTEXT.md) + +### Locked Decisions + +**DÖE-Ingest-Umfang** +- **D-01:** Initialer Backfill = **nur ab jetzt**. Beim ersten plattformweiten DÖE-Poll werden nur ab diesem Zeitpunkt veröffentlichte Ausschreibungen aufgenommen — kein historischer Import. Da die Tender-Daten global sind, sieht ein später aktivierender Mandant den seit Plattformstart aufgelaufenen Bestand. Passt zur späteren Backfill-Unterdrückung bei Benachrichtigungen (Phase 12). +- **D-02:** Nur **offene Ausschreibungen** (aktive Vergaben, auf die man bieten kann — tender/contract notices). Vergabeergebnisse, Zuschläge und Aufhebungen (award/result notices) werden in dieser Phase NICHT aufgenommen. +- **D-03:** Ingest lädt **ganz Deutschland global** (plattformweit einmal), **keine** Vor-Eingrenzung nach Region/CPV beim Ingest — Eingrenzung passiert erst pro Suchprofil (Phase 11). Bewusst wegen Multi-Tenant-Wiederverkauf. +- **D-04:** Standard-Poll-Intervall = **stündlich**, admin-konfigurierbar (INGEST-06). + +**Datenaufbewahrung** +- **D-05:** Ausschreibungen nach Ablauf der Abgabefrist werden **90 Tage** aufbewahrt, dann gelöscht. Vor Fristablauf: aktiv. Nach Fristablauf: als „abgelaufen" markiert, standardmäßig aus der aktiven Liste ausgeblendet, aber bis zur Löschung recherchierbar. + +### Claude's Discretion +- Exakte Prisma-Schema-Felder (OCDS-orientiert, Form gemäß ARCHITECTURE.md), Adapter-Interface (`TenderSourceAdapter`), Normalizer-Interna, Dedup-Key-Berechnung (OCID → Quelle:NoticeId → …) und contentHash-Änderungserkennung. +- DÖE-API-Client (native `fetch` + `fast-xml-parser`, CSV-Fallback), Pagination/Rate-Limit-Handling (Detail per Phase-Research live zu klären). +- Scheduler-Implementierung: **poll-once-fan-out-many, mehrmandantensicher — NICHT** das DKV-`findFirst()`-Single-Tenant-Muster. +- Modul-Registrierung im Marketplace nach DKV/Cert-Manager-Vorbild. + +### Deferred Ideas (OUT OF SCOPE) +- Vergabeergebnisse/Zuschläge/Aufhebungen (award notices) aufnehmen — spätere Erweiterung, außerhalb v1.1-Scope. +- Historischer Backfill (letzte 30 Tage / alles) — bei Bedarf später als Konfig-Option. +- Vor-Eingrenzung des Ingest nach Region/CPV — bewusst verworfen (widerspricht Multi-Tenant-Wiederverkauf); Eingrenzung bleibt Sache der Suchprofile. +- Aufbewahrungsdauer (90 Tage) später ggf. admin-konfigurierbar machen. + + + +## Phase Requirements + +| ID | Description | Research Support | +|----|-------------|------------------| +| CONFIG-01 | Modul im Marketplace registriert, pro Mandant aktivierbar (wie DKV-Fleet/Cert-Manager) | `tenders.seed.ts` follows verbatim `cert-manager.seed.ts`/`dkv.seed.ts` pattern (`isSystem: true`, category, i18n description). `module-loader.ts` MODULE_REGISTRY entry required or the page 404s even when activated (verified codebase pattern, see Architecture Patterns §1) | +| INGEST-01 | Zeitgesteuerter Abruf via DÖE OpenData API (eForms/OCDS, auth-frei) | Live-verified in this session: real endpoint `GET /api/notice-exports`, auth-free (no security scheme in spec, confirmed 200 with no credentials), format negotiation via `Accept` header or `format` query param — see DÖE API Live Findings | +| INGEST-06 | Poll-once-fan-out-many, Intervall pro Quelle admin-konfigurierbar | See Scheduler Design — day-cursor gated single global cron (not per-tenant), admin-configurable interval field per D-04 with an adapter-level no-op guard for the DÖE source's actual day-granularity | +| SCHEMA-01 | Einheitliches OCDS-orientiertes Schema, plattform-global | See Prisma Schema Fields — extends ARCHITECTURE.md's `Tender` model with live-verified nullable-field realities (deadline/value frequently absent in OCDS export) | +| SCHEMA-02 | Content-Hash-Änderungserkennung (Fristverlängerung, Aufhebung) | See Dedup & Change Detection — DÖE re-publishes a changed notice as a new `noticeVersion` under the same `ocid`; contentHash diff on re-poll triggers update path | + + +## Summary + +This research supersedes the "open verification point" flagged in `.planning/research/SUMMARY.md`, `ARCHITECTURE.md`, and `PITFALLS.md`: **the DÖE OpenData API's Swagger UI was live-queried this session** (its JS-rendered UI was bypassed by reading `swagger-initializer.js` directly, which reveals the real spec URL `/documentation/api/opendata`). The live OpenAPI 3.1 spec was fetched, and — critically — **a real export ZIP for a real past day was downloaded and inspected** (105 real German tender notices from 2026-07-20, in both OCDS and eForms-DE XML format). This is ground truth, not documentation inference. + +**The single most important finding, which changes the phase's architecture from what ARCHITECTURE.md assumed:** the DÖE OpenData API is **not** a paginated/incremental REST feed. It is a **daily batch-export API**: one call returns a ZIP of *all* notice versions processed on a given calendar day (`pubDay=YYYY-MM-DD`) or month (`pubMonth=YYYY-MM`) — there is no `since`/cursor/page parameter of any kind, and **the current day and any future day are rejected with HTTP 400** ("must lie in the past"). New data for "today" only becomes retrievable starting tomorrow. This means the `TenderSourceAdapter.fetchTenders(config, since?: Date)` signature sketched in ARCHITECTURE.md should track a **day cursor** (`lastIngestedDay: Date`, day-granularity), not an arbitrary timestamp — and the admin-configurable poll interval (D-04, default hourly, INGEST-06) is architecturally harmless to run hourly (idempotent day-cursor check: no-op if the next day isn't past yet) but will only ever produce *new* data once every ~24h for this source. This must be explicitly designed into the scheduler (cron tick frequency ≠ actual-fetch frequency for this specific source) and documented so a future contributor doesn't "fix" the apparent 23-hours-of-no-op as a bug. + +**Second critical finding:** live inspection of the real OCDS export shows that **`tender.tenderPeriod.endDate` (deadline) and `tender.value` (estimated value) are frequently `null` in the OCDS conversion — even for open ("tender"-tagged) notices — while the same information is present in free text inside `tender.description`/`lots[].description`, and, for at least one directly cross-checked notice, present as a proper structured field (`TenderSubmissionDeadlinePeriod/EndDate`) in the corresponding eForms-DE XML that OCDS silently drops.** This directly affects SCHEMA-01 (deadline/value are supposed to be first-class normalized fields) and D-05 (90-day post-deadline retention needs a deadline value to compute from). The recommendation below is to parse **eForms-DE XML as the primary structured-field source** for deadline/value/procedure-type, using OCDS JSON primarily for the `ocid` (stable dedup key across notice versions) and party/organization resolution — a reversal of ARCHITECTURE.md's blanket "prefer OCDS" guidance, based on this session's live cross-check, not assumption. + +**Third finding:** the export format is a ZIP archive (confirmed: real `Zip archive data` file, one JSON/XML file per notice inside). **No ZIP-handling library was included in STACK.md** — this is a genuine gap in the milestone-level stack research (it assumed pagination, not archive downloads). `adm-zip` is recommended below. + +**Fourth finding, directly informing D-02:** live data confirms `releases[].tag` is the correct discriminator for "open tender" (`tag: ["tender"]`, 74/105 = 70% of one real day's sample) vs. `["award"]` (17/105) vs. `["planning"]` (3/105) vs. **no `tag` field at all** (11/105 — untagged notices that still carry populated `awards`/`contracts` arrays, e.g. direct-award justification notices/"Vergabevermerke"). D-02's filter must therefore be: `tag` array contains `"tender"` **and** treat a missing `tag` conservatively (exclude, don't default to "open") since several of the untagged samples are in fact awarded/closed procurements. + +**Primary recommendation:** Build the `DoeOpenDataAdapter` around the single real endpoint `GET /api/notice-exports?pubDay=YYYY-MM-DD&format=eforms.zip` (eForms-DE XML as primary parse target for deadline/value; OCDS JSON fetched in parallel or CSV as secondary cross-check per STACK.md's original recommendation), with a day-cursor scheduler (not a since-timestamp), `adm-zip` for archive extraction, and D-02 filtering on `tag=["tender"]` with `awards`/`contracts` emptiness as a secondary guard. + +## Architectural Responsibility Map + +| Capability | Primary Tier | Secondary Tier | Rationale | +|------------|-------------|----------------|-----------| +| DÖE polling (HTTP fetch of daily ZIP) | API / Backend (`TenderSourceAdapter`) | — | Server-side scheduled job; no browser/CDN involvement | +| ZIP extraction + eForms-XML/OCDS parsing | API / Backend (`TenderNormalizerService`) | — | CPU/IO-bound parsing belongs in the Node process, not the DB or client | +| Dedup key + contentHash computation | API / Backend (`TenderNormalizerService`) | — | Pure computation on normalized fields before persistence | +| `Tender` persistence (global, no tenantId) | Database / Storage | — | Platform-wide reference data; RLS/tenant scoping deliberately does NOT apply here (see Pattern 3 in ARCHITECTURE.md) | +| Scheduler / cron lifecycle | API / Backend (`TenderSchedulerService` via `SchedulerRegistry`) | — | Server-only concern; no client involvement | +| Module registry self-seed + tenant activation gate | API / Backend (`ModuleRegistryService`, reused unmodified) | Database (`Module`/`TenantModuleActivation` tables) | Existing platform mechanism, not phase-specific | +| Admin poll-interval config UI | Frontend Server (Next.js `settings/page.tsx`) | API / Backend (`TendersController` admin routes) | Mirrors DKV's `InboxConfigForm` — form on the client, persisted via API | +| Marketplace listing/activation UI | Frontend Server (existing Marketplace page) | — | Unmodified platform feature; this phase only adds a `Module` row | + +## Standard Stack + +### Core (new to this phase; extends STACK.md's v1.1 delta) + +| Library | Version | Purpose | Why Standard | +|---------|---------|---------|--------------| +| `fast-xml-parser` | 5.10.1 [CITED: STACK.md, npm registry 2026-07-17] | Parse eForms-DE XML notices (primary structured-field source per this session's live finding) | Already selected in milestone STACK.md for this exact purpose; zero-dependency, pure-TS, CJS-safe | +| `adm-zip` | 0.6.0 [VERIFIED: npm registry, checked this session 2026-07-21] | Synchronous in-memory extraction of the DÖE daily export ZIP (typically tens of KB to a few MB — one day's sample was 161KB compressed / 105 files) | **New gap identified this session** — STACK.md did not anticipate the DÖE API returning an archive rather than a paginated feed. `adm-zip` is a simple, synchronous, CJS-native API (`new AdmZip(buffer).getEntries()`) well suited to fully-in-memory extraction of modest archives; MIT-licensed, 19.2M weekly downloads, actively maintained (github.com/cthackers/adm-zip). The automated `package-legitimacy check` seam flagged it `SUS` with reason `"too-new"` — manually confirmed as a **false positive**: the flagged date is the latest patch release (2026-07-10), not the package's origin; this is the same false-positive pattern STACK.md already documented for fast-xml-parser/csv-parse/playwright. `[ASSUMED]` provenance for the package-name choice itself (discovered via training knowledge, not an official doc) — planner should gate the `pnpm add adm-zip` step behind a `checkpoint:human-verify` per the package-legitimacy protocol. | +| `csv-parse` | 7.0.1 [CITED: STACK.md] | DÖE CSV export (`format=csv.zip`) as a cross-check/fallback format | Unchanged from STACK.md; confirmed this session that `csv.zip` is a real, working `format` enum value in the live spec | + +### Supporting +| Library | Version | Purpose | When to Use | +|---------|---------|---------|-------------| +| native `fetch` | Node 22 built-in | HTTP client for the DÖE endpoint | Already the established pattern (`icon-discovery.service.ts`, `ics.provider.ts`); DÖE requires no cookies/session state, so `fetch-cookie`/`tough-cookie` (needed for the scraping adapters in a later phase) are **not** needed for this phase | + +### Alternatives Considered +| Instead of | Could Use | Tradeoff | +|------------|-----------|----------| +| `adm-zip` (sync, in-memory) | `unzipper` (streaming) [VERIFIED: npm registry — 16.7M weekly downloads, same false-positive SUS flag] | Streaming extraction only pays off for very large archives; DÖE daily exports are small (low hundreds of files, sub-MB to low-MB compressed) — sync in-memory is simpler and matches the "wartbar/verständlich" project constraint | +| eForms-DE XML as primary parse target | OCDS JSON as primary parse target (ARCHITECTURE.md's original recommendation) | **Reversed this session based on live cross-check**: OCDS conversion measurably drops structured deadline data present in the source eForms-DE XML for at least one directly-verified notice. OCDS remains useful for `ocid` (dedup key) and resolved party/organization names, but should not be the sole source for deadline/value. | + +**Installation:** +```bash +pnpm --filter @tessera/api add fast-xml-parser adm-zip csv-parse +``` + +**Version verification:** confirmed live via `npm view version` this session (2026-07-21): `adm-zip@0.6.0` (published 2026-07-10), `unzipper@0.12.5` (published 2026-06-21). `fast-xml-parser`/`csv-parse` versions carried over from STACK.md (2026-07-17, 4 days prior — still current). + +## Package Legitimacy Audit + +| Package | Registry | Age (latest release) | Downloads | Source Repo | Verdict | Disposition | +|---------|----------|-----|-----------|-------------|---------|-------------| +| adm-zip | npm | 11 days (0.6.0, 2026-07-10) | 19.2M/wk | github.com/cthackers/adm-zip | SUS (`too-new` — false positive, see rationale above) | Flagged — planner adds `checkpoint:human-verify` before `pnpm add adm-zip`, per protocol | +| fast-xml-parser | npm | (per STACK.md, 2026-07-17) | — | — | SUS (per STACK.md, already assessed false-positive) | Approved (carried over from milestone research, no re-flag needed) | +| csv-parse | npm | (per STACK.md, 2026-07-17) | — | — | SUS (per STACK.md, already assessed false-positive) | Approved (carried over) | + +**Packages removed due to [SLOP] verdict:** none. +**Packages flagged as suspicious [SUS]:** `adm-zip` — planner must add a `checkpoint:human-verify` task before `pnpm add adm-zip` (spot-check the GitHub repo / Socket.dev score at implementation time, per the repo's existing dependency-hygiene bar already applied to the other v1.1 packages in STACK.md). + +*`adm-zip` was discovered via training knowledge (not an official doc), so per the package-name provenance rule it is tagged `[ASSUMED]` regardless of its clean npm registry / GitHub signals.* + +## Architecture Patterns + +### System Architecture Diagram + +``` +[TenderSchedulerService — 1 global cron, e.g. hourly per D-04] + | + v +[DoeOpenDataAdapter.fetchTenders(config, dayCursor)] + | 1. if dayCursor >= today(Europe/Berlin) -> no-op, return [] (nothing new yet) + | 2. else: GET /api/notice-exports?pubDay={dayCursor}&format=eforms.zip (+ ocds.zip for ocid/orgs) + v +[adm-zip: extract ZIP -> N per-notice XML (+ JSON) buffers] + v +[fast-xml-parser: parse each eForms-DE XML -> RawTenderRecord[]] + v +[TenderNormalizerService.normalize()] + | - filter: keep only tag=["tender"] (D-02); drop award/planning/untagged-with-awards + | - map fields -> Tender (title, buyer, cpvCodes, region/plz, deadlineAt, estimatedValue, procedureType, sourceUrl) + | - dedupKey: ocid (from paired OCDS fetch) -> fallback sourcePortal:noticeId + | - contentHash: sha256(title+deadline+value+status) + v +[TenderIngestionService.upsert()] -- prisma.tender.upsert({ where: { dedupKey } }) + | advance dayCursor by 1 day; loop while dayCursor still < today (catch-up on missed days) + v (only newly-created OR contentHash-changed rows this run) +[TenderMatchingService.matchDelta()] -- Phase 11+ (saved searches don't exist yet in Phase 10) +``` + +### Recommended Project Structure + +(Extends ARCHITECTURE.md's `apps/api/src/tenders/` layout — no changes needed, confirmed still correct after live verification.) + +``` +apps/api/src/tenders/ +├── tenders.module.ts +├── tenders.controller.ts # GET /tenders, GET /tenders/:id, admin source-config +├── tenders.seed.ts # seeds 'tender-radar' into ModuleRegistry (CONFIG-01) +├── tender-ingestion.service.ts # poll -> normalize -> upsert -> change-detect +├── tender-normalizer.service.ts # eForms-XML+OCDS -> Tender fields, dedupKey, contentHash +├── tender-scheduler.service.ts # SchedulerRegistry cron: 1 global job, day-cursor gated +├── tender.types.ts # RawTenderRecord, NormalizedTenderFields, SourceType +├── dto/ +│ ├── source-config.dto.ts # admin: pollIntervalMin, isActive +│ └── tender-query.dto.ts # pagination + filter query params (list/detail read, no saved-search yet) +└── adapters/ + ├── tender-source-adapter.interface.ts + └── doe-opendata.adapter.ts # THIS PHASE's only adapter +``` + +### Pattern 1: DÖE API — Day-Batch, Not Paginated Feed (live-verified this session) + +**What:** `GET https://oeffentlichevergabe.de/api/notice-exports` — the *entire* real API surface for OpenData bulk retrieval (confirmed: the live OpenAPI 3.1 spec at `/documentation/api/opendata` lists exactly one path). No auth (no `security` scheme in the spec; confirmed 200 with a plain unauthenticated `curl`). + +**Parameters (from the real spec, cross-verified against live requests):** +- `pubDay` (string, `YYYY-MM-DD`) — returns all notice *versions* processed on that calendar day. **Mutually exclusive with `pubMonth`.** +- `pubMonth` (string, `YYYY-MM`) — same, for a whole month (useful only for a manual/admin-triggered historical import — not used in this phase per D-01). +- `format` (query param, or via `Accept` header): `eforms.zip` (default), `ocds.zip`, `csv.zip`. + +**Live-verified boundary rules (confirmed via actual 400 responses, not just spec text):** +- `pubDay` = today → **HTTP 400**, message: `"The specified pubDay exceeds the allowed range. It must lie in the past."` +- `pubDay` = any future date → same 400. +- Earliest allowed: `pubDay >= 2022-12-01`, `pubMonth >= 2022-12` (per spec text — the Bekanntmachungsservice's operational start). +- Response headers on a real 200: `etag: "version-N"`, `cache-control: max-age=120`, `content-disposition: attachment; filename=notices_YYYY-MM-DD_OCDS.zip`. **No rate-limit headers of any kind** (`X-RateLimit-*` absent) — the `etag`/short cache-control instead suggest a caching layer in front of the origin; conditional `If-None-Match` requests are a viable politeness optimization but not required for correctness at day-granularity polling. + +**When to use:** This is the *only* way to get DÖE data — there is no "list notices since X" or paged-results endpoint. Design the adapter around "fetch one day's full batch," not "fetch new items since last check." + +**Trade-offs:** A day-cursor model means: (a) the very first poll after module activation on deployment day will return **zero** results until the next calendar day rolls over (today's data isn't retrievable yet) — this is expected API behavior, not a bug, and should be documented in the phase's UAT expectations; (b) catching up after downtime (e.g., scheduler was down for 3 days) is naturally supported — just loop `pubDay` from `lastIngestedDay + 1` up to `today - 1`, each as a separate request. + +### Pattern 2: OCDS Release Tag as the D-02 "Open Tender" Filter (live-verified) + +**What:** Live sample (105 real notices, 2026-07-20) breaks down as: + +| `releases[].tag` | Count | Meaning | +|---|---|---| +| `["tender"]` | 74 (70%) | Open/active procurement notice — **the only kind SCHEMA-01/D-02 wants** | +| `["award"]` | 17 (16%) | Award/result notice — explicitly out of scope (D-02) | +| `["planning"]` | 3 (3%) | Prior-information/planning notice — not yet a bookable tender — exclude | +| *(no `tag` field at all)* | 11 (11%) | **Live-verified edge case** — direct-award justification notices ("Vergabevermerke") with populated `awards`/`contracts` arrays but no `tag`. These are closed/awarded, not open. | + +**Filter logic (recommended, not yet in ARCHITECTURE.md):** +```typescript +function isOpenTenderNotice(release: OcdsRelease): boolean { + const tags = release.tag ?? []; + if (tags.includes('tender')) return true; + // Untagged notices with populated awards/contracts are NOT open — exclude. + // Untagged notices with neither (rare, if any) should also be excluded conservatively + // rather than defaulting to "open" — a false "open tender" shown to a bidder is worse + // than a missed one (which the coverage-transparency UI in Phase 11 already communicates). + return false; +} +``` + +**Why this matters:** A naive filter of "absence of `award` tag = open" would have wrongly included the 11 untagged-but-actually-closed notices found in this live sample (11% of one real day — not a negligible edge case). + +### Pattern 3: eForms-DE XML as Primary Structured-Field Source (live cross-check, corrects ARCHITECTURE.md) + +**What:** ARCHITECTURE.md recommended "prefer OCDS... for direct field alignment." Live cross-check of the **same real notice** (`25605686-1`) in both formats shows: + +- OCDS JSON: `tender.tenderPeriod` is **absent** (no deadline field at all); `tender.value` is **absent**; the deadline ("Angebotsfrist: Mittwoch, 26.08.2026, 10:00 Uhr") exists only as unstructured German prose inside `tender.description`. +- eForms-DE XML (same notice, same day, same export): contains a proper structured element — `2026-08-26+02:00` — and the document's `CustomizationID` (`eforms-sdk-0.1`) confirms this notice uses an old eForms SDK version (directly matching PITFALLS.md's Pitfall 17 — SDK version drift — now observed live, not just theorized). + +**Recommendation:** Fetch **both** formats per poll cycle (`eforms.zip` for structured deadline/value/procedure fields; `ocds.zip` for the `ocid` dedup key and resolved buyer/party names) rather than relying on OCDS alone. This is two HTTP requests per day-cursor tick instead of one — negligible cost at day-granularity polling. If phase scope favors a single-format MVP, prefer `eforms.zip` as primary and treat `ocid` absence as acceptable for v1 (fall back to `sourcePortal:noticeId` dedup key, already the documented fallback tier in ARCHITECTURE.md's Pattern 2) — but fetching both is the more complete and only modestly more expensive option, and is the recommended default. + +**Trade-off:** Two structurally different parsers now run per poll (fast-xml-parser for eForms-DE, plain `JSON.parse` for OCDS) — acceptable complexity, both already selected in STACK.md; no new parser library needed for this reversal, only a change in which one is authoritative for which fields. + +### Pattern 4: Nullable Deadline/Value as the Common Case, Not the Edge Case + +**What:** Of the 12 "tender"-tagged notices spot-checked in this session's live sample, **all 12** had `tenderPeriod.endDate = null` and `value = null` in the OCDS export. This is a stronger and more concrete version of PITFALLS.md's Pitfall 18 ("OCDS optional/conditional fields treated as always present") — live data suggests this is the **majority case for Unterschwelle notices**, not a rare edge case. + +**Recommendation for SCHEMA-01/D-05:** +- `deadlineAt` and `estimatedValue` **must** be nullable in the `Tender` Prisma model (already planned in ARCHITECTURE.md's schema sketch — confirmed necessary, not optional hardening). +- D-05's "90 days after deadline, then delete" retention rule needs an explicit fallback for `deadlineAt IS NULL` rows: **do not auto-expire them** on the deadline-based path (there is no deadline to expire from); either (a) leave them permanently active/searchable (simplest, matches "keine Vor-Eingrenzung" spirit — a human should judge relevance, not the retention job), or (b) apply a secondary fallback expiry based on `publishedAt + N days` for deadline-less rows only. Recommend (a) for the MVP phase — introduce (b) later if the table grows unmanageably, since retrofitting a stricter policy is easy but a wrongly-deleted no-deadline tender is not recoverable. +- The "nur offene" (still-open) view/filter (Phase 11, FILTER-04) will only be meaningful for the subset with a populated `deadlineAt` — this is a coverage-transparency concern (PITFALLS.md Pitfall 20) that now has a concrete, live-measured magnitude (100% of a 12-notice spot sample) to inform the Phase 11 UI copy, not just an abstract "some tenders lack this." + +## Don't Hand-Roll + +| Problem | Don't Build | Use Instead | Why | +|---------|-------------|-------------|-----| +| ZIP extraction | A manual DEFLATE/central-directory parser | `adm-zip` | Well-understood, tiny surface, sync API fits the modest archive sizes involved | +| Multi-tenant scheduling | A second scheduling mechanism | `SchedulerRegistry.addCronJob()` (same as `DkvSchedulerService`) | Already proven, dynamically updatable at runtime — reuse the mechanics, just replace the `findFirst()`-single-tenant *pattern*, not the underlying `cron` library | +| Module registry / marketplace activation | A new activation table or gating mechanism | `ModuleRegistryService` (unmodified) + `tenders.seed.ts` (new, mirrors `cert-manager.seed.ts`) | Already the load-bearing platform mechanism for CONFIG-01; do not special-case Tenders | +| OCID/dedup key computation | A custom hashing scheme from scratch | Priority-ordered key per ARCHITECTURE.md Pattern 2 (`ocid` → `sourcePortal:noticeId` → fuzzy fingerprint) | Already researched and live-confirmed: `ocid` is present and stable across notice versions (`ocds-mnwr74-{processId}`, distinct from the per-version notice/release `id`) | + +**Key insight:** This phase's genuinely new risk is not "how do we build a scheduler" (solved, mechanics reused from DKV) — it's **correctly interpreting the DÖE API's actual batch/day-cursor shape**, which this session's live verification now settles definitively instead of leaving as an assumption. + +## Runtime State Inventory + +Not applicable — this is a greenfield module addition (new Prisma models, new NestJS module, new module-registry row), not a rename/refactor/migration phase. No existing runtime state references the "Ausschreibungs-Radar"/"tender-radar" name that would need updating. + +## Common Pitfalls + +### Pitfall A: Treating the DÖE poll interval as "fetch frequency" instead of "check frequency" +**What goes wrong:** A naive implementation calls the DÖE endpoint every tick of the admin-configured interval (e.g., hourly per D-04), re-requesting the *same* day's ZIP 23 times before the day actually advances — wasted load, and worse, a developer "optimizing" this by shortening the interval further, or "fixing" the apparent no-op by re-implementing since-timestamp logic that the API doesn't actually support. +**Why it happens:** D-04 says "poll interval, admin-configurable, default hourly" — this reads like a typical incremental-feed poll interval, but the DÖE API's actual granularity is daily. +**How to avoid:** Decouple "cron tick frequency" (can stay hourly, satisfies D-04/INGEST-06 literally) from "actual upstream fetch" (gated by `dayCursor < today(Europe/Berlin)`). Document this decoupling explicitly in the adapter/scheduler code, not just in this research doc. +**Warning signs:** Scheduler logs show identical `pubDay` values fetched repeatedly within the same 24h window with unchanged results — expected, not a bug, but worth a debug-level (not warn/error) log line so it doesn't look broken during on-call review. + +### Pitfall B: Assuming OCDS alone is sufficient for SCHEMA-01's deadline/value fields +**What goes wrong:** Building the normalizer against OCDS only (per ARCHITECTURE.md's original guidance) ships a `Tender` catalog where deadline/value are null for the majority of below-threshold notices, even though the source data (eForms-DE XML) actually had it. +**How to avoid:** Parse eForms-DE XML as the primary field source per Pattern 3 above; treat OCDS as the `ocid`/party-resolution source, not the deadline/value source. +**Warning signs:** QA/UAT notices most "tender"-tagged records showing no deadline in the results list — check whether the eForms-XML export was even fetched/parsed before assuming this is unavoidable API limitation (it partly is — not every eForms document will have a machine-readable deadline either, since it can be embedded in a portal-specific free-text description — but the eForms path recovers meaningfully more than OCDS alone, confirmed on at least one directly cross-checked notice this session). + +### Pitfall C: Filtering "open tenders" (D-02) by tag absence instead of tag presence +**What goes wrong:** Assuming "no `award` tag = open tender" wrongly includes untagged-but-closed notices (11% of the live sample). +**How to avoid:** Filter positively on `tag.includes('tender')`; treat missing/other tags as excluded by default (Pattern 2 above). +**Warning signs:** Award-justification/"Vergabevermerk" notices (identifiable by populated `awards`/`contracts` arrays with no submission deadline) appearing in the "open, biddable" result list. + +### Pitfall D: Copying DkvSchedulerService's `findFirst()` single-tenant pattern (carried over from PITFALLS.md Pitfall 24, restated for this phase specifically) +**What goes wrong:** Already extensively documented in PITFALLS.md — restated here because this phase's scheduler is the first place it must be avoided, and because the DÖE source is **global** (not per-tenant, unlike DKV), the "multi-tenant" framing is slightly different: there IS only one DÖE poll config (platform-wide, per Pattern 3 in ARCHITECTURE.md), so `findFirst()` on `TenderSourcePollConfig` for the DÖE row specifically is **not** the anti-pattern here (there's genuinely only one DÖE config row to find) — the anti-pattern would be if a future per-tenant source (e.g., email-alert in Phase 14) reused this same `findFirst()` call path. **For Phase 10 specifically:** it's safe and correct to load the single DÖE `TenderSourcePollConfig` row via `findFirst()` (or better, `findUnique` on a fixed known slug like `sourceType: 'doe-opendata'`) since it is architecturally a singleton config — this is the one legitimate `findFirst()` in this phase; the acceptance-criterion "activating a 2nd tenant must not duplicate ingestion" is satisfied by the config being tenant-agnostic in the first place (Pattern 3), not by scheduler code that iterates tenants. +**Verification for this phase's two-tenant acceptance criterion (Success Criteria 4 & 5):** the test is NOT "does the scheduler correctly loop over 2 tenant configs" (there are none for DÖE) — it is "does activating the module for a 2nd tenant (`TenantModuleActivation` row) trigger zero additional DÖE HTTP calls, zero additional cron jobs, and zero additional `Tender` rows" — i.e., prove the *absence* of tenant-count-scaled behavior, not the presence of correct per-tenant iteration. + +## Code Examples + +### DÖE fetch — real, verified request shape +```typescript +// Source: live-verified this session against https://oeffentlichevergabe.de +// GET https://oeffentlichevergabe.de/api/notice-exports?pubDay=2026-07-20&format=eforms.zip +// -> 200, Content-Disposition: attachment; filename=notices_2026-07-20_eForms.zip +// -> binary ZIP body, ~105 files for a typical day (~160KB compressed for the OCDS variant) + +async function fetchDoeDay(pubDay: string, format: 'eforms.zip' | 'ocds.zip' | 'csv.zip') { + const url = `https://oeffentlichevergabe.de/api/notice-exports?pubDay=${pubDay}&format=${format}`; + const res = await fetch(url); + if (res.status === 400) { + // "must lie in the past" — pubDay is today or future; nothing to fetch yet this cycle + return null; + } + if (!res.ok) throw new Error(`DÖE fetch failed: ${res.status}`); + return Buffer.from(await res.arrayBuffer()); +} +``` + +### Day-cursor gate (replaces a `since: Date` timestamp cursor) +```typescript +// Source: this session's live finding — DÖE has no incremental/since parameter. +function nextDayToFetch(lastIngestedDay: Date | null): string | null { + const berlinToday = new Date().toLocaleDateString('en-CA', { timeZone: 'Europe/Berlin' }); // YYYY-MM-DD + const next = lastIngestedDay + ? new Date(lastIngestedDay.getTime() + 86_400_000).toISOString().slice(0, 10) + : new Date().toISOString().slice(0, 10); // D-01: "from now" — first eligible day is tomorrow relative to activation + return next < berlinToday ? next : null; // null = nothing new yet, no-op this tick +} +``` + +### D-02 open-tender filter (OCDS release tag) +```typescript +// Source: live-verified against a real 105-notice sample, 2026-07-20 +function isOpenTenderNotice(release: { tag?: string[]; awards?: unknown[]; contracts?: unknown[] }): boolean { + if (release.tag?.includes('tender')) return true; + return false; // conservative: missing tag + award/planning tags are all excluded +} +``` + +## State of the Art + +| Old Approach (assumed pre-verification) | Current Approach (live-verified this session) | When Changed | Impact | +|--------------------------------------------|-----------------------------------------------|---------------|--------| +| DÖE = paginated REST feed with a `since`/cursor param (implicit assumption in ARCHITECTURE.md's adapter signature) | DÖE = day/month-batch ZIP export, no pagination param exists, current/future days rejected (400) | Confirmed 2026-07-21, this session | Scheduler must be day-cursor gated, not since-timestamp gated; "pagination handling" (a phase-focus research question) resolves to "there is no pagination — there is a per-day archive" | +| "Prefer OCDS over eForms-DE" (ARCHITECTURE.md Pattern 2, based on the OCDS spec's cleaner structure) | eForms-DE XML has structured deadline data that OCDS conversion drops, at least for some notices | Confirmed 2026-07-21, this session, via direct cross-check of one real notice in both formats | Normalizer should parse eForms-DE as primary for deadline/value; OCDS retained for `ocid`/party names | + +**Deprecated/outdated:** None — this is a young API (operational since 2023-02-06 per prior research); no deprecated version churn observed in the live spec. + +## Assumptions Log + +| # | Claim | Section | Risk if Wrong | +|---|-------|---------|---------------| +| A1 | `adm-zip` is the right ZIP library choice (vs. `unzipper`/`jszip`/`yauzl`) | Standard Stack | Low — all are viable; `adm-zip`'s sync API is a convenience choice, not a hard technical requirement. If archives grow much larger than observed (161KB/105 files for one day), a streaming approach (`unzipper`) may be preferable — reassess if daily archive size grows past a few MB. | +| A2 | Fetching both `eforms.zip` and `ocds.zip` per poll cycle (two HTTP calls) is acceptable overhead vs. a single-format MVP | Pattern 3 | Low — at day-granularity polling (at most 1-2 real fetches/day per the day-cursor gate), two requests instead of one is negligible; if the planner descopes to single-format for MVP speed, document that deadline/value coverage will be measurably worse (per Pattern 4's 100%-null spot check) as an explicit MVP trade-off, not a silent gap | +| A3 | The 105-notice, 12-notice, and 11-notice samples (all from 2026-07-20) are representative of typical daily volume/tag distribution, not an anomalous day | Pattern 2, Pattern 4 | Medium — a single day's sample is directional, not statistically proven; if actual production volume/tag ratios differ significantly, the D-02 filter logic itself is still correct (tag-based), only the *magnitude* estimates (70% tender / 16% award / etc.) may shift | + +## Open Questions + +1. **Exact rate-limit tolerance of the DÖE API (undocumented)** + - What we know: no `X-RateLimit-*` headers observed on a real response; a `cache-control: max-age=120` + `etag` suggest a caching/CDN layer that can absorb repeated identical requests cheaply. + - What's unclear: whether rapid-fire requests (e.g., a catch-up loop fetching 30 missed days in quick succession after extended downtime) would trigger throttling — not tested this session to avoid abusing a public-good API mid-research. + - Recommendation: implement the catch-up loop (Pattern 1) with a small delay between requests (e.g., 1-2s, matching the "politeDelay" helper already recommended in STACK.md for the scraping adapters) even though DÖE itself carries "null" ToS risk — cheap insurance, no downside. + +2. **Whether eForms-DE deadline coverage is meaningfully better than OCDS across a larger sample** + - What we know: confirmed better on 1 directly cross-checked notice (deadline present in XML, absent in OCDS JSON and OCDS proper structured field). + - What's unclear: the aggregate improvement rate across hundreds/thousands of notices — this session's live check was necessarily a small spot-check, not a full-corpus audit. + - Recommendation: the phase's implementation should log a metric (e.g., `% of ingested tender-tagged notices with non-null deadlineAt`) from day one, so this can be measured empirically in production rather than re-guessed. + +## Environment Availability + +| Dependency | Required By | Available | Version | Fallback | +|------------|------------|-----------|---------|----------| +| Outbound internet access from the deployment host to `oeffentlichevergabe.de` | INGEST-01 (DÖE polling) | ✓ (verified from this dev/research environment — production host connectivity not independently verified this session) | — | If the production host lacks outbound HTTPS to this domain (e.g., restrictive egress firewall), the module cannot function at all — confirm with infra before phase execution if the API host is on the same network segment as the dev machine used for this research | +| Node.js `fetch` (native, Node 18+) | DÖE HTTP client | ✓ | Node 22-class per `@types/node@^22` (existing project baseline) | — | +| `adm-zip` / equivalent ZIP library | ZIP extraction | ✗ (not yet installed) | — | `pnpm add adm-zip` — no fallback needed, this is a build-time dependency, not an external service | + +**Missing dependencies with no fallback:** none beyond the standard "install the new npm package" step. +**Missing dependencies with fallback:** none applicable. + +## Validation Architecture + +### Test Framework +| Property | Value | +|----------|-------| +| Framework | Vitest 3.x (existing, per v1.0 STACK.md) | +| Config file | `apps/api/vitest.config.ts` (existing — confirm path during planning; not modified by this phase) | +| Quick run command | `pnpm --filter @tessera/api test -- tenders` (scoped to new test files) | +| Full suite command | `pnpm --filter @tessera/api test` | + +### Phase Requirements → Test Map +| Req ID | Behavior | Test Type | Automated Command | File Exists? | +|--------|----------|-----------|-------------------|-------------| +| CONFIG-01 | Module self-seeds into registry on boot; activatable per tenant | unit + integration | `pnpm --filter @tessera/api test -- tenders.seed` | ❌ Wave 0 | +| INGEST-01 | `DoeOpenDataAdapter` parses a real (or fixture-captured) eForms-DE/OCDS ZIP into `RawTenderRecord[]` | unit | `pnpm --filter @tessera/api test -- doe-opendata.adapter` | ❌ Wave 0 — fixture ZIP should be captured from this session's live download (`doe-2026-07-20-eforms.zip`/`doe-2026-07-20-ocds.zip`) rather than synthetic data, per this repo's stated preference for real fixtures over synthetic ones (matches the DKV PDF-parser precedent) | +| INGEST-06 | Two-tenant scheduler test: activating module for a 2nd tenant produces zero additional DÖE HTTP calls / cron jobs / `Tender` rows | integration | `pnpm --filter @tessera/api test -- tender-scheduler.service` | ❌ Wave 0 | +| SCHEMA-01 | Normalizer maps a real eForms-DE + OCDS notice pair into the expected `Tender` fields, including nullable deadline/value | unit | `pnpm --filter @tessera/api test -- tender-normalizer.service` | ❌ Wave 0 | +| SCHEMA-02 | Re-ingesting an updated notice (same `ocid`, changed `contentHash`) updates the existing row instead of duplicating | integration | `pnpm --filter @tessera/api test -- tender-ingestion.service` | ❌ Wave 0 | +| D-02 filter | `tag=["tender"]` included; `award`/`planning`/untagged-with-awards excluded | unit | `pnpm --filter @tessera/api test -- doe-opendata.adapter` (same fixture, asserts filtered-out count) | ❌ Wave 0 | + +### Sampling Rate +- **Per task commit:** `pnpm --filter @tessera/api test -- tenders` (scoped) +- **Per wave merge:** `pnpm --filter @tessera/api test` (full suite) +- **Phase gate:** Full suite green before `/gsd-verify-work` + +### Wave 0 Gaps +- [ ] `apps/api/src/tenders/__fixtures__/doe-2026-07-20-eforms.zip` and `doe-2026-07-20-ocds.zip` — real captured fixtures from this session's live download (recommend committing a trimmed version, e.g., 5-10 representative notices covering tender/award/planning/untagged tags, rather than the full 105-file archive, to keep the repo lean) +- [ ] `apps/api/src/tenders/tenders.seed.spec.ts`, `doe-opendata.adapter.spec.ts`, `tender-normalizer.service.spec.ts`, `tender-ingestion.service.spec.ts`, `tender-scheduler.service.spec.ts` — none exist yet, this is a greenfield module + +## Security Domain + +### Applicable ASVS Categories + +| ASVS Category | Applies | Standard Control | +|---------------|---------|-------------------| +| V2 Authentication | No | DÖE endpoint is auth-free by design (confirmed live); no credentials to protect for this specific integration | +| V3 Session Management | No | No session/cookie state involved in the DÖE adapter (unlike the later scraping adapters) | +| V4 Access Control | Yes | `GET /tenders` list/detail must be gated by `TenantModuleActivation` (module licensing) but NOT by `tenantId` row-filtering — this is the one controller where "tenant-gated" and "tenant-scoped" genuinely differ (per ARCHITECTURE.md's explicit callout); admin source-config routes (`POST /tenders/source-config`) must be `@Roles(Role.ADMIN, Role.SUPER_ADMIN)`-guarded like `DkvController` | +| V5 Input Validation | Yes | All parsed eForms-XML/OCDS-JSON fields are external, untrusted input — sanitize/escape before any future HTML rendering (Phase 11 UI); validate `pubDay`/`pubMonth` cursor values server-side before constructing the outbound DÖE URL (defense against cursor-corruption, low risk since these are internally computed, not user-supplied, but validate anyway) | +| V6 Cryptography | No | No credentials/secrets are stored for this phase (DÖE is auth-free) — `CalendarCryptoService`/AES-256-GCM pattern is reserved for later phases (email-alert inbox credentials, Phase 14) | + +### Known Threat Patterns for this stack + +| Pattern | STRIDE | Standard Mitigation | +|---------|--------|----------------------| +| Untrusted external content (buyer names, tender descriptions, CPV descriptions from eForms/OCDS) rendered unescaped in a future UI | Tampering / Injection | Sanitize/escape all ingested text fields before HTML interpolation — applies to Phase 11's UI, but the normalizer in this phase should not assume the data is "safe" just because it came from a government API; store raw, escape at render time | +| Zip-bomb / decompression-bomb via a maliciously large or deeply-nested ZIP response | Denial of Service | Low realistic risk given DÖE is a trusted government-operated source over HTTPS, but as defense-in-depth: `adm-zip`'s `getEntries()` reads the central directory without fully decompressing — validate total uncompressed size against a sane ceiling (e.g., reject if `entries.reduce(sum of entry.header.size) > 50MB`) before extracting each entry's buffer | +| Cursor/date-parameter injection into the outbound DÖE URL | Tampering | The `pubDay`/`pubMonth` cursor is entirely internally computed from `lastIngestedDay` (never user-supplied) — no admin-facing input flows directly into this URL, so injection risk is structurally absent as designed; keep it that way (don't add an admin "manually trigger for date X" feature without validating the date format server-side first) | + +## Sources + +### Primary (HIGH confidence) +- Live API verification, this session (2026-07-21): `https://oeffentlichevergabe.de/documentation/swagger-ui/opendata/swagger-initializer.js` (reveals real spec URL, bypassing the JS-rendered Swagger UI), `https://oeffentlichevergabe.de/documentation/api/opendata` (real OpenAPI 3.1 spec, fetched and parsed), `https://oeffentlichevergabe.de/api/notice-exports?pubDay=2026-07-20&format=ocds.zip` and `...&format=eforms.zip` (real 200 responses, 105 real notices downloaded and inspected), boundary tests for `pubDay=today` and `pubDay=future` (both confirmed 400) +- `npm view adm-zip version` / `npm view unzipper version` / `gsd-tools query package-legitimacy check` — direct registry + legitimacy-seam verification, this session +- Existing codebase (direct read, this session): `apps/api/src/dkv/dkv-scheduler.service.ts`, `apps/api/src/dkv/dkv.service.ts`, `apps/api/src/dkv/dkv.module.ts`, `apps/api/src/dkv/dkv.controller.ts`, `apps/api/src/dkv/dkv.seed.ts`, `apps/api/src/cert-manager/cert-manager.seed.ts`, `apps/api/src/module-registry/module-registry.service.ts`, `apps/api/src/prisma/prisma-tenant.extension.ts`, `apps/api/prisma/schema.prisma` (`DkvModuleConfig` model), `apps/web/src/lib/module-loader.ts`, `apps/web/src/app/(portal)/modules/dkv-fleet/settings/components/InboxConfigForm.tsx`, `apps/api/src/app.module.ts`, `apps/api/package.json` + +### Secondary (MEDIUM confidence) +- `.planning/research/SUMMARY.md`, `ARCHITECTURE.md`, `STACK.md`, `PITFALLS.md`, `ausschreibungs-portale-feasibility.md` (milestone-level research, 2026-07-16/17) — used as the binding baseline this document extends/corrects where live verification diverged + +### Tertiary (LOW confidence) +- None — this session prioritized live verification over inference specifically because the milestone research had flagged the DÖE pagination/rate-limit question as an explicit open point. + +## Metadata + +**Confidence breakdown:** +- DÖE API mechanics (endpoint, params, day-batch model, boundary rules): HIGH — directly observed via live requests this session, not inferred from documentation +- OCDS/eForms field completeness (deadline/value nullability, tag distribution): MEDIUM-HIGH — directly observed on a real sample, but the sample is one day (105 notices) and may not capture the full range of buyer/portal submission quality variance +- Platform integration (module registry, scheduler mechanics, Prisma conventions): HIGH — verified against existing, working codebase +- ZIP library choice (`adm-zip`): MEDIUM — sound technical choice, but package-name provenance is `[ASSUMED]` (training knowledge) per the package-legitimacy protocol; flagged for `checkpoint:human-verify` + +**Research date:** 2026-07-21 +**Valid until:** DÖE API mechanics — treat as stable for 90 days (young, actively-operated government API, no deprecation signals observed; re-verify if a phase re-plan happens much later). npm package versions — 30 days (fast-moving ecosystem norm).