schalli/tessera-ctl
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 14 Wiki Activity

docs(10): research DÖE OpenData API live verification and phase domain

Browse Source
This commit is contained in:
Schalli
2026-07-21 09:46:20 +02:00
parent fe55e7c054
commit 3a51d9bfed
1 changed files with 418 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.planning/phases/10-ausschreibungs-radar-foundation-d-e-ingestion/10-RESEARCH.md
+418
View File
@@ -0,0 +1,418 @@
# Phase 10: Ausschreibungs-Radar Foundation & DÖE Ingestion - Research
**Researched:** 2026-07-21
**Domain:** German public-procurement (DÖE OpenData) live API ingestion, OCDS/eForms-DE normalization, multi-tenant poll-once-fan-out-many scheduler, module-registry integration on the existing Tessera NestJS/Prisma platform
**Confidence:** HIGH (DÖE API mechanics — verified live against the production API this session, not just the JS-rendered Swagger UI); HIGH (platform integration — verified against existing `dkv/`/`module-registry/` codebase); MEDIUM (OCDS field completeness — verified against real live export, one day's sample; genuinely variable by buyer/portal)
<user_constraints>
## User Constraints (from CONTEXT.md)
### Locked Decisions
**DÖE-Ingest-Umfang**
- **D-01:** Initialer Backfill = **nur ab jetzt**. Beim ersten plattformweiten DÖE-Poll werden nur ab diesem Zeitpunkt veröffentlichte Ausschreibungen aufgenommen — kein historischer Import. Da die Tender-Daten global sind, sieht ein später aktivierender Mandant den seit Plattformstart aufgelaufenen Bestand. Passt zur späteren Backfill-Unterdrückung bei Benachrichtigungen (Phase 12).
- **D-02:** Nur **offene Ausschreibungen** (aktive Vergaben, auf die man bieten kann — tender/contract notices). Vergabeergebnisse, Zuschläge und Aufhebungen (award/result notices) werden in dieser Phase NICHT aufgenommen.
- **D-03:** Ingest lädt **ganz Deutschland global** (plattformweit einmal), **keine** Vor-Eingrenzung nach Region/CPV beim Ingest — Eingrenzung passiert erst pro Suchprofil (Phase 11). Bewusst wegen Multi-Tenant-Wiederverkauf.
- **D-04:** Standard-Poll-Intervall = **stündlich**, admin-konfigurierbar (INGEST-06).
**Datenaufbewahrung**
- **D-05:** Ausschreibungen nach Ablauf der Abgabefrist werden **90 Tage** aufbewahrt, dann gelöscht. Vor Fristablauf: aktiv. Nach Fristablauf: als „abgelaufen" markiert, standardmäßig aus der aktiven Liste ausgeblendet, aber bis zur Löschung recherchierbar.
### Claude's Discretion
- Exakte Prisma-Schema-Felder (OCDS-orientiert, Form gemäß ARCHITECTURE.md), Adapter-Interface (`TenderSourceAdapter`), Normalizer-Interna, Dedup-Key-Berechnung (OCID → Quelle:NoticeId → …) und contentHash-Änderungserkennung.
- DÖE-API-Client (native `fetch` + `fast-xml-parser`, CSV-Fallback), Pagination/Rate-Limit-Handling (Detail per Phase-Research live zu klären).
- Scheduler-Implementierung: **poll-once-fan-out-many, mehrmandantensicher — NICHT** das DKV-`findFirst()`-Single-Tenant-Muster.
- Modul-Registrierung im Marketplace nach DKV/Cert-Manager-Vorbild.
### Deferred Ideas (OUT OF SCOPE)
- Vergabeergebnisse/Zuschläge/Aufhebungen (award notices) aufnehmen — spätere Erweiterung, außerhalb v1.1-Scope.
- Historischer Backfill (letzte 30 Tage / alles) — bei Bedarf später als Konfig-Option.
- Vor-Eingrenzung des Ingest nach Region/CPV — bewusst verworfen (widerspricht Multi-Tenant-Wiederverkauf); Eingrenzung bleibt Sache der Suchprofile.
- Aufbewahrungsdauer (90 Tage) später ggf. admin-konfigurierbar machen.
</user_constraints>
<phase_requirements>
## Phase Requirements
| ID | Description | Research Support |
|----|-------------|------------------|
| CONFIG-01 | Modul im Marketplace registriert, pro Mandant aktivierbar (wie DKV-Fleet/Cert-Manager) | `tenders.seed.ts` follows verbatim `cert-manager.seed.ts`/`dkv.seed.ts` pattern (`isSystem: true`, category, i18n description). `module-loader.ts` MODULE_REGISTRY entry required or the page 404s even when activated (verified codebase pattern, see Architecture Patterns §1) |
| INGEST-01 | Zeitgesteuerter Abruf via DÖE OpenData API (eForms/OCDS, auth-frei) | Live-verified in this session: real endpoint `GET /api/notice-exports`, auth-free (no security scheme in spec, confirmed 200 with no credentials), format negotiation via `Accept` header or `format` query param — see DÖE API Live Findings |
| INGEST-06 | Poll-once-fan-out-many, Intervall pro Quelle admin-konfigurierbar | See Scheduler Design — day-cursor gated single global cron (not per-tenant), admin-configurable interval field per D-04 with an adapter-level no-op guard for the DÖE source's actual day-granularity |
| SCHEMA-01 | Einheitliches OCDS-orientiertes Schema, plattform-global | See Prisma Schema Fields — extends ARCHITECTURE.md's `Tender` model with live-verified nullable-field realities (deadline/value frequently absent in OCDS export) |
| SCHEMA-02 | Content-Hash-Änderungserkennung (Fristverlängerung, Aufhebung) | See Dedup & Change Detection — DÖE re-publishes a changed notice as a new `noticeVersion` under the same `ocid`; contentHash diff on re-poll triggers update path |
</phase_requirements>
## Summary
This research supersedes the "open verification point" flagged in `.planning/research/SUMMARY.md`, `ARCHITECTURE.md`, and `PITFALLS.md`: **the DÖE OpenData API's Swagger UI was live-queried this session** (its JS-rendered UI was bypassed by reading `swagger-initializer.js` directly, which reveals the real spec URL `/documentation/api/opendata`). The live OpenAPI 3.1 spec was fetched, and — critically — **a real export ZIP for a real past day was downloaded and inspected** (105 real German tender notices from 2026-07-20, in both OCDS and eForms-DE XML format). This is ground truth, not documentation inference.
**The single most important finding, which changes the phase's architecture from what ARCHITECTURE.md assumed:** the DÖE OpenData API is **not** a paginated/incremental REST feed. It is a **daily batch-export API**: one call returns a ZIP of *all* notice versions processed on a given calendar day (`pubDay=YYYY-MM-DD`) or month (`pubMonth=YYYY-MM`) — there is no `since`/cursor/page parameter of any kind, and **the current day and any future day are rejected with HTTP 400** ("must lie in the past"). New data for "today" only becomes retrievable starting tomorrow. This means the `TenderSourceAdapter.fetchTenders(config, since?: Date)` signature sketched in ARCHITECTURE.md should track a **day cursor** (`lastIngestedDay: Date`, day-granularity), not an arbitrary timestamp — and the admin-configurable poll interval (D-04, default hourly, INGEST-06) is architecturally harmless to run hourly (idempotent day-cursor check: no-op if the next day isn't past yet) but will only ever produce *new* data once every ~24h for this source. This must be explicitly designed into the scheduler (cron tick frequency ≠ actual-fetch frequency for this specific source) and documented so a future contributor doesn't "fix" the apparent 23-hours-of-no-op as a bug.
**Second critical finding:** live inspection of the real OCDS export shows that **`tender.tenderPeriod.endDate` (deadline) and `tender.value` (estimated value) are frequently `null` in the OCDS conversion — even for open ("tender"-tagged) notices — while the same information is present in free text inside `tender.description`/`lots[].description`, and, for at least one directly cross-checked notice, present as a proper structured field (`TenderSubmissionDeadlinePeriod/EndDate`) in the corresponding eForms-DE XML that OCDS silently drops.** This directly affects SCHEMA-01 (deadline/value are supposed to be first-class normalized fields) and D-05 (90-day post-deadline retention needs a deadline value to compute from). The recommendation below is to parse **eForms-DE XML as the primary structured-field source** for deadline/value/procedure-type, using OCDS JSON primarily for the `ocid` (stable dedup key across notice versions) and party/organization resolution — a reversal of ARCHITECTURE.md's blanket "prefer OCDS" guidance, based on this session's live cross-check, not assumption.
**Third finding:** the export format is a ZIP archive (confirmed: real `Zip archive data` file, one JSON/XML file per notice inside). **No ZIP-handling library was included in STACK.md** — this is a genuine gap in the milestone-level stack research (it assumed pagination, not archive downloads). `adm-zip` is recommended below.
**Fourth finding, directly informing D-02:** live data confirms `releases[].tag` is the correct discriminator for "open tender" (`tag: ["tender"]`, 74/105 = 70% of one real day's sample) vs. `["award"]` (17/105) vs. `["planning"]` (3/105) vs. **no `tag` field at all** (11/105 — untagged notices that still carry populated `awards`/`contracts` arrays, e.g. direct-award justification notices/"Vergabevermerke"). D-02's filter must therefore be: `tag` array contains `"tender"` **and** treat a missing `tag` conservatively (exclude, don't default to "open") since several of the untagged samples are in fact awarded/closed procurements.
**Primary recommendation:** Build the `DoeOpenDataAdapter` around the single real endpoint `GET /api/notice-exports?pubDay=YYYY-MM-DD&format=eforms.zip` (eForms-DE XML as primary parse target for deadline/value; OCDS JSON fetched in parallel or CSV as secondary cross-check per STACK.md's original recommendation), with a day-cursor scheduler (not a since-timestamp), `adm-zip` for archive extraction, and D-02 filtering on `tag=["tender"]` with `awards`/`contracts` emptiness as a secondary guard.
## Architectural Responsibility Map
| Capability | Primary Tier | Secondary Tier | Rationale |
|------------|-------------|----------------|-----------|
| DÖE polling (HTTP fetch of daily ZIP) | API / Backend (`TenderSourceAdapter`) | — | Server-side scheduled job; no browser/CDN involvement |
| ZIP extraction + eForms-XML/OCDS parsing | API / Backend (`TenderNormalizerService`) | — | CPU/IO-bound parsing belongs in the Node process, not the DB or client |
| Dedup key + contentHash computation | API / Backend (`TenderNormalizerService`) | — | Pure computation on normalized fields before persistence |
| `Tender` persistence (global, no tenantId) | Database / Storage | — | Platform-wide reference data; RLS/tenant scoping deliberately does NOT apply here (see Pattern 3 in ARCHITECTURE.md) |
| Scheduler / cron lifecycle | API / Backend (`TenderSchedulerService` via `SchedulerRegistry`) | — | Server-only concern; no client involvement |
| Module registry self-seed + tenant activation gate | API / Backend (`ModuleRegistryService`, reused unmodified) | Database (`Module`/`TenantModuleActivation` tables) | Existing platform mechanism, not phase-specific |
| Admin poll-interval config UI | Frontend Server (Next.js `settings/page.tsx`) | API / Backend (`TendersController` admin routes) | Mirrors DKV's `InboxConfigForm` — form on the client, persisted via API |
| Marketplace listing/activation UI | Frontend Server (existing Marketplace page) | — | Unmodified platform feature; this phase only adds a `Module` row |
## Standard Stack
### Core (new to this phase; extends STACK.md's v1.1 delta)
| Library | Version | Purpose | Why Standard |
|---------|---------|---------|--------------|
| `fast-xml-parser` | 5.10.1 [CITED: STACK.md, npm registry 2026-07-17] | Parse eForms-DE XML notices (primary structured-field source per this session's live finding) | Already selected in milestone STACK.md for this exact purpose; zero-dependency, pure-TS, CJS-safe |
| `adm-zip` | 0.6.0 [VERIFIED: npm registry, checked this session 2026-07-21] | Synchronous in-memory extraction of the DÖE daily export ZIP (typically tens of KB to a few MB — one day's sample was 161KB compressed / 105 files) | **New gap identified this session** — STACK.md did not anticipate the DÖE API returning an archive rather than a paginated feed. `adm-zip` is a simple, synchronous, CJS-native API (`new AdmZip(buffer).getEntries()`) well suited to fully-in-memory extraction of modest archives; MIT-licensed, 19.2M weekly downloads, actively maintained (github.com/cthackers/adm-zip). The automated `package-legitimacy check` seam flagged it `SUS` with reason `"too-new"` — manually confirmed as a **false positive**: the flagged date is the latest patch release (2026-07-10), not the package's origin; this is the same false-positive pattern STACK.md already documented for fast-xml-parser/csv-parse/playwright. `[ASSUMED]` provenance for the package-name choice itself (discovered via training knowledge, not an official doc) — planner should gate the `pnpm add adm-zip` step behind a `checkpoint:human-verify` per the package-legitimacy protocol. |
| `csv-parse` | 7.0.1 [CITED: STACK.md] | DÖE CSV export (`format=csv.zip`) as a cross-check/fallback format | Unchanged from STACK.md; confirmed this session that `csv.zip` is a real, working `format` enum value in the live spec |
### Supporting
| Library | Version | Purpose | When to Use |
|---------|---------|---------|-------------|
| native `fetch` | Node 22 built-in | HTTP client for the DÖE endpoint | Already the established pattern (`icon-discovery.service.ts`, `ics.provider.ts`); DÖE requires no cookies/session state, so `fetch-cookie`/`tough-cookie` (needed for the scraping adapters in a later phase) are **not** needed for this phase |
### Alternatives Considered
| Instead of | Could Use | Tradeoff |
|------------|-----------|----------|
| `adm-zip` (sync, in-memory) | `unzipper` (streaming) [VERIFIED: npm registry — 16.7M weekly downloads, same false-positive SUS flag] | Streaming extraction only pays off for very large archives; DÖE daily exports are small (low hundreds of files, sub-MB to low-MB compressed) — sync in-memory is simpler and matches the "wartbar/verständlich" project constraint |
| eForms-DE XML as primary parse target | OCDS JSON as primary parse target (ARCHITECTURE.md's original recommendation) | **Reversed this session based on live cross-check**: OCDS conversion measurably drops structured deadline data present in the source eForms-DE XML for at least one directly-verified notice. OCDS remains useful for `ocid` (dedup key) and resolved party/organization names, but should not be the sole source for deadline/value. |
**Installation:**
```bash
pnpm --filter @tessera/api add fast-xml-parser adm-zip csv-parse
```
**Version verification:** confirmed live via `npm view <pkg> version` this session (2026-07-21): `adm-zip@0.6.0` (published 2026-07-10), `unzipper@0.12.5` (published 2026-06-21). `fast-xml-parser`/`csv-parse` versions carried over from STACK.md (2026-07-17, 4 days prior — still current).
## Package Legitimacy Audit
| Package | Registry | Age (latest release) | Downloads | Source Repo | Verdict | Disposition |
|---------|----------|-----|-----------|-------------|---------|-------------|
| adm-zip | npm | 11 days (0.6.0, 2026-07-10) | 19.2M/wk | github.com/cthackers/adm-zip | SUS (`too-new` — false positive, see rationale above) | Flagged — planner adds `checkpoint:human-verify` before `pnpm add adm-zip`, per protocol |
| fast-xml-parser | npm | (per STACK.md, 2026-07-17) | — | — | SUS (per STACK.md, already assessed false-positive) | Approved (carried over from milestone research, no re-flag needed) |
| csv-parse | npm | (per STACK.md, 2026-07-17) | — | — | SUS (per STACK.md, already assessed false-positive) | Approved (carried over) |
**Packages removed due to [SLOP] verdict:** none.
**Packages flagged as suspicious [SUS]:** `adm-zip` — planner must add a `checkpoint:human-verify` task before `pnpm add adm-zip` (spot-check the GitHub repo / Socket.dev score at implementation time, per the repo's existing dependency-hygiene bar already applied to the other v1.1 packages in STACK.md).
*`adm-zip` was discovered via training knowledge (not an official doc), so per the package-name provenance rule it is tagged `[ASSUMED]` regardless of its clean npm registry / GitHub signals.*
## Architecture Patterns
### System Architecture Diagram
```
[TenderSchedulerService — 1 global cron, e.g. hourly per D-04]
|
v
[DoeOpenDataAdapter.fetchTenders(config, dayCursor)]
| 1. if dayCursor >= today(Europe/Berlin) -> no-op, return [] (nothing new yet)
| 2. else: GET /api/notice-exports?pubDay={dayCursor}&format=eforms.zip (+ ocds.zip for ocid/orgs)
v
[adm-zip: extract ZIP -> N per-notice XML (+ JSON) buffers]
v
[fast-xml-parser: parse each eForms-DE XML -> RawTenderRecord[]]
v
[TenderNormalizerService.normalize()]
| - filter: keep only tag=["tender"] (D-02); drop award/planning/untagged-with-awards
| - map fields -> Tender (title, buyer, cpvCodes, region/plz, deadlineAt, estimatedValue, procedureType, sourceUrl)
| - dedupKey: ocid (from paired OCDS fetch) -> fallback sourcePortal:noticeId
| - contentHash: sha256(title+deadline+value+status)
v
[TenderIngestionService.upsert()] -- prisma.tender.upsert({ where: { dedupKey } })
| advance dayCursor by 1 day; loop while dayCursor still < today (catch-up on missed days)
v (only newly-created OR contentHash-changed rows this run)
[TenderMatchingService.matchDelta()] -- Phase 11+ (saved searches don't exist yet in Phase 10)
```
### Recommended Project Structure
(Extends ARCHITECTURE.md's `apps/api/src/tenders/` layout — no changes needed, confirmed still correct after live verification.)
```
apps/api/src/tenders/
├── tenders.module.ts
├── tenders.controller.ts # GET /tenders, GET /tenders/:id, admin source-config
├── tenders.seed.ts # seeds 'tender-radar' into ModuleRegistry (CONFIG-01)
├── tender-ingestion.service.ts # poll -> normalize -> upsert -> change-detect
├── tender-normalizer.service.ts # eForms-XML+OCDS -> Tender fields, dedupKey, contentHash
├── tender-scheduler.service.ts # SchedulerRegistry cron: 1 global job, day-cursor gated
├── tender.types.ts # RawTenderRecord, NormalizedTenderFields, SourceType
├── dto/
│ ├── source-config.dto.ts # admin: pollIntervalMin, isActive
│ └── tender-query.dto.ts # pagination + filter query params (list/detail read, no saved-search yet)
└── adapters/
├── tender-source-adapter.interface.ts
└── doe-opendata.adapter.ts # THIS PHASE's only adapter
```
### Pattern 1: DÖE API — Day-Batch, Not Paginated Feed (live-verified this session)
**What:** `GET https://oeffentlichevergabe.de/api/notice-exports` — the *entire* real API surface for OpenData bulk retrieval (confirmed: the live OpenAPI 3.1 spec at `/documentation/api/opendata` lists exactly one path). No auth (no `security` scheme in the spec; confirmed 200 with a plain unauthenticated `curl`).
**Parameters (from the real spec, cross-verified against live requests):**
- `pubDay` (string, `YYYY-MM-DD`) — returns all notice *versions* processed on that calendar day. **Mutually exclusive with `pubMonth`.**
- `pubMonth` (string, `YYYY-MM`) — same, for a whole month (useful only for a manual/admin-triggered historical import — not used in this phase per D-01).
- `format` (query param, or via `Accept` header): `eforms.zip` (default), `ocds.zip`, `csv.zip`.
**Live-verified boundary rules (confirmed via actual 400 responses, not just spec text):**
- `pubDay` = today → **HTTP 400**, message: `"The specified pubDay exceeds the allowed range. It must lie in the past."`
- `pubDay` = any future date → same 400.
- Earliest allowed: `pubDay >= 2022-12-01`, `pubMonth >= 2022-12` (per spec text — the Bekanntmachungsservice's operational start).
- Response headers on a real 200: `etag: "version-N"`, `cache-control: max-age=120`, `content-disposition: attachment; filename=notices_YYYY-MM-DD_OCDS.zip`. **No rate-limit headers of any kind** (`X-RateLimit-*` absent) — the `etag`/short cache-control instead suggest a caching layer in front of the origin; conditional `If-None-Match` requests are a viable politeness optimization but not required for correctness at day-granularity polling.
**When to use:** This is the *only* way to get DÖE data — there is no "list notices since X" or paged-results endpoint. Design the adapter around "fetch one day's full batch," not "fetch new items since last check."
**Trade-offs:** A day-cursor model means: (a) the very first poll after module activation on deployment day will return **zero** results until the next calendar day rolls over (today's data isn't retrievable yet) — this is expected API behavior, not a bug, and should be documented in the phase's UAT expectations; (b) catching up after downtime (e.g., scheduler was down for 3 days) is naturally supported — just loop `pubDay` from `lastIngestedDay + 1` up to `today - 1`, each as a separate request.
### Pattern 2: OCDS Release Tag as the D-02 "Open Tender" Filter (live-verified)
**What:** Live sample (105 real notices, 2026-07-20) breaks down as:
| `releases[].tag` | Count | Meaning |
|---|---|---|
| `["tender"]` | 74 (70%) | Open/active procurement notice — **the only kind SCHEMA-01/D-02 wants** |
| `["award"]` | 17 (16%) | Award/result notice — explicitly out of scope (D-02) |
| `["planning"]` | 3 (3%) | Prior-information/planning notice — not yet a bookable tender — exclude |
| *(no `tag` field at all)* | 11 (11%) | **Live-verified edge case** — direct-award justification notices ("Vergabevermerke") with populated `awards`/`contracts` arrays but no `tag`. These are closed/awarded, not open. |
**Filter logic (recommended, not yet in ARCHITECTURE.md):**
```typescript
function isOpenTenderNotice(release: OcdsRelease): boolean {
const tags = release.tag ?? [];
if (tags.includes('tender')) return true;
// Untagged notices with populated awards/contracts are NOT open — exclude.
// Untagged notices with neither (rare, if any) should also be excluded conservatively
// rather than defaulting to "open" — a false "open tender" shown to a bidder is worse
// than a missed one (which the coverage-transparency UI in Phase 11 already communicates).
return false;
}
```
**Why this matters:** A naive filter of "absence of `award` tag = open" would have wrongly included the 11 untagged-but-actually-closed notices found in this live sample (11% of one real day — not a negligible edge case).
### Pattern 3: eForms-DE XML as Primary Structured-Field Source (live cross-check, corrects ARCHITECTURE.md)
**What:** ARCHITECTURE.md recommended "prefer OCDS... for direct field alignment." Live cross-check of the **same real notice** (`25605686-1`) in both formats shows:
- OCDS JSON: `tender.tenderPeriod` is **absent** (no deadline field at all); `tender.value` is **absent**; the deadline ("Angebotsfrist: Mittwoch, 26.08.2026, 10:00 Uhr") exists only as unstructured German prose inside `tender.description`.
- eForms-DE XML (same notice, same day, same export): contains a proper structured element — `<TenderSubmissionDeadlinePeriod><ns3:EndDate>2026-08-26+02:00</ns3:EndDate></TenderSubmissionDeadlinePeriod>` — and the document's `CustomizationID` (`eforms-sdk-0.1`) confirms this notice uses an old eForms SDK version (directly matching PITFALLS.md's Pitfall 17 — SDK version drift — now observed live, not just theorized).
**Recommendation:** Fetch **both** formats per poll cycle (`eforms.zip` for structured deadline/value/procedure fields; `ocds.zip` for the `ocid` dedup key and resolved buyer/party names) rather than relying on OCDS alone. This is two HTTP requests per day-cursor tick instead of one — negligible cost at day-granularity polling. If phase scope favors a single-format MVP, prefer `eforms.zip` as primary and treat `ocid` absence as acceptable for v1 (fall back to `sourcePortal:noticeId` dedup key, already the documented fallback tier in ARCHITECTURE.md's Pattern 2) — but fetching both is the more complete and only modestly more expensive option, and is the recommended default.
**Trade-off:** Two structurally different parsers now run per poll (fast-xml-parser for eForms-DE, plain `JSON.parse` for OCDS) — acceptable complexity, both already selected in STACK.md; no new parser library needed for this reversal, only a change in which one is authoritative for which fields.
### Pattern 4: Nullable Deadline/Value as the Common Case, Not the Edge Case
**What:** Of the 12 "tender"-tagged notices spot-checked in this session's live sample, **all 12** had `tenderPeriod.endDate = null` and `value = null` in the OCDS export. This is a stronger and more concrete version of PITFALLS.md's Pitfall 18 ("OCDS optional/conditional fields treated as always present") — live data suggests this is the **majority case for Unterschwelle notices**, not a rare edge case.
**Recommendation for SCHEMA-01/D-05:**
- `deadlineAt` and `estimatedValue` **must** be nullable in the `Tender` Prisma model (already planned in ARCHITECTURE.md's schema sketch — confirmed necessary, not optional hardening).
- D-05's "90 days after deadline, then delete" retention rule needs an explicit fallback for `deadlineAt IS NULL` rows: **do not auto-expire them** on the deadline-based path (there is no deadline to expire from); either (a) leave them permanently active/searchable (simplest, matches "keine Vor-Eingrenzung" spirit — a human should judge relevance, not the retention job), or (b) apply a secondary fallback expiry based on `publishedAt + N days` for deadline-less rows only. Recommend (a) for the MVP phase — introduce (b) later if the table grows unmanageably, since retrofitting a stricter policy is easy but a wrongly-deleted no-deadline tender is not recoverable.
- The "nur offene" (still-open) view/filter (Phase 11, FILTER-04) will only be meaningful for the subset with a populated `deadlineAt` — this is a coverage-transparency concern (PITFALLS.md Pitfall 20) that now has a concrete, live-measured magnitude (100% of a 12-notice spot sample) to inform the Phase 11 UI copy, not just an abstract "some tenders lack this."
## Don't Hand-Roll
| Problem | Don't Build | Use Instead | Why |
|---------|-------------|-------------|-----|
| ZIP extraction | A manual DEFLATE/central-directory parser | `adm-zip` | Well-understood, tiny surface, sync API fits the modest archive sizes involved |
| Multi-tenant scheduling | A second scheduling mechanism | `SchedulerRegistry.addCronJob()` (same as `DkvSchedulerService`) | Already proven, dynamically updatable at runtime — reuse the mechanics, just replace the `findFirst()`-single-tenant *pattern*, not the underlying `cron` library |
| Module registry / marketplace activation | A new activation table or gating mechanism | `ModuleRegistryService` (unmodified) + `tenders.seed.ts` (new, mirrors `cert-manager.seed.ts`) | Already the load-bearing platform mechanism for CONFIG-01; do not special-case Tenders |
| OCID/dedup key computation | A custom hashing scheme from scratch | Priority-ordered key per ARCHITECTURE.md Pattern 2 (`ocid` → `sourcePortal:noticeId` → fuzzy fingerprint) | Already researched and live-confirmed: `ocid` is present and stable across notice versions (`ocds-mnwr74-{processId}`, distinct from the per-version notice/release `id`) |
**Key insight:** This phase's genuinely new risk is not "how do we build a scheduler" (solved, mechanics reused from DKV) — it's **correctly interpreting the DÖE API's actual batch/day-cursor shape**, which this session's live verification now settles definitively instead of leaving as an assumption.
## Runtime State Inventory
Not applicable — this is a greenfield module addition (new Prisma models, new NestJS module, new module-registry row), not a rename/refactor/migration phase. No existing runtime state references the "Ausschreibungs-Radar"/"tender-radar" name that would need updating.
## Common Pitfalls
### Pitfall A: Treating the DÖE poll interval as "fetch frequency" instead of "check frequency"
**What goes wrong:** A naive implementation calls the DÖE endpoint every tick of the admin-configured interval (e.g., hourly per D-04), re-requesting the *same* day's ZIP 23 times before the day actually advances — wasted load, and worse, a developer "optimizing" this by shortening the interval further, or "fixing" the apparent no-op by re-implementing since-timestamp logic that the API doesn't actually support.
**Why it happens:** D-04 says "poll interval, admin-configurable, default hourly" — this reads like a typical incremental-feed poll interval, but the DÖE API's actual granularity is daily.
**How to avoid:** Decouple "cron tick frequency" (can stay hourly, satisfies D-04/INGEST-06 literally) from "actual upstream fetch" (gated by `dayCursor < today(Europe/Berlin)`). Document this decoupling explicitly in the adapter/scheduler code, not just in this research doc.
**Warning signs:** Scheduler logs show identical `pubDay` values fetched repeatedly within the same 24h window with unchanged results — expected, not a bug, but worth a debug-level (not warn/error) log line so it doesn't look broken during on-call review.
### Pitfall B: Assuming OCDS alone is sufficient for SCHEMA-01's deadline/value fields
**What goes wrong:** Building the normalizer against OCDS only (per ARCHITECTURE.md's original guidance) ships a `Tender` catalog where deadline/value are null for the majority of below-threshold notices, even though the source data (eForms-DE XML) actually had it.
**How to avoid:** Parse eForms-DE XML as the primary field source per Pattern 3 above; treat OCDS as the `ocid`/party-resolution source, not the deadline/value source.
**Warning signs:** QA/UAT notices most "tender"-tagged records showing no deadline in the results list — check whether the eForms-XML export was even fetched/parsed before assuming this is unavoidable API limitation (it partly is — not every eForms document will have a machine-readable deadline either, since it can be embedded in a portal-specific free-text description — but the eForms path recovers meaningfully more than OCDS alone, confirmed on at least one directly cross-checked notice this session).
### Pitfall C: Filtering "open tenders" (D-02) by tag absence instead of tag presence
**What goes wrong:** Assuming "no `award` tag = open tender" wrongly includes untagged-but-closed notices (11% of the live sample).
**How to avoid:** Filter positively on `tag.includes('tender')`; treat missing/other tags as excluded by default (Pattern 2 above).
**Warning signs:** Award-justification/"Vergabevermerk" notices (identifiable by populated `awards`/`contracts` arrays with no submission deadline) appearing in the "open, biddable" result list.
### Pitfall D: Copying DkvSchedulerService's `findFirst()` single-tenant pattern (carried over from PITFALLS.md Pitfall 24, restated for this phase specifically)
**What goes wrong:** Already extensively documented in PITFALLS.md — restated here because this phase's scheduler is the first place it must be avoided, and because the DÖE source is **global** (not per-tenant, unlike DKV), the "multi-tenant" framing is slightly different: there IS only one DÖE poll config (platform-wide, per Pattern 3 in ARCHITECTURE.md), so `findFirst()` on `TenderSourcePollConfig` for the DÖE row specifically is **not** the anti-pattern here (there's genuinely only one DÖE config row to find) — the anti-pattern would be if a future per-tenant source (e.g., email-alert in Phase 14) reused this same `findFirst()` call path. **For Phase 10 specifically:** it's safe and correct to load the single DÖE `TenderSourcePollConfig` row via `findFirst()` (or better, `findUnique` on a fixed known slug like `sourceType: 'doe-opendata'`) since it is architecturally a singleton config — this is the one legitimate `findFirst()` in this phase; the acceptance-criterion "activating a 2nd tenant must not duplicate ingestion" is satisfied by the config being tenant-agnostic in the first place (Pattern 3), not by scheduler code that iterates tenants.
**Verification for this phase's two-tenant acceptance criterion (Success Criteria 4 & 5):** the test is NOT "does the scheduler correctly loop over 2 tenant configs" (there are none for DÖE) — it is "does activating the module for a 2nd tenant (`TenantModuleActivation` row) trigger zero additional DÖE HTTP calls, zero additional cron jobs, and zero additional `Tender` rows" — i.e., prove the *absence* of tenant-count-scaled behavior, not the presence of correct per-tenant iteration.
## Code Examples
### DÖE fetch — real, verified request shape
```typescript
// Source: live-verified this session against https://oeffentlichevergabe.de
// GET https://oeffentlichevergabe.de/api/notice-exports?pubDay=2026-07-20&format=eforms.zip
// -> 200, Content-Disposition: attachment; filename=notices_2026-07-20_eForms.zip
// -> binary ZIP body, ~105 files for a typical day (~160KB compressed for the OCDS variant)
async function fetchDoeDay(pubDay: string, format: 'eforms.zip' | 'ocds.zip' | 'csv.zip') {
const url = `https://oeffentlichevergabe.de/api/notice-exports?pubDay=${pubDay}&format=${format}`;
const res = await fetch(url);
if (res.status === 400) {
// "must lie in the past" — pubDay is today or future; nothing to fetch yet this cycle
return null;
}
if (!res.ok) throw new Error(`DÖE fetch failed: ${res.status}`);
return Buffer.from(await res.arrayBuffer());
}
```
### Day-cursor gate (replaces a `since: Date` timestamp cursor)
```typescript
// Source: this session's live finding — DÖE has no incremental/since parameter.
function nextDayToFetch(lastIngestedDay: Date | null): string | null {
const berlinToday = new Date().toLocaleDateString('en-CA', { timeZone: 'Europe/Berlin' }); // YYYY-MM-DD
const next = lastIngestedDay
? new Date(lastIngestedDay.getTime() + 86_400_000).toISOString().slice(0, 10)
: new Date().toISOString().slice(0, 10); // D-01: "from now" — first eligible day is tomorrow relative to activation
return next < berlinToday ? next : null; // null = nothing new yet, no-op this tick
}
```
### D-02 open-tender filter (OCDS release tag)
```typescript
// Source: live-verified against a real 105-notice sample, 2026-07-20
function isOpenTenderNotice(release: { tag?: string[]; awards?: unknown[]; contracts?: unknown[] }): boolean {
if (release.tag?.includes('tender')) return true;
return false; // conservative: missing tag + award/planning tags are all excluded
}
```
## State of the Art
| Old Approach (assumed pre-verification) | Current Approach (live-verified this session) | When Changed | Impact |
|--------------------------------------------|-----------------------------------------------|---------------|--------|
| DÖE = paginated REST feed with a `since`/cursor param (implicit assumption in ARCHITECTURE.md's adapter signature) | DÖE = day/month-batch ZIP export, no pagination param exists, current/future days rejected (400) | Confirmed 2026-07-21, this session | Scheduler must be day-cursor gated, not since-timestamp gated; "pagination handling" (a phase-focus research question) resolves to "there is no pagination — there is a per-day archive" |
| "Prefer OCDS over eForms-DE" (ARCHITECTURE.md Pattern 2, based on the OCDS spec's cleaner structure) | eForms-DE XML has structured deadline data that OCDS conversion drops, at least for some notices | Confirmed 2026-07-21, this session, via direct cross-check of one real notice in both formats | Normalizer should parse eForms-DE as primary for deadline/value; OCDS retained for `ocid`/party names |
**Deprecated/outdated:** None — this is a young API (operational since 2023-02-06 per prior research); no deprecated version churn observed in the live spec.
## Assumptions Log
| # | Claim | Section | Risk if Wrong |
|---|-------|---------|---------------|
| A1 | `adm-zip` is the right ZIP library choice (vs. `unzipper`/`jszip`/`yauzl`) | Standard Stack | Low — all are viable; `adm-zip`'s sync API is a convenience choice, not a hard technical requirement. If archives grow much larger than observed (161KB/105 files for one day), a streaming approach (`unzipper`) may be preferable — reassess if daily archive size grows past a few MB. |
| A2 | Fetching both `eforms.zip` and `ocds.zip` per poll cycle (two HTTP calls) is acceptable overhead vs. a single-format MVP | Pattern 3 | Low — at day-granularity polling (at most 1-2 real fetches/day per the day-cursor gate), two requests instead of one is negligible; if the planner descopes to single-format for MVP speed, document that deadline/value coverage will be measurably worse (per Pattern 4's 100%-null spot check) as an explicit MVP trade-off, not a silent gap |
| A3 | The 105-notice, 12-notice, and 11-notice samples (all from 2026-07-20) are representative of typical daily volume/tag distribution, not an anomalous day | Pattern 2, Pattern 4 | Medium — a single day's sample is directional, not statistically proven; if actual production volume/tag ratios differ significantly, the D-02 filter logic itself is still correct (tag-based), only the *magnitude* estimates (70% tender / 16% award / etc.) may shift |
## Open Questions
1. **Exact rate-limit tolerance of the DÖE API (undocumented)**
- What we know: no `X-RateLimit-*` headers observed on a real response; a `cache-control: max-age=120` + `etag` suggest a caching/CDN layer that can absorb repeated identical requests cheaply.
- What's unclear: whether rapid-fire requests (e.g., a catch-up loop fetching 30 missed days in quick succession after extended downtime) would trigger throttling — not tested this session to avoid abusing a public-good API mid-research.
- Recommendation: implement the catch-up loop (Pattern 1) with a small delay between requests (e.g., 1-2s, matching the "politeDelay" helper already recommended in STACK.md for the scraping adapters) even though DÖE itself carries "null" ToS risk — cheap insurance, no downside.
2. **Whether eForms-DE deadline coverage is meaningfully better than OCDS across a larger sample**
- What we know: confirmed better on 1 directly cross-checked notice (deadline present in XML, absent in OCDS JSON and OCDS proper structured field).
- What's unclear: the aggregate improvement rate across hundreds/thousands of notices — this session's live check was necessarily a small spot-check, not a full-corpus audit.
- Recommendation: the phase's implementation should log a metric (e.g., `% of ingested tender-tagged notices with non-null deadlineAt`) from day one, so this can be measured empirically in production rather than re-guessed.
## Environment Availability
| Dependency | Required By | Available | Version | Fallback |
|------------|------------|-----------|---------|----------|
| Outbound internet access from the deployment host to `oeffentlichevergabe.de` | INGEST-01 (DÖE polling) | ✓ (verified from this dev/research environment — production host connectivity not independently verified this session) | — | If the production host lacks outbound HTTPS to this domain (e.g., restrictive egress firewall), the module cannot function at all — confirm with infra before phase execution if the API host is on the same network segment as the dev machine used for this research |
| Node.js `fetch` (native, Node 18+) | DÖE HTTP client | ✓ | Node 22-class per `@types/node@^22` (existing project baseline) | — |
| `adm-zip` / equivalent ZIP library | ZIP extraction | ✗ (not yet installed) | — | `pnpm add adm-zip` — no fallback needed, this is a build-time dependency, not an external service |
**Missing dependencies with no fallback:** none beyond the standard "install the new npm package" step.
**Missing dependencies with fallback:** none applicable.
## Validation Architecture
### Test Framework
| Property | Value |
|----------|-------|
| Framework | Vitest 3.x (existing, per v1.0 STACK.md) |
| Config file | `apps/api/vitest.config.ts` (existing — confirm path during planning; not modified by this phase) |
| Quick run command | `pnpm --filter @tessera/api test -- tenders` (scoped to new test files) |
| Full suite command | `pnpm --filter @tessera/api test` |
### Phase Requirements → Test Map
| Req ID | Behavior | Test Type | Automated Command | File Exists? |
|--------|----------|-----------|-------------------|-------------|
| CONFIG-01 | Module self-seeds into registry on boot; activatable per tenant | unit + integration | `pnpm --filter @tessera/api test -- tenders.seed` | ❌ Wave 0 |
| INGEST-01 | `DoeOpenDataAdapter` parses a real (or fixture-captured) eForms-DE/OCDS ZIP into `RawTenderRecord[]` | unit | `pnpm --filter @tessera/api test -- doe-opendata.adapter` | ❌ Wave 0 — fixture ZIP should be captured from this session's live download (`doe-2026-07-20-eforms.zip`/`doe-2026-07-20-ocds.zip`) rather than synthetic data, per this repo's stated preference for real fixtures over synthetic ones (matches the DKV PDF-parser precedent) |
| INGEST-06 | Two-tenant scheduler test: activating module for a 2nd tenant produces zero additional DÖE HTTP calls / cron jobs / `Tender` rows | integration | `pnpm --filter @tessera/api test -- tender-scheduler.service` | ❌ Wave 0 |
| SCHEMA-01 | Normalizer maps a real eForms-DE + OCDS notice pair into the expected `Tender` fields, including nullable deadline/value | unit | `pnpm --filter @tessera/api test -- tender-normalizer.service` | ❌ Wave 0 |
| SCHEMA-02 | Re-ingesting an updated notice (same `ocid`, changed `contentHash`) updates the existing row instead of duplicating | integration | `pnpm --filter @tessera/api test -- tender-ingestion.service` | ❌ Wave 0 |
| D-02 filter | `tag=["tender"]` included; `award`/`planning`/untagged-with-awards excluded | unit | `pnpm --filter @tessera/api test -- doe-opendata.adapter` (same fixture, asserts filtered-out count) | ❌ Wave 0 |
### Sampling Rate
- **Per task commit:** `pnpm --filter @tessera/api test -- tenders` (scoped)
- **Per wave merge:** `pnpm --filter @tessera/api test` (full suite)
- **Phase gate:** Full suite green before `/gsd-verify-work`
### Wave 0 Gaps
- [ ] `apps/api/src/tenders/__fixtures__/doe-2026-07-20-eforms.zip` and `doe-2026-07-20-ocds.zip` — real captured fixtures from this session's live download (recommend committing a trimmed version, e.g., 5-10 representative notices covering tender/award/planning/untagged tags, rather than the full 105-file archive, to keep the repo lean)
- [ ] `apps/api/src/tenders/tenders.seed.spec.ts`, `doe-opendata.adapter.spec.ts`, `tender-normalizer.service.spec.ts`, `tender-ingestion.service.spec.ts`, `tender-scheduler.service.spec.ts` — none exist yet, this is a greenfield module
## Security Domain
### Applicable ASVS Categories
| ASVS Category | Applies | Standard Control |
|---------------|---------|-------------------|
| V2 Authentication | No | DÖE endpoint is auth-free by design (confirmed live); no credentials to protect for this specific integration |
| V3 Session Management | No | No session/cookie state involved in the DÖE adapter (unlike the later scraping adapters) |
| V4 Access Control | Yes | `GET /tenders` list/detail must be gated by `TenantModuleActivation` (module licensing) but NOT by `tenantId` row-filtering — this is the one controller where "tenant-gated" and "tenant-scoped" genuinely differ (per ARCHITECTURE.md's explicit callout); admin source-config routes (`POST /tenders/source-config`) must be `@Roles(Role.ADMIN, Role.SUPER_ADMIN)`-guarded like `DkvController` |
| V5 Input Validation | Yes | All parsed eForms-XML/OCDS-JSON fields are external, untrusted input — sanitize/escape before any future HTML rendering (Phase 11 UI); validate `pubDay`/`pubMonth` cursor values server-side before constructing the outbound DÖE URL (defense against cursor-corruption, low risk since these are internally computed, not user-supplied, but validate anyway) |
| V6 Cryptography | No | No credentials/secrets are stored for this phase (DÖE is auth-free) — `CalendarCryptoService`/AES-256-GCM pattern is reserved for later phases (email-alert inbox credentials, Phase 14) |
### Known Threat Patterns for this stack
| Pattern | STRIDE | Standard Mitigation |
|---------|--------|----------------------|
| Untrusted external content (buyer names, tender descriptions, CPV descriptions from eForms/OCDS) rendered unescaped in a future UI | Tampering / Injection | Sanitize/escape all ingested text fields before HTML interpolation — applies to Phase 11's UI, but the normalizer in this phase should not assume the data is "safe" just because it came from a government API; store raw, escape at render time |
| Zip-bomb / decompression-bomb via a maliciously large or deeply-nested ZIP response | Denial of Service | Low realistic risk given DÖE is a trusted government-operated source over HTTPS, but as defense-in-depth: `adm-zip`'s `getEntries()` reads the central directory without fully decompressing — validate total uncompressed size against a sane ceiling (e.g., reject if `entries.reduce(sum of entry.header.size) > 50MB`) before extracting each entry's buffer |
| Cursor/date-parameter injection into the outbound DÖE URL | Tampering | The `pubDay`/`pubMonth` cursor is entirely internally computed from `lastIngestedDay` (never user-supplied) — no admin-facing input flows directly into this URL, so injection risk is structurally absent as designed; keep it that way (don't add an admin "manually trigger for date X" feature without validating the date format server-side first) |
## Sources
### Primary (HIGH confidence)
- Live API verification, this session (2026-07-21): `https://oeffentlichevergabe.de/documentation/swagger-ui/opendata/swagger-initializer.js` (reveals real spec URL, bypassing the JS-rendered Swagger UI), `https://oeffentlichevergabe.de/documentation/api/opendata` (real OpenAPI 3.1 spec, fetched and parsed), `https://oeffentlichevergabe.de/api/notice-exports?pubDay=2026-07-20&format=ocds.zip` and `...&format=eforms.zip` (real 200 responses, 105 real notices downloaded and inspected), boundary tests for `pubDay=today` and `pubDay=future` (both confirmed 400)
- `npm view adm-zip version` / `npm view unzipper version` / `gsd-tools query package-legitimacy check` — direct registry + legitimacy-seam verification, this session
- Existing codebase (direct read, this session): `apps/api/src/dkv/dkv-scheduler.service.ts`, `apps/api/src/dkv/dkv.service.ts`, `apps/api/src/dkv/dkv.module.ts`, `apps/api/src/dkv/dkv.controller.ts`, `apps/api/src/dkv/dkv.seed.ts`, `apps/api/src/cert-manager/cert-manager.seed.ts`, `apps/api/src/module-registry/module-registry.service.ts`, `apps/api/src/prisma/prisma-tenant.extension.ts`, `apps/api/prisma/schema.prisma` (`DkvModuleConfig` model), `apps/web/src/lib/module-loader.ts`, `apps/web/src/app/(portal)/modules/dkv-fleet/settings/components/InboxConfigForm.tsx`, `apps/api/src/app.module.ts`, `apps/api/package.json`
### Secondary (MEDIUM confidence)
- `.planning/research/SUMMARY.md`, `ARCHITECTURE.md`, `STACK.md`, `PITFALLS.md`, `ausschreibungs-portale-feasibility.md` (milestone-level research, 2026-07-16/17) — used as the binding baseline this document extends/corrects where live verification diverged
### Tertiary (LOW confidence)
- None — this session prioritized live verification over inference specifically because the milestone research had flagged the DÖE pagination/rate-limit question as an explicit open point.
## Metadata
**Confidence breakdown:**
- DÖE API mechanics (endpoint, params, day-batch model, boundary rules): HIGH — directly observed via live requests this session, not inferred from documentation
- OCDS/eForms field completeness (deadline/value nullability, tag distribution): MEDIUM-HIGH — directly observed on a real sample, but the sample is one day (105 notices) and may not capture the full range of buyer/portal submission quality variance
- Platform integration (module registry, scheduler mechanics, Prisma conventions): HIGH — verified against existing, working codebase
- ZIP library choice (`adm-zip`): MEDIUM — sound technical choice, but package-name provenance is `[ASSUMED]` (training knowledge) per the package-legitimacy protocol; flagged for `checkpoint:human-verify`
**Research date:** 2026-07-21
**Valid until:** DÖE API mechanics — treat as stable for 90 days (young, actively-operated government API, no deprecation signals observed; re-verify if a phase re-plan happens much later). npm package versions — 30 days (fast-moving ecosystem norm).