tessera-ctl/.planning/phases/10-ausschreibungs-radar-foundation-d-e-ingestion/10-RESEARCH.md

Phase 10: Ausschreibungs-Radar Foundation & DÖE Ingestion - Research

Researched: 2026-07-21 Domain: German public-procurement (DÖE OpenData) live API ingestion, OCDS/eForms-DE normalization, multi-tenant poll-once-fan-out-many scheduler, module-registry integration on the existing Tessera NestJS/Prisma platform Confidence: HIGH (DÖE API mechanics — verified live against the production API this session, not just the JS-rendered Swagger UI); HIGH (platform integration — verified against existing dkv//module-registry/ codebase); MEDIUM (OCDS field completeness — verified against real live export, one day's sample; genuinely variable by buyer/portal)

<user_constraints>

User Constraints (from CONTEXT.md)

Locked Decisions

DÖE-Ingest-Umfang

• D-01: Initialer Backfill = nur ab jetzt. Beim ersten plattformweiten DÖE-Poll werden nur ab diesem Zeitpunkt veröffentlichte Ausschreibungen aufgenommen — kein historischer Import. Da die Tender-Daten global sind, sieht ein später aktivierender Mandant den seit Plattformstart aufgelaufenen Bestand. Passt zur späteren Backfill-Unterdrückung bei Benachrichtigungen (Phase 12).
• D-02: Nur offene Ausschreibungen (aktive Vergaben, auf die man bieten kann — tender/contract notices). Vergabeergebnisse, Zuschläge und Aufhebungen (award/result notices) werden in dieser Phase NICHT aufgenommen.
• D-03: Ingest lädt ganz Deutschland global (plattformweit einmal), keine Vor-Eingrenzung nach Region/CPV beim Ingest — Eingrenzung passiert erst pro Suchprofil (Phase 11). Bewusst wegen Multi-Tenant-Wiederverkauf.
• D-04: Standard-Poll-Intervall = stündlich, admin-konfigurierbar (INGEST-06).

Datenaufbewahrung

• D-05: Ausschreibungen nach Ablauf der Abgabefrist werden 90 Tage aufbewahrt, dann gelöscht. Vor Fristablauf: aktiv. Nach Fristablauf: als „abgelaufen" markiert, standardmäßig aus der aktiven Liste ausgeblendet, aber bis zur Löschung recherchierbar.

Claude's Discretion

• Exakte Prisma-Schema-Felder (OCDS-orientiert, Form gemäß ARCHITECTURE.md), Adapter-Interface (TenderSourceAdapter), Normalizer-Interna, Dedup-Key-Berechnung (OCID → Quelle:NoticeId → …) und contentHash-Änderungserkennung.
• DÖE-API-Client (native fetch + fast-xml-parser, CSV-Fallback), Pagination/Rate-Limit-Handling (Detail per Phase-Research live zu klären).
• Scheduler-Implementierung: poll-once-fan-out-many, mehrmandantensicher — NICHT das DKV-findFirst()-Single-Tenant-Muster.
• Modul-Registrierung im Marketplace nach DKV/Cert-Manager-Vorbild.

Deferred Ideas (OUT OF SCOPE)

• Vergabeergebnisse/Zuschläge/Aufhebungen (award notices) aufnehmen — spätere Erweiterung, außerhalb v1.1-Scope.
• Historischer Backfill (letzte 30 Tage / alles) — bei Bedarf später als Konfig-Option.
• Vor-Eingrenzung des Ingest nach Region/CPV — bewusst verworfen (widerspricht Multi-Tenant-Wiederverkauf); Eingrenzung bleibt Sache der Suchprofile.
• Aufbewahrungsdauer (90 Tage) später ggf. admin-konfigurierbar machen. </user_constraints>

<phase_requirements>

Phase Requirements

ID	Description	Research Support
CONFIG-01	Modul im Marketplace registriert, pro Mandant aktivierbar (wie DKV-Fleet/Cert-Manager)	tenders.seed.ts follows verbatim cert-manager.seed.ts/dkv.seed.ts pattern (isSystem: true, category, i18n description). module-loader.ts MODULE_REGISTRY entry required or the page 404s even when activated (verified codebase pattern, see Architecture Patterns §1)
INGEST-01	Zeitgesteuerter Abruf via DÖE OpenData API (eForms/OCDS, auth-frei)	Live-verified in this session: real endpoint GET /api/notice-exports, auth-free (no security scheme in spec, confirmed 200 with no credentials), format negotiation via Accept header or format query param — see DÖE API Live Findings
INGEST-06	Poll-once-fan-out-many, Intervall pro Quelle admin-konfigurierbar	See Scheduler Design — day-cursor gated single global cron (not per-tenant), admin-configurable interval field per D-04 with an adapter-level no-op guard for the DÖE source's actual day-granularity
SCHEMA-01	Einheitliches OCDS-orientiertes Schema, plattform-global	See Prisma Schema Fields — extends ARCHITECTURE.md's Tender model with live-verified nullable-field realities (deadline/value frequently absent in OCDS export)
SCHEMA-02	Content-Hash-Änderungserkennung (Fristverlängerung, Aufhebung)	See Dedup & Change Detection — DÖE re-publishes a changed notice as a new noticeVersion under the same ocid; contentHash diff on re-poll triggers update path
</phase_requirements>

Summary

This research supersedes the "open verification point" flagged in .planning/research/SUMMARY.md, ARCHITECTURE.md, and PITFALLS.md: the DÖE OpenData API's Swagger UI was live-queried this session (its JS-rendered UI was bypassed by reading swagger-initializer.js directly, which reveals the real spec URL /documentation/api/opendata). The live OpenAPI 3.1 spec was fetched, and — critically — a real export ZIP for a real past day was downloaded and inspected (105 real German tender notices from 2026-07-20, in both OCDS and eForms-DE XML format). This is ground truth, not documentation inference.

The single most important finding, which changes the phase's architecture from what ARCHITECTURE.md assumed: the DÖE OpenData API is not a paginated/incremental REST feed. It is a daily batch-export API: one call returns a ZIP of all notice versions processed on a given calendar day (pubDay=YYYY-MM-DD) or month (pubMonth=YYYY-MM) — there is no since/cursor/page parameter of any kind, and the current day and any future day are rejected with HTTP 400 ("must lie in the past"). New data for "today" only becomes retrievable starting tomorrow. This means the TenderSourceAdapter.fetchTenders(config, since?: Date) signature sketched in ARCHITECTURE.md should track a day cursor (lastIngestedDay: Date, day-granularity), not an arbitrary timestamp — and the admin-configurable poll interval (D-04, default hourly, INGEST-06) is architecturally harmless to run hourly (idempotent day-cursor check: no-op if the next day isn't past yet) but will only ever produce new data once every ~24h for this source. This must be explicitly designed into the scheduler (cron tick frequency ≠ actual-fetch frequency for this specific source) and documented so a future contributor doesn't "fix" the apparent 23-hours-of-no-op as a bug.

Second critical finding: live inspection of the real OCDS export shows that tender.tenderPeriod.endDate (deadline) and tender.value (estimated value) are frequently null in the OCDS conversion — even for open ("tender"-tagged) notices — while the same information is present in free text inside tender.description/lots[].description, and, for at least one directly cross-checked notice, present as a proper structured field (TenderSubmissionDeadlinePeriod/EndDate) in the corresponding eForms-DE XML that OCDS silently drops. This directly affects SCHEMA-01 (deadline/value are supposed to be first-class normalized fields) and D-05 (90-day post-deadline retention needs a deadline value to compute from). The recommendation below is to parse eForms-DE XML as the primary structured-field source for deadline/value/procedure-type, using OCDS JSON primarily for the ocid (stable dedup key across notice versions) and party/organization resolution — a reversal of ARCHITECTURE.md's blanket "prefer OCDS" guidance, based on this session's live cross-check, not assumption.

Third finding: the export format is a ZIP archive (confirmed: real Zip archive data file, one JSON/XML file per notice inside). No ZIP-handling library was included in STACK.md — this is a genuine gap in the milestone-level stack research (it assumed pagination, not archive downloads). adm-zip is recommended below.

Fourth finding, directly informing D-02: live data confirms releases[].tag is the correct discriminator for "open tender" (tag: ["tender"], 74/105 = 70% of one real day's sample) vs. ["award"] (17/105) vs. ["planning"] (3/105) vs. no tag field at all (11/105 — untagged notices that still carry populated awards/contracts arrays, e.g. direct-award justification notices/"Vergabevermerke"). D-02's filter must therefore be: tag array contains "tender" and treat a missing tag conservatively (exclude, don't default to "open") since several of the untagged samples are in fact awarded/closed procurements.

Primary recommendation: Build the DoeOpenDataAdapter around the single real endpoint GET /api/notice-exports?pubDay=YYYY-MM-DD&format=eforms.zip (eForms-DE XML as primary parse target for deadline/value; OCDS JSON fetched in parallel or CSV as secondary cross-check per STACK.md's original recommendation), with a day-cursor scheduler (not a since-timestamp), adm-zip for archive extraction, and D-02 filtering on tag=["tender"] with awards/contracts emptiness as a secondary guard.

Architectural Responsibility Map

Capability	Primary Tier	Secondary Tier	Rationale
DÖE polling (HTTP fetch of daily ZIP)	API / Backend (TenderSourceAdapter)	—	Server-side scheduled job; no browser/CDN involvement
ZIP extraction + eForms-XML/OCDS parsing	API / Backend (TenderNormalizerService)	—	CPU/IO-bound parsing belongs in the Node process, not the DB or client
Dedup key + contentHash computation	API / Backend (TenderNormalizerService)	—	Pure computation on normalized fields before persistence
Tender persistence (global, no tenantId)	Database / Storage	—	Platform-wide reference data; RLS/tenant scoping deliberately does NOT apply here (see Pattern 3 in ARCHITECTURE.md)
Scheduler / cron lifecycle	API / Backend (TenderSchedulerService via SchedulerRegistry)	—	Server-only concern; no client involvement
Module registry self-seed + tenant activation gate	API / Backend (ModuleRegistryService, reused unmodified)	Database (Module/TenantModuleActivation tables)	Existing platform mechanism, not phase-specific
Admin poll-interval config UI	Frontend Server (Next.js settings/page.tsx)	API / Backend (TendersController admin routes)	Mirrors DKV's InboxConfigForm — form on the client, persisted via API
Marketplace listing/activation UI	Frontend Server (existing Marketplace page)	—	Unmodified platform feature; this phase only adds a Module row

Standard Stack

Core (new to this phase; extends STACK.md's v1.1 delta)

Library	Version	Purpose	Why Standard
fast-xml-parser	5.10.1 [CITED: STACK.md, npm registry 2026-07-17]	Parse eForms-DE XML notices (primary structured-field source per this session's live finding)	Already selected in milestone STACK.md for this exact purpose; zero-dependency, pure-TS, CJS-safe
adm-zip	0.6.0 [VERIFIED: npm registry, checked this session 2026-07-21]	Synchronous in-memory extraction of the DÖE daily export ZIP (typically tens of KB to a few MB — one day's sample was 161KB compressed / 105 files)	New gap identified this session — STACK.md did not anticipate the DÖE API returning an archive rather than a paginated feed. adm-zip is a simple, synchronous, CJS-native API (new AdmZip(buffer).getEntries()) well suited to fully-in-memory extraction of modest archives; MIT-licensed, 19.2M weekly downloads, actively maintained (github.com/cthackers/adm-zip). The automated package-legitimacy check seam flagged it SUS with reason "too-new" — manually confirmed as a false positive: the flagged date is the latest patch release (2026-07-10), not the package's origin; this is the same false-positive pattern STACK.md already documented for fast-xml-parser/csv-parse/playwright. [ASSUMED] provenance for the package-name choice itself (discovered via training knowledge, not an official doc) — planner should gate the pnpm add adm-zip step behind a checkpoint:human-verify per the package-legitimacy protocol.
csv-parse	7.0.1 [CITED: STACK.md]	DÖE CSV export (format=csv.zip) as a cross-check/fallback format	Unchanged from STACK.md; confirmed this session that csv.zip is a real, working format enum value in the live spec

Supporting

Library	Version	Purpose	When to Use
native fetch	Node 22 built-in	HTTP client for the DÖE endpoint	Already the established pattern (icon-discovery.service.ts, ics.provider.ts); DÖE requires no cookies/session state, so fetch-cookie/tough-cookie (needed for the scraping adapters in a later phase) are not needed for this phase

Alternatives Considered

Instead of	Could Use	Tradeoff
adm-zip (sync, in-memory)	unzipper (streaming) [VERIFIED: npm registry — 16.7M weekly downloads, same false-positive SUS flag]	Streaming extraction only pays off for very large archives; DÖE daily exports are small (low hundreds of files, sub-MB to low-MB compressed) — sync in-memory is simpler and matches the "wartbar/verständlich" project constraint
eForms-DE XML as primary parse target	OCDS JSON as primary parse target (ARCHITECTURE.md's original recommendation)	Reversed this session based on live cross-check: OCDS conversion measurably drops structured deadline data present in the source eForms-DE XML for at least one directly-verified notice. OCDS remains useful for ocid (dedup key) and resolved party/organization names, but should not be the sole source for deadline/value.

Installation:

pnpm --filter @tessera/api add fast-xml-parser adm-zip csv-parse

Version verification: confirmed live via npm view <pkg> version this session (2026-07-21): adm-zip@0.6.0 (published 2026-07-10), unzipper@0.12.5 (published 2026-06-21). fast-xml-parser/csv-parse versions carried over from STACK.md (2026-07-17, 4 days prior — still current).

Package Legitimacy Audit

Package	Registry	Age (latest release)	Downloads	Source Repo	Verdict	Disposition
adm-zip	npm	11 days (0.6.0, 2026-07-10)	19.2M/wk	github.com/cthackers/adm-zip	SUS (too-new — false positive, see rationale above)	Flagged — planner adds checkpoint:human-verify before pnpm add adm-zip, per protocol
fast-xml-parser	npm	(per STACK.md, 2026-07-17)	—	—	SUS (per STACK.md, already assessed false-positive)	Approved (carried over from milestone research, no re-flag needed)
csv-parse	npm	(per STACK.md, 2026-07-17)	—	—	SUS (per STACK.md, already assessed false-positive)	Approved (carried over)

Packages removed due to [SLOP] verdict: none. Packages flagged as suspicious [SUS]: adm-zip — planner must add a checkpoint:human-verify task before pnpm add adm-zip (spot-check the GitHub repo / Socket.dev score at implementation time, per the repo's existing dependency-hygiene bar already applied to the other v1.1 packages in STACK.md).

adm-zip was discovered via training knowledge (not an official doc), so per the package-name provenance rule it is tagged [ASSUMED] regardless of its clean npm registry / GitHub signals.

Architecture Patterns

System Architecture Diagram

[TenderSchedulerService — 1 global cron, e.g. hourly per D-04]
        |
        v
[DoeOpenDataAdapter.fetchTenders(config, dayCursor)]
        |  1. if dayCursor >= today(Europe/Berlin) -> no-op, return [] (nothing new yet)
        |  2. else: GET /api/notice-exports?pubDay={dayCursor}&format=eforms.zip  (+ ocds.zip for ocid/orgs)
        v
[adm-zip: extract ZIP -> N per-notice XML (+ JSON) buffers]
        v
[fast-xml-parser: parse each eForms-DE XML -> RawTenderRecord[]]
        v
[TenderNormalizerService.normalize()]
        |  - filter: keep only tag=["tender"] (D-02); drop award/planning/untagged-with-awards
        |  - map fields -> Tender (title, buyer, cpvCodes, region/plz, deadlineAt, estimatedValue, procedureType, sourceUrl)
        |  - dedupKey: ocid (from paired OCDS fetch) -> fallback sourcePortal:noticeId
        |  - contentHash: sha256(title+deadline+value+status)
        v
[TenderIngestionService.upsert()] -- prisma.tender.upsert({ where: { dedupKey } })
        |  advance dayCursor by 1 day; loop while dayCursor still < today (catch-up on missed days)
        v (only newly-created OR contentHash-changed rows this run)
[TenderMatchingService.matchDelta()]  -- Phase 11+ (saved searches don't exist yet in Phase 10)

Recommended Project Structure

(Extends ARCHITECTURE.md's apps/api/src/tenders/ layout — no changes needed, confirmed still correct after live verification.)

apps/api/src/tenders/
├── tenders.module.ts
├── tenders.controller.ts        # GET /tenders, GET /tenders/:id, admin source-config
├── tenders.seed.ts               # seeds 'tender-radar' into ModuleRegistry (CONFIG-01)
├── tender-ingestion.service.ts   # poll -> normalize -> upsert -> change-detect
├── tender-normalizer.service.ts  # eForms-XML+OCDS -> Tender fields, dedupKey, contentHash
├── tender-scheduler.service.ts   # SchedulerRegistry cron: 1 global job, day-cursor gated
├── tender.types.ts               # RawTenderRecord, NormalizedTenderFields, SourceType
├── dto/
│   ├── source-config.dto.ts      # admin: pollIntervalMin, isActive
│   └── tender-query.dto.ts       # pagination + filter query params (list/detail read, no saved-search yet)
└── adapters/
    ├── tender-source-adapter.interface.ts
    └── doe-opendata.adapter.ts    # THIS PHASE's only adapter

Pattern 1: DÖE API — Day-Batch, Not Paginated Feed (live-verified this session)

What: GET https://oeffentlichevergabe.de/api/notice-exports — the entire real API surface for OpenData bulk retrieval (confirmed: the live OpenAPI 3.1 spec at /documentation/api/opendata lists exactly one path). No auth (no security scheme in the spec; confirmed 200 with a plain unauthenticated curl).

Parameters (from the real spec, cross-verified against live requests):

• pubDay (string, YYYY-MM-DD) — returns all notice versions processed on that calendar day. Mutually exclusive with pubMonth.
• pubMonth (string, YYYY-MM) — same, for a whole month (useful only for a manual/admin-triggered historical import — not used in this phase per D-01).
• format (query param, or via Accept header): eforms.zip (default), ocds.zip, csv.zip.

Live-verified boundary rules (confirmed via actual 400 responses, not just spec text):

• pubDay = today → HTTP 400, message: "The specified pubDay exceeds the allowed range. It must lie in the past."
• pubDay = any future date → same 400.
• Earliest allowed: pubDay >= 2022-12-01, pubMonth >= 2022-12 (per spec text — the Bekanntmachungsservice's operational start).
• Response headers on a real 200: etag: "version-N", cache-control: max-age=120, content-disposition: attachment; filename=notices_YYYY-MM-DD_OCDS.zip. No rate-limit headers of any kind (X-RateLimit-* absent) — the etag/short cache-control instead suggest a caching layer in front of the origin; conditional If-None-Match requests are a viable politeness optimization but not required for correctness at day-granularity polling.

When to use: This is the only way to get DÖE data — there is no "list notices since X" or paged-results endpoint. Design the adapter around "fetch one day's full batch," not "fetch new items since last check."

Trade-offs: A day-cursor model means: (a) the very first poll after module activation on deployment day will return zero results until the next calendar day rolls over (today's data isn't retrievable yet) — this is expected API behavior, not a bug, and should be documented in the phase's UAT expectations; (b) catching up after downtime (e.g., scheduler was down for 3 days) is naturally supported — just loop pubDay from lastIngestedDay + 1 up to today - 1, each as a separate request.

Pattern 2: OCDS Release Tag as the D-02 "Open Tender" Filter (live-verified)

What: Live sample (105 real notices, 2026-07-20) breaks down as:

releases[].tag	Count	Meaning
["tender"]	74 (70%)	Open/active procurement notice — the only kind SCHEMA-01/D-02 wants
["award"]	17 (16%)	Award/result notice — explicitly out of scope (D-02)
["planning"]	3 (3%)	Prior-information/planning notice — not yet a bookable tender — exclude
(no tag field at all)	11 (11%)	Live-verified edge case — direct-award justification notices ("Vergabevermerke") with populated awards/contracts arrays but no tag. These are closed/awarded, not open.

Filter logic (recommended, not yet in ARCHITECTURE.md):

function isOpenTenderNotice(release: OcdsRelease): boolean {
  const tags = release.tag ?? [];
  if (tags.includes('tender')) return true;
  // Untagged notices with populated awards/contracts are NOT open — exclude.
  // Untagged notices with neither (rare, if any) should also be excluded conservatively
  // rather than defaulting to "open" — a false "open tender" shown to a bidder is worse
  // than a missed one (which the coverage-transparency UI in Phase 11 already communicates).
  return false;
}

Why this matters: A naive filter of "absence of award tag = open" would have wrongly included the 11 untagged-but-actually-closed notices found in this live sample (11% of one real day — not a negligible edge case).

Pattern 3: eForms-DE XML as Primary Structured-Field Source (live cross-check, corrects ARCHITECTURE.md)

What: ARCHITECTURE.md recommended "prefer OCDS... for direct field alignment." Live cross-check of the same real notice (25605686-1) in both formats shows:

• OCDS JSON: tender.tenderPeriod is absent (no deadline field at all); tender.value is absent; the deadline ("Angebotsfrist: Mittwoch, 26.08.2026, 10:00 Uhr") exists only as unstructured German prose inside tender.description.
• eForms-DE XML (same notice, same day, same export): contains a proper structured element — <TenderSubmissionDeadlinePeriod><ns3:EndDate>2026-08-26+02:00</ns3:EndDate></TenderSubmissionDeadlinePeriod> — and the document's CustomizationID (eforms-sdk-0.1) confirms this notice uses an old eForms SDK version (directly matching PITFALLS.md's Pitfall 17 — SDK version drift — now observed live, not just theorized).

Recommendation: Fetch both formats per poll cycle (eforms.zip for structured deadline/value/procedure fields; ocds.zip for the ocid dedup key and resolved buyer/party names) rather than relying on OCDS alone. This is two HTTP requests per day-cursor tick instead of one — negligible cost at day-granularity polling. If phase scope favors a single-format MVP, prefer eforms.zip as primary and treat ocid absence as acceptable for v1 (fall back to sourcePortal:noticeId dedup key, already the documented fallback tier in ARCHITECTURE.md's Pattern 2) — but fetching both is the more complete and only modestly more expensive option, and is the recommended default.

Trade-off: Two structurally different parsers now run per poll (fast-xml-parser for eForms-DE, plain JSON.parse for OCDS) — acceptable complexity, both already selected in STACK.md; no new parser library needed for this reversal, only a change in which one is authoritative for which fields.

Pattern 4: Nullable Deadline/Value as the Common Case, Not the Edge Case

What: Of the 12 "tender"-tagged notices spot-checked in this session's live sample, all 12 had tenderPeriod.endDate = null and value = null in the OCDS export. This is a stronger and more concrete version of PITFALLS.md's Pitfall 18 ("OCDS optional/conditional fields treated as always present") — live data suggests this is the majority case for Unterschwelle notices, not a rare edge case.

Recommendation for SCHEMA-01/D-05:

• deadlineAt and estimatedValue must be nullable in the Tender Prisma model (already planned in ARCHITECTURE.md's schema sketch — confirmed necessary, not optional hardening).
• D-05's "90 days after deadline, then delete" retention rule needs an explicit fallback for deadlineAt IS NULL rows: do not auto-expire them on the deadline-based path (there is no deadline to expire from); either (a) leave them permanently active/searchable (simplest, matches "keine Vor-Eingrenzung" spirit — a human should judge relevance, not the retention job), or (b) apply a secondary fallback expiry based on publishedAt + N days for deadline-less rows only. Recommend (a) for the MVP phase — introduce (b) later if the table grows unmanageably, since retrofitting a stricter policy is easy but a wrongly-deleted no-deadline tender is not recoverable.
• The "nur offene" (still-open) view/filter (Phase 11, FILTER-04) will only be meaningful for the subset with a populated deadlineAt — this is a coverage-transparency concern (PITFALLS.md Pitfall 20) that now has a concrete, live-measured magnitude (100% of a 12-notice spot sample) to inform the Phase 11 UI copy, not just an abstract "some tenders lack this."

Don't Hand-Roll

Problem	Don't Build	Use Instead	Why
ZIP extraction	A manual DEFLATE/central-directory parser	adm-zip	Well-understood, tiny surface, sync API fits the modest archive sizes involved
Multi-tenant scheduling	A second scheduling mechanism	SchedulerRegistry.addCronJob() (same as DkvSchedulerService)	Already proven, dynamically updatable at runtime — reuse the mechanics, just replace the findFirst()-single-tenant pattern, not the underlying cron library
Module registry / marketplace activation	A new activation table or gating mechanism	ModuleRegistryService (unmodified) + tenders.seed.ts (new, mirrors cert-manager.seed.ts)	Already the load-bearing platform mechanism for CONFIG-01; do not special-case Tenders
OCID/dedup key computation	A custom hashing scheme from scratch	Priority-ordered key per ARCHITECTURE.md Pattern 2 (ocid → sourcePortal:noticeId → fuzzy fingerprint)	Already researched and live-confirmed: ocid is present and stable across notice versions (ocds-mnwr74-{processId}, distinct from the per-version notice/release id)

Key insight: This phase's genuinely new risk is not "how do we build a scheduler" (solved, mechanics reused from DKV) — it's correctly interpreting the DÖE API's actual batch/day-cursor shape, which this session's live verification now settles definitively instead of leaving as an assumption.

Runtime State Inventory

Not applicable — this is a greenfield module addition (new Prisma models, new NestJS module, new module-registry row), not a rename/refactor/migration phase. No existing runtime state references the "Ausschreibungs-Radar"/"tender-radar" name that would need updating.

Common Pitfalls

Pitfall A: Treating the DÖE poll interval as "fetch frequency" instead of "check frequency"

What goes wrong: A naive implementation calls the DÖE endpoint every tick of the admin-configured interval (e.g., hourly per D-04), re-requesting the same day's ZIP 23 times before the day actually advances — wasted load, and worse, a developer "optimizing" this by shortening the interval further, or "fixing" the apparent no-op by re-implementing since-timestamp logic that the API doesn't actually support. Why it happens: D-04 says "poll interval, admin-configurable, default hourly" — this reads like a typical incremental-feed poll interval, but the DÖE API's actual granularity is daily. How to avoid: Decouple "cron tick frequency" (can stay hourly, satisfies D-04/INGEST-06 literally) from "actual upstream fetch" (gated by dayCursor < today(Europe/Berlin)). Document this decoupling explicitly in the adapter/scheduler code, not just in this research doc. Warning signs: Scheduler logs show identical pubDay values fetched repeatedly within the same 24h window with unchanged results — expected, not a bug, but worth a debug-level (not warn/error) log line so it doesn't look broken during on-call review.

Pitfall B: Assuming OCDS alone is sufficient for SCHEMA-01's deadline/value fields

What goes wrong: Building the normalizer against OCDS only (per ARCHITECTURE.md's original guidance) ships a Tender catalog where deadline/value are null for the majority of below-threshold notices, even though the source data (eForms-DE XML) actually had it. How to avoid: Parse eForms-DE XML as the primary field source per Pattern 3 above; treat OCDS as the ocid/party-resolution source, not the deadline/value source. Warning signs: QA/UAT notices most "tender"-tagged records showing no deadline in the results list — check whether the eForms-XML export was even fetched/parsed before assuming this is unavoidable API limitation (it partly is — not every eForms document will have a machine-readable deadline either, since it can be embedded in a portal-specific free-text description — but the eForms path recovers meaningfully more than OCDS alone, confirmed on at least one directly cross-checked notice this session).

Pitfall C: Filtering "open tenders" (D-02) by tag absence instead of tag presence

What goes wrong: Assuming "no award tag = open tender" wrongly includes untagged-but-closed notices (11% of the live sample). How to avoid: Filter positively on tag.includes('tender'); treat missing/other tags as excluded by default (Pattern 2 above). Warning signs: Award-justification/"Vergabevermerk" notices (identifiable by populated awards/contracts arrays with no submission deadline) appearing in the "open, biddable" result list.

Pitfall D: Copying DkvSchedulerService's findFirst() single-tenant pattern (carried over from PITFALLS.md Pitfall 24, restated for this phase specifically)

What goes wrong: Already extensively documented in PITFALLS.md — restated here because this phase's scheduler is the first place it must be avoided, and because the DÖE source is global (not per-tenant, unlike DKV), the "multi-tenant" framing is slightly different: there IS only one DÖE poll config (platform-wide, per Pattern 3 in ARCHITECTURE.md), so findFirst() on TenderSourcePollConfig for the DÖE row specifically is not the anti-pattern here (there's genuinely only one DÖE config row to find) — the anti-pattern would be if a future per-tenant source (e.g., email-alert in Phase 14) reused this same findFirst() call path. For Phase 10 specifically: it's safe and correct to load the single DÖE TenderSourcePollConfig row via findFirst() (or better, findUnique on a fixed known slug like sourceType: 'doe-opendata') since it is architecturally a singleton config — this is the one legitimate findFirst() in this phase; the acceptance-criterion "activating a 2nd tenant must not duplicate ingestion" is satisfied by the config being tenant-agnostic in the first place (Pattern 3), not by scheduler code that iterates tenants. Verification for this phase's two-tenant acceptance criterion (Success Criteria 4 & 5): the test is NOT "does the scheduler correctly loop over 2 tenant configs" (there are none for DÖE) — it is "does activating the module for a 2nd tenant (TenantModuleActivation row) trigger zero additional DÖE HTTP calls, zero additional cron jobs, and zero additional Tender rows" — i.e., prove the absence of tenant-count-scaled behavior, not the presence of correct per-tenant iteration.

Code Examples

DÖE fetch — real, verified request shape

// Source: live-verified this session against https://oeffentlichevergabe.de
// GET https://oeffentlichevergabe.de/api/notice-exports?pubDay=2026-07-20&format=eforms.zip
// -> 200, Content-Disposition: attachment; filename=notices_2026-07-20_eForms.zip
// -> binary ZIP body, ~105 files for a typical day (~160KB compressed for the OCDS variant)

async function fetchDoeDay(pubDay: string, format: 'eforms.zip' | 'ocds.zip' | 'csv.zip') {
  const url = `https://oeffentlichevergabe.de/api/notice-exports?pubDay=${pubDay}&format=${format}`;
  const res = await fetch(url);
  if (res.status === 400) {
    // "must lie in the past" — pubDay is today or future; nothing to fetch yet this cycle
    return null;
  }
  if (!res.ok) throw new Error(`DÖE fetch failed: ${res.status}`);
  return Buffer.from(await res.arrayBuffer());
}

Day-cursor gate (replaces a since: Date timestamp cursor)

// Source: this session's live finding — DÖE has no incremental/since parameter.
function nextDayToFetch(lastIngestedDay: Date | null): string | null {
  const berlinToday = new Date().toLocaleDateString('en-CA', { timeZone: 'Europe/Berlin' }); // YYYY-MM-DD
  const next = lastIngestedDay
    ? new Date(lastIngestedDay.getTime() + 86_400_000).toISOString().slice(0, 10)
    : new Date().toISOString().slice(0, 10); // D-01: "from now" — first eligible day is tomorrow relative to activation
  return next < berlinToday ? next : null; // null = nothing new yet, no-op this tick
}

D-02 open-tender filter (OCDS release tag)

// Source: live-verified against a real 105-notice sample, 2026-07-20
function isOpenTenderNotice(release: { tag?: string[]; awards?: unknown[]; contracts?: unknown[] }): boolean {
  if (release.tag?.includes('tender')) return true;
  return false; // conservative: missing tag + award/planning tags are all excluded
}

State of the Art

Old Approach (assumed pre-verification)	Current Approach (live-verified this session)	When Changed	Impact
DÖE = paginated REST feed with a since/cursor param (implicit assumption in ARCHITECTURE.md's adapter signature)	DÖE = day/month-batch ZIP export, no pagination param exists, current/future days rejected (400)	Confirmed 2026-07-21, this session	Scheduler must be day-cursor gated, not since-timestamp gated; "pagination handling" (a phase-focus research question) resolves to "there is no pagination — there is a per-day archive"
"Prefer OCDS over eForms-DE" (ARCHITECTURE.md Pattern 2, based on the OCDS spec's cleaner structure)	eForms-DE XML has structured deadline data that OCDS conversion drops, at least for some notices	Confirmed 2026-07-21, this session, via direct cross-check of one real notice in both formats	Normalizer should parse eForms-DE as primary for deadline/value; OCDS retained for ocid/party names

Deprecated/outdated: None — this is a young API (operational since 2023-02-06 per prior research); no deprecated version churn observed in the live spec.

Assumptions Log

#	Claim	Section	Risk if Wrong
A1	adm-zip is the right ZIP library choice (vs. unzipper/jszip/yauzl)	Standard Stack	Low — all are viable; adm-zip's sync API is a convenience choice, not a hard technical requirement. If archives grow much larger than observed (161KB/105 files for one day), a streaming approach (unzipper) may be preferable — reassess if daily archive size grows past a few MB.
A2	Fetching both eforms.zip and ocds.zip per poll cycle (two HTTP calls) is acceptable overhead vs. a single-format MVP	Pattern 3	Low — at day-granularity polling (at most 1-2 real fetches/day per the day-cursor gate), two requests instead of one is negligible; if the planner descopes to single-format for MVP speed, document that deadline/value coverage will be measurably worse (per Pattern 4's 100%-null spot check) as an explicit MVP trade-off, not a silent gap
A3	The 105-notice, 12-notice, and 11-notice samples (all from 2026-07-20) are representative of typical daily volume/tag distribution, not an anomalous day	Pattern 2, Pattern 4	Medium — a single day's sample is directional, not statistically proven; if actual production volume/tag ratios differ significantly, the D-02 filter logic itself is still correct (tag-based), only the magnitude estimates (70% tender / 16% award / etc.) may shift

Open Questions

1. Exact rate-limit tolerance of the DÖE API (undocumented)

   • What we know: no X-RateLimit-* headers observed on a real response; a cache-control: max-age=120 + etag suggest a caching/CDN layer that can absorb repeated identical requests cheaply.
   • What's unclear: whether rapid-fire requests (e.g., a catch-up loop fetching 30 missed days in quick succession after extended downtime) would trigger throttling — not tested this session to avoid abusing a public-good API mid-research.
   • Recommendation: implement the catch-up loop (Pattern 1) with a small delay between requests (e.g., 1-2s, matching the "politeDelay" helper already recommended in STACK.md for the scraping adapters) even though DÖE itself carries "null" ToS risk — cheap insurance, no downside.

2. Whether eForms-DE deadline coverage is meaningfully better than OCDS across a larger sample

   • What we know: confirmed better on 1 directly cross-checked notice (deadline present in XML, absent in OCDS JSON and OCDS proper structured field).
   • What's unclear: the aggregate improvement rate across hundreds/thousands of notices — this session's live check was necessarily a small spot-check, not a full-corpus audit.
   • Recommendation: the phase's implementation should log a metric (e.g., % of ingested tender-tagged notices with non-null deadlineAt) from day one, so this can be measured empirically in production rather than re-guessed.

Environment Availability

Dependency	Required By	Available	Version	Fallback
Outbound internet access from the deployment host to oeffentlichevergabe.de	INGEST-01 (DÖE polling)	✓ (verified from this dev/research environment — production host connectivity not independently verified this session)	—	If the production host lacks outbound HTTPS to this domain (e.g., restrictive egress firewall), the module cannot function at all — confirm with infra before phase execution if the API host is on the same network segment as the dev machine used for this research
Node.js fetch (native, Node 18+)	DÖE HTTP client	✓	Node 22-class per @types/node@^22 (existing project baseline)	—
adm-zip / equivalent ZIP library	ZIP extraction	✗ (not yet installed)	—	pnpm add adm-zip — no fallback needed, this is a build-time dependency, not an external service

Missing dependencies with no fallback: none beyond the standard "install the new npm package" step. Missing dependencies with fallback: none applicable.

Validation Architecture

Test Framework

Property	Value
Framework	Vitest 3.x (existing, per v1.0 STACK.md)
Config file	apps/api/vitest.config.ts (existing — confirm path during planning; not modified by this phase)
Quick run command	pnpm --filter @tessera/api test -- tenders (scoped to new test files)
Full suite command	pnpm --filter @tessera/api test

Phase Requirements → Test Map

Req ID	Behavior	Test Type	Automated Command	File Exists?
CONFIG-01	Module self-seeds into registry on boot; activatable per tenant	unit + integration	pnpm --filter @tessera/api test -- tenders.seed	❌ Wave 0
INGEST-01	DoeOpenDataAdapter parses a real (or fixture-captured) eForms-DE/OCDS ZIP into RawTenderRecord[]	unit	pnpm --filter @tessera/api test -- doe-opendata.adapter	❌ Wave 0 — fixture ZIP should be captured from this session's live download (doe-2026-07-20-eforms.zip/doe-2026-07-20-ocds.zip) rather than synthetic data, per this repo's stated preference for real fixtures over synthetic ones (matches the DKV PDF-parser precedent)
INGEST-06	Two-tenant scheduler test: activating module for a 2nd tenant produces zero additional DÖE HTTP calls / cron jobs / Tender rows	integration	pnpm --filter @tessera/api test -- tender-scheduler.service	❌ Wave 0
SCHEMA-01	Normalizer maps a real eForms-DE + OCDS notice pair into the expected Tender fields, including nullable deadline/value	unit	pnpm --filter @tessera/api test -- tender-normalizer.service	❌ Wave 0
SCHEMA-02	Re-ingesting an updated notice (same ocid, changed contentHash) updates the existing row instead of duplicating	integration	pnpm --filter @tessera/api test -- tender-ingestion.service	❌ Wave 0
D-02 filter	tag=["tender"] included; award/planning/untagged-with-awards excluded	unit	pnpm --filter @tessera/api test -- doe-opendata.adapter (same fixture, asserts filtered-out count)	❌ Wave 0

Sampling Rate

• Per task commit: pnpm --filter @tessera/api test -- tenders (scoped)
• Per wave merge: pnpm --filter @tessera/api test (full suite)
• Phase gate: Full suite green before /gsd-verify-work

Wave 0 Gaps

• apps/api/src/tenders/__fixtures__/doe-2026-07-20-eforms.zip and doe-2026-07-20-ocds.zip — real captured fixtures from this session's live download (recommend committing a trimmed version, e.g., 5-10 representative notices covering tender/award/planning/untagged tags, rather than the full 105-file archive, to keep the repo lean)
• apps/api/src/tenders/tenders.seed.spec.ts, doe-opendata.adapter.spec.ts, tender-normalizer.service.spec.ts, tender-ingestion.service.spec.ts, tender-scheduler.service.spec.ts — none exist yet, this is a greenfield module

Security Domain

Applicable ASVS Categories

ASVS Category	Applies	Standard Control
V2 Authentication	No	DÖE endpoint is auth-free by design (confirmed live); no credentials to protect for this specific integration
V3 Session Management	No	No session/cookie state involved in the DÖE adapter (unlike the later scraping adapters)
V4 Access Control	Yes	GET /tenders list/detail must be gated by TenantModuleActivation (module licensing) but NOT by tenantId row-filtering — this is the one controller where "tenant-gated" and "tenant-scoped" genuinely differ (per ARCHITECTURE.md's explicit callout); admin source-config routes (POST /tenders/source-config) must be @Roles(Role.ADMIN, Role.SUPER_ADMIN)-guarded like DkvController
V5 Input Validation	Yes	All parsed eForms-XML/OCDS-JSON fields are external, untrusted input — sanitize/escape before any future HTML rendering (Phase 11 UI); validate pubDay/pubMonth cursor values server-side before constructing the outbound DÖE URL (defense against cursor-corruption, low risk since these are internally computed, not user-supplied, but validate anyway)
V6 Cryptography	No	No credentials/secrets are stored for this phase (DÖE is auth-free) — CalendarCryptoService/AES-256-GCM pattern is reserved for later phases (email-alert inbox credentials, Phase 14)

Known Threat Patterns for this stack

Pattern	STRIDE	Standard Mitigation
Untrusted external content (buyer names, tender descriptions, CPV descriptions from eForms/OCDS) rendered unescaped in a future UI	Tampering / Injection	Sanitize/escape all ingested text fields before HTML interpolation — applies to Phase 11's UI, but the normalizer in this phase should not assume the data is "safe" just because it came from a government API; store raw, escape at render time
Zip-bomb / decompression-bomb via a maliciously large or deeply-nested ZIP response	Denial of Service	Low realistic risk given DÖE is a trusted government-operated source over HTTPS, but as defense-in-depth: adm-zip's getEntries() reads the central directory without fully decompressing — validate total uncompressed size against a sane ceiling (e.g., reject if entries.reduce(sum of entry.header.size) > 50MB) before extracting each entry's buffer
Cursor/date-parameter injection into the outbound DÖE URL	Tampering	The pubDay/pubMonth cursor is entirely internally computed from lastIngestedDay (never user-supplied) — no admin-facing input flows directly into this URL, so injection risk is structurally absent as designed; keep it that way (don't add an admin "manually trigger for date X" feature without validating the date format server-side first)

Sources

Primary (HIGH confidence)

• Live API verification, this session (2026-07-21): https://oeffentlichevergabe.de/documentation/swagger-ui/opendata/swagger-initializer.js (reveals real spec URL, bypassing the JS-rendered Swagger UI), https://oeffentlichevergabe.de/documentation/api/opendata (real OpenAPI 3.1 spec, fetched and parsed), https://oeffentlichevergabe.de/api/notice-exports?pubDay=2026-07-20&format=ocds.zip and ...&format=eforms.zip (real 200 responses, 105 real notices downloaded and inspected), boundary tests for pubDay=today and pubDay=future (both confirmed 400)
• npm view adm-zip version / npm view unzipper version / gsd-tools query package-legitimacy check — direct registry + legitimacy-seam verification, this session
• Existing codebase (direct read, this session): apps/api/src/dkv/dkv-scheduler.service.ts, apps/api/src/dkv/dkv.service.ts, apps/api/src/dkv/dkv.module.ts, apps/api/src/dkv/dkv.controller.ts, apps/api/src/dkv/dkv.seed.ts, apps/api/src/cert-manager/cert-manager.seed.ts, apps/api/src/module-registry/module-registry.service.ts, apps/api/src/prisma/prisma-tenant.extension.ts, apps/api/prisma/schema.prisma (DkvModuleConfig model), apps/web/src/lib/module-loader.ts, apps/web/src/app/(portal)/modules/dkv-fleet/settings/components/InboxConfigForm.tsx, apps/api/src/app.module.ts, apps/api/package.json

Secondary (MEDIUM confidence)

• .planning/research/SUMMARY.md, ARCHITECTURE.md, STACK.md, PITFALLS.md, ausschreibungs-portale-feasibility.md (milestone-level research, 2026-07-16/17) — used as the binding baseline this document extends/corrects where live verification diverged

Tertiary (LOW confidence)

• None — this session prioritized live verification over inference specifically because the milestone research had flagged the DÖE pagination/rate-limit question as an explicit open point.

Metadata

Confidence breakdown:

• DÖE API mechanics (endpoint, params, day-batch model, boundary rules): HIGH — directly observed via live requests this session, not inferred from documentation
• OCDS/eForms field completeness (deadline/value nullability, tag distribution): MEDIUM-HIGH — directly observed on a real sample, but the sample is one day (105 notices) and may not capture the full range of buyer/portal submission quality variance
• Platform integration (module registry, scheduler mechanics, Prisma conventions): HIGH — verified against existing, working codebase
• ZIP library choice (adm-zip): MEDIUM — sound technical choice, but package-name provenance is [ASSUMED] (training knowledge) per the package-legitimacy protocol; flagged for checkpoint:human-verify

Research date: 2026-07-21 Valid until: DÖE API mechanics — treat as stable for 90 days (young, actively-operated government API, no deprecation signals observed; re-verify if a phase re-plan happens much later). npm package versions — 30 days (fast-moving ecosystem norm).