--- phase: 13-scraping-adapters-cross-source-dedup plan: 05 subsystem: backend tags: [cheerio, html-scraping, adapter-pattern, cosinex, dtvp, encoding] requires: - phase: 13-scraping-adapters-cross-source-dedup plan: 01 provides: SourceType open union (includes 'cosinex-dtvp'), RawTenderRecord contract - phase: 13-scraping-adapters-cross-source-dedup plan: 02 provides: TenderSourceAdapter.portals[], SourceRegistry with DeniedPortalError denylist gate - phase: 13-scraping-adapters-cross-source-dedup plan: 04 provides: cheerio dependency (already installed, package-legitimacy-approved), reference adapter pattern (fetch/parse/error-isolation/registration shape) provides: - "CosinexAdapter — a SEPARATE single-portal HTML adapter for the cosinex/DTVP Vergabemarktplatz satellite (sourceType='cosinex-dtvp'), structurally distinct from NetServerAdapter" - "Live-captured cosinex-search.html fixture (20 rows, transcoded to UTF-8) for stable, network-free adapter tests" - "CosinexAdapter registered as a TendersModule provider + at DI boot via SourceRegistry.register(); cosinex-dtvp TenderSourcePollConfig seeded (isActive: false)" affects: [phase-14-admin-source-activation-ui] tech-stack: added: [] patterns: - "Single-portal HTML adapter (contrast with NetServerAdapter's config-driven multi-portal shape): one class, one hardcoded base URL, one listing fetch+parse call — the fetch/parse/per-row-try-catch/registration shape repeats from 13-04, portal fan-out does not apply since cosinex/DTVP is one portal" - "Explicit charset decoding via arrayBuffer() + TextDecoder(charset) instead of Response.text() — required whenever a source serves a non-UTF-8 charset (cosinex/DTVP: ISO-8859-1 with raw Latin-1 bytes, not HTML entities), since Response.text() always UTF-8-decodes per the WHATWG Fetch spec regardless of the declared Content-Type charset" - "
-separated table-cell splitting via cell.contents().each() node-type walk (text vs.
vs. nested tag) rather than cell.html().split(/
/) — avoids re-parsing HTML fragments out of context and correctly text-extracts nested tags (e.g. an TNW procedure-type abbreviation)" key-files: created: - apps/api/src/tenders/adapters/cosinex.adapter.ts - apps/api/src/tenders/adapters/cosinex.adapter.spec.ts - apps/api/src/tenders/__fixtures__/cosinex-search.html modified: - apps/api/src/tenders/tenders.module.ts key-decisions: - "Selectors fully populated, NOT deferred as 'needs-JS' — 13-05-PLAN.md's D-01 best-effort fallback anticipated a JS-dependent listing (Open Question 1), but live inspection (2026-07-23) found the 'Aktuelle Bekanntmachungen' results table (table.csx-new-table) is fully server-rendered HTML with real rows, real dates, and a real pid=-bearing detail link per row. The plan's contingency (adapter skeleton + [] fallback + 'needs-JS, deferred' documentation) was not needed; this is a strictly better outcome than the plan's worst case." - "sourceUrl IS a genuine per-notice deep link (unlike NetServerAdapter's search-results-URL fallback, 13-04) — each row's 'Aktion' column links to /Satellite/public/company/projectForwarding.do?pid=, giving both a stable sourceNoticeId (the pid) and a real navigable sourceUrl. This resolves 13-RESEARCH.md's open question about cosinex deep-linkability in a strictly better way than NetServer's portal-only fallback." - "Rule 1 bugfix: decode the fetch response via arrayBuffer()+TextDecoder('iso-8859-1'), not res.text() — cosinex/DTVP's live HTTP response header is 'Content-Type: text/html;charset=ISO-8859-1' and umlauts are raw Latin-1 bytes (0xE4/0xF6/0xFC/0xDF), not ä-style HTML entities (unlike NetServer). Per the WHATWG Fetch spec, Response.text() unconditionally UTF-8-decodes the body regardless of the declared charset, which would silently mojibake every umlaut buyer name/title. This was caught during fixture capture (grep/file treated the raw curl output as binary — a tell that it wasn't UTF-8) and fixed before any test was written, not discovered via a failing test." - "On-disk fixture is transcoded to UTF-8 (iconv -f ISO-8859-1 -t UTF-8) rather than stored as raw Latin-1 bytes — keeps the spec's readFileSync(path, 'utf8') convention identical to netserver-search.html. The adapter's real ISO-8859-1 decode path is exercised separately by two dedicated tests that stub fetch's arrayBuffer() with Latin-1-encoded bytes (Buffer.from(html, 'latin1')), including a raw-umlaut-byte round-trip assertion." - "dayCursor is accepted for interface conformance but not used as a query filter, same rationale as NetServerAdapter (13-04) — the cosinex listing has no documented incremental/date-range parameter; it is a 'most recent first' paginated table (page 1 of ~331 live, 2026-07-23). First-page-only fetch is accepted as sufficient for the MVP proof; pagination is out of scope for this plan." - "cosinex-dtvp TenderSourcePollConfig seeded with isActive: false — this plan builds, registers, and tests the adapter; it does not force-activate live polling (D-02: activation is a later admin/seed decision, Phase 14 UI), matching the ai-netserver precedent." requirements-completed: [INGEST-03] coverage: - id: D1 description: "A SEPARATE CosinexAdapter (not a NetServerAdapter reuse) declares sourceType='cosinex-dtvp' and portals=['cosinex-dtvp'], parsing the cosinex/DTVP 'Aktuelle Bekanntmachungen' listing into RawTenderRecord[] with sourcePortal/sourceNoticeId/sourceUrl set correctly per row" requirement: INGEST-03 verification: - kind: unit ref: "apps/api/src/tenders/adapters/cosinex.adapter.spec.ts — 'declares sourceType cosinex-dtvp and the single cosinex-dtvp portal', 'parses all rows of the fixture into RawTenderRecord[]...' (16/16 tests pass)" status: pass human_judgment: false - id: D2 description: "Adapter parses the live-captured cosinex table into RawTenderRecord[] with correct publishedAt/deadlineAt Date-or-null mapping (including the 'nv'/nicht-vorhanden no-deadline case), nested- procedure-type extraction, and umlaut-correct buyer names — server-rendered, no JS-dependency fallback needed (D-01 resolved favorably)" requirement: INGEST-03 verification: - kind: unit ref: "apps/api/src/tenders/adapters/cosinex.adapter.spec.ts — 'parses a populated publishedAt and an nv deadline to null', 'parses a populated deadline row to a non-null ISO deadlineAt', 'extracts a nested- procedure-type abbreviation', 'decodes a real umlaut buyer name correctly (Münster AöR)'" status: pass human_judgment: false - id: D3 description: "Parse-Totalausfall (empty HTML, no matching table/rows, malformed row without a pid-bearing detail link, listing fetch throwing, listing responding non-2xx) returns [] instead of throwing (D-01 error tolerance)" requirement: INGEST-03 verification: - kind: unit ref: "apps/api/src/tenders/adapters/cosinex.adapter.spec.ts — 'returns [] for empty HTML', 'returns [] for HTML with no matching result table', 'skips a row without a pid-bearing detail link', 'returns [] when fetch throws', 'returns [] without throwing when the listing responds non-2xx'" status: pass human_judgment: false - id: D4 description: "CosinexAdapter correctly decodes the source's real ISO-8859-1 charset (arrayBuffer + TextDecoder, not Response.text()) rather than mojibaking umlaut content — a Rule 1 bugfix discovered before any test was written" requirement: INGEST-03 verification: - kind: unit ref: "apps/api/src/tenders/adapters/cosinex.adapter.spec.ts — 'fetches and parses the listing, decoding the real fixture bytes correctly', 'decodes raw Latin-1 umlaut bytes correctly (Rule 1 fix...)'" status: pass human_judgment: false - id: D5 description: "CosinexAdapter is a TendersModule provider and is registered with SourceRegistry at DI boot alongside DoeOpenDataAdapter/NetServerAdapter; the INGEST-07 denylist gate structurally applies (cosinex-dtvp is not denylisted, so registration succeeds); project-wide typecheck and the full tenders test slice remain green" requirement: INGEST-03 verification: - kind: unit ref: "cd apps/api && npx tsc --noEmit -p tsconfig.json (clean)" status: pass - kind: unit ref: "cd apps/api && npx vitest run src/tenders (21 files, 221/221 pass, includes source-registry.spec.ts 6/6 unchanged and cosinex.adapter.spec.ts 16/16 new)" status: pass human_judgment: false duration: 40min completed: 2026-07-23 status: complete --- # Phase 13 Plan 05: cosinex/DTVP-Adapter Summary **A SEPARATE, single-portal `CosinexAdapter` (cheerio-based) parses the cosinex/DTVP "Aktuelle Bekanntmachungen" listing into `RawTenderRecord[]` — the results table turned out to be fully server-rendered (not JS-dependent as the plan's D-01 fallback anticipated), yielding real per-notice deep links and requiring an explicit ISO-8859-1 charset-decode fix — the third real ingestion source completing INGEST-03 and the phase's `activePortalCount >= 2` dedup-observability precondition (already met by 13-04) with a full three-source set.** ## Performance - **Duration:** ~40 min - **Tasks:** 2/2 completed - **Files modified:** 4 (3 created, 1 modified) ## Accomplishments - **cosinex/DTVP is server-rendered, not JS-dependent (D-01 resolved favorably)**: a live GET against `https://www.dtvp.de/Satellite/company/welcome.do` (HTTP 200, 40 KB, 2026-07-23) returned a fully populated `` with real `` result rows, real publish/deadline dates, and a real `pid=`-bearing per-notice deep link (`/Satellite/public/company/projectForwarding.do?pid=`) — better than NetServer's JS-only navigation (13-04), which had no per-row `` at all. No "needs-JS, deferred" stub was needed; all selectors are fully populated. - `CosinexAdapter`: `sourceType='cosinex-dtvp'`, `portals=['cosinex-dtvp']`, hardcoded `COSINEX_BASE_URL` constant (SSRF guard, T-13-05-01), fetches the listing via native `fetch` + `AbortController` 15s timeout + `redirect: 'follow'`, parses with cheerio. - **Rule 1 bugfix, caught before writing any test**: the live response header is `Content-Type: text/html;charset=ISO-8859-1` and umlauts are raw Latin-1 bytes (not `ä`-style entities like NetServer uses). Per the WHATWG Fetch spec, `Response.text()` always UTF-8-decodes regardless of the declared charset — using it here would have silently mojibaked every umlaut buyer name/title. Fixed by reading `arrayBuffer()` and decoding explicitly via `TextDecoder('iso-8859-1')`. - Fault tolerance per D-01/Pitfall 2: try/catch around the whole listing parse (`cheerio.load()` wrapped, returns `[]` for a totally unparsable page), try/catch per row (skip + `logger.warn`, never aborts the rest), and the outer `fetchTenders()` returns `[]` on fetch-throw or non-2xx instead of propagating. - Extracts: `publishedAt`/`deadlineAt` (from ``, with the deadline column's `nv` case correctly falling through to `null`), title, a `
`-split legal-framework/procedure-type pair (including nested-`` abbreviations like "TNW" — text-extracted, never raw HTML, V5), and buyer name. - Live-captured fixture (`__fixtures__/cosinex-search.html`, 20 real rows, 2026-07-23) transcoded from ISO-8859-1 to UTF-8 on disk for a simple `readFileSync(..., 'utf8')` convention (matching `netserver-search.html`); the adapter's real ISO-8859-1 decode path is exercised separately via two dedicated tests that stub `fetch`'s `arrayBuffer()` with Latin-1-encoded bytes. - 16 spec tests, all green: sourceType/portals declaration, full-fixture parse with correct `sourcePortal`/`sourceNoticeId`(pid)/`sourceUrl`(real deep link), `nv`-deadline-to-null and populated-deadline-to-ISO-date mapping, nested-`` procedure-type extraction, umlaut buyer-name decoding (both from the UTF-8 fixture and via the raw-Latin-1-byte round trip), empty/no-table-HTML `[]`-fallback, missing-pid-row skip, fetch-throw/non-2xx `[]`-fallback, no-axios guard, no-input-interpolated-URL guard. - `CosinexAdapter` registered as a `TendersModule` provider and at DI boot via `SourceRegistry.register()` (denylist gate structurally applies — `cosinex-dtvp` is not denylisted); `cosinex-dtvp` `TenderSourcePollConfig` seeded with `isActive: false` (activation deferred to admin/Phase 14, per D-02). - `npx tsc --noEmit` clean; `src/tenders` slice: 21 files, 221/221 tests pass (205 pre-existing + 16 new). ## Task Commits Each task was committed atomically: 1. **Task 1: cosinex/DTVP-Adapter + HTML-Fixture + Spec** - `12fc5ac` (feat) 2. **Task 2: cosinex-Adapter im Modul registrieren** - `6fe0dd0` (feat) ## Files Created/Modified - `apps/api/src/tenders/adapters/cosinex.adapter.ts` - single-portal `CosinexAdapter` (`fetchTenders`, `parseSearchResults`, `parseCosinexDateTime`/`splitByBr` helpers) - `apps/api/src/tenders/adapters/cosinex.adapter.spec.ts` - 16 unit tests against the fixture + mocked fetch (including encoding-conversion tests) - `apps/api/src/tenders/__fixtures__/cosinex-search.html` - live-captured cosinex/DTVP results table (20 rows, transcoded to UTF-8) - `apps/api/src/tenders/tenders.module.ts` - `CosinexAdapter` provider + `onModuleInit` registration + `cosinex-dtvp` poll-config seed (`isActive: false`) ## Decisions Made - See `key-decisions` in frontmatter — most notably: selectors fully populated rather than deferred (D-01 resolved favorably by live inspection), genuine per-notice deep links (better than NetServer's fallback), and the ISO-8859-1 charset-decode Rule 1 fix. ## Deviations from Plan ### Auto-fixed Issues **1. [Rule 1 - Bug] Decode fetch response via arrayBuffer()+TextDecoder('iso-8859-1'), not Response.text()** - **Found during:** Task 1 (fixture capture — `grep`/`file` treated the raw `curl` output as "ISO-8859 text", a tell that it wasn't UTF-8, before any adapter code was written) - **Issue:** cosinex/DTVP serves `Content-Type: text/html;charset=ISO-8859-1` with raw Latin-1 bytes for umlauts. The WHATWG Fetch spec's `Response.text()` always UTF-8-decodes the body regardless of the declared Content-Type charset — using it (as `NetServerAdapter`'s pattern does, since NetServer's source happens to use HTML entities instead) would have silently mojibaked every umlaut buyer name and title. - **Fix:** `fetchListingHtml()` reads `res.arrayBuffer()` and decodes explicitly with `new TextDecoder('iso-8859-1').decode(buffer)`. - **Files modified:** `apps/api/src/tenders/adapters/cosinex.adapter.ts` - **Verification:** Two dedicated spec tests — one exercising the real fixture's umlaut row via the mocked `arrayBuffer()` fetch path, one with a synthetic raw-Latin-1-byte round trip (`'Landkreis München'`) — both assert the correctly decoded string. - **Committed in:** `12fc5ac` (Task 1 commit) --- **Total deviations:** 1 auto-fixed (1 Rule 1 bugfix). **Impact on plan:** Necessary for correctness (any umlaut-bearing buyer name or title would otherwise be silently corrupted in production). No scope creep — same files as planned (`cosinex.adapter.ts`/`.spec.ts`, the fixture, `tenders.module.ts`). ## Issues Encountered None beyond the Rule 1 encoding fix above. Live network access was available in this environment, so the fixture was captured genuinely live (matching the plan's "live-gecapturte HTML-Fixture" requirement) and the D-01 "needs-JS?" open question was resolved definitively by direct inspection rather than left as an assumption. ## User Setup Required None. `cosinex-dtvp` remains `isActive: false` — no live polling will start until an admin explicitly activates it (Phase 14 UI) or a manual DB update is made. No test/prod server was touched (local dev API only, host-side Vitest); the one live fetch performed was the read-only fixture capture itself. ## Next Phase Readiness - Phase 13's three planned real adapters (DÖE, NetServer, cosinex/DTVP) are now all built, tested, and registered with `SourceRegistry` — `activePortalCount` can reach 3 once an admin activates `ai-netserver`/`cosinex-dtvp` (Phase 14 UI), giving the cross-source dedup core (13-03) its first genuinely multi-source live exercise. - **Same normalizer follow-up flagged by 13-04 applies here too, and is explicitly out of this plan's scope** (only `cosinex.adapter.ts`/`.spec.ts`, the fixture, and `tenders.module.ts` were planned): `TenderNormalizerService` does not yet understand `CosinexAdapter`'s generic `ocdsPayload` field bag (`{title, buyerName, procedureType, legalFramework, deadlineAt}` — same shape convention as `NetServerAdapter`'s bag, differing field values). Extending the normalizer for non-DÖE-shaped payloads (ideally handling both the NetServer and cosinex bags together, since they already share a shape) is a natural next plan. - Manual/live verification (13-VALIDATION.md) — an optional live poll against `dtvp.de` to confirm real records ingest end-to-end through `pollDueSources` — remains a manual step, not executed here (no `cosinex-dtvp` activation was performed; the adapter's live-fetch path was exercised for the fixture capture itself, confirming reachability and correct decoding, but not through the full poll/ingest pipeline). - No blockers. This was the final plan of Phase 13. ## Known Stubs - **`ocdsPayload` generic bag is not yet consumed by `TenderNormalizerService`** — identical boundary/rationale to 13-04-SUMMARY.md's Known Stubs entry, now also applying to `CosinexAdapter`. `activePortalCount >= 2` (the dedup-gate signal) is unaffected (it counts active poll configs, not correctly-normalized records), but real cross-source dedup value from NetServer/cosinex records will only materialize once the normalizer is extended AND the respective poll configs are activated. This is a deliberate, documented scope boundary — not a bug — matching both this plan's and 13-04's `files_modified` list exactly. --- *Phase: 13-scraping-adapters-cross-source-dedup* *Completed: 2026-07-23* ## Self-Check: PASSED All created/modified files verified present on disk; both task commit hashes (12fc5ac, 6fe0dd0) verified in git log.