tessera-ctl/.planning/phases/13-scraping-adapters-cross-source-dedup/13-05-SUMMARY.md

phase, plan, subsystem, tags, requires, provides, affects, tech-stack, key-files, key-decisions, requirements-completed, coverage, duration, completed, status

phase	plan	subsystem	tags	requires	provides	affects	tech-stack	key-files	key-decisions	requirements-completed	coverage	duration	completed	status
13-scraping-adapters-cross-source-dedup	05	backend	cheerio html-scraping adapter-pattern cosinex dtvp encoding	phase plan provides 13-scraping-adapters-cross-source-dedup 01 SourceType open union (includes 'cosinex-dtvp'), RawTenderRecord contract phase plan provides 13-scraping-adapters-cross-source-dedup 02 TenderSourceAdapter.portals[], SourceRegistry with DeniedPortalError denylist gate phase plan provides 13-scraping-adapters-cross-source-dedup 04 cheerio dependency (already installed, package-legitimacy-approved), reference adapter pattern (fetch/parse/error-isolation/registration shape)	CosinexAdapter — a SEPARATE single-portal HTML adapter for the cosinex/DTVP Vergabemarktplatz satellite (sourceType='cosinex-dtvp'), structurally distinct from NetServerAdapter Live-captured cosinex-search.html fixture (20 rows, transcoded to UTF-8) for stable, network-free adapter tests CosinexAdapter registered as a TendersModule provider + at DI boot via SourceRegistry.register(); cosinex-dtvp TenderSourcePollConfig seeded (isActive: false)	phase-14-admin-source-activation-ui	added patterns Single-portal HTML adapter (contrast with NetServerAdapter's config-driven multi-portal shape): one class, one hardcoded base URL, one listing fetch+parse call — the fetch/parse/per-row-try-catch/registration shape repeats from 13-04, portal fan-out does not apply since cosinex/DTVP is one portal Explicit charset decoding via arrayBuffer() + TextDecoder(charset) instead of Response.text() — required whenever a source serves a non-UTF-8 charset (cosinex/DTVP: ISO-8859-1 with raw Latin-1 bytes, not HTML entities), since Response.text() always UTF-8-decodes per the WHATWG Fetch spec regardless of the declared Content-Type charset <br>-separated table-cell splitting via cell.contents().each() node-type walk (text vs. <br> vs. nested tag) rather than cell.html().split(/<br>/) — avoids re-parsing HTML fragments out of context and correctly text-extracts nested tags (e.g. an <abbr title=...>TNW</abbr> procedure-type abbreviation)	created modified apps/api/src/tenders/adapters/cosinex.adapter.ts apps/api/src/tenders/adapters/cosinex.adapter.spec.ts apps/api/src/tenders/__fixtures__/cosinex-search.html apps/api/src/tenders/tenders.module.ts	Selectors fully populated, NOT deferred as 'needs-JS' — 13-05-PLAN.md's D-01 best-effort fallback anticipated a JS-dependent listing (Open Question 1), but live inspection (2026-07-23) found the 'Aktuelle Bekanntmachungen' results table (table.csx-new-table) is fully server-rendered HTML with real <tr> rows, real dates, and a real pid=-bearing detail link per row. The plan's contingency (adapter skeleton + [] fallback + 'needs-JS, deferred' documentation) was not needed; this is a strictly better outcome than the plan's worst case. sourceUrl IS a genuine per-notice deep link (unlike NetServerAdapter's search-results-URL fallback, 13-04) — each row's 'Aktion' column links to /Satellite/public/company/projectForwarding.do?pid=<id>, giving both a stable sourceNoticeId (the pid) and a real navigable sourceUrl. This resolves 13-RESEARCH.md's open question about cosinex deep-linkability in a strictly better way than NetServer's portal-only fallback. Rule 1 bugfix: decode the fetch response via arrayBuffer()+TextDecoder('iso-8859-1'), not res.text() — cosinex/DTVP's live HTTP response header is 'Content-Type: text/html;charset=ISO-8859-1' and umlauts are raw Latin-1 bytes (0xE4/0xF6/0xFC/0xDF), not ä-style HTML entities (unlike NetServer). Per the WHATWG Fetch spec, Response.text() unconditionally UTF-8-decodes the body regardless of the declared charset, which would silently mojibake every umlaut buyer name/title. This was caught during fixture capture (grep/file treated the raw curl output as binary — a tell that it wasn't UTF-8) and fixed before any test was written, not discovered via a failing test. On-disk fixture is transcoded to UTF-8 (iconv -f ISO-8859-1 -t UTF-8) rather than stored as raw Latin-1 bytes — keeps the spec's readFileSync(path, 'utf8') convention identical to netserver-search.html. The adapter's real ISO-8859-1 decode path is exercised separately by two dedicated tests that stub fetch's arrayBuffer() with Latin-1-encoded bytes (Buffer.from(html, 'latin1')), including a raw-umlaut-byte round-trip assertion. dayCursor is accepted for interface conformance but not used as a query filter, same rationale as NetServerAdapter (13-04) — the cosinex listing has no documented incremental/date-range parameter; it is a 'most recent first' paginated table (page 1 of ~331 live, 2026-07-23). First-page-only fetch is accepted as sufficient for the MVP proof; pagination is out of scope for this plan. cosinex-dtvp TenderSourcePollConfig seeded with isActive: false — this plan builds, registers, and tests the adapter; it does not force-activate live polling (D-02: activation is a later admin/seed decision, Phase 14 UI), matching the ai-netserver precedent.	INGEST-03	id description requirement verification human_judgment D1 A SEPARATE CosinexAdapter (not a NetServerAdapter reuse) declares sourceType='cosinex-dtvp' and portals=['cosinex-dtvp'], parsing the cosinex/DTVP 'Aktuelle Bekanntmachungen' listing into RawTenderRecord[] with sourcePortal/sourceNoticeId/sourceUrl set correctly per row INGEST-03 kind ref status unit apps/api/src/tenders/adapters/cosinex.adapter.spec.ts — 'declares sourceType cosinex-dtvp and the single cosinex-dtvp portal', 'parses all rows of the fixture into RawTenderRecord[]...' (16/16 tests pass) pass false id description requirement verification human_judgment D2 Adapter parses the live-captured cosinex table into RawTenderRecord[] with correct publishedAt/deadlineAt Date-or-null mapping (including the 'nv'/nicht-vorhanden no-deadline case), nested-<abbr> procedure-type extraction, and umlaut-correct buyer names — server-rendered, no JS-dependency fallback needed (D-01 resolved favorably) INGEST-03 kind ref status unit apps/api/src/tenders/adapters/cosinex.adapter.spec.ts — 'parses a populated publishedAt and an nv deadline to null', 'parses a populated deadline row to a non-null ISO deadlineAt', 'extracts a nested-<abbr> procedure-type abbreviation', 'decodes a real umlaut buyer name correctly (Münster AöR)' pass false id description requirement verification human_judgment D3 Parse-Totalausfall (empty HTML, no matching table/rows, malformed row without a pid-bearing detail link, listing fetch throwing, listing responding non-2xx) returns [] instead of throwing (D-01 error tolerance) INGEST-03 kind ref status unit apps/api/src/tenders/adapters/cosinex.adapter.spec.ts — 'returns [] for empty HTML', 'returns [] for HTML with no matching result table', 'skips a row without a pid-bearing detail link', 'returns [] when fetch throws', 'returns [] without throwing when the listing responds non-2xx' pass false id description requirement verification human_judgment D4 CosinexAdapter correctly decodes the source's real ISO-8859-1 charset (arrayBuffer + TextDecoder, not Response.text()) rather than mojibaking umlaut content — a Rule 1 bugfix discovered before any test was written INGEST-03 kind ref status unit apps/api/src/tenders/adapters/cosinex.adapter.spec.ts — 'fetches and parses the listing, decoding the real fixture bytes correctly', 'decodes raw Latin-1 umlaut bytes correctly (Rule 1 fix...)' pass false id description requirement verification human_judgment D5 CosinexAdapter is a TendersModule provider and is registered with SourceRegistry at DI boot alongside DoeOpenDataAdapter/NetServerAdapter; the INGEST-07 denylist gate structurally applies (cosinex-dtvp is not denylisted, so registration succeeds); project-wide typecheck and the full tenders test slice remain green INGEST-03 kind ref status unit cd apps/api && npx tsc --noEmit -p tsconfig.json (clean) pass kind ref status unit cd apps/api && npx vitest run src/tenders (21 files, 221/221 pass, includes source-registry.spec.ts 6/6 unchanged and cosinex.adapter.spec.ts 16/16 new) pass false	40min	2026-07-23	complete

Phase 13 Plan 05: cosinex/DTVP-Adapter Summary

A SEPARATE, single-portal CosinexAdapter (cheerio-based) parses the cosinex/DTVP "Aktuelle Bekanntmachungen" listing into RawTenderRecord[] — the results table turned out to be fully server-rendered (not JS-dependent as the plan's D-01 fallback anticipated), yielding real per-notice deep links and requiring an explicit ISO-8859-1 charset-decode fix — the third real ingestion source completing INGEST-03 and the phase's activePortalCount >= 2 dedup-observability precondition (already met by 13-04) with a full three-source set.

Performance

• Duration: ~40 min
• Tasks: 2/2 completed
• Files modified: 4 (3 created, 1 modified)

Accomplishments

• cosinex/DTVP is server-rendered, not JS-dependent (D-01 resolved favorably): a live GET against https://www.dtvp.de/Satellite/company/welcome.do (HTTP 200, 40 KB, 2026-07-23) returned a fully populated <table class="csx-new-table"> with real <tr> result rows, real publish/deadline dates, and a real pid=-bearing per-notice deep link (/Satellite/public/company/projectForwarding.do?pid=<id>) — better than NetServer's JS-only navigation (13-04), which had no per-row <a href> at all. No "needs-JS, deferred" stub was needed; all selectors are fully populated.
• CosinexAdapter: sourceType='cosinex-dtvp', portals=['cosinex-dtvp'], hardcoded COSINEX_BASE_URL constant (SSRF guard, T-13-05-01), fetches the listing via native fetch + AbortController 15s timeout + redirect: 'follow', parses with cheerio.
• Rule 1 bugfix, caught before writing any test: the live response header is Content-Type: text/html;charset=ISO-8859-1 and umlauts are raw Latin-1 bytes (not ä-style entities like NetServer uses). Per the WHATWG Fetch spec, Response.text() always UTF-8-decodes regardless of the declared charset — using it here would have silently mojibaked every umlaut buyer name/title. Fixed by reading arrayBuffer() and decoding explicitly via TextDecoder('iso-8859-1').
• Fault tolerance per D-01/Pitfall 2: try/catch around the whole listing parse (cheerio.load() wrapped, returns [] for a totally unparsable page), try/catch per row (skip + logger.warn, never aborts the rest), and the outer fetchTenders() returns [] on fetch-throw or non-2xx instead of propagating.
• Extracts: publishedAt/deadlineAt (from <abbr title="dd.MM.yyyy um HH:mm Uhr">, with the deadline column's <abbr title="...nicht vorhanden...">nv</abbr> case correctly falling through to null), title, a <br>-split legal-framework/procedure-type pair (including nested-<abbr> abbreviations like "TNW" — text-extracted, never raw HTML, V5), and buyer name.
• Live-captured fixture (__fixtures__/cosinex-search.html, 20 real rows, 2026-07-23) transcoded from ISO-8859-1 to UTF-8 on disk for a simple readFileSync(..., 'utf8') convention (matching netserver-search.html); the adapter's real ISO-8859-1 decode path is exercised separately via two dedicated tests that stub fetch's arrayBuffer() with Latin-1-encoded bytes.
• 16 spec tests, all green: sourceType/portals declaration, full-fixture parse with correct sourcePortal/sourceNoticeId(pid)/sourceUrl(real deep link), nv-deadline-to-null and populated-deadline-to-ISO-date mapping, nested-<abbr> procedure-type extraction, umlaut buyer-name decoding (both from the UTF-8 fixture and via the raw-Latin-1-byte round trip), empty/no-table-HTML []-fallback, missing-pid-row skip, fetch-throw/non-2xx []-fallback, no-axios guard, no-input-interpolated-URL guard.
• CosinexAdapter registered as a TendersModule provider and at DI boot via SourceRegistry.register() (denylist gate structurally applies — cosinex-dtvp is not denylisted); cosinex-dtvp TenderSourcePollConfig seeded with isActive: false (activation deferred to admin/Phase 14, per D-02).
• npx tsc --noEmit clean; src/tenders slice: 21 files, 221/221 tests pass (205 pre-existing + 16 new).

Task Commits

Each task was committed atomically:

1. Task 1: cosinex/DTVP-Adapter + HTML-Fixture + Spec - 12fc5ac (feat)
2. Task 2: cosinex-Adapter im Modul registrieren - 6fe0dd0 (feat)

Files Created/Modified

• apps/api/src/tenders/adapters/cosinex.adapter.ts - single-portal CosinexAdapter (fetchTenders, parseSearchResults, parseCosinexDateTime/splitByBr helpers)
• apps/api/src/tenders/adapters/cosinex.adapter.spec.ts - 16 unit tests against the fixture + mocked fetch (including encoding-conversion tests)
• apps/api/src/tenders/__fixtures__/cosinex-search.html - live-captured cosinex/DTVP results table (20 rows, transcoded to UTF-8)
• apps/api/src/tenders/tenders.module.ts - CosinexAdapter provider + onModuleInit registration + cosinex-dtvp poll-config seed (isActive: false)

Decisions Made

• See key-decisions in frontmatter — most notably: selectors fully populated rather than deferred (D-01 resolved favorably by live inspection), genuine per-notice deep links (better than NetServer's fallback), and the ISO-8859-1 charset-decode Rule 1 fix.

Deviations from Plan

Auto-fixed Issues

1. [Rule 1 - Bug] Decode fetch response via arrayBuffer()+TextDecoder('iso-8859-1'), not Response.text()

• Found during: Task 1 (fixture capture — grep/file treated the raw curl output as "ISO-8859 text", a tell that it wasn't UTF-8, before any adapter code was written)
• Issue: cosinex/DTVP serves Content-Type: text/html;charset=ISO-8859-1 with raw Latin-1 bytes for umlauts. The WHATWG Fetch spec's Response.text() always UTF-8-decodes the body regardless of the declared Content-Type charset — using it (as NetServerAdapter's pattern does, since NetServer's source happens to use HTML entities instead) would have silently mojibaked every umlaut buyer name and title.
• Fix: fetchListingHtml() reads res.arrayBuffer() and decodes explicitly with new TextDecoder('iso-8859-1').decode(buffer).
• Files modified: apps/api/src/tenders/adapters/cosinex.adapter.ts
• Verification: Two dedicated spec tests — one exercising the real fixture's umlaut row via the mocked arrayBuffer() fetch path, one with a synthetic raw-Latin-1-byte round trip ('Landkreis München') — both assert the correctly decoded string.
• Committed in: 12fc5ac (Task 1 commit)

---

Total deviations: 1 auto-fixed (1 Rule 1 bugfix). Impact on plan: Necessary for correctness (any umlaut-bearing buyer name or title would otherwise be silently corrupted in production). No scope creep — same files as planned (cosinex.adapter.ts/.spec.ts, the fixture, tenders.module.ts).

Issues Encountered

None beyond the Rule 1 encoding fix above. Live network access was available in this environment, so the fixture was captured genuinely live (matching the plan's "live-gecapturte HTML-Fixture" requirement) and the D-01 "needs-JS?" open question was resolved definitively by direct inspection rather than left as an assumption.

User Setup Required

None. cosinex-dtvp remains isActive: false — no live polling will start until an admin explicitly activates it (Phase 14 UI) or a manual DB update is made. No test/prod server was touched (local dev API only, host-side Vitest); the one live fetch performed was the read-only fixture capture itself.

Next Phase Readiness

• Phase 13's three planned real adapters (DÖE, NetServer, cosinex/DTVP) are now all built, tested, and registered with SourceRegistry — activePortalCount can reach 3 once an admin activates ai-netserver/cosinex-dtvp (Phase 14 UI), giving the cross-source dedup core (13-03) its first genuinely multi-source live exercise.
• Same normalizer follow-up flagged by 13-04 applies here too, and is explicitly out of this plan's scope (only cosinex.adapter.ts/.spec.ts, the fixture, and tenders.module.ts were planned): TenderNormalizerService does not yet understand CosinexAdapter's generic ocdsPayload field bag ({title, buyerName, procedureType, legalFramework, deadlineAt} — same shape convention as NetServerAdapter's bag, differing field values). Extending the normalizer for non-DÖE-shaped payloads (ideally handling both the NetServer and cosinex bags together, since they already share a shape) is a natural next plan.
• Manual/live verification (13-VALIDATION.md) — an optional live poll against dtvp.de to confirm real records ingest end-to-end through pollDueSources — remains a manual step, not executed here (no cosinex-dtvp activation was performed; the adapter's live-fetch path was exercised for the fixture capture itself, confirming reachability and correct decoding, but not through the full poll/ingest pipeline).
• No blockers. This was the final plan of Phase 13.

Known Stubs

• ocdsPayload generic bag is not yet consumed by TenderNormalizerService — identical boundary/rationale to 13-04-SUMMARY.md's Known Stubs entry, now also applying to CosinexAdapter. activePortalCount >= 2 (the dedup-gate signal) is unaffected (it counts active poll configs, not correctly-normalized records), but real cross-source dedup value from NetServer/cosinex records will only materialize once the normalizer is extended AND the respective poll configs are activated. This is a deliberate, documented scope boundary — not a bug — matching both this plan's and 13-04's files_modified list exactly.

---

Phase: 13-scraping-adapters-cross-source-dedup Completed: 2026-07-23

Self-Check: PASSED

All created/modified files verified present on disk; both task commit hashes (12fc5ac, 6fe0dd0) verified in git log.