diff --git a/.planning/REQUIREMENTS.md b/.planning/REQUIREMENTS.md index 07b221c..5142fcb 100644 --- a/.planning/REQUIREMENTS.md +++ b/.planning/REQUIREMENTS.md @@ -12,7 +12,7 @@ - [x] **INGEST-01**: Das System ruft Ausschreibungen zeitgesteuert über die DÖE OpenData-API (oeffentlichevergabe.de, eForms/OCDS, auth-frei) ab. - [x] **INGEST-02**: Das System importiert Ausschreibungen von Administration-Intelligence-NetServer-Portalen (lhs-vpbw, tender24, vergabe.landbw) über einen konfigurierbaren Adapter. -- [ ] **INGEST-03**: Das System importiert Ausschreibungen vom cosinex-Vergabemarktplatz (DTVP) über einen Adapter. +- [x] **INGEST-03**: Das System importiert Ausschreibungen vom cosinex-Vergabemarktplatz (DTVP) über einen Adapter. - [ ] **INGEST-04**: Das System importiert Ausschreibungen aus RSS-Feeds (subreport-elvis, service.bund.de). - [ ] **INGEST-05**: Das System liest Portal-Benachrichtigungs-E-Mails aus einem konfigurierten Postfach ein (nutzt bestehende DKV-Inbox-Infrastruktur) und extrahiert daraus Ausschreibungen. - [x] **INGEST-06**: Jede Quelle wird einmal zentral pro Zeitplan abgefragt (poll-once-fan-out-many), Intervall pro Quelle im Admin-Bereich konfigurierbar; das Ergebnis wird an alle passenden Mandanten-Suchprofile verteilt. @@ -97,7 +97,7 @@ | NOTIFY-03 | Phase 12 | Complete | | NOTIFY-04 | Phase 12 | Complete | | INGEST-02 | Phase 13 | Complete | -| INGEST-03 | Phase 13 | Pending | +| INGEST-03 | Phase 13 | Complete | | INGEST-07 | Phase 13 | Complete | | SCHEMA-03 | Phase 13 | Complete | | INGEST-04 | Phase 14 | Pending | diff --git a/.planning/ROADMAP.md b/.planning/ROADMAP.md index e730495..e070924 100644 --- a/.planning/ROADMAP.md +++ b/.planning/ROADMAP.md @@ -439,7 +439,7 @@ Plans: 3. A tender that appears via both DÖE and a scraping adapter shows up once in the results list (fuzzy fingerprint dedup on buyer+title+CPV+deadline+value), with links to all of its source portals -- dedup logic only activates once a second source is live 4. Attempting to register vergabe24 or aumass as a poll source is refused by the system itself (adapter registry denylist enforced in code), not just documented as forbidden -**Plans**: 5/6 plans executed +**Plans**: 6/6 plans executed Plans: @@ -447,7 +447,7 @@ Plans: - [x] 13-02-PLAN.md — SourceRegistry + harte Denylist-Gate (vergabe24/aumass) + Adapter-Interface-Generalisierung (INGEST-07) - [x] 13-03-PLAN.md — 3-Stufen-Dedup-Resolver + pollDueSources Fan-out (catch-per-source) + Modul-Wiring (SCHEMA-03) - [x] 13-04-PLAN.md — NetServer-Adapter (config-getrieben, 3 Portale) + Package-Legitimacy-Checkpoint (INGEST-02) -- [ ] 13-05-PLAN.md — cosinex/DTVP-Adapter (separat, best-effort) (INGEST-03) +- [x] 13-05-PLAN.md — cosinex/DTVP-Adapter (separat, best-effort) (INGEST-03) - [x] 13-06-PLAN.md — Read-Endpoint include sources[] + TenderDetail Multi-Source-Links (SCHEMA-03 Anzeige) ### Phase 14: RSS, Email-Alert Ingestion & Module Rollout @@ -486,5 +486,5 @@ Phases execute in numeric order: 1 -> 2 -> 3 -> 4 -> 5 -> 6 -> 7 -> 8 -> 9 -> 10 | 10. Ausschreibungs-Radar Foundation & DÖE Ingestion | 6/6 | Complete | 2026-07-21 | | 11. Filter Engine, Results UI & Saved Searches | 6/6 | In Progress| | | 12. Tender Notifications | 4/4 | In Progress| | -| 13. Scraping Adapters & Cross-Source Deduplication | 5/6 | In Progress| | +| 13. Scraping Adapters & Cross-Source Deduplication | 6/6 | In Progress| | | 14. RSS, Email-Alert Ingestion & Module Rollout | 0/TBD | Not started | - | diff --git a/.planning/STATE.md b/.planning/STATE.md index 35979c9..5d9bd24 100644 --- a/.planning/STATE.md +++ b/.planning/STATE.md @@ -4,16 +4,16 @@ milestone: v1.1 milestone_name: Ausschreibungs-Radar current_phase: 13 current_phase_name: scraping-adapters-cross-source-dedup -status: executing -stopped_at: Completed 13-04-PLAN.md -last_updated: "2026-07-23T07:15:59.330Z" +status: verifying +stopped_at: Completed 13-05-PLAN.md +last_updated: "2026-07-23T07:51:24.572Z" last_activity: 2026-07-23 last_activity_desc: Phase 13 execution started progress: total_phases: 13 - completed_phases: 11 + completed_phases: 12 total_plans: 62 - completed_plans: 60 + completed_plans: 61 --- # Project State @@ -29,10 +29,10 @@ See: .planning/PROJECT.md (updated 2026-07-17) Phase: 13 (scraping-adapters-cross-source-dedup) — EXECUTING Plan: 6 of 6 -Status: Ready to execute +Status: Phase complete — ready for verification Last activity: 2026-07-23 — Phase 13 execution started -Progress: [██████████] 97% +Progress: [██████████] 98% ## Performance Metrics @@ -95,6 +95,7 @@ Progress: [██████████] 97% | Phase 13 P03 | 45min | 3 tasks | 5 files | | Phase 13 P06 | 15min | 2 tasks | 5 files | | Phase 13 P04 | 55min | 3 tasks | 6 files | +| Phase 13 P05 | 40min | 2 tasks | 4 files | ## Accumulated Context @@ -203,6 +204,8 @@ Recent decisions affecting current work: - [Phase ?]: Fingerprint always computed on tender.create regardless of dedupActive, so DÖE-only tenders become fingerprint-matchable the instant a 2nd source activates without further backfill - [Phase ?]: 13-06: portalLabel()-Map bleibt lokal in TenderDetail.tsx (nicht geteilt) — einziger Consumer bisher; Fallback auf sourceUrl-Block bei fehlendem/leerem sources[]. - [Phase ?]: 13-04: cheerio (nicht node-html-parser) fuer NetServer-HTML-Parsing gewaehlt — Legitimacy-Gate flaggte node-html-parser als 'too-new' (Fehlmessung: Latest-Version-Datum statt Package-Alter); cheerio bestand das Gate sauber, Human-Approval eingeholt +- [Phase ?]: cosinex/DTVP-Selektoren voll befuellt statt needs-JS-deferred (D-01) — Trefferliste ist server-gerendert, live geprueft 2026-07-23 +- [Phase ?]: cosinex/DTVP liefert echte Notice-Deep-Links (pid=) — besser als NetServer-Fallback (Such-URL); Rule-1-Fix: arrayBuffer()+TextDecoder('iso-8859-1') statt res.text(), da Portal ISO-8859-1 mit rohen Latin-1-Bytes sendet ### Pending Todos @@ -240,7 +243,7 @@ Items acknowledged and carried forward from previous milestone close: ## Session Continuity -Last session: 2026-07-23T07:15:59.317Z -Stopped at: Completed 13-04-PLAN.md +Last session: 2026-07-23T07:51:24.556Z +Stopped at: Completed 13-05-PLAN.md Resume file: None Last activity: 2026-07-14 - Built LDAP per-user exclude/denylist filter (9d1323f), migration applied on live DB, verified via Playwright: sync deactivated 4 excluded service accounts (administrator/krbtgt/guest/dns-ldap), 2 real LDAP users stay active, 0 wrongly created diff --git a/.planning/phases/13-scraping-adapters-cross-source-dedup/13-05-SUMMARY.md b/.planning/phases/13-scraping-adapters-cross-source-dedup/13-05-SUMMARY.md new file mode 100644 index 0000000..5d32246 --- /dev/null +++ b/.planning/phases/13-scraping-adapters-cross-source-dedup/13-05-SUMMARY.md @@ -0,0 +1,176 @@ +--- +phase: 13-scraping-adapters-cross-source-dedup +plan: 05 +subsystem: backend +tags: [cheerio, html-scraping, adapter-pattern, cosinex, dtvp, encoding] + +requires: + - phase: 13-scraping-adapters-cross-source-dedup + plan: 01 + provides: SourceType open union (includes 'cosinex-dtvp'), RawTenderRecord contract + - phase: 13-scraping-adapters-cross-source-dedup + plan: 02 + provides: TenderSourceAdapter.portals[], SourceRegistry with DeniedPortalError denylist gate + - phase: 13-scraping-adapters-cross-source-dedup + plan: 04 + provides: cheerio dependency (already installed, package-legitimacy-approved), reference adapter pattern (fetch/parse/error-isolation/registration shape) +provides: + - "CosinexAdapter — a SEPARATE single-portal HTML adapter for the cosinex/DTVP Vergabemarktplatz satellite (sourceType='cosinex-dtvp'), structurally distinct from NetServerAdapter" + - "Live-captured cosinex-search.html fixture (20 rows, transcoded to UTF-8) for stable, network-free adapter tests" + - "CosinexAdapter registered as a TendersModule provider + at DI boot via SourceRegistry.register(); cosinex-dtvp TenderSourcePollConfig seeded (isActive: false)" +affects: [phase-14-admin-source-activation-ui] + +tech-stack: + added: [] + patterns: + - "Single-portal HTML adapter (contrast with NetServerAdapter's config-driven multi-portal shape): one class, one hardcoded base URL, one listing fetch+parse call — the fetch/parse/per-row-try-catch/registration shape repeats from 13-04, portal fan-out does not apply since cosinex/DTVP is one portal" + - "Explicit charset decoding via arrayBuffer() + TextDecoder(charset) instead of Response.text() — required whenever a source serves a non-UTF-8 charset (cosinex/DTVP: ISO-8859-1 with raw Latin-1 bytes, not HTML entities), since Response.text() always UTF-8-decodes per the WHATWG Fetch spec regardless of the declared Content-Type charset" + - "
-separated table-cell splitting via cell.contents().each() node-type walk (text vs.
vs. nested tag) rather than cell.html().split(/
/) — avoids re-parsing HTML fragments out of context and correctly text-extracts nested tags (e.g. an TNW procedure-type abbreviation)" + +key-files: + created: + - apps/api/src/tenders/adapters/cosinex.adapter.ts + - apps/api/src/tenders/adapters/cosinex.adapter.spec.ts + - apps/api/src/tenders/__fixtures__/cosinex-search.html + modified: + - apps/api/src/tenders/tenders.module.ts + +key-decisions: + - "Selectors fully populated, NOT deferred as 'needs-JS' — 13-05-PLAN.md's D-01 best-effort fallback anticipated a JS-dependent listing (Open Question 1), but live inspection (2026-07-23) found the 'Aktuelle Bekanntmachungen' results table (table.csx-new-table) is fully server-rendered HTML with real rows, real dates, and a real pid=-bearing detail link per row. The plan's contingency (adapter skeleton + [] fallback + 'needs-JS, deferred' documentation) was not needed; this is a strictly better outcome than the plan's worst case." + - "sourceUrl IS a genuine per-notice deep link (unlike NetServerAdapter's search-results-URL fallback, 13-04) — each row's 'Aktion' column links to /Satellite/public/company/projectForwarding.do?pid=, giving both a stable sourceNoticeId (the pid) and a real navigable sourceUrl. This resolves 13-RESEARCH.md's open question about cosinex deep-linkability in a strictly better way than NetServer's portal-only fallback." + - "Rule 1 bugfix: decode the fetch response via arrayBuffer()+TextDecoder('iso-8859-1'), not res.text() — cosinex/DTVP's live HTTP response header is 'Content-Type: text/html;charset=ISO-8859-1' and umlauts are raw Latin-1 bytes (0xE4/0xF6/0xFC/0xDF), not ä-style HTML entities (unlike NetServer). Per the WHATWG Fetch spec, Response.text() unconditionally UTF-8-decodes the body regardless of the declared charset, which would silently mojibake every umlaut buyer name/title. This was caught during fixture capture (grep/file treated the raw curl output as binary — a tell that it wasn't UTF-8) and fixed before any test was written, not discovered via a failing test." + - "On-disk fixture is transcoded to UTF-8 (iconv -f ISO-8859-1 -t UTF-8) rather than stored as raw Latin-1 bytes — keeps the spec's readFileSync(path, 'utf8') convention identical to netserver-search.html. The adapter's real ISO-8859-1 decode path is exercised separately by two dedicated tests that stub fetch's arrayBuffer() with Latin-1-encoded bytes (Buffer.from(html, 'latin1')), including a raw-umlaut-byte round-trip assertion." + - "dayCursor is accepted for interface conformance but not used as a query filter, same rationale as NetServerAdapter (13-04) — the cosinex listing has no documented incremental/date-range parameter; it is a 'most recent first' paginated table (page 1 of ~331 live, 2026-07-23). First-page-only fetch is accepted as sufficient for the MVP proof; pagination is out of scope for this plan." + - "cosinex-dtvp TenderSourcePollConfig seeded with isActive: false — this plan builds, registers, and tests the adapter; it does not force-activate live polling (D-02: activation is a later admin/seed decision, Phase 14 UI), matching the ai-netserver precedent." + +requirements-completed: [INGEST-03] + +coverage: + - id: D1 + description: "A SEPARATE CosinexAdapter (not a NetServerAdapter reuse) declares sourceType='cosinex-dtvp' and portals=['cosinex-dtvp'], parsing the cosinex/DTVP 'Aktuelle Bekanntmachungen' listing into RawTenderRecord[] with sourcePortal/sourceNoticeId/sourceUrl set correctly per row" + requirement: INGEST-03 + verification: + - kind: unit + ref: "apps/api/src/tenders/adapters/cosinex.adapter.spec.ts — 'declares sourceType cosinex-dtvp and the single cosinex-dtvp portal', 'parses all rows of the fixture into RawTenderRecord[]...' (16/16 tests pass)" + status: pass + human_judgment: false + - id: D2 + description: "Adapter parses the live-captured cosinex table into RawTenderRecord[] with correct publishedAt/deadlineAt Date-or-null mapping (including the 'nv'/nicht-vorhanden no-deadline case), nested- procedure-type extraction, and umlaut-correct buyer names — server-rendered, no JS-dependency fallback needed (D-01 resolved favorably)" + requirement: INGEST-03 + verification: + - kind: unit + ref: "apps/api/src/tenders/adapters/cosinex.adapter.spec.ts — 'parses a populated publishedAt and an nv deadline to null', 'parses a populated deadline row to a non-null ISO deadlineAt', 'extracts a nested- procedure-type abbreviation', 'decodes a real umlaut buyer name correctly (Münster AöR)'" + status: pass + human_judgment: false + - id: D3 + description: "Parse-Totalausfall (empty HTML, no matching table/rows, malformed row without a pid-bearing detail link, listing fetch throwing, listing responding non-2xx) returns [] instead of throwing (D-01 error tolerance)" + requirement: INGEST-03 + verification: + - kind: unit + ref: "apps/api/src/tenders/adapters/cosinex.adapter.spec.ts — 'returns [] for empty HTML', 'returns [] for HTML with no matching result table', 'skips a row without a pid-bearing detail link', 'returns [] when fetch throws', 'returns [] without throwing when the listing responds non-2xx'" + status: pass + human_judgment: false + - id: D4 + description: "CosinexAdapter correctly decodes the source's real ISO-8859-1 charset (arrayBuffer + TextDecoder, not Response.text()) rather than mojibaking umlaut content — a Rule 1 bugfix discovered before any test was written" + requirement: INGEST-03 + verification: + - kind: unit + ref: "apps/api/src/tenders/adapters/cosinex.adapter.spec.ts — 'fetches and parses the listing, decoding the real fixture bytes correctly', 'decodes raw Latin-1 umlaut bytes correctly (Rule 1 fix...)'" + status: pass + human_judgment: false + - id: D5 + description: "CosinexAdapter is a TendersModule provider and is registered with SourceRegistry at DI boot alongside DoeOpenDataAdapter/NetServerAdapter; the INGEST-07 denylist gate structurally applies (cosinex-dtvp is not denylisted, so registration succeeds); project-wide typecheck and the full tenders test slice remain green" + requirement: INGEST-03 + verification: + - kind: unit + ref: "cd apps/api && npx tsc --noEmit -p tsconfig.json (clean)" + status: pass + - kind: unit + ref: "cd apps/api && npx vitest run src/tenders (21 files, 221/221 pass, includes source-registry.spec.ts 6/6 unchanged and cosinex.adapter.spec.ts 16/16 new)" + status: pass + human_judgment: false + +duration: 40min +completed: 2026-07-23 +status: complete +--- + +# Phase 13 Plan 05: cosinex/DTVP-Adapter Summary + +**A SEPARATE, single-portal `CosinexAdapter` (cheerio-based) parses the cosinex/DTVP "Aktuelle Bekanntmachungen" listing into `RawTenderRecord[]` — the results table turned out to be fully server-rendered (not JS-dependent as the plan's D-01 fallback anticipated), yielding real per-notice deep links and requiring an explicit ISO-8859-1 charset-decode fix — the third real ingestion source completing INGEST-03 and the phase's `activePortalCount >= 2` dedup-observability precondition (already met by 13-04) with a full three-source set.** + +## Performance + +- **Duration:** ~40 min +- **Tasks:** 2/2 completed +- **Files modified:** 4 (3 created, 1 modified) + +## Accomplishments +- **cosinex/DTVP is server-rendered, not JS-dependent (D-01 resolved favorably)**: a live GET against `https://www.dtvp.de/Satellite/company/welcome.do` (HTTP 200, 40 KB, 2026-07-23) returned a fully populated `` with real `` result rows, real publish/deadline dates, and a real `pid=`-bearing per-notice deep link (`/Satellite/public/company/projectForwarding.do?pid=`) — better than NetServer's JS-only navigation (13-04), which had no per-row `` at all. No "needs-JS, deferred" stub was needed; all selectors are fully populated. +- `CosinexAdapter`: `sourceType='cosinex-dtvp'`, `portals=['cosinex-dtvp']`, hardcoded `COSINEX_BASE_URL` constant (SSRF guard, T-13-05-01), fetches the listing via native `fetch` + `AbortController` 15s timeout + `redirect: 'follow'`, parses with cheerio. +- **Rule 1 bugfix, caught before writing any test**: the live response header is `Content-Type: text/html;charset=ISO-8859-1` and umlauts are raw Latin-1 bytes (not `ä`-style entities like NetServer uses). Per the WHATWG Fetch spec, `Response.text()` always UTF-8-decodes regardless of the declared charset — using it here would have silently mojibaked every umlaut buyer name/title. Fixed by reading `arrayBuffer()` and decoding explicitly via `TextDecoder('iso-8859-1')`. +- Fault tolerance per D-01/Pitfall 2: try/catch around the whole listing parse (`cheerio.load()` wrapped, returns `[]` for a totally unparsable page), try/catch per row (skip + `logger.warn`, never aborts the rest), and the outer `fetchTenders()` returns `[]` on fetch-throw or non-2xx instead of propagating. +- Extracts: `publishedAt`/`deadlineAt` (from ``, with the deadline column's `nv` case correctly falling through to `null`), title, a `
`-split legal-framework/procedure-type pair (including nested-`` abbreviations like "TNW" — text-extracted, never raw HTML, V5), and buyer name. +- Live-captured fixture (`__fixtures__/cosinex-search.html`, 20 real rows, 2026-07-23) transcoded from ISO-8859-1 to UTF-8 on disk for a simple `readFileSync(..., 'utf8')` convention (matching `netserver-search.html`); the adapter's real ISO-8859-1 decode path is exercised separately via two dedicated tests that stub `fetch`'s `arrayBuffer()` with Latin-1-encoded bytes. +- 16 spec tests, all green: sourceType/portals declaration, full-fixture parse with correct `sourcePortal`/`sourceNoticeId`(pid)/`sourceUrl`(real deep link), `nv`-deadline-to-null and populated-deadline-to-ISO-date mapping, nested-`` procedure-type extraction, umlaut buyer-name decoding (both from the UTF-8 fixture and via the raw-Latin-1-byte round trip), empty/no-table-HTML `[]`-fallback, missing-pid-row skip, fetch-throw/non-2xx `[]`-fallback, no-axios guard, no-input-interpolated-URL guard. +- `CosinexAdapter` registered as a `TendersModule` provider and at DI boot via `SourceRegistry.register()` (denylist gate structurally applies — `cosinex-dtvp` is not denylisted); `cosinex-dtvp` `TenderSourcePollConfig` seeded with `isActive: false` (activation deferred to admin/Phase 14, per D-02). +- `npx tsc --noEmit` clean; `src/tenders` slice: 21 files, 221/221 tests pass (205 pre-existing + 16 new). + +## Task Commits + +Each task was committed atomically: + +1. **Task 1: cosinex/DTVP-Adapter + HTML-Fixture + Spec** - `12fc5ac` (feat) +2. **Task 2: cosinex-Adapter im Modul registrieren** - `6fe0dd0` (feat) + +## Files Created/Modified +- `apps/api/src/tenders/adapters/cosinex.adapter.ts` - single-portal `CosinexAdapter` (`fetchTenders`, `parseSearchResults`, `parseCosinexDateTime`/`splitByBr` helpers) +- `apps/api/src/tenders/adapters/cosinex.adapter.spec.ts` - 16 unit tests against the fixture + mocked fetch (including encoding-conversion tests) +- `apps/api/src/tenders/__fixtures__/cosinex-search.html` - live-captured cosinex/DTVP results table (20 rows, transcoded to UTF-8) +- `apps/api/src/tenders/tenders.module.ts` - `CosinexAdapter` provider + `onModuleInit` registration + `cosinex-dtvp` poll-config seed (`isActive: false`) + +## Decisions Made +- See `key-decisions` in frontmatter — most notably: selectors fully populated rather than deferred (D-01 resolved favorably by live inspection), genuine per-notice deep links (better than NetServer's fallback), and the ISO-8859-1 charset-decode Rule 1 fix. + +## Deviations from Plan + +### Auto-fixed Issues + +**1. [Rule 1 - Bug] Decode fetch response via arrayBuffer()+TextDecoder('iso-8859-1'), not Response.text()** +- **Found during:** Task 1 (fixture capture — `grep`/`file` treated the raw `curl` output as "ISO-8859 text", a tell that it wasn't UTF-8, before any adapter code was written) +- **Issue:** cosinex/DTVP serves `Content-Type: text/html;charset=ISO-8859-1` with raw Latin-1 bytes for umlauts. The WHATWG Fetch spec's `Response.text()` always UTF-8-decodes the body regardless of the declared Content-Type charset — using it (as `NetServerAdapter`'s pattern does, since NetServer's source happens to use HTML entities instead) would have silently mojibaked every umlaut buyer name and title. +- **Fix:** `fetchListingHtml()` reads `res.arrayBuffer()` and decodes explicitly with `new TextDecoder('iso-8859-1').decode(buffer)`. +- **Files modified:** `apps/api/src/tenders/adapters/cosinex.adapter.ts` +- **Verification:** Two dedicated spec tests — one exercising the real fixture's umlaut row via the mocked `arrayBuffer()` fetch path, one with a synthetic raw-Latin-1-byte round trip (`'Landkreis München'`) — both assert the correctly decoded string. +- **Committed in:** `12fc5ac` (Task 1 commit) + +--- + +**Total deviations:** 1 auto-fixed (1 Rule 1 bugfix). +**Impact on plan:** Necessary for correctness (any umlaut-bearing buyer name or title would otherwise be silently corrupted in production). No scope creep — same files as planned (`cosinex.adapter.ts`/`.spec.ts`, the fixture, `tenders.module.ts`). + +## Issues Encountered + +None beyond the Rule 1 encoding fix above. Live network access was available in this environment, so the fixture was captured genuinely live (matching the plan's "live-gecapturte HTML-Fixture" requirement) and the D-01 "needs-JS?" open question was resolved definitively by direct inspection rather than left as an assumption. + +## User Setup Required + +None. `cosinex-dtvp` remains `isActive: false` — no live polling will start until an admin explicitly activates it (Phase 14 UI) or a manual DB update is made. No test/prod server was touched (local dev API only, host-side Vitest); the one live fetch performed was the read-only fixture capture itself. + +## Next Phase Readiness +- Phase 13's three planned real adapters (DÖE, NetServer, cosinex/DTVP) are now all built, tested, and registered with `SourceRegistry` — `activePortalCount` can reach 3 once an admin activates `ai-netserver`/`cosinex-dtvp` (Phase 14 UI), giving the cross-source dedup core (13-03) its first genuinely multi-source live exercise. +- **Same normalizer follow-up flagged by 13-04 applies here too, and is explicitly out of this plan's scope** (only `cosinex.adapter.ts`/`.spec.ts`, the fixture, and `tenders.module.ts` were planned): `TenderNormalizerService` does not yet understand `CosinexAdapter`'s generic `ocdsPayload` field bag (`{title, buyerName, procedureType, legalFramework, deadlineAt}` — same shape convention as `NetServerAdapter`'s bag, differing field values). Extending the normalizer for non-DÖE-shaped payloads (ideally handling both the NetServer and cosinex bags together, since they already share a shape) is a natural next plan. +- Manual/live verification (13-VALIDATION.md) — an optional live poll against `dtvp.de` to confirm real records ingest end-to-end through `pollDueSources` — remains a manual step, not executed here (no `cosinex-dtvp` activation was performed; the adapter's live-fetch path was exercised for the fixture capture itself, confirming reachability and correct decoding, but not through the full poll/ingest pipeline). +- No blockers. This was the final plan of Phase 13. + +## Known Stubs + +- **`ocdsPayload` generic bag is not yet consumed by `TenderNormalizerService`** — identical boundary/rationale to 13-04-SUMMARY.md's Known Stubs entry, now also applying to `CosinexAdapter`. `activePortalCount >= 2` (the dedup-gate signal) is unaffected (it counts active poll configs, not correctly-normalized records), but real cross-source dedup value from NetServer/cosinex records will only materialize once the normalizer is extended AND the respective poll configs are activated. This is a deliberate, documented scope boundary — not a bug — matching both this plan's and 13-04's `files_modified` list exactly. + +--- +*Phase: 13-scraping-adapters-cross-source-dedup* +*Completed: 2026-07-23* + +## Self-Check: PASSED + +All created/modified files verified present on disk; both task commit hashes (12fc5ac, 6fe0dd0) verified in git log.