docs(13): phase verification — gaps_found (normalizer stub blocks criteria 1/2/3)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

.planning/phases/13-scraping-adapters-cross-source-dedup/13-VERIFICATION.md

@@ -0,0 +1,174 @@
+---
+phase: 13-scraping-adapters-cross-source-dedup
+verified: 2026-07-23T10:15:00Z
+status: gaps_found
+score: 1/4 must-haves verified
+behavior_unverified: 0
+overrides_applied: 0
+gaps:
+  - truth: "Tenders from AI-AG NetServer portals (lhs-vpbw, tender24, vergabe.landbw) appear as additional, correctly normalized results via one config-driven adapter"
+    status: failed
+    reason: >
+      NetServerAdapter correctly parses NetServer HTML into RawTenderRecord[] (14/14 unit tests
+      pass), but the shared TenderNormalizerService.normalize() only understands the DÖE
+      eForms-DE/OCDS payload shape (getOcdsTender()/getOcdsBuyer() look for `.tender`/`.buyer`
+      keys). NetServerAdapter deliberately puts a FLAT generic bag
+      `{title, buyerName, procedureType, legalFramework, deadlineAt}` into `ocdsPayload` (own code
+      comment: "NetServer has no eForms/OCDS structure ... a future normalizer extension
+      (out of scope for this plan) can map them"). Because the bag has no `.tender`/`.buyer`
+      keys, every NetServer record would normalize to title='Unbenannte Ausschreibung',
+      buyerName=null, cpvCodes=[], deadlineAt=null, estimatedValue=null — i.e. NOT
+      "correctly normalized". This is explicitly self-documented as "Known Stubs" in
+      13-04-SUMMARY.md, not resolved anywhere in Phase 13, and not scheduled in Phase 14
+      (Phase 14 scope per ROADMAP is RSS/email/admin-UI/i18n, no normalizer work).
+    artifacts:
+      - path: "apps/api/src/tenders/tender-normalizer.service.ts"
+        issue: "getOcdsTender()/getOcdsBuyer() only read `.tender`/`.buyer` keys; has no branch for the generic ocdsPayload bag NetServerAdapter/CosinexAdapter produce"
+      - path: "apps/api/src/tenders/adapters/netserver.adapter.ts"
+        issue: "Puts {title, buyerName, procedureType, legalFramework, deadlineAt} directly into ocdsPayload (own comment marks this as needing a future normalizer extension)"
+    missing:
+      - "Extend TenderNormalizerService (or add a per-sourceType normalization strategy) so ai-netserver's generic ocdsPayload bag maps to real title/buyerName/deadlineAt/procedureType instead of falling through to defaults"
+  - truth: "Tenders from the cosinex Vergabemarktplatz (DTVP) appear as additional, correctly normalized results via a separate config-driven adapter"
+    status: failed
+    reason: >
+      Identical root cause to the NetServer gap above. CosinexAdapter correctly parses the
+      live cosinex/DTVP results table into RawTenderRecord[] (16/16 unit tests pass, including
+      a real ISO-8859-1 charset-decode fix), but puts the same generic
+      {title, buyerName, procedureType, legalFramework, deadlineAt} bag into ocdsPayload.
+      TenderNormalizerService does not understand this shape either, so cosinex records would
+      normalize incorrectly for the same reason. Self-documented as "Known Stubs" in
+      13-05-SUMMARY.md, not resolved in this phase, not scheduled in Phase 14.
+    artifacts:
+      - path: "apps/api/src/tenders/adapters/cosinex.adapter.ts"
+        issue: "Same generic ocdsPayload bag convention as NetServerAdapter — normalizer cannot map it"
+    missing:
+      - "Same normalizer extension as the NetServer gap, ideally handling both bag-shaped sources together"
+  - truth: "A tender that appears via both DÖE and a scraping adapter shows up once in the results list (fuzzy fingerprint dedup on buyer+title+CPV+deadline+value), with links to all of its source portals -- dedup logic only activates once a second source is live"
+    status: partial
+    reason: >
+      The DEDUP MECHANISM itself is solidly built and unit-tested: TenderDedupService's
+      three-tier resolver (OCID -> source:noticeId -> fingerprint), the dedupActive=(activePortalCount>=2)
+      gate, pollDueSources' findMany fan-out with catch-per-source isolation, the 2851-row
+      backfill, and the TenderDetail multi-source-link UI are all present, wired, and covered by
+      passing tests (tender-dedup.service.spec.ts 8/8, tender-ingestion.service.spec.ts fan-out
+      tests, tenders.controller.spec.ts, TenderDetail.test.tsx). HOWEVER, the truth as stated
+      ("a tender that appears via both DÖE and a scraping adapter shows up once ... fuzzy
+      fingerprint dedup on buyer+title+CPV+deadline+value") is undermined transitively by the two
+      gaps above: because NetServer/cosinex records normalize with a generic/empty
+      title+buyer+deadline ('Unbenannte Ausschreibung', null buyer, null deadline), their computed
+      fingerprint will NOT match the fingerprint of the corresponding real DÖE tender (which has
+      the actual title/buyer/deadline). So even once a second source is activated, real-world
+      DÖE<->NetServer or DÖE<->cosinex duplicates will NOT be recognized as the same tender —
+      the fingerprint tier will practically never produce a true match for these two sources. The
+      mechanism is verified in isolation with well-formed NormalizedTenderFields; the end-to-end
+      achievement described by the ROADMAP truth is not currently achievable for the two new
+      adapters because of the normalizer gap.
+    artifacts:
+      - path: "apps/api/src/tenders/tender-dedup.service.ts"
+        issue: "Correct in isolation; fingerprint computed from NormalizedTenderFields, which will be malformed for NetServer/cosinex until the normalizer gap above is closed"
+    missing:
+      - "Same normalizer fix as gaps 1/2 is a hard prerequisite for this truth to hold end-to-end for any pair other than DÖE<->DÖE"
+      - "A live/manual confirmation once ai-netserver or cosinex-dtvp poll config is activated (TenderSourcePollConfig.isActive=true) and a real overlapping tender is observed to merge into one entry with two source links"
+deferred: []
+human_verification:
+  - test: "Activate the ai-netserver or cosinex-dtvp TenderSourcePollConfig row (isActive=true) after the normalizer gap is fixed, let pollDueSources run one tick, and confirm in the DB/UI that a tender known to exist on both DÖE and the new source collapses to ONE Tender with two TenderSource rows/links in TenderDetail"
+    expected: "One results-list entry; TenderDetail shows both a DÖE link and a tender24/lhs-vpbw/vergabe.landbw/DTVP link"
+    why_human: "Requires live network access to the real portals, a real overlapping notice, and a running scheduler tick against a real DB — cannot be proven by static code/unit-test inspection alone"
+  - test: "Confirm local API container was rebuilt/restarted after Plans 13-04/13-05 so the ai-netserver/cosinex-dtvp TenderSourcePollConfig seed rows actually exist in the local DB"
+    expected: "SELECT sourceType, isActive FROM TenderSourcePollConfig shows 3 rows (doe-opendata, ai-netserver, cosinex-dtvp), the latter two isActive=false"
+    why_human: "As of this verification the local dev DB only has the doe-opendata row — onModuleInit's seed code exists and is correct, but the API process has not been restarted/rebuilt to run it since Plans 13-04/13-05 landed"
+---
+
+# Phase 13: Scraping Adapters & Cross-Source Deduplication Verification Report
+
+**Phase Goal:** Die Plattform erweitert die Ausschreibungs-Abdeckung um AI-AG-NetServer- und cosinex-Portal-Adapter und fasst Ausschreibungen, die auf mehreren Quellen auftauchen, zu einem Eintrag zusammen — während sie das Scrapen der AGB-verbotenen Portale strukturell verweigert.
+**Verified:** 2026-07-23T10:15:00Z
+**Status:** gaps_found
+**Re-verification:** No — initial verification
+
+## Goal Achievement
+
+### Observable Truths (ROADMAP Success Criteria)
+
+| # | Truth | Status | Evidence |
+|---|-------|--------|----------|
+| 1 | NetServer-Portale (lhs-vpbw, tender24, vergabe.landbw) erscheinen als zusätzliche, **korrekt normalisierte** Treffer über EINEN config-getriebenen Adapter | ✗ FAILED | `netserver.adapter.ts` parst HTML korrekt zu `RawTenderRecord[]` (14/14 Tests grün), aber `tender-normalizer.service.ts` versteht die generische `ocdsPayload`-Bag nicht (`getOcdsTender()`/`getOcdsBuyer()` lesen nur `.tender`/`.buyer`) — jeder NetServer-Datensatz würde zu `title='Unbenannte Ausschreibung'`, `buyerName=null`, `cpvCodes=[]`, `deadlineAt=null` normalisiert. Selbst dokumentiert als "Known Stubs" in 13-04-SUMMARY.md. |
+| 2 | cosinex/DTVP (DTVP) erscheint als zusätzlicher Treffer über einen SEPARATEN config-getriebenen Adapter, **korrekt normalisiert** | ✗ FAILED | Gleiche Grundursache wie #1: `cosinex.adapter.ts` parst korrekt (16/16 Tests grün, inkl. echtem ISO-8859-1-Encoding-Fix), aber dieselbe generische Bag wird vom Normalizer nicht verstanden. Selbst dokumentiert als "Known Stubs" in 13-05-SUMMARY.md. |
+| 3 | Ein über DÖE + einem Scraping-Adapter gesehener Tender erscheint EINMAL mit Links zu allen Quell-Portalen (Fuzzy-Fingerprint-Dedup); Dedup greift erst ab 2. aktiver Quelle | ⚠️ PARTIAL (siehe gaps) | Resolver-Mechanik (`tender-dedup.service.ts`, 3-Stufen, `dedupActive`-Gate), Fan-out (`pollDueSources` via `findMany`), Backfill (2851/2851) und Multi-Source-UI (`TenderDetail.tsx`) sind alle vorhanden, verdrahtet und isoliert getestet — VERIFIZIERT auf Mechanik-Ebene. ABER: da NetServer/cosinex-Datensätze wegen Truth 1/2 mit generischem/leerem Titel+Buyer+Frist normalisiert würden, würde ihr Fingerprint NIE mit dem echten DÖE-Gegenstück kollidieren — reale Cross-Source-Dedup-Treffer zwischen DÖE↔NetServer/cosinex sind praktisch unerreichbar, bis der Normalizer-Gap geschlossen ist. |
+| 4 | Registrierung von vergabe24/aumass als Poll-Quelle wird vom System selbst (Code, nicht nur Doku) verweigert | ✓ VERIFIED | `source-registry.ts`: `SourceRegistry.register()` iteriert `adapter.portals` und wirft `DeniedPortalError` bei jedem Treffer gegen `DENYLISTED_PORTALS = ['vergabe24','aumass']`, auch bei gemischtem Array (kein Teil-Register). 6/6 Tests grün, inkl. explizitem vergabe24-Test, aumass-Test und Mixed-Array-Test. |
+
+**Score:** 1/4 truths verified (0 present-behavior-unverified; 3 failed/partial, all rooted in one shared cause: the normalizer does not consume the generic `ocdsPayload` bag the two new adapters intentionally produce)
+
+### Required Artifacts
+
+| Artifact | Expected | Status | Details |
+|----------|----------|--------|---------|
+| `apps/api/src/tenders/tender-fingerprint.ts` | NULL-tolerant pure fingerprint fn | ✓ VERIFIED | 8/8 unit tests pass; sha256([buyer,title,cpvDivisionKey,deadlineKey,valueBucket]) |
+| `apps/api/prisma/schema.prisma` (TenderSource, Tender.fingerprint) | 1:n model + additive column | ✓ VERIFIED | `model TenderSource` present, `@@unique([sourcePortal, sourceNoticeId])`, `Tender.fingerprint` nullable + `@@index`; migration applied locally |
+| `apps/api/src/tenders/source-registry.ts` | Denylist gate | ✓ VERIFIED | `DeniedPortalError`, `DENYLISTED_PORTALS`, wired into `TendersModule.onModuleInit` |
+| `apps/api/src/tenders/adapters/tender-source-adapter.interface.ts` | Generalized `portals[]` contract | ✓ VERIFIED | `readonly portals: readonly string[]` present; DÖE/NetServer/cosinex all declare it |
+| `apps/api/src/tenders/tender-dedup.service.ts` | 3-tier resolver | ✓ VERIFIED | OCID → source:noticeId → fingerprint, `dedupActive` hard gate on tier 3, SCHEMA-02 field-refresh preserved |
+| `apps/api/src/tenders/tender-ingestion.service.ts` (`pollDueSources`) | `findMany` fan-out, catch-per-source | ✓ VERIFIED | `prisma.tenderSourcePollConfig.findMany({where:{isActive:true}})`, try/catch per config inside the loop |
+| `apps/api/src/tenders/adapters/netserver.adapter.ts` | Config-driven, 3 portals, 1 class | ✓ VERIFIED (as HTML-parser) / ✗ (as normalized-tender producer) | `portals = ['tender24','lhs-vpbw','vergabe.landbw']`; parse layer solid, but output not consumable by normalizer (see Truth 1) |
+| `apps/api/src/tenders/adapters/cosinex.adapter.ts` | Separate single-portal adapter | ✓ VERIFIED (as HTML-parser) / ✗ (as normalized-tender producer) | `portals = ['cosinex-dtvp']`; same caveat as above (see Truth 2) |
+| `apps/api/src/tenders/tenders.controller.ts` (`getTender`) | includes `sources[]` | ✓ VERIFIED | `include: { sources: { select: {...} } }`, route order unchanged (`:id` after all static routes) |
+| `apps/web/.../TenderDetail.tsx` | Renders all source links | ✓ VERIFIED | `tender.sources.map(...)` with `portalLabel()`, fallback to legacy single `sourceUrl` block; 2 dedicated tests pass |
+
+### Key Link Verification
+
+| From | To | Via | Status | Details |
+|------|-----|-----|--------|---------|
+| `netserver.adapter.ts` / `cosinex.adapter.ts` | `tender-normalizer.service.ts` | `RawTenderRecord.ocdsPayload` -> `normalize()` | ✗ NOT_WIRED (semantically) | Data physically flows (no crash), but `normalize()`'s field extraction contract (`.tender`/`.buyer`) does not match the flat bag shape the adapters produce — a silent, no-error data-quality break, not a compile/runtime error |
+| `tender-ingestion.service.ts` (`pollDueSources`) | `source-registry.ts` | `registry.get(config.sourceType)` | ✓ WIRED | Confirmed in code + `tender-ingestion.service.spec.ts` fan-out tests |
+| `tenders.module.ts` (`onModuleInit`) | `source-registry.ts` | `sourceRegistry.register(doeAdapter/netServerAdapter/cosinexAdapter)` | ✓ WIRED | All 3 adapters registered at DI boot; denylist gate structurally applies here |
+| `tender-dedup.service.ts` | `prisma.tenderSource` / `prisma.tender` | `upsert`/`create` on match/no-match | ✓ WIRED | Composite-key upsert target verified by dedicated test |
+| `tenders.controller.ts` (`getTender`) | `apps/web` `TenderDetail.tsx` | `sources[]` field on API response | ✓ WIRED | Backend include + frontend consumption both tested |
+
+### Behavioral Spot-Checks / Test Suite Results (run live during this verification)
+
+| Check | Command | Result | Status |
+|-------|---------|--------|--------|
+| API full test suite | `cd apps/api && npx vitest run` | 25 files, **284/284 pass** | ✓ PASS |
+| API typecheck | `cd apps/api && npx tsc --noEmit -p tsconfig.json` | clean, exit 0 | ✓ PASS |
+| Web full test suite | `cd apps/web && npx vitest run` | 23 files, **133/133 pass** | ✓ PASS |
+| Web typecheck | `cd apps/web && npx tsc --noEmit` | clean, exit 0 | ✓ PASS |
+| Denylist refusal (source-registry.spec.ts) | (part of full suite) | 6/6 pass | ✓ PASS |
+| Dedup resolver (tender-dedup.service.spec.ts) | (part of full suite) | 8/8 pass | ✓ PASS |
+| Local DB — TenderSource backfill | `psql -c 'SELECT count(*) FROM "TenderSource"'` | 2851 | ✓ PASS (matches `SELECT count(*) FROM "Tender"` = 2851) |
+| Local DB — fingerprint backfill | `psql -c 'SELECT count(*) FROM "Tender" WHERE fingerprint IS NOT NULL'` | 2851 | ✓ PASS |
+| Local DB — poll configs seeded | `psql -c 'SELECT "sourceType","isActive" FROM "TenderSourcePollConfig"'` | Only `doe-opendata` row present | ⚠️ Local API process not yet restarted since Plans 13-04/13-05 landed — seed code for `ai-netserver`/`cosinex-dtvp` exists but hasn't run against this DB (see human_verification) |
+| Debt markers (TBD/FIXME/XXX) in phase files | `grep -n -E "TBD|FIXME|XXX"` across 8 key files | none found | ✓ PASS |
+
+### Requirements Coverage
+
+| Requirement | Source Plan | Description | Status | Evidence |
+|-------------|------------|-------------|--------|----------|
+| INGEST-02 | 13-04 | Import von AI-AG-NetServer-Portalen über konfigurierbaren Adapter | ✗ BLOCKED | Adapter fetches/parses correctly, but records would not be usably imported (garbage-normalized) — see Truth 1 |
+| INGEST-03 | 13-05 | Import vom cosinex-Vergabemarktplatz über Adapter | ✗ BLOCKED | Same root cause — see Truth 2 |
+| INGEST-07 | 13-02 | vergabe24/aumass hart als Denylist, keine automatische Registrierung möglich | ✓ SATISFIED | `source-registry.spec.ts`, code-level `DeniedPortalError` |
+| SCHEMA-03 | 13-01/03/06 | Dedup zu einem Eintrag mit mehreren Quell-Links; greift erst ab 2. aktiver Quelle | ⚠️ PARTIALLY SATISFIED | Schema, resolver, fan-out, backfill, and display are all correctly built and tested for the *mechanism*; the practical DÖE↔NetServer/cosinex cross-source match is blocked by the same normalizer gap (Truth 3) |
+
+REQUIREMENTS.md currently marks all four as `[x]`/"Complete" — this verification finds INGEST-02 and INGEST-03 not actually satisfiable end-to-end yet, and SCHEMA-03 only partially achieved for the two new sources (DÖE-internal re-poll/change-detection dedup, which predates this phase, remains fully intact — see below).
+
+### SCHEMA-02 Regression Check
+
+Confirmed NOT regressed: `tender-dedup.service.spec.ts` has two dedicated tests ("OCID match with a changed contentHash still updates...", "...UNCHANGED contentHash does NOT call tender.update"), and the pre-existing `tender-ingestion.service.spec.ts` SCHEMA-02 suite (insert-once, update-in-place-on-changed-hash) was migrated to the new `resolve()` hand-off and remains green. The mutable-field list (title, buyerName, cpvCodes, cpvDivisions, region, plz, bundesland, deadlineAt, estimatedValue, procedureType, sourceUrl, contentHash) was copied verbatim from the old `tender.upsert` UPDATE branch.
+
+### Anti-Patterns Found
+
+None — no `TBD`/`FIXME`/`XXX`/`TODO`/`HACK`/`PLACEHOLDER` markers in the 8 key phase files scanned. The normalizer gap is not a code-smell/stub pattern; it is a structurally absent capability, self-disclosed transparently in both SUMMARY.md files' "Known Stubs" sections rather than hidden.
+
+### Human Verification Required
+
+1. **Real-world cross-source dedup confirmation** — Test: after the normalizer gap is fixed, activate `ai-netserver` or `cosinex-dtvp` (`TenderSourcePollConfig.isActive=true`), run one poll tick, and confirm a real known-overlapping tender collapses to one `Tender` row with two `TenderSource` links visible in `TenderDetail`. Expected: one results-list entry, two clickable source links. Why human: needs live portal access, a real overlapping notice, and a running scheduler tick — not provable from static code.
+2. **Poll-config seed freshness** — Test: confirm the local API process has been rebuilt/restarted since Plans 13-04/13-05 so `ai-netserver`/`cosinex-dtvp` `TenderSourcePollConfig` rows actually exist locally (currently only `doe-opendata` is present in the DB). Why human: requires a deploy/restart action outside this verification's scope (per project convention, Claude does not restart local Docker services).
+
+### Gaps Summary
+
+Phase 13 built a genuinely solid, well-tested **core**: the `TenderSource` schema + 2851-row backfill, the NULL-tolerant fingerprint function, the `SourceRegistry` with a real code-level denylist gate (Success Criterion 4 fully achieved), the three-tier dedup resolver with `dedupActive` hard-gating (SCHEMA-03 mechanism), the `findMany` fan-out with catch-per-source isolation, and the multi-source-link display in `TenderDetail`. All of this is unit-tested (284/284 API, 133/133 web) and both typechecks are clean.
+
+However, both new HTML adapters (`NetServerAdapter`, `CosinexAdapter`) deliberately punt on a required, self-documented follow-up: they carry their parsed fields in a flat generic `ocdsPayload` bag that `TenderNormalizerService` — hardcoded to the DÖE eForms/OCDS shape — cannot read. Both `13-04-SUMMARY.md` and `13-05-SUMMARY.md` disclose this transparently under "Known Stubs" as "out of this plan's `files_modified` scope" and "a natural follow-up plan" — but no such follow-up plan exists in Phase 13, and Phase 14's roadmap scope (RSS/email ingestion, admin source UI, i18n) does not cover it either. Since the poll configs for both new sources are seeded `isActive: false`, this gap is currently dormant in production — but it directly blocks 3 of the phase's 4 ROADMAP success criteria from being true as written ("appear as additional, **correctly normalized** results" x2, and real DÖE↔scraper cross-source dedup matching). Recommend a closure plan that extends `TenderNormalizerService` (or introduces a per-sourceType normalization strategy) to map the NetServer/cosinex generic bag into `NormalizedTenderFields` before this phase can be considered to have achieved its stated goal.
+
+---
+*Verified: 2026-07-23T10:15:00Z*
+*Verifier: Claude (gsd-verifier)*