diff --git a/.planning/phases/13-scraping-adapters-cross-source-dedup/13-VERIFICATION.md b/.planning/phases/13-scraping-adapters-cross-source-dedup/13-VERIFICATION.md new file mode 100644 index 0000000..15e46a9 --- /dev/null +++ b/.planning/phases/13-scraping-adapters-cross-source-dedup/13-VERIFICATION.md @@ -0,0 +1,174 @@ +--- +phase: 13-scraping-adapters-cross-source-dedup +verified: 2026-07-23T10:15:00Z +status: gaps_found +score: 1/4 must-haves verified +behavior_unverified: 0 +overrides_applied: 0 +gaps: + - truth: "Tenders from AI-AG NetServer portals (lhs-vpbw, tender24, vergabe.landbw) appear as additional, correctly normalized results via one config-driven adapter" + status: failed + reason: > + NetServerAdapter correctly parses NetServer HTML into RawTenderRecord[] (14/14 unit tests + pass), but the shared TenderNormalizerService.normalize() only understands the DÖE + eForms-DE/OCDS payload shape (getOcdsTender()/getOcdsBuyer() look for `.tender`/`.buyer` + keys). NetServerAdapter deliberately puts a FLAT generic bag + `{title, buyerName, procedureType, legalFramework, deadlineAt}` into `ocdsPayload` (own code + comment: "NetServer has no eForms/OCDS structure ... a future normalizer extension + (out of scope for this plan) can map them"). Because the bag has no `.tender`/`.buyer` + keys, every NetServer record would normalize to title='Unbenannte Ausschreibung', + buyerName=null, cpvCodes=[], deadlineAt=null, estimatedValue=null — i.e. NOT + "correctly normalized". This is explicitly self-documented as "Known Stubs" in + 13-04-SUMMARY.md, not resolved anywhere in Phase 13, and not scheduled in Phase 14 + (Phase 14 scope per ROADMAP is RSS/email/admin-UI/i18n, no normalizer work). + artifacts: + - path: "apps/api/src/tenders/tender-normalizer.service.ts" + issue: "getOcdsTender()/getOcdsBuyer() only read `.tender`/`.buyer` keys; has no branch for the generic ocdsPayload bag NetServerAdapter/CosinexAdapter produce" + - path: "apps/api/src/tenders/adapters/netserver.adapter.ts" + issue: "Puts {title, buyerName, procedureType, legalFramework, deadlineAt} directly into ocdsPayload (own comment marks this as needing a future normalizer extension)" + missing: + - "Extend TenderNormalizerService (or add a per-sourceType normalization strategy) so ai-netserver's generic ocdsPayload bag maps to real title/buyerName/deadlineAt/procedureType instead of falling through to defaults" + - truth: "Tenders from the cosinex Vergabemarktplatz (DTVP) appear as additional, correctly normalized results via a separate config-driven adapter" + status: failed + reason: > + Identical root cause to the NetServer gap above. CosinexAdapter correctly parses the + live cosinex/DTVP results table into RawTenderRecord[] (16/16 unit tests pass, including + a real ISO-8859-1 charset-decode fix), but puts the same generic + {title, buyerName, procedureType, legalFramework, deadlineAt} bag into ocdsPayload. + TenderNormalizerService does not understand this shape either, so cosinex records would + normalize incorrectly for the same reason. Self-documented as "Known Stubs" in + 13-05-SUMMARY.md, not resolved in this phase, not scheduled in Phase 14. + artifacts: + - path: "apps/api/src/tenders/adapters/cosinex.adapter.ts" + issue: "Same generic ocdsPayload bag convention as NetServerAdapter — normalizer cannot map it" + missing: + - "Same normalizer extension as the NetServer gap, ideally handling both bag-shaped sources together" + - truth: "A tender that appears via both DÖE and a scraping adapter shows up once in the results list (fuzzy fingerprint dedup on buyer+title+CPV+deadline+value), with links to all of its source portals -- dedup logic only activates once a second source is live" + status: partial + reason: > + The DEDUP MECHANISM itself is solidly built and unit-tested: TenderDedupService's + three-tier resolver (OCID -> source:noticeId -> fingerprint), the dedupActive=(activePortalCount>=2) + gate, pollDueSources' findMany fan-out with catch-per-source isolation, the 2851-row + backfill, and the TenderDetail multi-source-link UI are all present, wired, and covered by + passing tests (tender-dedup.service.spec.ts 8/8, tender-ingestion.service.spec.ts fan-out + tests, tenders.controller.spec.ts, TenderDetail.test.tsx). HOWEVER, the truth as stated + ("a tender that appears via both DÖE and a scraping adapter shows up once ... fuzzy + fingerprint dedup on buyer+title+CPV+deadline+value") is undermined transitively by the two + gaps above: because NetServer/cosinex records normalize with a generic/empty + title+buyer+deadline ('Unbenannte Ausschreibung', null buyer, null deadline), their computed + fingerprint will NOT match the fingerprint of the corresponding real DÖE tender (which has + the actual title/buyer/deadline). So even once a second source is activated, real-world + DÖE<->NetServer or DÖE<->cosinex duplicates will NOT be recognized as the same tender — + the fingerprint tier will practically never produce a true match for these two sources. The + mechanism is verified in isolation with well-formed NormalizedTenderFields; the end-to-end + achievement described by the ROADMAP truth is not currently achievable for the two new + adapters because of the normalizer gap. + artifacts: + - path: "apps/api/src/tenders/tender-dedup.service.ts" + issue: "Correct in isolation; fingerprint computed from NormalizedTenderFields, which will be malformed for NetServer/cosinex until the normalizer gap above is closed" + missing: + - "Same normalizer fix as gaps 1/2 is a hard prerequisite for this truth to hold end-to-end for any pair other than DÖE<->DÖE" + - "A live/manual confirmation once ai-netserver or cosinex-dtvp poll config is activated (TenderSourcePollConfig.isActive=true) and a real overlapping tender is observed to merge into one entry with two source links" +deferred: [] +human_verification: + - test: "Activate the ai-netserver or cosinex-dtvp TenderSourcePollConfig row (isActive=true) after the normalizer gap is fixed, let pollDueSources run one tick, and confirm in the DB/UI that a tender known to exist on both DÖE and the new source collapses to ONE Tender with two TenderSource rows/links in TenderDetail" + expected: "One results-list entry; TenderDetail shows both a DÖE link and a tender24/lhs-vpbw/vergabe.landbw/DTVP link" + why_human: "Requires live network access to the real portals, a real overlapping notice, and a running scheduler tick against a real DB — cannot be proven by static code/unit-test inspection alone" + - test: "Confirm local API container was rebuilt/restarted after Plans 13-04/13-05 so the ai-netserver/cosinex-dtvp TenderSourcePollConfig seed rows actually exist in the local DB" + expected: "SELECT sourceType, isActive FROM TenderSourcePollConfig shows 3 rows (doe-opendata, ai-netserver, cosinex-dtvp), the latter two isActive=false" + why_human: "As of this verification the local dev DB only has the doe-opendata row — onModuleInit's seed code exists and is correct, but the API process has not been restarted/rebuilt to run it since Plans 13-04/13-05 landed" +--- + +# Phase 13: Scraping Adapters & Cross-Source Deduplication Verification Report + +**Phase Goal:** Die Plattform erweitert die Ausschreibungs-Abdeckung um AI-AG-NetServer- und cosinex-Portal-Adapter und fasst Ausschreibungen, die auf mehreren Quellen auftauchen, zu einem Eintrag zusammen — während sie das Scrapen der AGB-verbotenen Portale strukturell verweigert. +**Verified:** 2026-07-23T10:15:00Z +**Status:** gaps_found +**Re-verification:** No — initial verification + +## Goal Achievement + +### Observable Truths (ROADMAP Success Criteria) + +| # | Truth | Status | Evidence | +|---|-------|--------|----------| +| 1 | NetServer-Portale (lhs-vpbw, tender24, vergabe.landbw) erscheinen als zusätzliche, **korrekt normalisierte** Treffer über EINEN config-getriebenen Adapter | ✗ FAILED | `netserver.adapter.ts` parst HTML korrekt zu `RawTenderRecord[]` (14/14 Tests grün), aber `tender-normalizer.service.ts` versteht die generische `ocdsPayload`-Bag nicht (`getOcdsTender()`/`getOcdsBuyer()` lesen nur `.tender`/`.buyer`) — jeder NetServer-Datensatz würde zu `title='Unbenannte Ausschreibung'`, `buyerName=null`, `cpvCodes=[]`, `deadlineAt=null` normalisiert. Selbst dokumentiert als "Known Stubs" in 13-04-SUMMARY.md. | +| 2 | cosinex/DTVP (DTVP) erscheint als zusätzlicher Treffer über einen SEPARATEN config-getriebenen Adapter, **korrekt normalisiert** | ✗ FAILED | Gleiche Grundursache wie #1: `cosinex.adapter.ts` parst korrekt (16/16 Tests grün, inkl. echtem ISO-8859-1-Encoding-Fix), aber dieselbe generische Bag wird vom Normalizer nicht verstanden. Selbst dokumentiert als "Known Stubs" in 13-05-SUMMARY.md. | +| 3 | Ein über DÖE + einem Scraping-Adapter gesehener Tender erscheint EINMAL mit Links zu allen Quell-Portalen (Fuzzy-Fingerprint-Dedup); Dedup greift erst ab 2. aktiver Quelle | ⚠️ PARTIAL (siehe gaps) | Resolver-Mechanik (`tender-dedup.service.ts`, 3-Stufen, `dedupActive`-Gate), Fan-out (`pollDueSources` via `findMany`), Backfill (2851/2851) und Multi-Source-UI (`TenderDetail.tsx`) sind alle vorhanden, verdrahtet und isoliert getestet — VERIFIZIERT auf Mechanik-Ebene. ABER: da NetServer/cosinex-Datensätze wegen Truth 1/2 mit generischem/leerem Titel+Buyer+Frist normalisiert würden, würde ihr Fingerprint NIE mit dem echten DÖE-Gegenstück kollidieren — reale Cross-Source-Dedup-Treffer zwischen DÖE↔NetServer/cosinex sind praktisch unerreichbar, bis der Normalizer-Gap geschlossen ist. | +| 4 | Registrierung von vergabe24/aumass als Poll-Quelle wird vom System selbst (Code, nicht nur Doku) verweigert | ✓ VERIFIED | `source-registry.ts`: `SourceRegistry.register()` iteriert `adapter.portals` und wirft `DeniedPortalError` bei jedem Treffer gegen `DENYLISTED_PORTALS = ['vergabe24','aumass']`, auch bei gemischtem Array (kein Teil-Register). 6/6 Tests grün, inkl. explizitem vergabe24-Test, aumass-Test und Mixed-Array-Test. | + +**Score:** 1/4 truths verified (0 present-behavior-unverified; 3 failed/partial, all rooted in one shared cause: the normalizer does not consume the generic `ocdsPayload` bag the two new adapters intentionally produce) + +### Required Artifacts + +| Artifact | Expected | Status | Details | +|----------|----------|--------|---------| +| `apps/api/src/tenders/tender-fingerprint.ts` | NULL-tolerant pure fingerprint fn | ✓ VERIFIED | 8/8 unit tests pass; sha256([buyer,title,cpvDivisionKey,deadlineKey,valueBucket]) | +| `apps/api/prisma/schema.prisma` (TenderSource, Tender.fingerprint) | 1:n model + additive column | ✓ VERIFIED | `model TenderSource` present, `@@unique([sourcePortal, sourceNoticeId])`, `Tender.fingerprint` nullable + `@@index`; migration applied locally | +| `apps/api/src/tenders/source-registry.ts` | Denylist gate | ✓ VERIFIED | `DeniedPortalError`, `DENYLISTED_PORTALS`, wired into `TendersModule.onModuleInit` | +| `apps/api/src/tenders/adapters/tender-source-adapter.interface.ts` | Generalized `portals[]` contract | ✓ VERIFIED | `readonly portals: readonly string[]` present; DÖE/NetServer/cosinex all declare it | +| `apps/api/src/tenders/tender-dedup.service.ts` | 3-tier resolver | ✓ VERIFIED | OCID → source:noticeId → fingerprint, `dedupActive` hard gate on tier 3, SCHEMA-02 field-refresh preserved | +| `apps/api/src/tenders/tender-ingestion.service.ts` (`pollDueSources`) | `findMany` fan-out, catch-per-source | ✓ VERIFIED | `prisma.tenderSourcePollConfig.findMany({where:{isActive:true}})`, try/catch per config inside the loop | +| `apps/api/src/tenders/adapters/netserver.adapter.ts` | Config-driven, 3 portals, 1 class | ✓ VERIFIED (as HTML-parser) / ✗ (as normalized-tender producer) | `portals = ['tender24','lhs-vpbw','vergabe.landbw']`; parse layer solid, but output not consumable by normalizer (see Truth 1) | +| `apps/api/src/tenders/adapters/cosinex.adapter.ts` | Separate single-portal adapter | ✓ VERIFIED (as HTML-parser) / ✗ (as normalized-tender producer) | `portals = ['cosinex-dtvp']`; same caveat as above (see Truth 2) | +| `apps/api/src/tenders/tenders.controller.ts` (`getTender`) | includes `sources[]` | ✓ VERIFIED | `include: { sources: { select: {...} } }`, route order unchanged (`:id` after all static routes) | +| `apps/web/.../TenderDetail.tsx` | Renders all source links | ✓ VERIFIED | `tender.sources.map(...)` with `portalLabel()`, fallback to legacy single `sourceUrl` block; 2 dedicated tests pass | + +### Key Link Verification + +| From | To | Via | Status | Details | +|------|-----|-----|--------|---------| +| `netserver.adapter.ts` / `cosinex.adapter.ts` | `tender-normalizer.service.ts` | `RawTenderRecord.ocdsPayload` -> `normalize()` | ✗ NOT_WIRED (semantically) | Data physically flows (no crash), but `normalize()`'s field extraction contract (`.tender`/`.buyer`) does not match the flat bag shape the adapters produce — a silent, no-error data-quality break, not a compile/runtime error | +| `tender-ingestion.service.ts` (`pollDueSources`) | `source-registry.ts` | `registry.get(config.sourceType)` | ✓ WIRED | Confirmed in code + `tender-ingestion.service.spec.ts` fan-out tests | +| `tenders.module.ts` (`onModuleInit`) | `source-registry.ts` | `sourceRegistry.register(doeAdapter/netServerAdapter/cosinexAdapter)` | ✓ WIRED | All 3 adapters registered at DI boot; denylist gate structurally applies here | +| `tender-dedup.service.ts` | `prisma.tenderSource` / `prisma.tender` | `upsert`/`create` on match/no-match | ✓ WIRED | Composite-key upsert target verified by dedicated test | +| `tenders.controller.ts` (`getTender`) | `apps/web` `TenderDetail.tsx` | `sources[]` field on API response | ✓ WIRED | Backend include + frontend consumption both tested | + +### Behavioral Spot-Checks / Test Suite Results (run live during this verification) + +| Check | Command | Result | Status | +|-------|---------|--------|--------| +| API full test suite | `cd apps/api && npx vitest run` | 25 files, **284/284 pass** | ✓ PASS | +| API typecheck | `cd apps/api && npx tsc --noEmit -p tsconfig.json` | clean, exit 0 | ✓ PASS | +| Web full test suite | `cd apps/web && npx vitest run` | 23 files, **133/133 pass** | ✓ PASS | +| Web typecheck | `cd apps/web && npx tsc --noEmit` | clean, exit 0 | ✓ PASS | +| Denylist refusal (source-registry.spec.ts) | (part of full suite) | 6/6 pass | ✓ PASS | +| Dedup resolver (tender-dedup.service.spec.ts) | (part of full suite) | 8/8 pass | ✓ PASS | +| Local DB — TenderSource backfill | `psql -c 'SELECT count(*) FROM "TenderSource"'` | 2851 | ✓ PASS (matches `SELECT count(*) FROM "Tender"` = 2851) | +| Local DB — fingerprint backfill | `psql -c 'SELECT count(*) FROM "Tender" WHERE fingerprint IS NOT NULL'` | 2851 | ✓ PASS | +| Local DB — poll configs seeded | `psql -c 'SELECT "sourceType","isActive" FROM "TenderSourcePollConfig"'` | Only `doe-opendata` row present | ⚠️ Local API process not yet restarted since Plans 13-04/13-05 landed — seed code for `ai-netserver`/`cosinex-dtvp` exists but hasn't run against this DB (see human_verification) | +| Debt markers (TBD/FIXME/XXX) in phase files | `grep -n -E "TBD|FIXME|XXX"` across 8 key files | none found | ✓ PASS | + +### Requirements Coverage + +| Requirement | Source Plan | Description | Status | Evidence | +|-------------|------------|-------------|--------|----------| +| INGEST-02 | 13-04 | Import von AI-AG-NetServer-Portalen über konfigurierbaren Adapter | ✗ BLOCKED | Adapter fetches/parses correctly, but records would not be usably imported (garbage-normalized) — see Truth 1 | +| INGEST-03 | 13-05 | Import vom cosinex-Vergabemarktplatz über Adapter | ✗ BLOCKED | Same root cause — see Truth 2 | +| INGEST-07 | 13-02 | vergabe24/aumass hart als Denylist, keine automatische Registrierung möglich | ✓ SATISFIED | `source-registry.spec.ts`, code-level `DeniedPortalError` | +| SCHEMA-03 | 13-01/03/06 | Dedup zu einem Eintrag mit mehreren Quell-Links; greift erst ab 2. aktiver Quelle | ⚠️ PARTIALLY SATISFIED | Schema, resolver, fan-out, backfill, and display are all correctly built and tested for the *mechanism*; the practical DÖE↔NetServer/cosinex cross-source match is blocked by the same normalizer gap (Truth 3) | + +REQUIREMENTS.md currently marks all four as `[x]`/"Complete" — this verification finds INGEST-02 and INGEST-03 not actually satisfiable end-to-end yet, and SCHEMA-03 only partially achieved for the two new sources (DÖE-internal re-poll/change-detection dedup, which predates this phase, remains fully intact — see below). + +### SCHEMA-02 Regression Check + +Confirmed NOT regressed: `tender-dedup.service.spec.ts` has two dedicated tests ("OCID match with a changed contentHash still updates...", "...UNCHANGED contentHash does NOT call tender.update"), and the pre-existing `tender-ingestion.service.spec.ts` SCHEMA-02 suite (insert-once, update-in-place-on-changed-hash) was migrated to the new `resolve()` hand-off and remains green. The mutable-field list (title, buyerName, cpvCodes, cpvDivisions, region, plz, bundesland, deadlineAt, estimatedValue, procedureType, sourceUrl, contentHash) was copied verbatim from the old `tender.upsert` UPDATE branch. + +### Anti-Patterns Found + +None — no `TBD`/`FIXME`/`XXX`/`TODO`/`HACK`/`PLACEHOLDER` markers in the 8 key phase files scanned. The normalizer gap is not a code-smell/stub pattern; it is a structurally absent capability, self-disclosed transparently in both SUMMARY.md files' "Known Stubs" sections rather than hidden. + +### Human Verification Required + +1. **Real-world cross-source dedup confirmation** — Test: after the normalizer gap is fixed, activate `ai-netserver` or `cosinex-dtvp` (`TenderSourcePollConfig.isActive=true`), run one poll tick, and confirm a real known-overlapping tender collapses to one `Tender` row with two `TenderSource` links visible in `TenderDetail`. Expected: one results-list entry, two clickable source links. Why human: needs live portal access, a real overlapping notice, and a running scheduler tick — not provable from static code. +2. **Poll-config seed freshness** — Test: confirm the local API process has been rebuilt/restarted since Plans 13-04/13-05 so `ai-netserver`/`cosinex-dtvp` `TenderSourcePollConfig` rows actually exist locally (currently only `doe-opendata` is present in the DB). Why human: requires a deploy/restart action outside this verification's scope (per project convention, Claude does not restart local Docker services). + +### Gaps Summary + +Phase 13 built a genuinely solid, well-tested **core**: the `TenderSource` schema + 2851-row backfill, the NULL-tolerant fingerprint function, the `SourceRegistry` with a real code-level denylist gate (Success Criterion 4 fully achieved), the three-tier dedup resolver with `dedupActive` hard-gating (SCHEMA-03 mechanism), the `findMany` fan-out with catch-per-source isolation, and the multi-source-link display in `TenderDetail`. All of this is unit-tested (284/284 API, 133/133 web) and both typechecks are clean. + +However, both new HTML adapters (`NetServerAdapter`, `CosinexAdapter`) deliberately punt on a required, self-documented follow-up: they carry their parsed fields in a flat generic `ocdsPayload` bag that `TenderNormalizerService` — hardcoded to the DÖE eForms/OCDS shape — cannot read. Both `13-04-SUMMARY.md` and `13-05-SUMMARY.md` disclose this transparently under "Known Stubs" as "out of this plan's `files_modified` scope" and "a natural follow-up plan" — but no such follow-up plan exists in Phase 13, and Phase 14's roadmap scope (RSS/email ingestion, admin source UI, i18n) does not cover it either. Since the poll configs for both new sources are seeded `isActive: false`, this gap is currently dormant in production — but it directly blocks 3 of the phase's 4 ROADMAP success criteria from being true as written ("appear as additional, **correctly normalized** results" x2, and real DÖE↔scraper cross-source dedup matching). Recommend a closure plan that extends `TenderNormalizerService` (or introduces a per-sourceType normalization strategy) to map the NetServer/cosinex generic bag into `NormalizedTenderFields` before this phase can be considered to have achieved its stated goal. + +--- +*Verified: 2026-07-23T10:15:00Z* +*Verifier: Claude (gsd-verifier)*