From 7de0714b308f2f9d048dcead384ddf5006fd978e Mon Sep 17 00:00:00 2001 From: Schalli Date: Thu, 23 Jul 2026 10:37:28 +0200 Subject: [PATCH] =?UTF-8?q?docs(13):=20re-verify=20after=20normalizer=20fi?= =?UTF-8?q?x=20=E2=80=94=203/4,=20Truth=203=20deferred=20(option=20C)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Normalizer-Gap (quick 260723-e7i) closed criteria 1/2; re-verification lifts Phase 13 from 1/4 to 3/4. Truth 3 (cross-source fingerprint dedup) stays PARTIAL: newly-found cpvDivisions asymmetry (scrapers always [], ~79% of DÖE tenders populated) makes exact-match fingerprint miss most real overlaps. User decision: accept as dormant deferral — dedup is inert (both scrapers isActive=false), decide with live data once a 2nd source is activated. Co-Authored-By: Claude Opus 4.8 --- .../13-VERIFICATION.md | 237 +++++++++--------- 1 file changed, 124 insertions(+), 113 deletions(-) diff --git a/.planning/phases/13-scraping-adapters-cross-source-dedup/13-VERIFICATION.md b/.planning/phases/13-scraping-adapters-cross-source-dedup/13-VERIFICATION.md index 15e46a9..6d43bed 100644 --- a/.planning/phases/13-scraping-adapters-cross-source-dedup/13-VERIFICATION.md +++ b/.planning/phases/13-scraping-adapters-cross-source-dedup/13-VERIFICATION.md @@ -1,90 +1,112 @@ --- phase: 13-scraping-adapters-cross-source-dedup -verified: 2026-07-23T10:15:00Z +verified: 2026-07-23T10:30:00Z status: gaps_found -score: 1/4 must-haves verified +score: 3/4 must-haves verified behavior_unverified: 0 overrides_applied: 0 +re_verification: + previous_status: gaps_found + previous_score: 1/4 + gaps_closed: + - "Tenders from AI-AG NetServer portals appear as additional, correctly normalized results via one config-driven adapter" + - "Tenders from the cosinex Vergabemarktplatz (DTVP) appear as additional, correctly normalized results via a separate config-driven adapter" + gaps_remaining: + - "A tender that appears via both DÖE and a scraping adapter shows up once in the results list (fuzzy fingerprint dedup on buyer+title+CPV+deadline+value) — NEW residual finding: cpvDivisions asymmetry (scraper always [], DÖE usually populated) makes the fingerprint tier fail to collapse the majority of real cross-source duplicates even after the normalizer fix" + regressions: [] gaps: - - truth: "Tenders from AI-AG NetServer portals (lhs-vpbw, tender24, vergabe.landbw) appear as additional, correctly normalized results via one config-driven adapter" - status: failed - reason: > - NetServerAdapter correctly parses NetServer HTML into RawTenderRecord[] (14/14 unit tests - pass), but the shared TenderNormalizerService.normalize() only understands the DÖE - eForms-DE/OCDS payload shape (getOcdsTender()/getOcdsBuyer() look for `.tender`/`.buyer` - keys). NetServerAdapter deliberately puts a FLAT generic bag - `{title, buyerName, procedureType, legalFramework, deadlineAt}` into `ocdsPayload` (own code - comment: "NetServer has no eForms/OCDS structure ... a future normalizer extension - (out of scope for this plan) can map them"). Because the bag has no `.tender`/`.buyer` - keys, every NetServer record would normalize to title='Unbenannte Ausschreibung', - buyerName=null, cpvCodes=[], deadlineAt=null, estimatedValue=null — i.e. NOT - "correctly normalized". This is explicitly self-documented as "Known Stubs" in - 13-04-SUMMARY.md, not resolved anywhere in Phase 13, and not scheduled in Phase 14 - (Phase 14 scope per ROADMAP is RSS/email/admin-UI/i18n, no normalizer work). - artifacts: - - path: "apps/api/src/tenders/tender-normalizer.service.ts" - issue: "getOcdsTender()/getOcdsBuyer() only read `.tender`/`.buyer` keys; has no branch for the generic ocdsPayload bag NetServerAdapter/CosinexAdapter produce" - - path: "apps/api/src/tenders/adapters/netserver.adapter.ts" - issue: "Puts {title, buyerName, procedureType, legalFramework, deadlineAt} directly into ocdsPayload (own comment marks this as needing a future normalizer extension)" - missing: - - "Extend TenderNormalizerService (or add a per-sourceType normalization strategy) so ai-netserver's generic ocdsPayload bag maps to real title/buyerName/deadlineAt/procedureType instead of falling through to defaults" - - truth: "Tenders from the cosinex Vergabemarktplatz (DTVP) appear as additional, correctly normalized results via a separate config-driven adapter" - status: failed - reason: > - Identical root cause to the NetServer gap above. CosinexAdapter correctly parses the - live cosinex/DTVP results table into RawTenderRecord[] (16/16 unit tests pass, including - a real ISO-8859-1 charset-decode fix), but puts the same generic - {title, buyerName, procedureType, legalFramework, deadlineAt} bag into ocdsPayload. - TenderNormalizerService does not understand this shape either, so cosinex records would - normalize incorrectly for the same reason. Self-documented as "Known Stubs" in - 13-05-SUMMARY.md, not resolved in this phase, not scheduled in Phase 14. - artifacts: - - path: "apps/api/src/tenders/adapters/cosinex.adapter.ts" - issue: "Same generic ocdsPayload bag convention as NetServerAdapter — normalizer cannot map it" - missing: - - "Same normalizer extension as the NetServer gap, ideally handling both bag-shaped sources together" - truth: "A tender that appears via both DÖE and a scraping adapter shows up once in the results list (fuzzy fingerprint dedup on buyer+title+CPV+deadline+value), with links to all of its source portals -- dedup logic only activates once a second source is live" status: partial reason: > - The DEDUP MECHANISM itself is solidly built and unit-tested: TenderDedupService's - three-tier resolver (OCID -> source:noticeId -> fingerprint), the dedupActive=(activePortalCount>=2) - gate, pollDueSources' findMany fan-out with catch-per-source isolation, the 2851-row - backfill, and the TenderDetail multi-source-link UI are all present, wired, and covered by - passing tests (tender-dedup.service.spec.ts 8/8, tender-ingestion.service.spec.ts fan-out - tests, tenders.controller.spec.ts, TenderDetail.test.tsx). HOWEVER, the truth as stated - ("a tender that appears via both DÖE and a scraping adapter shows up once ... fuzzy - fingerprint dedup on buyer+title+CPV+deadline+value") is undermined transitively by the two - gaps above: because NetServer/cosinex records normalize with a generic/empty - title+buyer+deadline ('Unbenannte Ausschreibung', null buyer, null deadline), their computed - fingerprint will NOT match the fingerprint of the corresponding real DÖE tender (which has - the actual title/buyer/deadline). So even once a second source is activated, real-world - DÖE<->NetServer or DÖE<->cosinex duplicates will NOT be recognized as the same tender — - the fingerprint tier will practically never produce a true match for these two sources. The - mechanism is verified in isolation with well-formed NormalizedTenderFields; the end-to-end - achievement described by the ROADMAP truth is not currently achievable for the two new - adapters because of the normalizer gap. + The previously-identified root cause (NetServer/cosinex normalizing to title/buyerName/ + deadlineAt defaults) is CLOSED by quick task 260723-e7i — verified live in this + re-verification (see Truth table). However, a SEPARATE, previously-undetected issue + remains: tenderFingerprint()'s canonical string is a straight concatenation of + [buyerName, title, cpvDivisionKey, deadlineKey, valueBucket] hashed with sha256 — it is + an EXACT match on all five segments, not a weighted/fuzzy comparison despite the + "Hauptgewicht" (main weight) language in 13-RESEARCH.md. NetServerAdapter and + CosinexAdapter structurally cannot supply CPV codes (verified by reading both adapters' + HTML column extraction: neither the NetServer 6-column table nor the cosinex + csx-new-table has a CPV column at all), so TenderNormalizerService.normalizeBag() + correctly and necessarily always sets cpvDivisions=[] for these two sources (this is + not a normalizer bug — there is nothing to extract). Live DB check + (`SELECT count(*) FILTER (WHERE array_length("cpvDivisions",1) IS NULL OR + array_length("cpvDivisions",1)=0)` -> 601/2851 = ~21%) shows ~79% of existing + DÖE-sourced tenders carry a non-empty cpvDivisions. Because cpvDivisionKey('') != + cpvDivisionKey(['45']) (or any other non-empty division), the sha256 hash differs + whenever the DÖE-side counterpart has CPV codes -- i.e. for an estimated ~79% of real + overlapping tenders, the fingerprint tier will NOT collapse the DÖE and NetServer/cosinex + records into one Tender, even once a second source is activated and even though + title/buyerName/deadlineAt now correctly match. tender-dedup.service.spec.ts's only + cross-source fingerprint test (`fingerprint match (no OCID/noticeId overlap)`) does not + exercise this asymmetry -- it spreads `...BASE` (cpvDivisions: ['45']) into both + synthetic records, so both sides always share the same non-empty CPV segment. No test + in the suite proves a NetServer/cosinex record (cpvDivisions=[]) actually collides with + its real DÖE counterpart (cpvDivisions=['45'] or similar). artifacts: - - path: "apps/api/src/tenders/tender-dedup.service.ts" - issue: "Correct in isolation; fingerprint computed from NormalizedTenderFields, which will be malformed for NetServer/cosinex until the normalizer gap above is closed" + - path: "apps/api/src/tenders/tender-fingerprint.ts" + issue: "cpvDivisionKey is one of 5 hashed segments with no null/empty-tolerant matching for the CPV dimension specifically (unlike deadlineAt/estimatedValue, which ARE null-tolerant by design) -- an empty array on one side and a populated array on the other always produces a different hash" + - path: "apps/api/src/tenders/tender-normalizer.service.ts" + issue: "normalizeBag() correctly sets cpvDivisions=[] (there is no CPV data in the scraped HTML to map) -- this is the correct behavior for the available source data, but it is the direct cause of the fingerprint mismatch described above" missing: - - "Same normalizer fix as gaps 1/2 is a hard prerequisite for this truth to hold end-to-end for any pair other than DÖE<->DÖE" - - "A live/manual confirmation once ai-netserver or cosinex-dtvp poll config is activated (TenderSourcePollConfig.isActive=true) and a real overlapping tender is observed to merge into one entry with two source links" + - "A developer decision on how to handle the CPV-availability asymmetry for fingerprint matching: e.g. treat an empty cpvDivisions on either side as a wildcard/skip for that segment instead of an exact-match requirement, or accept the reduced (~21%) practical match rate for NetServer/cosinex cross-source dedup as a known limitation of these two sources" + - "A live/manual confirmation once ai-netserver or cosinex-dtvp poll config is activated (TenderSourcePollConfig.isActive=true) and a real overlapping tender is observed to merge into one entry with two source links (unchanged from prior verification -- still not possible to test locally, see Human Verification)" deferred: [] human_verification: - - test: "Activate the ai-netserver or cosinex-dtvp TenderSourcePollConfig row (isActive=true) after the normalizer gap is fixed, let pollDueSources run one tick, and confirm in the DB/UI that a tender known to exist on both DÖE and the new source collapses to ONE Tender with two TenderSource rows/links in TenderDetail" - expected: "One results-list entry; TenderDetail shows both a DÖE link and a tender24/lhs-vpbw/vergabe.landbw/DTVP link" + - test: "Activate the ai-netserver or cosinex-dtvp TenderSourcePollConfig row (isActive=true), let pollDueSources run one tick, and confirm in the DB/UI that a tender known to exist on both DÖE and the new source collapses to ONE Tender with two TenderSource rows/links in TenderDetail. Given the CPV-asymmetry finding above, prioritize testing with a real overlapping notice that happens to have NO CPV code on the DÖE side, since that is currently the only case the fingerprint tier can realistically catch for these two sources." + expected: "One results-list entry; TenderDetail shows both a DÖE link and a tender24/lhs-vpbw/vergabe.landbw/DTVP link — but only reliably for DÖE counterparts with no CPV code, per the gap above" why_human: "Requires live network access to the real portals, a real overlapping notice, and a running scheduler tick against a real DB — cannot be proven by static code/unit-test inspection alone" - - test: "Confirm local API container was rebuilt/restarted after Plans 13-04/13-05 so the ai-netserver/cosinex-dtvp TenderSourcePollConfig seed rows actually exist in the local DB" + - test: "Confirm local API container was rebuilt/restarted so the ai-netserver/cosinex-dtvp TenderSourcePollConfig seed rows actually exist in the local DB" expected: "SELECT sourceType, isActive FROM TenderSourcePollConfig shows 3 rows (doe-opendata, ai-netserver, cosinex-dtvp), the latter two isActive=false" - why_human: "As of this verification the local dev DB only has the doe-opendata row — onModuleInit's seed code exists and is correct, but the API process has not been restarted/rebuilt to run it since Plans 13-04/13-05 landed" + why_human: "Re-checked live in this verification (docker exec psql): still only 1 row (doe-opendata). The API process has not been restarted/rebuilt since Plans 13-04/13-05 landed -- unrelated to and not resolved by the normalizer quick-fix. Per project convention, Claude does not restart local Docker services." +residual_gap_decision: + date: 2026-07-23 + chosen: "C — accept as dormant deferral (no code change now)" + by: user + rationale: > + The cpvDivisions fingerprint asymmetry (Truth 3) is real but the whole + cross-source dedup mechanism is currently inert: both scraper poll configs + are seeded isActive=false, so only DÖE runs and the fingerprint tier is + never exercised in production. Whether CPV-less matching (Option A: a second + buyer+title+deadline fingerprint tier) has acceptable precision or produces + too many false merges can only be judged against real overlapping notices, + not statically. Decision deferred to when a second source is actually + activated. Options considered: A = CPV-less second fingerprint tier + (new column + migration + backfill 2851 + 4th resolver tier); B = drop + CPV+value from the primary fingerprint (simple, higher false-merge risk); + C = leave dormant (chosen). Truth 3 stays PARTIAL by design, not by neglect. --- # Phase 13: Scraping Adapters & Cross-Source Deduplication Verification Report **Phase Goal:** Die Plattform erweitert die Ausschreibungs-Abdeckung um AI-AG-NetServer- und cosinex-Portal-Adapter und fasst Ausschreibungen, die auf mehreren Quellen auftauchen, zu einem Eintrag zusammen — während sie das Scrapen der AGB-verbotenen Portale strukturell verweigert. -**Verified:** 2026-07-23T10:15:00Z +**Verified:** 2026-07-23T10:30:00Z **Status:** gaps_found -**Re-verification:** No — initial verification +**Re-verification:** Yes — after gap closure (quick task 260723-e7i) + +## What Changed Since Last Verification + +Quick task 260723-e7i (commits `82c9a48`, `9881005`) added a per-`sourceType` dispatch to +`TenderNormalizerService.normalize()`. Read live in this re-verification +(`apps/api/src/tenders/tender-normalizer.service.ts`): + +- `normalize()` switches on `raw.sourceType`: `ai-netserver`/`cosinex-dtvp` → `normalizeBag()`; + `doe-opendata`/default → `normalizeDoe()` (default never throws). +- `normalizeBag()` reads the flat `ocdsPayload` bag `{title, buyerName, procedureType, + legalFramework, deadlineAt}` and maps `title` (fallback `'Unbenannte Ausschreibung'`), + `buyerName`/`procedureType` (null-tolerant, no invented fallback), `deadlineAt` via the + existing `parseOcdsDate()`. `cpvCodes`/`cpvDivisions` = `[]`, `region`/`plz`/`bundesland`/ + `estimatedValue` = `null` (there is genuinely no such data in the NetServer/cosinex HTML — + confirmed by reading both adapters' column extraction). +- A shared `assemble()` computes `status`/`dedupKey`/`contentHash`/`publishedAt` identically for + both paths — `normalizeDoe()` is a byte-identical extraction of the pre-existing DÖE logic. + +**This closes the previously-identified root cause for Truths 1 and 2** — NetServer and +cosinex records now carry their real title/buyer/procedureType/deadline instead of falling +through to defaults. It **partially** closes Truth 3: title/buyer/deadline now flow correctly +into the fingerprint, but re-verification surfaced a **new, previously-undetected** issue +specific to the CPV dimension of the fingerprint that still blocks most real-world +cross-source matches (see Gap below). ## Goal Achievement @@ -92,83 +114,72 @@ human_verification: | # | Truth | Status | Evidence | |---|-------|--------|----------| -| 1 | NetServer-Portale (lhs-vpbw, tender24, vergabe.landbw) erscheinen als zusätzliche, **korrekt normalisierte** Treffer über EINEN config-getriebenen Adapter | ✗ FAILED | `netserver.adapter.ts` parst HTML korrekt zu `RawTenderRecord[]` (14/14 Tests grün), aber `tender-normalizer.service.ts` versteht die generische `ocdsPayload`-Bag nicht (`getOcdsTender()`/`getOcdsBuyer()` lesen nur `.tender`/`.buyer`) — jeder NetServer-Datensatz würde zu `title='Unbenannte Ausschreibung'`, `buyerName=null`, `cpvCodes=[]`, `deadlineAt=null` normalisiert. Selbst dokumentiert als "Known Stubs" in 13-04-SUMMARY.md. | -| 2 | cosinex/DTVP (DTVP) erscheint als zusätzlicher Treffer über einen SEPARATEN config-getriebenen Adapter, **korrekt normalisiert** | ✗ FAILED | Gleiche Grundursache wie #1: `cosinex.adapter.ts` parst korrekt (16/16 Tests grün, inkl. echtem ISO-8859-1-Encoding-Fix), aber dieselbe generische Bag wird vom Normalizer nicht verstanden. Selbst dokumentiert als "Known Stubs" in 13-05-SUMMARY.md. | -| 3 | Ein über DÖE + einem Scraping-Adapter gesehener Tender erscheint EINMAL mit Links zu allen Quell-Portalen (Fuzzy-Fingerprint-Dedup); Dedup greift erst ab 2. aktiver Quelle | ⚠️ PARTIAL (siehe gaps) | Resolver-Mechanik (`tender-dedup.service.ts`, 3-Stufen, `dedupActive`-Gate), Fan-out (`pollDueSources` via `findMany`), Backfill (2851/2851) und Multi-Source-UI (`TenderDetail.tsx`) sind alle vorhanden, verdrahtet und isoliert getestet — VERIFIZIERT auf Mechanik-Ebene. ABER: da NetServer/cosinex-Datensätze wegen Truth 1/2 mit generischem/leerem Titel+Buyer+Frist normalisiert würden, würde ihr Fingerprint NIE mit dem echten DÖE-Gegenstück kollidieren — reale Cross-Source-Dedup-Treffer zwischen DÖE↔NetServer/cosinex sind praktisch unerreichbar, bis der Normalizer-Gap geschlossen ist. | -| 4 | Registrierung von vergabe24/aumass als Poll-Quelle wird vom System selbst (Code, nicht nur Doku) verweigert | ✓ VERIFIED | `source-registry.ts`: `SourceRegistry.register()` iteriert `adapter.portals` und wirft `DeniedPortalError` bei jedem Treffer gegen `DENYLISTED_PORTALS = ['vergabe24','aumass']`, auch bei gemischtem Array (kein Teil-Register). 6/6 Tests grün, inkl. explizitem vergabe24-Test, aumass-Test und Mixed-Array-Test. | +| 1 | NetServer-Portale (lhs-vpbw, tender24, vergabe.landbw) erscheinen als zusätzliche, **korrekt normalisierte** Treffer über EINEN config-getriebenen Adapter | ✓ VERIFIED (flipped) | `normalizeBag()` now maps `ai-netserver` bag records into real `title`/`buyerName`/`procedureType`/`deadlineAt` (verified live: `tender-normalizer.service.spec.ts` "ai-netserver: maps the flat bag into NormalizedTenderFields" test asserts `title==='Bauleistung X'`, `buyerName==='Stadt Y'`, `deadlineAt` ISO round-trip). `cpvCodes=[]`/`region`/`plz`/`bundesland`/`estimatedValue=null` is CORRECT (not a stub) — the NetServer HTML table structurally has no such columns (confirmed reading `netserver.adapter.ts`: 6 cells = published/title/buyer/procedureType/legalFramework/deadline, no CPV). Full API suite green (288/288), tsc clean. | +| 2 | cosinex/DTVP (DTVP) erscheint als zusätzlicher Treffer über einen SEPARATEN config-getriebenen Adapter, **korrekt normalisiert** | ✓ VERIFIED (flipped) | Identical fix applies via the same `normalizeBag()` dispatch for `cosinex-dtvp`. Verified live: "cosinex-dtvp: maps an identical bag shape identically" test passes. `cosinex.adapter.ts`'s csx-new-table columns (published/deadline/title/type/buyer/action) likewise have no CPV column — `cpvCodes=[]` is correct, not a gap. | +| 3 | Ein über DÖE + einem Scraping-Adapter gesehener Tender erscheint EINMAL mit Links zu allen Quell-Portalen (Fuzzy-Fingerprint-Dedup); Dedup greift erst ab 2. aktiver Quelle | ⚠️ PARTIAL (still, for a NEW reason) | Mechanism (resolver, `dedupActive` gate, fan-out, backfill, multi-source UI) remains fully built/tested — unchanged from prior verification. The originally-identified blocker (title/buyer/deadline defaults) IS closed. BUT: `tenderFingerprint()` hashes `cpvDivisionKey` as an exact-match segment; NetServer/cosinex always produce `cpvDivisions=[]` (no CPV data exists in their HTML) while ~79% of existing DÖE tenders (2250/2851 live, `array_length>0`) have non-empty `cpvDivisions`. For any such tender, the scraper-side fingerprint hash will NEVER equal the DÖE-side hash, so the fingerprint tier will not collapse them — "shows up once" will fail for an estimated majority of real overlapping notices. No existing test exercises this asymmetry (the one cross-source fingerprint test in `tender-dedup.service.spec.ts` gives both synthetic records the same non-empty `cpvDivisions`). See Gaps. | +| 4 | Registrierung von vergabe24/aumass als Poll-Quelle wird vom System selbst (Code, nicht nur Doku) verweigert | ✓ VERIFIED (re-confirmed, no regression) | `source-registry.spec.ts` 6/6 pass (unchanged); `source-registry.ts` untouched by the quick-fix (only `tender-normalizer.service.ts`/`.spec.ts` were modified — confirmed via `git log` on the file). | -**Score:** 1/4 truths verified (0 present-behavior-unverified; 3 failed/partial, all rooted in one shared cause: the normalizer does not consume the generic `ocdsPayload` bag the two new adapters intentionally produce) +**Score:** 3/4 truths verified (0 present-behavior-unverified; 1 partial — mechanism fully built and now fed correct title/buyer/deadline data, but a newly-identified CPV-availability asymmetry in the fingerprint still blocks most real cross-source matches) -### Required Artifacts +### Required Artifacts (delta since prior verification) | Artifact | Expected | Status | Details | |----------|----------|--------|---------| -| `apps/api/src/tenders/tender-fingerprint.ts` | NULL-tolerant pure fingerprint fn | ✓ VERIFIED | 8/8 unit tests pass; sha256([buyer,title,cpvDivisionKey,deadlineKey,valueBucket]) | -| `apps/api/prisma/schema.prisma` (TenderSource, Tender.fingerprint) | 1:n model + additive column | ✓ VERIFIED | `model TenderSource` present, `@@unique([sourcePortal, sourceNoticeId])`, `Tender.fingerprint` nullable + `@@index`; migration applied locally | -| `apps/api/src/tenders/source-registry.ts` | Denylist gate | ✓ VERIFIED | `DeniedPortalError`, `DENYLISTED_PORTALS`, wired into `TendersModule.onModuleInit` | -| `apps/api/src/tenders/adapters/tender-source-adapter.interface.ts` | Generalized `portals[]` contract | ✓ VERIFIED | `readonly portals: readonly string[]` present; DÖE/NetServer/cosinex all declare it | -| `apps/api/src/tenders/tender-dedup.service.ts` | 3-tier resolver | ✓ VERIFIED | OCID → source:noticeId → fingerprint, `dedupActive` hard gate on tier 3, SCHEMA-02 field-refresh preserved | -| `apps/api/src/tenders/tender-ingestion.service.ts` (`pollDueSources`) | `findMany` fan-out, catch-per-source | ✓ VERIFIED | `prisma.tenderSourcePollConfig.findMany({where:{isActive:true}})`, try/catch per config inside the loop | -| `apps/api/src/tenders/adapters/netserver.adapter.ts` | Config-driven, 3 portals, 1 class | ✓ VERIFIED (as HTML-parser) / ✗ (as normalized-tender producer) | `portals = ['tender24','lhs-vpbw','vergabe.landbw']`; parse layer solid, but output not consumable by normalizer (see Truth 1) | -| `apps/api/src/tenders/adapters/cosinex.adapter.ts` | Separate single-portal adapter | ✓ VERIFIED (as HTML-parser) / ✗ (as normalized-tender producer) | `portals = ['cosinex-dtvp']`; same caveat as above (see Truth 2) | -| `apps/api/src/tenders/tenders.controller.ts` (`getTender`) | includes `sources[]` | ✓ VERIFIED | `include: { sources: { select: {...} } }`, route order unchanged (`:id` after all static routes) | -| `apps/web/.../TenderDetail.tsx` | Renders all source links | ✓ VERIFIED | `tender.sources.map(...)` with `portalLabel()`, fallback to legacy single `sourceUrl` block; 2 dedicated tests pass | +| `apps/api/src/tenders/tender-normalizer.service.ts` | Per-sourceType dispatch mapping the generic bag into `NormalizedTenderFields` | ✓ VERIFIED | `normalize()` switches on `raw.sourceType`; `normalizeBag()` present, `normalizeDoe()` unchanged (byte-identical DÖE extraction preserved under a renamed private method); shared `assemble()` extracted, `computeContentHash` not duplicated | +| `apps/api/src/tenders/tender-normalizer.service.spec.ts` | Tests for both bag sources + null fields + DÖE regression | ✓ VERIFIED | 14/14 tests pass (10 pre-existing DÖE + 4 new bag tests); DÖE fixture-based regression tests untouched and green | -### Key Link Verification +All other artifacts from the prior verification (fingerprint fn, schema, source-registry, adapters, dedup resolver, controller, TenderDetail) are unchanged by this quick task and were re-confirmed passing as part of the full-suite run below (no regression). + +### Key Link Verification (delta) | From | To | Via | Status | Details | |------|-----|-----|--------|---------| -| `netserver.adapter.ts` / `cosinex.adapter.ts` | `tender-normalizer.service.ts` | `RawTenderRecord.ocdsPayload` -> `normalize()` | ✗ NOT_WIRED (semantically) | Data physically flows (no crash), but `normalize()`'s field extraction contract (`.tender`/`.buyer`) does not match the flat bag shape the adapters produce — a silent, no-error data-quality break, not a compile/runtime error | -| `tender-ingestion.service.ts` (`pollDueSources`) | `source-registry.ts` | `registry.get(config.sourceType)` | ✓ WIRED | Confirmed in code + `tender-ingestion.service.spec.ts` fan-out tests | -| `tenders.module.ts` (`onModuleInit`) | `source-registry.ts` | `sourceRegistry.register(doeAdapter/netServerAdapter/cosinexAdapter)` | ✓ WIRED | All 3 adapters registered at DI boot; denylist gate structurally applies here | -| `tender-dedup.service.ts` | `prisma.tenderSource` / `prisma.tender` | `upsert`/`create` on match/no-match | ✓ WIRED | Composite-key upsert target verified by dedicated test | -| `tenders.controller.ts` (`getTender`) | `apps/web` `TenderDetail.tsx` | `sources[]` field on API response | ✓ WIRED | Backend include + frontend consumption both tested | +| `netserver.adapter.ts` / `cosinex.adapter.ts` | `tender-normalizer.service.ts` | `RawTenderRecord.ocdsPayload` -> `normalize()` | ✓ WIRED (flipped from NOT_WIRED) | `normalizeBag()` now correctly extracts the flat bag; verified by dedicated bag tests and by the field-authority correctness argument above (empty CPV is a correct reflection of absent source data, not a mapping bug) | +| `tender-normalizer.service.ts` (`normalizeBag` output) | `tender-fingerprint.ts` (`tenderFingerprint`) | `NormalizedTenderFields.cpvDivisions` -> `cpvDivisionKey()` | ⚠️ PARTIAL — semantically mismatched | Data flows without error, but the exact-match hash semantics mean a structurally-always-empty `cpvDivisions` on the scraper side will not collide with a populated `cpvDivisions` on the DÖE side for ~79% of existing tenders — see Gap | -### Behavioral Spot-Checks / Test Suite Results (run live during this verification) +### Behavioral Spot-Checks / Test Suite Results (run live during this re-verification) | Check | Command | Result | Status | |-------|---------|--------|--------| -| API full test suite | `cd apps/api && npx vitest run` | 25 files, **284/284 pass** | ✓ PASS | +| API full test suite | `cd apps/api && npx vitest run` | 25 files, **288/288 pass** (up from 284; +4 new bag-mapping tests) | ✓ PASS | | API typecheck | `cd apps/api && npx tsc --noEmit -p tsconfig.json` | clean, exit 0 | ✓ PASS | -| Web full test suite | `cd apps/web && npx vitest run` | 23 files, **133/133 pass** | ✓ PASS | -| Web typecheck | `cd apps/web && npx tsc --noEmit` | clean, exit 0 | ✓ PASS | -| Denylist refusal (source-registry.spec.ts) | (part of full suite) | 6/6 pass | ✓ PASS | -| Dedup resolver (tender-dedup.service.spec.ts) | (part of full suite) | 8/8 pass | ✓ PASS | -| Local DB — TenderSource backfill | `psql -c 'SELECT count(*) FROM "TenderSource"'` | 2851 | ✓ PASS (matches `SELECT count(*) FROM "Tender"` = 2851) | -| Local DB — fingerprint backfill | `psql -c 'SELECT count(*) FROM "Tender" WHERE fingerprint IS NOT NULL'` | 2851 | ✓ PASS | -| Local DB — poll configs seeded | `psql -c 'SELECT "sourceType","isActive" FROM "TenderSourcePollConfig"'` | Only `doe-opendata` row present | ⚠️ Local API process not yet restarted since Plans 13-04/13-05 landed — seed code for `ai-netserver`/`cosinex-dtvp` exists but hasn't run against this DB (see human_verification) | -| Debt markers (TBD/FIXME/XXX) in phase files | `grep -n -E "TBD|FIXME|XXX"` across 8 key files | none found | ✓ PASS | +| Normalizer spec only | (part of full suite) | `tender-normalizer.service.spec.ts` 14/14 pass, incl. new `describe('generic ocdsPayload bag ...')` block | ✓ PASS | +| Dedup resolver (unchanged) | (part of full suite) | `tender-dedup.service.spec.ts` 8/8 pass | ✓ PASS | +| Fingerprint fn (unchanged) | (part of full suite) | `tender-fingerprint.spec.ts` 8/8 pass | ✓ PASS | +| Denylist refusal (unchanged) | (part of full suite) | `source-registry.spec.ts` 6/6 pass | ✓ PASS | +| Live DB — CPV-empty ratio (new check, this re-verification) | `docker exec tessera-ctl-db-1 psql -U tessera -d tessera -c "SELECT count(*) total, count(*) FILTER (WHERE array_length(cpvDivisions,1) IS NULL OR array_length(cpvDivisions,1)=0) empty_cpv FROM Tender"` | 2851 total, 601 empty_cpv (~21%) | ⚠️ Confirms the fingerprint CPV-mismatch gap above — ~79% of existing tenders have non-empty cpvDivisions | +| Live DB — poll configs seeded (re-checked, unchanged) | `docker exec tessera-ctl-db-1 psql -U tessera -d tessera -c 'SELECT "sourceType","isActive" FROM "TenderSourcePollConfig"'` | Only `doe-opendata` row present | ⚠️ Still not restarted since Plans 13-04/13-05 — unaffected by this quick task, human_verification item carried forward unchanged | +| Debt markers (TBD/FIXME/XXX/TODO/HACK/PLACEHOLDER) in modified files | `grep -n -E "TBD|FIXME|XXX|TODO|HACK|PLACEHOLDER"` on `tender-normalizer.service.ts`/`.spec.ts` | none found | ✓ PASS | +| Git history confirms scope | `git log --oneline -- apps/api/src/tenders/tender-normalizer.service.ts` | `82c9a48 feat(quick-260723-e7i): per-sourceType dispatch in TenderNormalizerService` is the latest commit on the file | ✓ PASS | ### Requirements Coverage | Requirement | Source Plan | Description | Status | Evidence | |-------------|------------|-------------|--------|----------| -| INGEST-02 | 13-04 | Import von AI-AG-NetServer-Portalen über konfigurierbaren Adapter | ✗ BLOCKED | Adapter fetches/parses correctly, but records would not be usably imported (garbage-normalized) — see Truth 1 | -| INGEST-03 | 13-05 | Import vom cosinex-Vergabemarktplatz über Adapter | ✗ BLOCKED | Same root cause — see Truth 2 | -| INGEST-07 | 13-02 | vergabe24/aumass hart als Denylist, keine automatische Registrierung möglich | ✓ SATISFIED | `source-registry.spec.ts`, code-level `DeniedPortalError` | -| SCHEMA-03 | 13-01/03/06 | Dedup zu einem Eintrag mit mehreren Quell-Links; greift erst ab 2. aktiver Quelle | ⚠️ PARTIALLY SATISFIED | Schema, resolver, fan-out, backfill, and display are all correctly built and tested for the *mechanism*; the practical DÖE↔NetServer/cosinex cross-source match is blocked by the same normalizer gap (Truth 3) | +| INGEST-02 | 13-04 + quick-260723-e7i | Import von AI-AG-NetServer-Portalen über konfigurierbaren Adapter | ✓ SATISFIED | Adapter fetches/parses correctly AND records now normalize correctly (Truth 1 flipped) | +| INGEST-03 | 13-05 + quick-260723-e7i | Import vom cosinex-Vergabemarktplatz über Adapter | ✓ SATISFIED | Same fix applies (Truth 2 flipped) | +| INGEST-07 | 13-02 | vergabe24/aumass hart als Denylist, keine automatische Registrierung möglich | ✓ SATISFIED (re-confirmed) | Unchanged, 6/6 tests green | +| SCHEMA-03 | 13-01/03/06 + quick-260723-e7i | Dedup zu einem Eintrag mit mehreren Quell-Links; greift erst ab 2. aktiver Quelle | ⚠️ PARTIALLY SATISFIED (still) | Mechanism + now-correct title/buyer/deadline inputs are in place, but the CPV-availability asymmetry in the fingerprint blocks the practical "shows up once" outcome for most real DÖE↔scraper duplicates | -REQUIREMENTS.md currently marks all four as `[x]`/"Complete" — this verification finds INGEST-02 and INGEST-03 not actually satisfiable end-to-end yet, and SCHEMA-03 only partially achieved for the two new sources (DÖE-internal re-poll/change-detection dedup, which predates this phase, remains fully intact — see below). - -### SCHEMA-02 Regression Check - -Confirmed NOT regressed: `tender-dedup.service.spec.ts` has two dedicated tests ("OCID match with a changed contentHash still updates...", "...UNCHANGED contentHash does NOT call tender.update"), and the pre-existing `tender-ingestion.service.spec.ts` SCHEMA-02 suite (insert-once, update-in-place-on-changed-hash) was migrated to the new `resolve()` hand-off and remains green. The mutable-field list (title, buyerName, cpvCodes, cpvDivisions, region, plz, bundesland, deadlineAt, estimatedValue, procedureType, sourceUrl, contentHash) was copied verbatim from the old `tender.upsert` UPDATE branch. +REQUIREMENTS.md currently marks all four as `[x]`/"Complete" (Phase 13). This re-verification confirms INGEST-02 and INGEST-03 are now genuinely satisfiable end-to-end. SCHEMA-03 remains only partially achieved for the two new sources specifically because of the CPV-fingerprint asymmetry documented above — this is a narrower, more precise finding than the prior verification's broader "everything defaults" blocker, but it is not yet fully closed. ### Anti-Patterns Found -None — no `TBD`/`FIXME`/`XXX`/`TODO`/`HACK`/`PLACEHOLDER` markers in the 8 key phase files scanned. The normalizer gap is not a code-smell/stub pattern; it is a structurally absent capability, self-disclosed transparently in both SUMMARY.md files' "Known Stubs" sections rather than hidden. +None in the two files modified by the quick fix (`tender-normalizer.service.ts`, `tender-normalizer.service.spec.ts`) — no `TBD`/`FIXME`/`XXX`/`TODO`/`HACK`/`PLACEHOLDER` markers. ### Human Verification Required -1. **Real-world cross-source dedup confirmation** — Test: after the normalizer gap is fixed, activate `ai-netserver` or `cosinex-dtvp` (`TenderSourcePollConfig.isActive=true`), run one poll tick, and confirm a real known-overlapping tender collapses to one `Tender` row with two `TenderSource` links visible in `TenderDetail`. Expected: one results-list entry, two clickable source links. Why human: needs live portal access, a real overlapping notice, and a running scheduler tick — not provable from static code. -2. **Poll-config seed freshness** — Test: confirm the local API process has been rebuilt/restarted since Plans 13-04/13-05 so `ai-netserver`/`cosinex-dtvp` `TenderSourcePollConfig` rows actually exist locally (currently only `doe-opendata` is present in the DB). Why human: requires a deploy/restart action outside this verification's scope (per project convention, Claude does not restart local Docker services). +1. **Real-world cross-source dedup confirmation** (carried forward, refined) — after `ai-netserver`/`cosinex-dtvp` is activated and a poll tick runs, confirm a real overlapping notice collapses to one `Tender` with two `TenderSource` links. Given the CPV-asymmetry finding, this is now more likely to succeed specifically for a DÖE counterpart that happens to have no CPV code — not a general guarantee. Why human: needs live portal access, a real overlapping notice, and a running scheduler tick. +2. **Poll-config seed freshness** (carried forward, unchanged, re-checked live) — local API process still needs a rebuild/restart so `ai-netserver`/`cosinex-dtvp` `TenderSourcePollConfig` rows exist (confirmed via `docker exec psql`: still only 1 row). Why human: requires a deploy/restart action outside verification scope. ### Gaps Summary -Phase 13 built a genuinely solid, well-tested **core**: the `TenderSource` schema + 2851-row backfill, the NULL-tolerant fingerprint function, the `SourceRegistry` with a real code-level denylist gate (Success Criterion 4 fully achieved), the three-tier dedup resolver with `dedupActive` hard-gating (SCHEMA-03 mechanism), the `findMany` fan-out with catch-per-source isolation, and the multi-source-link display in `TenderDetail`. All of this is unit-tested (284/284 API, 133/133 web) and both typechecks are clean. +Quick task 260723-e7i successfully closed the root cause originally blocking Success Criteria 1 and 2: `TenderNormalizerService` now correctly maps the NetServer/cosinex flat `ocdsPayload` bag into real `title`/`buyerName`/`procedureType`/`deadlineAt`, confirmed by 4 new passing unit tests and a clean 288/288 full-suite run + clean `tsc`. Truths 1, 2, and 4 are now fully VERIFIED. -However, both new HTML adapters (`NetServerAdapter`, `CosinexAdapter`) deliberately punt on a required, self-documented follow-up: they carry their parsed fields in a flat generic `ocdsPayload` bag that `TenderNormalizerService` — hardcoded to the DÖE eForms/OCDS shape — cannot read. Both `13-04-SUMMARY.md` and `13-05-SUMMARY.md` disclose this transparently under "Known Stubs" as "out of this plan's `files_modified` scope" and "a natural follow-up plan" — but no such follow-up plan exists in Phase 13, and Phase 14's roadmap scope (RSS/email ingestion, admin source UI, i18n) does not cover it either. Since the poll configs for both new sources are seeded `isActive: false`, this gap is currently dormant in production — but it directly blocks 3 of the phase's 4 ROADMAP success criteria from being true as written ("appear as additional, **correctly normalized** results" x2, and real DÖE↔scraper cross-source dedup matching). Recommend a closure plan that extends `TenderNormalizerService` (or introduces a per-sourceType normalization strategy) to map the NetServer/cosinex generic bag into `NormalizedTenderFields` before this phase can be considered to have achieved its stated goal. +Success Criterion 3 (cross-source dedup) is closer to achieved than before, but this re-verification surfaced a **new, previously-undetected** residual issue distinct from the one just fixed: `tenderFingerprint()` requires an EXACT match on `cpvDivisionKey`, and NetServer/cosinex structurally can never populate `cpvDivisions` (there is no CPV column in either portal's search-results HTML). A live DB check shows ~79% of existing DÖE tenders carry non-empty `cpvDivisions`, meaning the fingerprint tier will fail to collapse the majority of real DÖE↔NetServer/cosinex duplicates even with correct title/buyer/deadline data. This is provable from code + live DB stats, not something requiring live portal access — it is a developer decision (e.g., treat an empty `cpvDivisions` as a wildcard segment rather than requiring exact equality) that has not yet been made. Recommend a small follow-up plan or an explicit accepted-limitation override before closing SCHEMA-03/Success Criterion 3 for good. + +The two previously-identified human_verification items (live overlap confirmation once a second source is active; local poll-config seed freshness) are unaffected by the quick fix and are carried forward unchanged — re-checked live in this session, the local DB still shows only the `doe-opendata` poll config row. --- -*Verified: 2026-07-23T10:15:00Z* +*Verified: 2026-07-23T10:30:00Z* *Verifier: Claude (gsd-verifier)*