tessera-ctl/.planning/phases/13-scraping-adapters-cross-source-dedup/13-VERIFICATION.md

phase, verified, status, score, behavior_unverified, overrides_applied, re_verification, gaps, deferred, human_verification, residual_gap_decision

phase	verified	status	score	behavior_unverified	overrides_applied	re_verification	gaps	deferred	human_verification	residual_gap_decision
13-scraping-adapters-cross-source-dedup	2026-07-23T10:30:00Z	gaps_found	3/4 must-haves verified	0	0	previous_status previous_score gaps_closed gaps_remaining regressions gaps_found 1/4 Tenders from AI-AG NetServer portals appear as additional, correctly normalized results via one config-driven adapter Tenders from the cosinex Vergabemarktplatz (DTVP) appear as additional, correctly normalized results via a separate config-driven adapter A tender that appears via both DÖE and a scraping adapter shows up once in the results list (fuzzy fingerprint dedup on buyer+title+CPV+deadline+value) — NEW residual finding: cpvDivisions asymmetry (scraper always [], DÖE usually populated) makes the fingerprint tier fail to collapse the majority of real cross-source duplicates even after the normalizer fix	truth status reason artifacts missing A tender that appears via both DÖE and a scraping adapter shows up once in the results list (fuzzy fingerprint dedup on buyer+title+CPV+deadline+value), with links to all of its source portals -- dedup logic only activates once a second source is live partial The previously-identified root cause (NetServer/cosinex normalizing to title/buyerName/ deadlineAt defaults) is CLOSED by quick task 260723-e7i — verified live in this re-verification (see Truth table). However, a SEPARATE, previously-undetected issue remains: tenderFingerprint()'s canonical string is a straight concatenation of [buyerName, title, cpvDivisionKey, deadlineKey, valueBucket] hashed with sha256 — it is an EXACT match on all five segments, not a weighted/fuzzy comparison despite the "Hauptgewicht" (main weight) language in 13-RESEARCH.md. NetServerAdapter and CosinexAdapter structurally cannot supply CPV codes (verified by reading both adapters' HTML column extraction: neither the NetServer 6-column table nor the cosinex csx-new-table has a CPV column at all), so TenderNormalizerService.normalizeBag() correctly and necessarily always sets cpvDivisions=[] for these two sources (this is not a normalizer bug — there is nothing to extract). Live DB check (`SELECT count(*) FILTER (WHERE array_length("cpvDivisions",1) IS NULL OR array_length("cpvDivisions",1)=0)` -> 601/2851 = ~21%) shows ~79% of existing DÖE-sourced tenders carry a non-empty cpvDivisions. Because cpvDivisionKey('') != cpvDivisionKey(['45']) (or any other non-empty division), the sha256 hash differs whenever the DÖE-side counterpart has CPV codes -- i.e. for an estimated ~79% of real overlapping tenders, the fingerprint tier will NOT collapse the DÖE and NetServer/cosinex records into one Tender, even once a second source is activated and even though title/buyerName/deadlineAt now correctly match. tender-dedup.service.spec.ts's only cross-source fingerprint test (`fingerprint match (no OCID/noticeId overlap)`) does not exercise this asymmetry -- it spreads `...BASE` (cpvDivisions: ['45']) into both synthetic records, so both sides always share the same non-empty CPV segment. No test in the suite proves a NetServer/cosinex record (cpvDivisions=[]) actually collides with its real DÖE counterpart (cpvDivisions=['45'] or similar). path issue apps/api/src/tenders/tender-fingerprint.ts cpvDivisionKey is one of 5 hashed segments with no null/empty-tolerant matching for the CPV dimension specifically (unlike deadlineAt/estimatedValue, which ARE null-tolerant by design) -- an empty array on one side and a populated array on the other always produces a different hash path issue apps/api/src/tenders/tender-normalizer.service.ts normalizeBag() correctly sets cpvDivisions=[] (there is no CPV data in the scraped HTML to map) -- this is the correct behavior for the available source data, but it is the direct cause of the fingerprint mismatch described above A developer decision on how to handle the CPV-availability asymmetry for fingerprint matching: e.g. treat an empty cpvDivisions on either side as a wildcard/skip for that segment instead of an exact-match requirement, or accept the reduced (~21%) practical match rate for NetServer/cosinex cross-source dedup as a known limitation of these two sources A live/manual confirmation once ai-netserver or cosinex-dtvp poll config is activated (TenderSourcePollConfig.isActive=true) and a real overlapping tender is observed to merge into one entry with two source links (unchanged from prior verification -- still not possible to test locally, see Human Verification)		test expected why_human Activate the ai-netserver or cosinex-dtvp TenderSourcePollConfig row (isActive=true), let pollDueSources run one tick, and confirm in the DB/UI that a tender known to exist on both DÖE and the new source collapses to ONE Tender with two TenderSource rows/links in TenderDetail. Given the CPV-asymmetry finding above, prioritize testing with a real overlapping notice that happens to have NO CPV code on the DÖE side, since that is currently the only case the fingerprint tier can realistically catch for these two sources. One results-list entry; TenderDetail shows both a DÖE link and a tender24/lhs-vpbw/vergabe.landbw/DTVP link — but only reliably for DÖE counterparts with no CPV code, per the gap above Requires live network access to the real portals, a real overlapping notice, and a running scheduler tick against a real DB — cannot be proven by static code/unit-test inspection alone test expected why_human Confirm local API container was rebuilt/restarted so the ai-netserver/cosinex-dtvp TenderSourcePollConfig seed rows actually exist in the local DB SELECT sourceType, isActive FROM TenderSourcePollConfig shows 3 rows (doe-opendata, ai-netserver, cosinex-dtvp), the latter two isActive=false Re-checked live in this verification (docker exec psql): still only 1 row (doe-opendata). The API process has not been restarted/rebuilt since Plans 13-04/13-05 landed -- unrelated to and not resolved by the normalizer quick-fix. Per project convention, Claude does not restart local Docker services.	date chosen by rationale 2026-07-23 C — accept as dormant deferral (no code change now) user The cpvDivisions fingerprint asymmetry (Truth 3) is real but the whole cross-source dedup mechanism is currently inert: both scraper poll configs are seeded isActive=false, so only DÖE runs and the fingerprint tier is never exercised in production. Whether CPV-less matching (Option A: a second buyer+title+deadline fingerprint tier) has acceptable precision or produces too many false merges can only be judged against real overlapping notices, not statically. Decision deferred to when a second source is actually activated. Options considered: A = CPV-less second fingerprint tier (new column + migration + backfill 2851 + 4th resolver tier); B = drop CPV+value from the primary fingerprint (simple, higher false-merge risk); C = leave dormant (chosen). Truth 3 stays PARTIAL by design, not by neglect.

Phase 13: Scraping Adapters & Cross-Source Deduplication Verification Report

Phase Goal: Die Plattform erweitert die Ausschreibungs-Abdeckung um AI-AG-NetServer- und cosinex-Portal-Adapter und fasst Ausschreibungen, die auf mehreren Quellen auftauchen, zu einem Eintrag zusammen — während sie das Scrapen der AGB-verbotenen Portale strukturell verweigert. Verified: 2026-07-23T10:30:00Z Status: gaps_found Re-verification: Yes — after gap closure (quick task 260723-e7i)

What Changed Since Last Verification

Quick task 260723-e7i (commits 82c9a48, 9881005) added a per-sourceType dispatch to TenderNormalizerService.normalize(). Read live in this re-verification (apps/api/src/tenders/tender-normalizer.service.ts):

• normalize() switches on raw.sourceType: ai-netserver/cosinex-dtvp → normalizeBag(); doe-opendata/default → normalizeDoe() (default never throws).
• normalizeBag() reads the flat ocdsPayload bag {title, buyerName, procedureType, legalFramework, deadlineAt} and maps title (fallback 'Unbenannte Ausschreibung'), buyerName/procedureType (null-tolerant, no invented fallback), deadlineAt via the existing parseOcdsDate(). cpvCodes/cpvDivisions = [], region/plz/bundesland/ estimatedValue = null (there is genuinely no such data in the NetServer/cosinex HTML — confirmed by reading both adapters' column extraction).
• A shared assemble() computes status/dedupKey/contentHash/publishedAt identically for both paths — normalizeDoe() is a byte-identical extraction of the pre-existing DÖE logic.

This closes the previously-identified root cause for Truths 1 and 2 — NetServer and cosinex records now carry their real title/buyer/procedureType/deadline instead of falling through to defaults. It partially closes Truth 3: title/buyer/deadline now flow correctly into the fingerprint, but re-verification surfaced a new, previously-undetected issue specific to the CPV dimension of the fingerprint that still blocks most real-world cross-source matches (see Gap below).

Goal Achievement

Observable Truths (ROADMAP Success Criteria)

#	Truth	Status	Evidence
1	NetServer-Portale (lhs-vpbw, tender24, vergabe.landbw) erscheinen als zusätzliche, korrekt normalisierte Treffer über EINEN config-getriebenen Adapter	✓ VERIFIED (flipped)	normalizeBag() now maps ai-netserver bag records into real title/buyerName/procedureType/deadlineAt (verified live: tender-normalizer.service.spec.ts "ai-netserver: maps the flat bag into NormalizedTenderFields" test asserts title==='Bauleistung X', buyerName==='Stadt Y', deadlineAt ISO round-trip). cpvCodes=[]/region/plz/bundesland/estimatedValue=null is CORRECT (not a stub) — the NetServer HTML table structurally has no such columns (confirmed reading netserver.adapter.ts: 6 cells = published/title/buyer/procedureType/legalFramework/deadline, no CPV). Full API suite green (288/288), tsc clean.
2	cosinex/DTVP (DTVP) erscheint als zusätzlicher Treffer über einen SEPARATEN config-getriebenen Adapter, korrekt normalisiert	✓ VERIFIED (flipped)	Identical fix applies via the same normalizeBag() dispatch for cosinex-dtvp. Verified live: "cosinex-dtvp: maps an identical bag shape identically" test passes. cosinex.adapter.ts's csx-new-table columns (published/deadline/title/type/buyer/action) likewise have no CPV column — cpvCodes=[] is correct, not a gap.
3	Ein über DÖE + einem Scraping-Adapter gesehener Tender erscheint EINMAL mit Links zu allen Quell-Portalen (Fuzzy-Fingerprint-Dedup); Dedup greift erst ab 2. aktiver Quelle	⚠️ PARTIAL (still, for a NEW reason)	Mechanism (resolver, dedupActive gate, fan-out, backfill, multi-source UI) remains fully built/tested — unchanged from prior verification. The originally-identified blocker (title/buyer/deadline defaults) IS closed. BUT: tenderFingerprint() hashes cpvDivisionKey as an exact-match segment; NetServer/cosinex always produce cpvDivisions=[] (no CPV data exists in their HTML) while ~79% of existing DÖE tenders (2250/2851 live, array_length>0) have non-empty cpvDivisions. For any such tender, the scraper-side fingerprint hash will NEVER equal the DÖE-side hash, so the fingerprint tier will not collapse them — "shows up once" will fail for an estimated majority of real overlapping notices. No existing test exercises this asymmetry (the one cross-source fingerprint test in tender-dedup.service.spec.ts gives both synthetic records the same non-empty cpvDivisions). See Gaps.
4	Registrierung von vergabe24/aumass als Poll-Quelle wird vom System selbst (Code, nicht nur Doku) verweigert	✓ VERIFIED (re-confirmed, no regression)	source-registry.spec.ts 6/6 pass (unchanged); source-registry.ts untouched by the quick-fix (only tender-normalizer.service.ts/.spec.ts were modified — confirmed via git log on the file).

Score: 3/4 truths verified (0 present-behavior-unverified; 1 partial — mechanism fully built and now fed correct title/buyer/deadline data, but a newly-identified CPV-availability asymmetry in the fingerprint still blocks most real cross-source matches)

Required Artifacts (delta since prior verification)

Artifact	Expected	Status	Details
apps/api/src/tenders/tender-normalizer.service.ts	Per-sourceType dispatch mapping the generic bag into NormalizedTenderFields	✓ VERIFIED	normalize() switches on raw.sourceType; normalizeBag() present, normalizeDoe() unchanged (byte-identical DÖE extraction preserved under a renamed private method); shared assemble() extracted, computeContentHash not duplicated
apps/api/src/tenders/tender-normalizer.service.spec.ts	Tests for both bag sources + null fields + DÖE regression	✓ VERIFIED	14/14 tests pass (10 pre-existing DÖE + 4 new bag tests); DÖE fixture-based regression tests untouched and green

All other artifacts from the prior verification (fingerprint fn, schema, source-registry, adapters, dedup resolver, controller, TenderDetail) are unchanged by this quick task and were re-confirmed passing as part of the full-suite run below (no regression).

Key Link Verification (delta)

From	To	Via	Status	Details
netserver.adapter.ts / cosinex.adapter.ts	tender-normalizer.service.ts	RawTenderRecord.ocdsPayload -> normalize()	✓ WIRED (flipped from NOT_WIRED)	normalizeBag() now correctly extracts the flat bag; verified by dedicated bag tests and by the field-authority correctness argument above (empty CPV is a correct reflection of absent source data, not a mapping bug)
tender-normalizer.service.ts (normalizeBag output)	tender-fingerprint.ts (tenderFingerprint)	NormalizedTenderFields.cpvDivisions -> cpvDivisionKey()	⚠️ PARTIAL — semantically mismatched	Data flows without error, but the exact-match hash semantics mean a structurally-always-empty cpvDivisions on the scraper side will not collide with a populated cpvDivisions on the DÖE side for ~79% of existing tenders — see Gap

Behavioral Spot-Checks / Test Suite Results (run live during this re-verification)

Check	Command	Result	Status
API full test suite	cd apps/api && npx vitest run	25 files, 288/288 pass (up from 284; +4 new bag-mapping tests)	✓ PASS
API typecheck	cd apps/api && npx tsc --noEmit -p tsconfig.json	clean, exit 0	✓ PASS
Normalizer spec only	(part of full suite)	tender-normalizer.service.spec.ts 14/14 pass, incl. new describe('generic ocdsPayload bag ...') block	✓ PASS
Dedup resolver (unchanged)	(part of full suite)	tender-dedup.service.spec.ts 8/8 pass	✓ PASS
Fingerprint fn (unchanged)	(part of full suite)	tender-fingerprint.spec.ts 8/8 pass	✓ PASS
Denylist refusal (unchanged)	(part of full suite)	source-registry.spec.ts 6/6 pass	✓ PASS
Live DB — CPV-empty ratio (new check, this re-verification)	docker exec tessera-ctl-db-1 psql -U tessera -d tessera -c "SELECT count(*) total, count(*) FILTER (WHERE array_length(cpvDivisions,1) IS NULL OR array_length(cpvDivisions,1)=0) empty_cpv FROM Tender"	2851 total, 601 empty_cpv (~21%)	⚠️ Confirms the fingerprint CPV-mismatch gap above — ~79% of existing tenders have non-empty cpvDivisions
Live DB — poll configs seeded (re-checked, unchanged)	docker exec tessera-ctl-db-1 psql -U tessera -d tessera -c 'SELECT "sourceType","isActive" FROM "TenderSourcePollConfig"'	Only doe-opendata row present	⚠️ Still not restarted since Plans 13-04/13-05 — unaffected by this quick task, human_verification item carried forward unchanged
Debt markers (TBD/FIXME/XXX/TODO/HACK/PLACEHOLDER) in modified files	`grep -n -E "TBD	FIXME	XXX
Git history confirms scope	git log --oneline -- apps/api/src/tenders/tender-normalizer.service.ts	82c9a48 feat(quick-260723-e7i): per-sourceType dispatch in TenderNormalizerService is the latest commit on the file	✓ PASS

Requirements Coverage

Requirement	Source Plan	Description	Status	Evidence
INGEST-02	13-04 + quick-260723-e7i	Import von AI-AG-NetServer-Portalen über konfigurierbaren Adapter	✓ SATISFIED	Adapter fetches/parses correctly AND records now normalize correctly (Truth 1 flipped)
INGEST-03	13-05 + quick-260723-e7i	Import vom cosinex-Vergabemarktplatz über Adapter	✓ SATISFIED	Same fix applies (Truth 2 flipped)
INGEST-07	13-02	vergabe24/aumass hart als Denylist, keine automatische Registrierung möglich	✓ SATISFIED (re-confirmed)	Unchanged, 6/6 tests green
SCHEMA-03	13-01/03/06 + quick-260723-e7i	Dedup zu einem Eintrag mit mehreren Quell-Links; greift erst ab 2. aktiver Quelle	⚠️ PARTIALLY SATISFIED (still)	Mechanism + now-correct title/buyer/deadline inputs are in place, but the CPV-availability asymmetry in the fingerprint blocks the practical "shows up once" outcome for most real DÖE↔scraper duplicates

REQUIREMENTS.md currently marks all four as [x]/"Complete" (Phase 13). This re-verification confirms INGEST-02 and INGEST-03 are now genuinely satisfiable end-to-end. SCHEMA-03 remains only partially achieved for the two new sources specifically because of the CPV-fingerprint asymmetry documented above — this is a narrower, more precise finding than the prior verification's broader "everything defaults" blocker, but it is not yet fully closed.

Anti-Patterns Found

None in the two files modified by the quick fix (tender-normalizer.service.ts, tender-normalizer.service.spec.ts) — no TBD/FIXME/XXX/TODO/HACK/PLACEHOLDER markers.

Human Verification Required

1. Real-world cross-source dedup confirmation (carried forward, refined) — after ai-netserver/cosinex-dtvp is activated and a poll tick runs, confirm a real overlapping notice collapses to one Tender with two TenderSource links. Given the CPV-asymmetry finding, this is now more likely to succeed specifically for a DÖE counterpart that happens to have no CPV code — not a general guarantee. Why human: needs live portal access, a real overlapping notice, and a running scheduler tick.
2. Poll-config seed freshness (carried forward, unchanged, re-checked live) — local API process still needs a rebuild/restart so ai-netserver/cosinex-dtvp TenderSourcePollConfig rows exist (confirmed via docker exec psql: still only 1 row). Why human: requires a deploy/restart action outside verification scope.

Gaps Summary

Quick task 260723-e7i successfully closed the root cause originally blocking Success Criteria 1 and 2: TenderNormalizerService now correctly maps the NetServer/cosinex flat ocdsPayload bag into real title/buyerName/procedureType/deadlineAt, confirmed by 4 new passing unit tests and a clean 288/288 full-suite run + clean tsc. Truths 1, 2, and 4 are now fully VERIFIED.

Success Criterion 3 (cross-source dedup) is closer to achieved than before, but this re-verification surfaced a new, previously-undetected residual issue distinct from the one just fixed: tenderFingerprint() requires an EXACT match on cpvDivisionKey, and NetServer/cosinex structurally can never populate cpvDivisions (there is no CPV column in either portal's search-results HTML). A live DB check shows ~79% of existing DÖE tenders carry non-empty cpvDivisions, meaning the fingerprint tier will fail to collapse the majority of real DÖE↔NetServer/cosinex duplicates even with correct title/buyer/deadline data. This is provable from code + live DB stats, not something requiring live portal access — it is a developer decision (e.g., treat an empty cpvDivisions as a wildcard segment rather than requiring exact equality) that has not yet been made. Recommend a small follow-up plan or an explicit accepted-limitation override before closing SCHEMA-03/Success Criterion 3 for good.

The two previously-identified human_verification items (live overlap confirmation once a second source is active; local poll-config seed freshness) are unaffected by the quick fix and are carried forward unchanged — re-checked live in this session, the local DB still shows only the doe-opendata poll config row.

---

Verified: 2026-07-23T10:30:00Z Verifier: Claude (gsd-verifier)