tessera-ctl/.planning/phases/13-scraping-adapters-cross-source-dedup/13-VERIFICATION.md

phase, verified, status, score, behavior_unverified, overrides_applied, gaps, deferred, human_verification

phase	verified	status	score	behavior_unverified	overrides_applied	gaps	deferred	human_verification
13-scraping-adapters-cross-source-dedup	2026-07-23T10:15:00Z	gaps_found	1/4 must-haves verified	0	0	truth status reason artifacts missing Tenders from AI-AG NetServer portals (lhs-vpbw, tender24, vergabe.landbw) appear as additional, correctly normalized results via one config-driven adapter failed NetServerAdapter correctly parses NetServer HTML into RawTenderRecord[] (14/14 unit tests pass), but the shared TenderNormalizerService.normalize() only understands the DÖE eForms-DE/OCDS payload shape (getOcdsTender()/getOcdsBuyer() look for `.tender`/`.buyer` keys). NetServerAdapter deliberately puts a FLAT generic bag `{title, buyerName, procedureType, legalFramework, deadlineAt}` into `ocdsPayload` (own code comment: "NetServer has no eForms/OCDS structure ... a future normalizer extension (out of scope for this plan) can map them"). Because the bag has no `.tender`/`.buyer` keys, every NetServer record would normalize to title='Unbenannte Ausschreibung', buyerName=null, cpvCodes=[], deadlineAt=null, estimatedValue=null — i.e. NOT "correctly normalized". This is explicitly self-documented as "Known Stubs" in 13-04-SUMMARY.md, not resolved anywhere in Phase 13, and not scheduled in Phase 14 (Phase 14 scope per ROADMAP is RSS/email/admin-UI/i18n, no normalizer work). path issue apps/api/src/tenders/tender-normalizer.service.ts getOcdsTender()/getOcdsBuyer() only read `.tender`/`.buyer` keys; has no branch for the generic ocdsPayload bag NetServerAdapter/CosinexAdapter produce path issue apps/api/src/tenders/adapters/netserver.adapter.ts Puts {title, buyerName, procedureType, legalFramework, deadlineAt} directly into ocdsPayload (own comment marks this as needing a future normalizer extension) Extend TenderNormalizerService (or add a per-sourceType normalization strategy) so ai-netserver's generic ocdsPayload bag maps to real title/buyerName/deadlineAt/procedureType instead of falling through to defaults truth status reason artifacts missing Tenders from the cosinex Vergabemarktplatz (DTVP) appear as additional, correctly normalized results via a separate config-driven adapter failed Identical root cause to the NetServer gap above. CosinexAdapter correctly parses the live cosinex/DTVP results table into RawTenderRecord[] (16/16 unit tests pass, including a real ISO-8859-1 charset-decode fix), but puts the same generic {title, buyerName, procedureType, legalFramework, deadlineAt} bag into ocdsPayload. TenderNormalizerService does not understand this shape either, so cosinex records would normalize incorrectly for the same reason. Self-documented as "Known Stubs" in 13-05-SUMMARY.md, not resolved in this phase, not scheduled in Phase 14. path issue apps/api/src/tenders/adapters/cosinex.adapter.ts Same generic ocdsPayload bag convention as NetServerAdapter — normalizer cannot map it Same normalizer extension as the NetServer gap, ideally handling both bag-shaped sources together truth status reason artifacts missing A tender that appears via both DÖE and a scraping adapter shows up once in the results list (fuzzy fingerprint dedup on buyer+title+CPV+deadline+value), with links to all of its source portals -- dedup logic only activates once a second source is live partial The DEDUP MECHANISM itself is solidly built and unit-tested: TenderDedupService's three-tier resolver (OCID -> source:noticeId -> fingerprint), the dedupActive=(activePortalCount>=2) gate, pollDueSources' findMany fan-out with catch-per-source isolation, the 2851-row backfill, and the TenderDetail multi-source-link UI are all present, wired, and covered by passing tests (tender-dedup.service.spec.ts 8/8, tender-ingestion.service.spec.ts fan-out tests, tenders.controller.spec.ts, TenderDetail.test.tsx). HOWEVER, the truth as stated ("a tender that appears via both DÖE and a scraping adapter shows up once ... fuzzy fingerprint dedup on buyer+title+CPV+deadline+value") is undermined transitively by the two gaps above: because NetServer/cosinex records normalize with a generic/empty title+buyer+deadline ('Unbenannte Ausschreibung', null buyer, null deadline), their computed fingerprint will NOT match the fingerprint of the corresponding real DÖE tender (which has the actual title/buyer/deadline). So even once a second source is activated, real-world DÖE<->NetServer or DÖE<->cosinex duplicates will NOT be recognized as the same tender — the fingerprint tier will practically never produce a true match for these two sources. The mechanism is verified in isolation with well-formed NormalizedTenderFields; the end-to-end achievement described by the ROADMAP truth is not currently achievable for the two new adapters because of the normalizer gap. path issue apps/api/src/tenders/tender-dedup.service.ts Correct in isolation; fingerprint computed from NormalizedTenderFields, which will be malformed for NetServer/cosinex until the normalizer gap above is closed Same normalizer fix as gaps 1/2 is a hard prerequisite for this truth to hold end-to-end for any pair other than DÖE<->DÖE A live/manual confirmation once ai-netserver or cosinex-dtvp poll config is activated (TenderSourcePollConfig.isActive=true) and a real overlapping tender is observed to merge into one entry with two source links		test expected why_human Activate the ai-netserver or cosinex-dtvp TenderSourcePollConfig row (isActive=true) after the normalizer gap is fixed, let pollDueSources run one tick, and confirm in the DB/UI that a tender known to exist on both DÖE and the new source collapses to ONE Tender with two TenderSource rows/links in TenderDetail One results-list entry; TenderDetail shows both a DÖE link and a tender24/lhs-vpbw/vergabe.landbw/DTVP link Requires live network access to the real portals, a real overlapping notice, and a running scheduler tick against a real DB — cannot be proven by static code/unit-test inspection alone test expected why_human Confirm local API container was rebuilt/restarted after Plans 13-04/13-05 so the ai-netserver/cosinex-dtvp TenderSourcePollConfig seed rows actually exist in the local DB SELECT sourceType, isActive FROM TenderSourcePollConfig shows 3 rows (doe-opendata, ai-netserver, cosinex-dtvp), the latter two isActive=false As of this verification the local dev DB only has the doe-opendata row — onModuleInit's seed code exists and is correct, but the API process has not been restarted/rebuilt to run it since Plans 13-04/13-05 landed

Phase 13: Scraping Adapters & Cross-Source Deduplication Verification Report

Phase Goal: Die Plattform erweitert die Ausschreibungs-Abdeckung um AI-AG-NetServer- und cosinex-Portal-Adapter und fasst Ausschreibungen, die auf mehreren Quellen auftauchen, zu einem Eintrag zusammen — während sie das Scrapen der AGB-verbotenen Portale strukturell verweigert. Verified: 2026-07-23T10:15:00Z Status: gaps_found Re-verification: No — initial verification

Goal Achievement

Observable Truths (ROADMAP Success Criteria)

#	Truth	Status	Evidence
1	NetServer-Portale (lhs-vpbw, tender24, vergabe.landbw) erscheinen als zusätzliche, korrekt normalisierte Treffer über EINEN config-getriebenen Adapter	✗ FAILED	netserver.adapter.ts parst HTML korrekt zu RawTenderRecord[] (14/14 Tests grün), aber tender-normalizer.service.ts versteht die generische ocdsPayload-Bag nicht (getOcdsTender()/getOcdsBuyer() lesen nur .tender/.buyer) — jeder NetServer-Datensatz würde zu title='Unbenannte Ausschreibung', buyerName=null, cpvCodes=[], deadlineAt=null normalisiert. Selbst dokumentiert als "Known Stubs" in 13-04-SUMMARY.md.
2	cosinex/DTVP (DTVP) erscheint als zusätzlicher Treffer über einen SEPARATEN config-getriebenen Adapter, korrekt normalisiert	✗ FAILED	Gleiche Grundursache wie #1: cosinex.adapter.ts parst korrekt (16/16 Tests grün, inkl. echtem ISO-8859-1-Encoding-Fix), aber dieselbe generische Bag wird vom Normalizer nicht verstanden. Selbst dokumentiert als "Known Stubs" in 13-05-SUMMARY.md.
3	Ein über DÖE + einem Scraping-Adapter gesehener Tender erscheint EINMAL mit Links zu allen Quell-Portalen (Fuzzy-Fingerprint-Dedup); Dedup greift erst ab 2. aktiver Quelle	⚠️ PARTIAL (siehe gaps)	Resolver-Mechanik (tender-dedup.service.ts, 3-Stufen, dedupActive-Gate), Fan-out (pollDueSources via findMany), Backfill (2851/2851) und Multi-Source-UI (TenderDetail.tsx) sind alle vorhanden, verdrahtet und isoliert getestet — VERIFIZIERT auf Mechanik-Ebene. ABER: da NetServer/cosinex-Datensätze wegen Truth 1/2 mit generischem/leerem Titel+Buyer+Frist normalisiert würden, würde ihr Fingerprint NIE mit dem echten DÖE-Gegenstück kollidieren — reale Cross-Source-Dedup-Treffer zwischen DÖE↔NetServer/cosinex sind praktisch unerreichbar, bis der Normalizer-Gap geschlossen ist.
4	Registrierung von vergabe24/aumass als Poll-Quelle wird vom System selbst (Code, nicht nur Doku) verweigert	✓ VERIFIED	source-registry.ts: SourceRegistry.register() iteriert adapter.portals und wirft DeniedPortalError bei jedem Treffer gegen DENYLISTED_PORTALS = ['vergabe24','aumass'], auch bei gemischtem Array (kein Teil-Register). 6/6 Tests grün, inkl. explizitem vergabe24-Test, aumass-Test und Mixed-Array-Test.

Score: 1/4 truths verified (0 present-behavior-unverified; 3 failed/partial, all rooted in one shared cause: the normalizer does not consume the generic ocdsPayload bag the two new adapters intentionally produce)

Required Artifacts

Artifact	Expected	Status	Details
apps/api/src/tenders/tender-fingerprint.ts	NULL-tolerant pure fingerprint fn	✓ VERIFIED	8/8 unit tests pass; sha256([buyer,title,cpvDivisionKey,deadlineKey,valueBucket])
apps/api/prisma/schema.prisma (TenderSource, Tender.fingerprint)	1:n model + additive column	✓ VERIFIED	model TenderSource present, @@unique([sourcePortal, sourceNoticeId]), Tender.fingerprint nullable + @@index; migration applied locally
apps/api/src/tenders/source-registry.ts	Denylist gate	✓ VERIFIED	DeniedPortalError, DENYLISTED_PORTALS, wired into TendersModule.onModuleInit
apps/api/src/tenders/adapters/tender-source-adapter.interface.ts	Generalized portals[] contract	✓ VERIFIED	readonly portals: readonly string[] present; DÖE/NetServer/cosinex all declare it
apps/api/src/tenders/tender-dedup.service.ts	3-tier resolver	✓ VERIFIED	OCID → source:noticeId → fingerprint, dedupActive hard gate on tier 3, SCHEMA-02 field-refresh preserved
apps/api/src/tenders/tender-ingestion.service.ts (pollDueSources)	findMany fan-out, catch-per-source	✓ VERIFIED	prisma.tenderSourcePollConfig.findMany({where:{isActive:true}}), try/catch per config inside the loop
apps/api/src/tenders/adapters/netserver.adapter.ts	Config-driven, 3 portals, 1 class	✓ VERIFIED (as HTML-parser) / ✗ (as normalized-tender producer)	portals = ['tender24','lhs-vpbw','vergabe.landbw']; parse layer solid, but output not consumable by normalizer (see Truth 1)
apps/api/src/tenders/adapters/cosinex.adapter.ts	Separate single-portal adapter	✓ VERIFIED (as HTML-parser) / ✗ (as normalized-tender producer)	portals = ['cosinex-dtvp']; same caveat as above (see Truth 2)
apps/api/src/tenders/tenders.controller.ts (getTender)	includes sources[]	✓ VERIFIED	include: { sources: { select: {...} } }, route order unchanged (:id after all static routes)
apps/web/.../TenderDetail.tsx	Renders all source links	✓ VERIFIED	tender.sources.map(...) with portalLabel(), fallback to legacy single sourceUrl block; 2 dedicated tests pass

Key Link Verification

From	To	Via	Status	Details
netserver.adapter.ts / cosinex.adapter.ts	tender-normalizer.service.ts	RawTenderRecord.ocdsPayload -> normalize()	✗ NOT_WIRED (semantically)	Data physically flows (no crash), but normalize()'s field extraction contract (.tender/.buyer) does not match the flat bag shape the adapters produce — a silent, no-error data-quality break, not a compile/runtime error
tender-ingestion.service.ts (pollDueSources)	source-registry.ts	registry.get(config.sourceType)	✓ WIRED	Confirmed in code + tender-ingestion.service.spec.ts fan-out tests
tenders.module.ts (onModuleInit)	source-registry.ts	sourceRegistry.register(doeAdapter/netServerAdapter/cosinexAdapter)	✓ WIRED	All 3 adapters registered at DI boot; denylist gate structurally applies here
tender-dedup.service.ts	prisma.tenderSource / prisma.tender	upsert/create on match/no-match	✓ WIRED	Composite-key upsert target verified by dedicated test
tenders.controller.ts (getTender)	apps/web TenderDetail.tsx	sources[] field on API response	✓ WIRED	Backend include + frontend consumption both tested

Behavioral Spot-Checks / Test Suite Results (run live during this verification)

Check	Command	Result	Status
API full test suite	cd apps/api && npx vitest run	25 files, 284/284 pass	✓ PASS
API typecheck	cd apps/api && npx tsc --noEmit -p tsconfig.json	clean, exit 0	✓ PASS
Web full test suite	cd apps/web && npx vitest run	23 files, 133/133 pass	✓ PASS
Web typecheck	cd apps/web && npx tsc --noEmit	clean, exit 0	✓ PASS
Denylist refusal (source-registry.spec.ts)	(part of full suite)	6/6 pass	✓ PASS
Dedup resolver (tender-dedup.service.spec.ts)	(part of full suite)	8/8 pass	✓ PASS
Local DB — TenderSource backfill	psql -c 'SELECT count(*) FROM "TenderSource"'	2851	✓ PASS (matches SELECT count(*) FROM "Tender" = 2851)
Local DB — fingerprint backfill	psql -c 'SELECT count(*) FROM "Tender" WHERE fingerprint IS NOT NULL'	2851	✓ PASS
Local DB — poll configs seeded	psql -c 'SELECT "sourceType","isActive" FROM "TenderSourcePollConfig"'	Only doe-opendata row present	⚠️ Local API process not yet restarted since Plans 13-04/13-05 landed — seed code for ai-netserver/cosinex-dtvp exists but hasn't run against this DB (see human_verification)
Debt markers (TBD/FIXME/XXX) in phase files	`grep -n -E "TBD	FIXME	XXX"` across 8 key files

Requirements Coverage

Requirement	Source Plan	Description	Status	Evidence
INGEST-02	13-04	Import von AI-AG-NetServer-Portalen über konfigurierbaren Adapter	✗ BLOCKED	Adapter fetches/parses correctly, but records would not be usably imported (garbage-normalized) — see Truth 1
INGEST-03	13-05	Import vom cosinex-Vergabemarktplatz über Adapter	✗ BLOCKED	Same root cause — see Truth 2
INGEST-07	13-02	vergabe24/aumass hart als Denylist, keine automatische Registrierung möglich	✓ SATISFIED	source-registry.spec.ts, code-level DeniedPortalError
SCHEMA-03	13-01/03/06	Dedup zu einem Eintrag mit mehreren Quell-Links; greift erst ab 2. aktiver Quelle	⚠️ PARTIALLY SATISFIED	Schema, resolver, fan-out, backfill, and display are all correctly built and tested for the mechanism; the practical DÖE↔NetServer/cosinex cross-source match is blocked by the same normalizer gap (Truth 3)

REQUIREMENTS.md currently marks all four as [x]/"Complete" — this verification finds INGEST-02 and INGEST-03 not actually satisfiable end-to-end yet, and SCHEMA-03 only partially achieved for the two new sources (DÖE-internal re-poll/change-detection dedup, which predates this phase, remains fully intact — see below).

SCHEMA-02 Regression Check

Confirmed NOT regressed: tender-dedup.service.spec.ts has two dedicated tests ("OCID match with a changed contentHash still updates...", "...UNCHANGED contentHash does NOT call tender.update"), and the pre-existing tender-ingestion.service.spec.ts SCHEMA-02 suite (insert-once, update-in-place-on-changed-hash) was migrated to the new resolve() hand-off and remains green. The mutable-field list (title, buyerName, cpvCodes, cpvDivisions, region, plz, bundesland, deadlineAt, estimatedValue, procedureType, sourceUrl, contentHash) was copied verbatim from the old tender.upsert UPDATE branch.

Anti-Patterns Found

None — no TBD/FIXME/XXX/TODO/HACK/PLACEHOLDER markers in the 8 key phase files scanned. The normalizer gap is not a code-smell/stub pattern; it is a structurally absent capability, self-disclosed transparently in both SUMMARY.md files' "Known Stubs" sections rather than hidden.

Human Verification Required

1. Real-world cross-source dedup confirmation — Test: after the normalizer gap is fixed, activate ai-netserver or cosinex-dtvp (TenderSourcePollConfig.isActive=true), run one poll tick, and confirm a real known-overlapping tender collapses to one Tender row with two TenderSource links visible in TenderDetail. Expected: one results-list entry, two clickable source links. Why human: needs live portal access, a real overlapping notice, and a running scheduler tick — not provable from static code.
2. Poll-config seed freshness — Test: confirm the local API process has been rebuilt/restarted since Plans 13-04/13-05 so ai-netserver/cosinex-dtvp TenderSourcePollConfig rows actually exist locally (currently only doe-opendata is present in the DB). Why human: requires a deploy/restart action outside this verification's scope (per project convention, Claude does not restart local Docker services).

Gaps Summary

Phase 13 built a genuinely solid, well-tested core: the TenderSource schema + 2851-row backfill, the NULL-tolerant fingerprint function, the SourceRegistry with a real code-level denylist gate (Success Criterion 4 fully achieved), the three-tier dedup resolver with dedupActive hard-gating (SCHEMA-03 mechanism), the findMany fan-out with catch-per-source isolation, and the multi-source-link display in TenderDetail. All of this is unit-tested (284/284 API, 133/133 web) and both typechecks are clean.

However, both new HTML adapters (NetServerAdapter, CosinexAdapter) deliberately punt on a required, self-documented follow-up: they carry their parsed fields in a flat generic ocdsPayload bag that TenderNormalizerService — hardcoded to the DÖE eForms/OCDS shape — cannot read. Both 13-04-SUMMARY.md and 13-05-SUMMARY.md disclose this transparently under "Known Stubs" as "out of this plan's files_modified scope" and "a natural follow-up plan" — but no such follow-up plan exists in Phase 13, and Phase 14's roadmap scope (RSS/email ingestion, admin source UI, i18n) does not cover it either. Since the poll configs for both new sources are seeded isActive: false, this gap is currently dormant in production — but it directly blocks 3 of the phase's 4 ROADMAP success criteria from being true as written ("appear as additional, correctly normalized results" x2, and real DÖE↔scraper cross-source dedup matching). Recommend a closure plan that extends TenderNormalizerService (or introduces a per-sourceType normalization strategy) to map the NetServer/cosinex generic bag into NormalizedTenderFields before this phase can be considered to have achieved its stated goal.

---

Verified: 2026-07-23T10:15:00Z Verifier: Claude (gsd-verifier)