---
phase: 13-scraping-adapters-cross-source-dedup
verified: 2026-07-23T10:30:00Z
status: gaps_found
score: 3/4 must-haves verified
behavior_unverified: 0
overrides_applied: 0
re_verification:
  previous_status: gaps_found
  previous_score: 1/4
  gaps_closed:
    - "Tenders from AI-AG NetServer portals appear as additional, correctly normalized results via one config-driven adapter"
    - "Tenders from the cosinex Vergabemarktplatz (DTVP) appear as additional, correctly normalized results via a separate config-driven adapter"
  gaps_remaining:
    - "A tender that appears via both DÖE and a scraping adapter shows up once in the results list (fuzzy fingerprint dedup on buyer+title+CPV+deadline+value) — NEW residual finding: cpvDivisions asymmetry (scraper always [], DÖE usually populated) makes the fingerprint tier fail to collapse the majority of real cross-source duplicates even after the normalizer fix"
  regressions: []
gaps:
  - truth: "A tender that appears via both DÖE and a scraping adapter shows up once in the results list (fuzzy fingerprint dedup on buyer+title+CPV+deadline+value), with links to all of its source portals -- dedup logic only activates once a second source is live"
    status: partial
    reason: >
      The previously-identified root cause (NetServer/cosinex normalizing to title/buyerName/
      deadlineAt defaults) is CLOSED by quick task 260723-e7i — verified live in this
      re-verification (see Truth table). However, a SEPARATE, previously-undetected issue
      remains: tenderFingerprint()'s canonical string is a straight concatenation of
      [buyerName, title, cpvDivisionKey, deadlineKey, valueBucket] hashed with sha256 — it is
      an EXACT match on all five segments, not a weighted/fuzzy comparison despite the
      "Hauptgewicht" (main weight) language in 13-RESEARCH.md. NetServerAdapter and
      CosinexAdapter structurally cannot supply CPV codes (verified by reading both adapters'
      HTML column extraction: neither the NetServer 6-column table nor the cosinex
      csx-new-table has a CPV column at all), so TenderNormalizerService.normalizeBag()
      correctly and necessarily always sets cpvDivisions=[] for these two sources (this is
      not a normalizer bug — there is nothing to extract). Live DB check
      (`SELECT count(*) FILTER (WHERE array_length("cpvDivisions",1) IS NULL OR
      array_length("cpvDivisions",1)=0)` -> 601/2851 = ~21%) shows ~79% of existing
      DÖE-sourced tenders carry a non-empty cpvDivisions. Because cpvDivisionKey('') !=
      cpvDivisionKey(['45']) (or any other non-empty division), the sha256 hash differs
      whenever the DÖE-side counterpart has CPV codes -- i.e. for an estimated ~79% of real
      overlapping tenders, the fingerprint tier will NOT collapse the DÖE and NetServer/cosinex
      records into one Tender, even once a second source is activated and even though
      title/buyerName/deadlineAt now correctly match. tender-dedup.service.spec.ts's only
      cross-source fingerprint test (`fingerprint match (no OCID/noticeId overlap)`) does not
      exercise this asymmetry -- it spreads `...BASE` (cpvDivisions: ['45']) into both
      synthetic records, so both sides always share the same non-empty CPV segment. No test
      in the suite proves a NetServer/cosinex record (cpvDivisions=[]) actually collides with
      its real DÖE counterpart (cpvDivisions=['45'] or similar).
    artifacts:
      - path: "apps/api/src/tenders/tender-fingerprint.ts"
        issue: "cpvDivisionKey is one of 5 hashed segments with no null/empty-tolerant matching for the CPV dimension specifically (unlike deadlineAt/estimatedValue, which ARE null-tolerant by design) -- an empty array on one side and a populated array on the other always produces a different hash"
      - path: "apps/api/src/tenders/tender-normalizer.service.ts"
        issue: "normalizeBag() correctly sets cpvDivisions=[] (there is no CPV data in the scraped HTML to map) -- this is the correct behavior for the available source data, but it is the direct cause of the fingerprint mismatch described above"
    missing:
      - "A developer decision on how to handle the CPV-availability asymmetry for fingerprint matching: e.g. treat an empty cpvDivisions on either side as a wildcard/skip for that segment instead of an exact-match requirement, or accept the reduced (~21%) practical match rate for NetServer/cosinex cross-source dedup as a known limitation of these two sources"
      - "A live/manual confirmation once ai-netserver or cosinex-dtvp poll config is activated (TenderSourcePollConfig.isActive=true) and a real overlapping tender is observed to merge into one entry with two source links (unchanged from prior verification -- still not possible to test locally, see Human Verification)"
deferred: []
human_verification:
  - test: "Activate the ai-netserver or cosinex-dtvp TenderSourcePollConfig row (isActive=true), let pollDueSources run one tick, and confirm in the DB/UI that a tender known to exist on both DÖE and the new source collapses to ONE Tender with two TenderSource rows/links in TenderDetail. Given the CPV-asymmetry finding above, prioritize testing with a real overlapping notice that happens to have NO CPV code on the DÖE side, since that is currently the only case the fingerprint tier can realistically catch for these two sources."
    expected: "One results-list entry; TenderDetail shows both a DÖE link and a tender24/lhs-vpbw/vergabe.landbw/DTVP link — but only reliably for DÖE counterparts with no CPV code, per the gap above"
    why_human: "Requires live network access to the real portals, a real overlapping notice, and a running scheduler tick against a real DB — cannot be proven by static code/unit-test inspection alone"
  - test: "Confirm local API container was rebuilt/restarted so the ai-netserver/cosinex-dtvp TenderSourcePollConfig seed rows actually exist in the local DB"
    expected: "SELECT sourceType, isActive FROM TenderSourcePollConfig shows 3 rows (doe-opendata, ai-netserver, cosinex-dtvp), the latter two isActive=false"
    why_human: "Re-checked live in this verification (docker exec psql): still only 1 row (doe-opendata). The API process has not been restarted/rebuilt since Plans 13-04/13-05 landed -- unrelated to and not resolved by the normalizer quick-fix. Per project convention, Claude does not restart local Docker services."
residual_gap_decision:
  date: 2026-07-23
  chosen: "C — accept as dormant deferral (no code change now)"
  by: user
  rationale: >
    The cpvDivisions fingerprint asymmetry (Truth 3) is real but the whole
    cross-source dedup mechanism is currently inert: both scraper poll configs
    are seeded isActive=false, so only DÖE runs and the fingerprint tier is
    never exercised in production. Whether CPV-less matching (Option A: a second
    buyer+title+deadline fingerprint tier) has acceptable precision or produces
    too many false merges can only be judged against real overlapping notices,
    not statically. Decision deferred to when a second source is actually
    activated. Options considered: A = CPV-less second fingerprint tier
    (new column + migration + backfill 2851 + 4th resolver tier); B = drop
    CPV+value from the primary fingerprint (simple, higher false-merge risk);
    C = leave dormant (chosen). Truth 3 stays PARTIAL by design, not by neglect.
---

# Phase 13: Scraping Adapters & Cross-Source Deduplication Verification Report

**Phase Goal:** Die Plattform erweitert die Ausschreibungs-Abdeckung um AI-AG-NetServer- und cosinex-Portal-Adapter und fasst Ausschreibungen, die auf mehreren Quellen auftauchen, zu einem Eintrag zusammen — während sie das Scrapen der AGB-verbotenen Portale strukturell verweigert.
**Verified:** 2026-07-23T10:30:00Z
**Status:** gaps_found
**Re-verification:** Yes — after gap closure (quick task 260723-e7i)

## What Changed Since Last Verification

Quick task 260723-e7i (commits `82c9a48`, `9881005`) added a per-`sourceType` dispatch to
`TenderNormalizerService.normalize()`. Read live in this re-verification
(`apps/api/src/tenders/tender-normalizer.service.ts`):

- `normalize()` switches on `raw.sourceType`: `ai-netserver`/`cosinex-dtvp` → `normalizeBag()`;
  `doe-opendata`/default → `normalizeDoe()` (default never throws).
- `normalizeBag()` reads the flat `ocdsPayload` bag `{title, buyerName, procedureType,
  legalFramework, deadlineAt}` and maps `title` (fallback `'Unbenannte Ausschreibung'`),
  `buyerName`/`procedureType` (null-tolerant, no invented fallback), `deadlineAt` via the
  existing `parseOcdsDate()`. `cpvCodes`/`cpvDivisions` = `[]`, `region`/`plz`/`bundesland`/
  `estimatedValue` = `null` (there is genuinely no such data in the NetServer/cosinex HTML —
  confirmed by reading both adapters' column extraction).
- A shared `assemble()` computes `status`/`dedupKey`/`contentHash`/`publishedAt` identically for
  both paths — `normalizeDoe()` is a byte-identical extraction of the pre-existing DÖE logic.

**This closes the previously-identified root cause for Truths 1 and 2** — NetServer and
cosinex records now carry their real title/buyer/procedureType/deadline instead of falling
through to defaults. It **partially** closes Truth 3: title/buyer/deadline now flow correctly
into the fingerprint, but re-verification surfaced a **new, previously-undetected** issue
specific to the CPV dimension of the fingerprint that still blocks most real-world
cross-source matches (see Gap below).

## Goal Achievement

### Observable Truths (ROADMAP Success Criteria)

| # | Truth | Status | Evidence |
|---|-------|--------|----------|
| 1 | NetServer-Portale (lhs-vpbw, tender24, vergabe.landbw) erscheinen als zusätzliche, **korrekt normalisierte** Treffer über EINEN config-getriebenen Adapter | ✓ VERIFIED (flipped) | `normalizeBag()` now maps `ai-netserver` bag records into real `title`/`buyerName`/`procedureType`/`deadlineAt` (verified live: `tender-normalizer.service.spec.ts` "ai-netserver: maps the flat bag into NormalizedTenderFields" test asserts `title==='Bauleistung X'`, `buyerName==='Stadt Y'`, `deadlineAt` ISO round-trip). `cpvCodes=[]`/`region`/`plz`/`bundesland`/`estimatedValue=null` is CORRECT (not a stub) — the NetServer HTML table structurally has no such columns (confirmed reading `netserver.adapter.ts`: 6 cells = published/title/buyer/procedureType/legalFramework/deadline, no CPV). Full API suite green (288/288), tsc clean. |
| 2 | cosinex/DTVP (DTVP) erscheint als zusätzlicher Treffer über einen SEPARATEN config-getriebenen Adapter, **korrekt normalisiert** | ✓ VERIFIED (flipped) | Identical fix applies via the same `normalizeBag()` dispatch for `cosinex-dtvp`. Verified live: "cosinex-dtvp: maps an identical bag shape identically" test passes. `cosinex.adapter.ts`'s csx-new-table columns (published/deadline/title/type/buyer/action) likewise have no CPV column — `cpvCodes=[]` is correct, not a gap. |
| 3 | Ein über DÖE + einem Scraping-Adapter gesehener Tender erscheint EINMAL mit Links zu allen Quell-Portalen (Fuzzy-Fingerprint-Dedup); Dedup greift erst ab 2. aktiver Quelle | ⚠️ PARTIAL (still, for a NEW reason) | Mechanism (resolver, `dedupActive` gate, fan-out, backfill, multi-source UI) remains fully built/tested — unchanged from prior verification. The originally-identified blocker (title/buyer/deadline defaults) IS closed. BUT: `tenderFingerprint()` hashes `cpvDivisionKey` as an exact-match segment; NetServer/cosinex always produce `cpvDivisions=[]` (no CPV data exists in their HTML) while ~79% of existing DÖE tenders (2250/2851 live, `array_length>0`) have non-empty `cpvDivisions`. For any such tender, the scraper-side fingerprint hash will NEVER equal the DÖE-side hash, so the fingerprint tier will not collapse them — "shows up once" will fail for an estimated majority of real overlapping notices. No existing test exercises this asymmetry (the one cross-source fingerprint test in `tender-dedup.service.spec.ts` gives both synthetic records the same non-empty `cpvDivisions`). See Gaps. |
| 4 | Registrierung von vergabe24/aumass als Poll-Quelle wird vom System selbst (Code, nicht nur Doku) verweigert | ✓ VERIFIED (re-confirmed, no regression) | `source-registry.spec.ts` 6/6 pass (unchanged); `source-registry.ts` untouched by the quick-fix (only `tender-normalizer.service.ts`/`.spec.ts` were modified — confirmed via `git log` on the file). |

**Score:** 3/4 truths verified (0 present-behavior-unverified; 1 partial — mechanism fully built and now fed correct title/buyer/deadline data, but a newly-identified CPV-availability asymmetry in the fingerprint still blocks most real cross-source matches)

### Required Artifacts (delta since prior verification)

| Artifact | Expected | Status | Details |
|----------|----------|--------|---------|
| `apps/api/src/tenders/tender-normalizer.service.ts` | Per-sourceType dispatch mapping the generic bag into `NormalizedTenderFields` | ✓ VERIFIED | `normalize()` switches on `raw.sourceType`; `normalizeBag()` present, `normalizeDoe()` unchanged (byte-identical DÖE extraction preserved under a renamed private method); shared `assemble()` extracted, `computeContentHash` not duplicated |
| `apps/api/src/tenders/tender-normalizer.service.spec.ts` | Tests for both bag sources + null fields + DÖE regression | ✓ VERIFIED | 14/14 tests pass (10 pre-existing DÖE + 4 new bag tests); DÖE fixture-based regression tests untouched and green |

All other artifacts from the prior verification (fingerprint fn, schema, source-registry, adapters, dedup resolver, controller, TenderDetail) are unchanged by this quick task and were re-confirmed passing as part of the full-suite run below (no regression).

### Key Link Verification (delta)

| From | To | Via | Status | Details |
|------|-----|-----|--------|---------|
| `netserver.adapter.ts` / `cosinex.adapter.ts` | `tender-normalizer.service.ts` | `RawTenderRecord.ocdsPayload` -> `normalize()` | ✓ WIRED (flipped from NOT_WIRED) | `normalizeBag()` now correctly extracts the flat bag; verified by dedicated bag tests and by the field-authority correctness argument above (empty CPV is a correct reflection of absent source data, not a mapping bug) |
| `tender-normalizer.service.ts` (`normalizeBag` output) | `tender-fingerprint.ts` (`tenderFingerprint`) | `NormalizedTenderFields.cpvDivisions` -> `cpvDivisionKey()` | ⚠️ PARTIAL — semantically mismatched | Data flows without error, but the exact-match hash semantics mean a structurally-always-empty `cpvDivisions` on the scraper side will not collide with a populated `cpvDivisions` on the DÖE side for ~79% of existing tenders — see Gap |

### Behavioral Spot-Checks / Test Suite Results (run live during this re-verification)

| Check | Command | Result | Status |
|-------|---------|--------|--------|
| API full test suite | `cd apps/api && npx vitest run` | 25 files, **288/288 pass** (up from 284; +4 new bag-mapping tests) | ✓ PASS |
| API typecheck | `cd apps/api && npx tsc --noEmit -p tsconfig.json` | clean, exit 0 | ✓ PASS |
| Normalizer spec only | (part of full suite) | `tender-normalizer.service.spec.ts` 14/14 pass, incl. new `describe('generic ocdsPayload bag ...')` block | ✓ PASS |
| Dedup resolver (unchanged) | (part of full suite) | `tender-dedup.service.spec.ts` 8/8 pass | ✓ PASS |
| Fingerprint fn (unchanged) | (part of full suite) | `tender-fingerprint.spec.ts` 8/8 pass | ✓ PASS |
| Denylist refusal (unchanged) | (part of full suite) | `source-registry.spec.ts` 6/6 pass | ✓ PASS |
| Live DB — CPV-empty ratio (new check, this re-verification) | `docker exec tessera-ctl-db-1 psql -U tessera -d tessera -c "SELECT count(*) total, count(*) FILTER (WHERE array_length(cpvDivisions,1) IS NULL OR array_length(cpvDivisions,1)=0) empty_cpv FROM Tender"` | 2851 total, 601 empty_cpv (~21%) | ⚠️ Confirms the fingerprint CPV-mismatch gap above — ~79% of existing tenders have non-empty cpvDivisions |
| Live DB — poll configs seeded (re-checked, unchanged) | `docker exec tessera-ctl-db-1 psql -U tessera -d tessera -c 'SELECT "sourceType","isActive" FROM "TenderSourcePollConfig"'` | Only `doe-opendata` row present | ⚠️ Still not restarted since Plans 13-04/13-05 — unaffected by this quick task, human_verification item carried forward unchanged |
| Debt markers (TBD/FIXME/XXX/TODO/HACK/PLACEHOLDER) in modified files | `grep -n -E "TBD|FIXME|XXX|TODO|HACK|PLACEHOLDER"` on `tender-normalizer.service.ts`/`.spec.ts` | none found | ✓ PASS |
| Git history confirms scope | `git log --oneline -- apps/api/src/tenders/tender-normalizer.service.ts` | `82c9a48 feat(quick-260723-e7i): per-sourceType dispatch in TenderNormalizerService` is the latest commit on the file | ✓ PASS |

### Requirements Coverage

| Requirement | Source Plan | Description | Status | Evidence |
|-------------|------------|-------------|--------|----------|
| INGEST-02 | 13-04 + quick-260723-e7i | Import von AI-AG-NetServer-Portalen über konfigurierbaren Adapter | ✓ SATISFIED | Adapter fetches/parses correctly AND records now normalize correctly (Truth 1 flipped) |
| INGEST-03 | 13-05 + quick-260723-e7i | Import vom cosinex-Vergabemarktplatz über Adapter | ✓ SATISFIED | Same fix applies (Truth 2 flipped) |
| INGEST-07 | 13-02 | vergabe24/aumass hart als Denylist, keine automatische Registrierung möglich | ✓ SATISFIED (re-confirmed) | Unchanged, 6/6 tests green |
| SCHEMA-03 | 13-01/03/06 + quick-260723-e7i | Dedup zu einem Eintrag mit mehreren Quell-Links; greift erst ab 2. aktiver Quelle | ⚠️ PARTIALLY SATISFIED (still) | Mechanism + now-correct title/buyer/deadline inputs are in place, but the CPV-availability asymmetry in the fingerprint blocks the practical "shows up once" outcome for most real DÖE↔scraper duplicates |

REQUIREMENTS.md currently marks all four as `[x]`/"Complete" (Phase 13). This re-verification confirms INGEST-02 and INGEST-03 are now genuinely satisfiable end-to-end. SCHEMA-03 remains only partially achieved for the two new sources specifically because of the CPV-fingerprint asymmetry documented above — this is a narrower, more precise finding than the prior verification's broader "everything defaults" blocker, but it is not yet fully closed.

### Anti-Patterns Found

None in the two files modified by the quick fix (`tender-normalizer.service.ts`, `tender-normalizer.service.spec.ts`) — no `TBD`/`FIXME`/`XXX`/`TODO`/`HACK`/`PLACEHOLDER` markers.

### Human Verification Required

1. **Real-world cross-source dedup confirmation** (carried forward, refined) — after `ai-netserver`/`cosinex-dtvp` is activated and a poll tick runs, confirm a real overlapping notice collapses to one `Tender` with two `TenderSource` links. Given the CPV-asymmetry finding, this is now more likely to succeed specifically for a DÖE counterpart that happens to have no CPV code — not a general guarantee. Why human: needs live portal access, a real overlapping notice, and a running scheduler tick.
2. **Poll-config seed freshness** (carried forward, unchanged, re-checked live) — local API process still needs a rebuild/restart so `ai-netserver`/`cosinex-dtvp` `TenderSourcePollConfig` rows exist (confirmed via `docker exec psql`: still only 1 row). Why human: requires a deploy/restart action outside verification scope.

### Gaps Summary

Quick task 260723-e7i successfully closed the root cause originally blocking Success Criteria 1 and 2: `TenderNormalizerService` now correctly maps the NetServer/cosinex flat `ocdsPayload` bag into real `title`/`buyerName`/`procedureType`/`deadlineAt`, confirmed by 4 new passing unit tests and a clean 288/288 full-suite run + clean `tsc`. Truths 1, 2, and 4 are now fully VERIFIED.

Success Criterion 3 (cross-source dedup) is closer to achieved than before, but this re-verification surfaced a **new, previously-undetected** residual issue distinct from the one just fixed: `tenderFingerprint()` requires an EXACT match on `cpvDivisionKey`, and NetServer/cosinex structurally can never populate `cpvDivisions` (there is no CPV column in either portal's search-results HTML). A live DB check shows ~79% of existing DÖE tenders carry non-empty `cpvDivisions`, meaning the fingerprint tier will fail to collapse the majority of real DÖE↔NetServer/cosinex duplicates even with correct title/buyer/deadline data. This is provable from code + live DB stats, not something requiring live portal access — it is a developer decision (e.g., treat an empty `cpvDivisions` as a wildcard segment rather than requiring exact equality) that has not yet been made. Recommend a small follow-up plan or an explicit accepted-limitation override before closing SCHEMA-03/Success Criterion 3 for good.

The two previously-identified human_verification items (live overlap confirmation once a second source is active; local poll-config seed freshness) are unaffected by the quick fix and are carried forward unchanged — re-checked live in this session, the local DB still shows only the `doe-opendata` poll config row.

---
*Verified: 2026-07-23T10:30:00Z*
*Verifier: Claude (gsd-verifier)*