tessera-ctl/.planning/phases/13-scraping-adapters-cross-source-dedup/13-03-SUMMARY.md

phase, plan, subsystem, tags, requires, provides, affects, tech-stack, key-files, key-decisions, patterns-established, requirements-completed, coverage, duration, completed, status

phase	plan	subsystem	tags	requires	provides	affects	tech-stack	key-files	key-decisions	patterns-established	requirements-completed	coverage	duration	completed	status
13-scraping-adapters-cross-source-dedup	03	backend	nestjs dedup prisma fan-out error-isolation	phase plan provides 13-scraping-adapters-cross-source-dedup 01 tenderFingerprint(), TenderSource model, Tender.fingerprint, SourceType open union phase plan provides 13-scraping-adapters-cross-source-dedup 02 SourceRegistry (get/activeAdapters), TenderSourceAdapter.portals[], DeniedPortalError denylist gate	TenderDedupService.resolve(n, {dedupActive}) — three-tier cross-source dedup resolver (OCID -> source:noticeId -> fingerprint), fingerprint tier hard-gated by dedupActive (D-05) pollDueSources fan-out over ALL active TenderSourcePollConfig rows (findMany, not findUnique) with catch-per-source error isolation SourceRegistry + TenderDedupService registered as TendersModule providers; DoeOpenDataAdapter registered with the registry at DI boot (onModuleInit)	13-04-netserver-adapter 13-05-cosinex-adapter 13-06-multi-source-display	added patterns Dedup resolver owns both the create-vs-attach decision AND the SCHEMA-02 mutable-field refresh — the old direct tender.upsert UPDATE branch moved into resolve()'s match path unchanged in semantics dedupActive computed once per tick from configs.length (activePortalCount >= 2) and threaded through every resolve() call in the fan-out loop catch-per-source: each config's day-cursor-gate/fetch/normalize/resolve block runs in its own try/catch inside the fan-out for-loop, nested inside the existing outer tick-level try/catch	created modified apps/api/src/tenders/tender-dedup.service.ts apps/api/src/tenders/tender-dedup.service.spec.ts apps/api/src/tenders/tender-ingestion.service.ts apps/api/src/tenders/tender-ingestion.service.spec.ts apps/api/src/tenders/tenders.module.ts	Tier 2 (source:noticeId via TenderSource lookup) is NOT gated by dedupActive — it is the same-source re-poll identity check, independent from cross-source merging, and must remain active even with a single source (idempotent re-poll). Fingerprint is always computed and stored on tender.create (n.fingerprint ?? tenderFingerprint(n)), regardless of dedupActive — a DÖE-only tender still gets a fingerprint so it becomes matchable the moment a 2nd source activates, without a further backfill. SCHEMA-02 mutable-field list is copied verbatim from the pre-existing tender.upsert UPDATE block (title, buyerName, cpvCodes, cpvDivisions, region, plz, bundesland, deadlineAt, estimatedValue, procedureType, sourceUrl, contentHash) into resolve()'s match branch, gated by a contentHash comparison instead of always running on every upsert call — functionally identical outcome, explicit no-op test added. pruneExpiredTenders() runs once per tick (not once per source) — anyDayFetched is now an across-all-sources flag, matching the pre-existing once-per-tick retention semantics.	Fake TenderDedupService.resolve() in tender-ingestion.service.spec.ts mirrors the real service's dedupKey-upsert contract (created vs. updated) without re-testing tier logic — keeps the fan-out spec focused on orchestration, not dedup internals (covered separately by tender-dedup.service.spec.ts).	SCHEMA-03	id description requirement verification human_judgment D1 dedupActive=false: two normalized records with the same fingerprint but different source:noticeId create TWO Tender rows (fingerprint tier skipped, D-05 inert-proof, Erfolgskriterium 3) SCHEMA-03 kind ref status unit apps/api/src/tenders/tender-dedup.service.spec.ts — 'creates TWO Tender rows for records with the same fingerprint but different source:noticeId, when dedupActive=false' pass false id description requirement verification human_judgment D2 dedupActive=true: OCID match, idempotent source:noticeId re-poll, and fingerprint match all attach an additional TenderSource to the existing Tender (created=false, D-03 merge); tenderSource.upsert targets the @@unique([sourcePortal, sourceNoticeId]) composite key SCHEMA-03 kind ref status unit apps/api/src/tenders/tender-dedup.service.spec.ts — OCID-match/idempotent-reseen/fingerprint-match/no-match/composite-key tests (8/8 pass) pass false id description requirement verification human_judgment D3 SCHEMA-02 change-detection preserved through resolve(): a matched Tender whose incoming contentHash differs gets its mutable fields (title, deadlineAt, contentHash, etc.) updated; an unchanged contentHash does NOT call tender.update — no Phase-10 regression SCHEMA-03 kind ref status unit apps/api/src/tenders/tender-dedup.service.spec.ts — 'OCID match with a changed contentHash still updates...' + 'OCID match with an UNCHANGED contentHash does NOT call tender.update' pass kind ref status unit apps/api/src/tenders/tender-ingestion.service.spec.ts — existing SCHEMA-02 re-poll suite (insert-once, update-in-place-on-changed-hash) kept green through the resolve() hand-off pass false id description requirement verification human_judgment D4 pollDueSources fans out over ALL active TenderSourcePollConfig rows (findMany, not findFirst/findUnique) with catch-per-source error isolation — one broken/blocking source does not abort polling of the others, nor does it throw out of the tick SCHEMA-03 kind ref status unit apps/api/src/tenders/tender-ingestion.service.spec.ts — 'polls every active config...', 'skips a config whose sourceType has no registered adapter, without throwing', 'a throwing source does not prevent another active source from being polled' (14/14 pass) pass false id description requirement verification human_judgment D5 dedupActive is bound to activePortalCount >= 2 and threaded into every resolve() call; delta-only matchDelta boundary (D-07) preserved across the fan-out (only genuinely-created IDs across all sources) SCHEMA-03 kind ref status unit apps/api/src/tenders/tender-ingestion.service.spec.ts — 'passes dedupActive=false...single config', 'passes dedupActive=true...two configs', existing D-07 delta-only suite kept green pass false id description requirement verification human_judgment D6 SourceRegistry + TenderDedupService are TendersModule providers; DoeOpenDataAdapter is registered with the registry in onModuleInit (denylist gate applies at DI boot); DI graph resolves cleanly SCHEMA-03 kind ref status unit cd apps/api && npx tsc --noEmit -p tsconfig.json (clean) pass kind ref status unit cd apps/api && npx vitest run src/tenders (19 files, 190/190 pass) pass kind ref status unit cd apps/api && npx vitest run (full suite, 23 files, 253/253 pass) pass false	45min	2026-07-23	complete

Phase 13 Plan 03: Dedup Resolver + Fan-out Wiring Summary

Three-tier TenderDedupService.resolve() (OCID -> source:noticeId -> fingerprint, fingerprint tier hard-gated by activePortalCount >= 2) now drives pollDueSources, which fans out over every active TenderSourcePollConfig with catch-per-source error isolation — replacing the DÖE-only findUnique/direct tender.upsert, while preserving SCHEMA-02 change detection byte-for-byte.

Performance

• Duration: 45 min
• Started: 2026-07-23T08:45:00Z
• Completed: 2026-07-23T09:30:00Z
• Tasks: 3
• Files modified: 5 (2 created, 3 modified)

Accomplishments

• TenderDedupService.resolve(n, {dedupActive}): OCID -> source:noticeId -> fingerprint (D-04), fingerprint tier only reachable when dedupActive is true (D-05). Match -> tenderSource.upsert attaches an additional source to the existing Tender (created: false, D-03 merge) + SCHEMA-02 mutable-field refresh when contentHash differs. No match -> tender.create (with computed fingerprint) + tenderSource.create (created: true). Plain PrismaService, no forTenant()/RLS. TDD RED->GREEN, 8/8 tests green, including the D-05 inert-proof (fingerprint-colliding records with dedupActive=false stay two Tenders) and two dedicated SCHEMA-02-preservation tests.
• pollDueSources rebuilt on findMany({ isActive: true }) fan-out (poll-once-fan-out-many, NOT findFirst/findUnique). Each config's day-cursor-gate/fetch/normalize/resolve block runs inside its own try/catch — a throwing source (portal down) no longer kills the tick or blocks other sources (D-01). dedupActive = configs.length >= 2 computed once per tick and threaded through every resolve() call. pruneExpiredTenders() and matchDelta (D-07 delta-only boundary) run once across the whole tick, unchanged in intent. Existing SCHEMA-02/D-07/retention/day-cursor spec suites updated to the new (prisma, registry, normalizer, matching, dedup) constructor shape and kept green; four new fan-out/catch-per-source/dedupActive-gate tests added. 14/14 pass.
• TendersModule: SourceRegistry + TenderDedupService added as providers; onModuleInit registers DoeOpenDataAdapter with the registry before the scheduler's first tick can fire — the DI-boot-time point where the INGEST-07 denylist gate (D-06) structurally applies. Explicitly documented as the sole 13-04/13-05 extension point for future adapter registrations.
• npx tsc --noEmit clean; src/tenders slice 19 files/190 tests green; full API suite 23 files/253 tests green.

Task Commits

Each task was committed atomically:

1. Task 1a: RED — failing TenderDedupService spec - 605ea4c (test)
2. Task 1b: GREEN — TenderDedupService three-tier resolver - 1cd2fcf (feat)
3. Task 2: pollDueSources fan-out + catch-per-source + dedup hook - d453dbb (feat)
4. Task 3: Module wiring — SourceRegistry + TenderDedupService providers - 1d4f9cf (feat)

Task 1 (tdd="true") produced two commits (test -> feat) per protocol; no refactor commit was needed — the first implementation passed all 8 tests cleanly.

Files Created/Modified

• apps/api/src/tenders/tender-dedup.service.ts - @Injectable() TenderDedupService, three-tier resolve() resolver
• apps/api/src/tenders/tender-dedup.service.spec.ts - 8 unit tests (D-05 inert-proof, tier1/2/3 match, no-match create, composite-key upsert target, SCHEMA-02 preservation x2)
• apps/api/src/tenders/tender-ingestion.service.ts - pollDueSources rebuilt as findMany fan-out with catch-per-source + dedup.resolve() hand-off; constructor now takes (prisma, registry, normalizer, matching, dedup)
• apps/api/src/tenders/tender-ingestion.service.spec.ts - fake registry/dedup collaborators; existing suites migrated to the new constructor shape; 4 new tests (fan-out, adapter-missing skip, catch-per-source isolation, dedupActive gate x2)
• apps/api/src/tenders/tenders.module.ts - SourceRegistry + TenderDedupService providers; onModuleInit registers DoeOpenDataAdapter

Decisions Made

• Tier 2 (source:noticeId) is unconditional, not gated by dedupActive — it's the same-source re-poll identity check (idempotency), orthogonal to cross-source fingerprint merging. Gating it would have broken idempotent re-polls of a single active source.
• Fingerprint is always computed on create, independent of dedupActive — so a DÖE-only tender is immediately fingerprint-matchable the instant a second source activates, with no further backfill needed (13-01's backfill already covers the pre-existing 2851 rows; this covers every NEW tender ingested going forward).
• SCHEMA-02 mutable-field list copied verbatim from the old tender.upsert UPDATE branch into resolve()'s match path, now gated by an explicit contentHash !== comparison (previously implicit in every upsert call) — behaviorally identical, with an added explicit "unchanged hash -> no update call" test that didn't exist before.
• pruneExpiredTenders()/matchDelta run once per tick, not once per source — anyDayFetched and newTenderIds are tick-scoped across the whole fan-out loop, preserving the pre-existing once-per-tick retention/matching cadence.

Deviations from Plan

None — plan executed exactly as written. All four must_haves.truths are met and directly tested:

1. dedupActive=false inertness — tested (D1 above).
2. 2nd-source fingerprint match attaches TenderSource instead of creating a new Tender — tested (D2).
3. pollDueSources fans out via findMany({isActive:true}) with catch-per-source — tested (D4).
4. SCHEMA-02 change detection preserved through the move into resolve() — tested (D3), with both the original ingestion-service spec suite kept green AND two new dedicated dedup-service tests.

Issues Encountered

None. Local (non-worktree, main branch) execution; no migrations touched in this plan (13-01 already applied the schema). No Docker/DB interaction required — pure TypeScript service/wiring plan, verified entirely via Vitest on host.

User Setup Required

None. No external service configuration, no migration, no environment changes.

Next Phase Readiness

• TenderDedupService.resolve() and the SourceRegistry-driven fan-out are the exact seams 13-04 (NetServer adapter) and 13-05 (cosinex adapter) need: adding a new adapter is (a) implement TenderSourceAdapter, (b) add one providers entry + one registry.register(...) call in tenders.module.ts's onModuleInit, (c) seed a TenderSourcePollConfig row for the new sourceType. No further changes to pollDueSources or TenderDedupService are needed — both are already source-count-agnostic.
• Cross-source dedup end-to-end (real DÖE + NetServer overlap producing ONE Tender with two TenderSource links) remains a manual/live verification per 13-VALIDATION.md — this plan's automated coverage proves the resolver and fan-out logic in isolation; the live confirmation needs a second active source (13-04) to observe.
• No blockers.

---

Phase: 13-scraping-adapters-cross-source-dedup Completed: 2026-07-23

Self-Check: PASSED

All created/modified files verified present on disk; all 4 task commit hashes (605ea4c, 1cd2fcf, d453dbb, 1d4f9cf) verified in git log.