--- phase: 13-scraping-adapters-cross-source-dedup plan: 03 subsystem: backend tags: [nestjs, dedup, prisma, fan-out, error-isolation] requires: - phase: 13-scraping-adapters-cross-source-dedup plan: 01 provides: tenderFingerprint(), TenderSource model, Tender.fingerprint, SourceType open union - phase: 13-scraping-adapters-cross-source-dedup plan: 02 provides: SourceRegistry (get/activeAdapters), TenderSourceAdapter.portals[], DeniedPortalError denylist gate provides: - "TenderDedupService.resolve(n, {dedupActive}) — three-tier cross-source dedup resolver (OCID -> source:noticeId -> fingerprint), fingerprint tier hard-gated by dedupActive (D-05)" - "pollDueSources fan-out over ALL active TenderSourcePollConfig rows (findMany, not findUnique) with catch-per-source error isolation" - "SourceRegistry + TenderDedupService registered as TendersModule providers; DoeOpenDataAdapter registered with the registry at DI boot (onModuleInit)" affects: [13-04-netserver-adapter, 13-05-cosinex-adapter, 13-06-multi-source-display] tech-stack: added: [] patterns: - "Dedup resolver owns both the create-vs-attach decision AND the SCHEMA-02 mutable-field refresh — the old direct tender.upsert UPDATE branch moved into resolve()'s match path unchanged in semantics" - "dedupActive computed once per tick from configs.length (activePortalCount >= 2) and threaded through every resolve() call in the fan-out loop" - "catch-per-source: each config's day-cursor-gate/fetch/normalize/resolve block runs in its own try/catch inside the fan-out for-loop, nested inside the existing outer tick-level try/catch" key-files: created: - apps/api/src/tenders/tender-dedup.service.ts - apps/api/src/tenders/tender-dedup.service.spec.ts modified: - apps/api/src/tenders/tender-ingestion.service.ts - apps/api/src/tenders/tender-ingestion.service.spec.ts - apps/api/src/tenders/tenders.module.ts key-decisions: - "Tier 2 (source:noticeId via TenderSource lookup) is NOT gated by dedupActive — it is the same-source re-poll identity check, independent from cross-source merging, and must remain active even with a single source (idempotent re-poll)." - "Fingerprint is always computed and stored on tender.create (n.fingerprint ?? tenderFingerprint(n)), regardless of dedupActive — a DÖE-only tender still gets a fingerprint so it becomes matchable the moment a 2nd source activates, without a further backfill." - "SCHEMA-02 mutable-field list is copied verbatim from the pre-existing tender.upsert UPDATE block (title, buyerName, cpvCodes, cpvDivisions, region, plz, bundesland, deadlineAt, estimatedValue, procedureType, sourceUrl, contentHash) into resolve()'s match branch, gated by a contentHash comparison instead of always running on every upsert call — functionally identical outcome, explicit no-op test added." - "pruneExpiredTenders() runs once per tick (not once per source) — anyDayFetched is now an across-all-sources flag, matching the pre-existing once-per-tick retention semantics." patterns-established: - "Fake TenderDedupService.resolve() in tender-ingestion.service.spec.ts mirrors the real service's dedupKey-upsert contract (created vs. updated) without re-testing tier logic — keeps the fan-out spec focused on orchestration, not dedup internals (covered separately by tender-dedup.service.spec.ts)." requirements-completed: [SCHEMA-03] coverage: - id: D1 description: "dedupActive=false: two normalized records with the same fingerprint but different source:noticeId create TWO Tender rows (fingerprint tier skipped, D-05 inert-proof, Erfolgskriterium 3)" requirement: SCHEMA-03 verification: - kind: unit ref: "apps/api/src/tenders/tender-dedup.service.spec.ts — 'creates TWO Tender rows for records with the same fingerprint but different source:noticeId, when dedupActive=false'" status: pass human_judgment: false - id: D2 description: "dedupActive=true: OCID match, idempotent source:noticeId re-poll, and fingerprint match all attach an additional TenderSource to the existing Tender (created=false, D-03 merge); tenderSource.upsert targets the @@unique([sourcePortal, sourceNoticeId]) composite key" requirement: SCHEMA-03 verification: - kind: unit ref: "apps/api/src/tenders/tender-dedup.service.spec.ts — OCID-match/idempotent-reseen/fingerprint-match/no-match/composite-key tests (8/8 pass)" status: pass human_judgment: false - id: D3 description: "SCHEMA-02 change-detection preserved through resolve(): a matched Tender whose incoming contentHash differs gets its mutable fields (title, deadlineAt, contentHash, etc.) updated; an unchanged contentHash does NOT call tender.update — no Phase-10 regression" requirement: SCHEMA-03 verification: - kind: unit ref: "apps/api/src/tenders/tender-dedup.service.spec.ts — 'OCID match with a changed contentHash still updates...' + 'OCID match with an UNCHANGED contentHash does NOT call tender.update'" status: pass - kind: unit ref: "apps/api/src/tenders/tender-ingestion.service.spec.ts — existing SCHEMA-02 re-poll suite (insert-once, update-in-place-on-changed-hash) kept green through the resolve() hand-off" status: pass human_judgment: false - id: D4 description: "pollDueSources fans out over ALL active TenderSourcePollConfig rows (findMany, not findFirst/findUnique) with catch-per-source error isolation — one broken/blocking source does not abort polling of the others, nor does it throw out of the tick" requirement: SCHEMA-03 verification: - kind: unit ref: "apps/api/src/tenders/tender-ingestion.service.spec.ts — 'polls every active config...', 'skips a config whose sourceType has no registered adapter, without throwing', 'a throwing source does not prevent another active source from being polled' (14/14 pass)" status: pass human_judgment: false - id: D5 description: "dedupActive is bound to activePortalCount >= 2 and threaded into every resolve() call; delta-only matchDelta boundary (D-07) preserved across the fan-out (only genuinely-created IDs across all sources)" requirement: SCHEMA-03 verification: - kind: unit ref: "apps/api/src/tenders/tender-ingestion.service.spec.ts — 'passes dedupActive=false...single config', 'passes dedupActive=true...two configs', existing D-07 delta-only suite kept green" status: pass human_judgment: false - id: D6 description: "SourceRegistry + TenderDedupService are TendersModule providers; DoeOpenDataAdapter is registered with the registry in onModuleInit (denylist gate applies at DI boot); DI graph resolves cleanly" requirement: SCHEMA-03 verification: - kind: unit ref: "cd apps/api && npx tsc --noEmit -p tsconfig.json (clean)" status: pass - kind: unit ref: "cd apps/api && npx vitest run src/tenders (19 files, 190/190 pass)" status: pass - kind: unit ref: "cd apps/api && npx vitest run (full suite, 23 files, 253/253 pass)" status: pass human_judgment: false duration: 45min completed: 2026-07-23 status: complete --- # Phase 13 Plan 03: Dedup Resolver + Fan-out Wiring Summary **Three-tier `TenderDedupService.resolve()` (OCID -> source:noticeId -> fingerprint, fingerprint tier hard-gated by `activePortalCount >= 2`) now drives `pollDueSources`, which fans out over every active `TenderSourcePollConfig` with catch-per-source error isolation — replacing the DÖE-only `findUnique`/direct `tender.upsert`, while preserving SCHEMA-02 change detection byte-for-byte.** ## Performance - **Duration:** 45 min - **Started:** 2026-07-23T08:45:00Z - **Completed:** 2026-07-23T09:30:00Z - **Tasks:** 3 - **Files modified:** 5 (2 created, 3 modified) ## Accomplishments - `TenderDedupService.resolve(n, {dedupActive})`: OCID -> source:noticeId -> fingerprint (D-04), fingerprint tier only reachable when `dedupActive` is true (D-05). Match -> `tenderSource.upsert` attaches an additional source to the existing Tender (`created: false`, D-03 merge) + SCHEMA-02 mutable-field refresh when `contentHash` differs. No match -> `tender.create` (with computed `fingerprint`) + `tenderSource.create` (`created: true`). Plain `PrismaService`, no `forTenant()`/RLS. TDD RED->GREEN, 8/8 tests green, including the D-05 inert-proof (fingerprint-colliding records with `dedupActive=false` stay two Tenders) and two dedicated SCHEMA-02-preservation tests. - `pollDueSources` rebuilt on `findMany({ isActive: true })` fan-out (poll-once-fan-out-many, NOT findFirst/findUnique). Each config's day-cursor-gate/fetch/normalize/resolve block runs inside its own `try/catch` — a throwing source (portal down) no longer kills the tick or blocks other sources (D-01). `dedupActive = configs.length >= 2` computed once per tick and threaded through every `resolve()` call. `pruneExpiredTenders()` and `matchDelta` (D-07 delta-only boundary) run once across the whole tick, unchanged in intent. Existing SCHEMA-02/D-07/retention/day-cursor spec suites updated to the new `(prisma, registry, normalizer, matching, dedup)` constructor shape and kept green; four new fan-out/catch-per-source/dedupActive-gate tests added. 14/14 pass. - `TendersModule`: `SourceRegistry` + `TenderDedupService` added as providers; `onModuleInit` registers `DoeOpenDataAdapter` with the registry before the scheduler's first tick can fire — the DI-boot-time point where the INGEST-07 denylist gate (D-06) structurally applies. Explicitly documented as the sole 13-04/13-05 extension point for future adapter registrations. - `npx tsc --noEmit` clean; `src/tenders` slice 19 files/190 tests green; full API suite 23 files/253 tests green. ## Task Commits Each task was committed atomically: 1. **Task 1a: RED — failing TenderDedupService spec** - `605ea4c` (test) 2. **Task 1b: GREEN — TenderDedupService three-tier resolver** - `1cd2fcf` (feat) 3. **Task 2: pollDueSources fan-out + catch-per-source + dedup hook** - `d453dbb` (feat) 4. **Task 3: Module wiring — SourceRegistry + TenderDedupService providers** - `1d4f9cf` (feat) _Task 1 (tdd="true") produced two commits (test -> feat) per protocol; no refactor commit was needed — the first implementation passed all 8 tests cleanly._ ## Files Created/Modified - `apps/api/src/tenders/tender-dedup.service.ts` - `@Injectable() TenderDedupService`, three-tier `resolve()` resolver - `apps/api/src/tenders/tender-dedup.service.spec.ts` - 8 unit tests (D-05 inert-proof, tier1/2/3 match, no-match create, composite-key upsert target, SCHEMA-02 preservation x2) - `apps/api/src/tenders/tender-ingestion.service.ts` - `pollDueSources` rebuilt as `findMany` fan-out with catch-per-source + `dedup.resolve()` hand-off; constructor now takes `(prisma, registry, normalizer, matching, dedup)` - `apps/api/src/tenders/tender-ingestion.service.spec.ts` - fake `registry`/`dedup` collaborators; existing suites migrated to the new constructor shape; 4 new tests (fan-out, adapter-missing skip, catch-per-source isolation, dedupActive gate x2) - `apps/api/src/tenders/tenders.module.ts` - `SourceRegistry` + `TenderDedupService` providers; `onModuleInit` registers `DoeOpenDataAdapter` ## Decisions Made - **Tier 2 (source:noticeId) is unconditional, not gated by `dedupActive`** — it's the same-source re-poll identity check (idempotency), orthogonal to cross-source fingerprint merging. Gating it would have broken idempotent re-polls of a single active source. - **Fingerprint is always computed on create**, independent of `dedupActive` — so a DÖE-only tender is immediately fingerprint-matchable the instant a second source activates, with no further backfill needed (13-01's backfill already covers the pre-existing 2851 rows; this covers every NEW tender ingested going forward). - **SCHEMA-02 mutable-field list copied verbatim** from the old `tender.upsert` UPDATE branch into `resolve()`'s match path, now gated by an explicit `contentHash !==` comparison (previously implicit in every upsert call) — behaviorally identical, with an added explicit "unchanged hash -> no update call" test that didn't exist before. - **`pruneExpiredTenders()`/`matchDelta` run once per tick, not once per source** — `anyDayFetched` and `newTenderIds` are tick-scoped across the whole fan-out loop, preserving the pre-existing once-per-tick retention/matching cadence. ## Deviations from Plan None — plan executed exactly as written. All four `must_haves.truths` are met and directly tested: 1. `dedupActive=false` inertness — tested (D1 above). 2. 2nd-source fingerprint match attaches TenderSource instead of creating a new Tender — tested (D2). 3. `pollDueSources` fans out via `findMany({isActive:true})` with catch-per-source — tested (D4). 4. SCHEMA-02 change detection preserved through the move into `resolve()` — tested (D3), with both the original ingestion-service spec suite kept green AND two new dedicated dedup-service tests. ## Issues Encountered None. Local (non-worktree, `main` branch) execution; no migrations touched in this plan (13-01 already applied the schema). No Docker/DB interaction required — pure TypeScript service/wiring plan, verified entirely via Vitest on host. ## User Setup Required None. No external service configuration, no migration, no environment changes. ## Next Phase Readiness - `TenderDedupService.resolve()` and the `SourceRegistry`-driven fan-out are the exact seams 13-04 (NetServer adapter) and 13-05 (cosinex adapter) need: adding a new adapter is (a) implement `TenderSourceAdapter`, (b) add one `providers` entry + one `registry.register(...)` call in `tenders.module.ts`'s `onModuleInit`, (c) seed a `TenderSourcePollConfig` row for the new `sourceType`. No further changes to `pollDueSources` or `TenderDedupService` are needed — both are already source-count-agnostic. - Cross-source dedup end-to-end (real DÖE + NetServer overlap producing ONE Tender with two TenderSource links) remains a **manual/live verification** per 13-VALIDATION.md — this plan's automated coverage proves the resolver and fan-out logic in isolation; the live confirmation needs a second active source (13-04) to observe. - No blockers. --- *Phase: 13-scraping-adapters-cross-source-dedup* *Completed: 2026-07-23* ## Self-Check: PASSED All created/modified files verified present on disk; all 4 task commit hashes (605ea4c, 1cd2fcf, d453dbb, 1d4f9cf) verified in git log.