tessera-ctl/.planning/phases/13-scraping-adapters-cross-source-dedup/13-03-SUMMARY.md

---
phase: 13-scraping-adapters-cross-source-dedup
plan: 03
subsystem: backend
tags: [nestjs, dedup, prisma, fan-out, error-isolation]

requires:
  - phase: 13-scraping-adapters-cross-source-dedup
    plan: 01
    provides: tenderFingerprint(), TenderSource model, Tender.fingerprint, SourceType open union
  - phase: 13-scraping-adapters-cross-source-dedup
    plan: 02
    provides: SourceRegistry (get/activeAdapters), TenderSourceAdapter.portals[], DeniedPortalError denylist gate
provides:
  - "TenderDedupService.resolve(n, {dedupActive}) — three-tier cross-source dedup resolver (OCID -> source:noticeId -> fingerprint), fingerprint tier hard-gated by dedupActive (D-05)"
  - "pollDueSources fan-out over ALL active TenderSourcePollConfig rows (findMany, not findUnique) with catch-per-source error isolation"
  - "SourceRegistry + TenderDedupService registered as TendersModule providers; DoeOpenDataAdapter registered with the registry at DI boot (onModuleInit)"
affects: [13-04-netserver-adapter, 13-05-cosinex-adapter, 13-06-multi-source-display]

tech-stack:
  added: []
  patterns:
    - "Dedup resolver owns both the create-vs-attach decision AND the SCHEMA-02 mutable-field refresh — the old direct tender.upsert UPDATE branch moved into resolve()'s match path unchanged in semantics"
    - "dedupActive computed once per tick from configs.length (activePortalCount >= 2) and threaded through every resolve() call in the fan-out loop"
    - "catch-per-source: each config's day-cursor-gate/fetch/normalize/resolve block runs in its own try/catch inside the fan-out for-loop, nested inside the existing outer tick-level try/catch"

key-files:
  created:
    - apps/api/src/tenders/tender-dedup.service.ts
    - apps/api/src/tenders/tender-dedup.service.spec.ts
  modified:
    - apps/api/src/tenders/tender-ingestion.service.ts
    - apps/api/src/tenders/tender-ingestion.service.spec.ts
    - apps/api/src/tenders/tenders.module.ts

key-decisions:
  - "Tier 2 (source:noticeId via TenderSource lookup) is NOT gated by dedupActive — it is the same-source re-poll identity check, independent from cross-source merging, and must remain active even with a single source (idempotent re-poll)."
  - "Fingerprint is always computed and stored on tender.create (n.fingerprint ?? tenderFingerprint(n)), regardless of dedupActive — a DÖE-only tender still gets a fingerprint so it becomes matchable the moment a 2nd source activates, without a further backfill."
  - "SCHEMA-02 mutable-field list is copied verbatim from the pre-existing tender.upsert UPDATE block (title, buyerName, cpvCodes, cpvDivisions, region, plz, bundesland, deadlineAt, estimatedValue, procedureType, sourceUrl, contentHash) into resolve()'s match branch, gated by a contentHash comparison instead of always running on every upsert call — functionally identical outcome, explicit no-op test added."
  - "pruneExpiredTenders() runs once per tick (not once per source) — anyDayFetched is now an across-all-sources flag, matching the pre-existing once-per-tick retention semantics."

patterns-established:
  - "Fake TenderDedupService.resolve() in tender-ingestion.service.spec.ts mirrors the real service's dedupKey-upsert contract (created vs. updated) without re-testing tier logic — keeps the fan-out spec focused on orchestration, not dedup internals (covered separately by tender-dedup.service.spec.ts)."

requirements-completed: [SCHEMA-03]

coverage:
  - id: D1
    description: "dedupActive=false: two normalized records with the same fingerprint but different source:noticeId create TWO Tender rows (fingerprint tier skipped, D-05 inert-proof, Erfolgskriterium 3)"
    requirement: SCHEMA-03
    verification:
      - kind: unit
        ref: "apps/api/src/tenders/tender-dedup.service.spec.ts — 'creates TWO Tender rows for records with the same fingerprint but different source:noticeId, when dedupActive=false'"
        status: pass
    human_judgment: false
  - id: D2
    description: "dedupActive=true: OCID match, idempotent source:noticeId re-poll, and fingerprint match all attach an additional TenderSource to the existing Tender (created=false, D-03 merge); tenderSource.upsert targets the @@unique([sourcePortal, sourceNoticeId]) composite key"
    requirement: SCHEMA-03
    verification:
      - kind: unit
        ref: "apps/api/src/tenders/tender-dedup.service.spec.ts — OCID-match/idempotent-reseen/fingerprint-match/no-match/composite-key tests (8/8 pass)"
        status: pass
    human_judgment: false
  - id: D3
    description: "SCHEMA-02 change-detection preserved through resolve(): a matched Tender whose incoming contentHash differs gets its mutable fields (title, deadlineAt, contentHash, etc.) updated; an unchanged contentHash does NOT call tender.update — no Phase-10 regression"
    requirement: SCHEMA-03
    verification:
      - kind: unit
        ref: "apps/api/src/tenders/tender-dedup.service.spec.ts — 'OCID match with a changed contentHash still updates...' + 'OCID match with an UNCHANGED contentHash does NOT call tender.update'"
        status: pass
      - kind: unit
        ref: "apps/api/src/tenders/tender-ingestion.service.spec.ts — existing SCHEMA-02 re-poll suite (insert-once, update-in-place-on-changed-hash) kept green through the resolve() hand-off"
        status: pass
    human_judgment: false
  - id: D4
    description: "pollDueSources fans out over ALL active TenderSourcePollConfig rows (findMany, not findFirst/findUnique) with catch-per-source error isolation — one broken/blocking source does not abort polling of the others, nor does it throw out of the tick"
    requirement: SCHEMA-03
    verification:
      - kind: unit
        ref: "apps/api/src/tenders/tender-ingestion.service.spec.ts — 'polls every active config...', 'skips a config whose sourceType has no registered adapter, without throwing', 'a throwing source does not prevent another active source from being polled' (14/14 pass)"
        status: pass
    human_judgment: false
  - id: D5
    description: "dedupActive is bound to activePortalCount >= 2 and threaded into every resolve() call; delta-only matchDelta boundary (D-07) preserved across the fan-out (only genuinely-created IDs across all sources)"
    requirement: SCHEMA-03
    verification:
      - kind: unit
        ref: "apps/api/src/tenders/tender-ingestion.service.spec.ts — 'passes dedupActive=false...single config', 'passes dedupActive=true...two configs', existing D-07 delta-only suite kept green"
        status: pass
    human_judgment: false
  - id: D6
    description: "SourceRegistry + TenderDedupService are TendersModule providers; DoeOpenDataAdapter is registered with the registry in onModuleInit (denylist gate applies at DI boot); DI graph resolves cleanly"
    requirement: SCHEMA-03
    verification:
      - kind: unit
        ref: "cd apps/api && npx tsc --noEmit -p tsconfig.json (clean)"
        status: pass
      - kind: unit
        ref: "cd apps/api && npx vitest run src/tenders (19 files, 190/190 pass)"
        status: pass
      - kind: unit
        ref: "cd apps/api && npx vitest run (full suite, 23 files, 253/253 pass)"
        status: pass
    human_judgment: false

duration: 45min
completed: 2026-07-23
status: complete
---

# Phase 13 Plan 03: Dedup Resolver + Fan-out Wiring Summary

**Three-tier `TenderDedupService.resolve()` (OCID -> source:noticeId -> fingerprint, fingerprint tier hard-gated by `activePortalCount >= 2`) now drives `pollDueSources`, which fans out over every active `TenderSourcePollConfig` with catch-per-source error isolation — replacing the DÖE-only `findUnique`/direct `tender.upsert`, while preserving SCHEMA-02 change detection byte-for-byte.**

## Performance

- **Duration:** 45 min
- **Started:** 2026-07-23T08:45:00Z
- **Completed:** 2026-07-23T09:30:00Z
- **Tasks:** 3
- **Files modified:** 5 (2 created, 3 modified)

## Accomplishments
- `TenderDedupService.resolve(n, {dedupActive})`: OCID -> source:noticeId -> fingerprint (D-04), fingerprint tier only reachable when `dedupActive` is true (D-05). Match -> `tenderSource.upsert` attaches an additional source to the existing Tender (`created: false`, D-03 merge) + SCHEMA-02 mutable-field refresh when `contentHash` differs. No match -> `tender.create` (with computed `fingerprint`) + `tenderSource.create` (`created: true`). Plain `PrismaService`, no `forTenant()`/RLS. TDD RED->GREEN, 8/8 tests green, including the D-05 inert-proof (fingerprint-colliding records with `dedupActive=false` stay two Tenders) and two dedicated SCHEMA-02-preservation tests.
- `pollDueSources` rebuilt on `findMany({ isActive: true })` fan-out (poll-once-fan-out-many, NOT findFirst/findUnique). Each config's day-cursor-gate/fetch/normalize/resolve block runs inside its own `try/catch` — a throwing source (portal down) no longer kills the tick or blocks other sources (D-01). `dedupActive = configs.length >= 2` computed once per tick and threaded through every `resolve()` call. `pruneExpiredTenders()` and `matchDelta` (D-07 delta-only boundary) run once across the whole tick, unchanged in intent. Existing SCHEMA-02/D-07/retention/day-cursor spec suites updated to the new `(prisma, registry, normalizer, matching, dedup)` constructor shape and kept green; four new fan-out/catch-per-source/dedupActive-gate tests added. 14/14 pass.
- `TendersModule`: `SourceRegistry` + `TenderDedupService` added as providers; `onModuleInit` registers `DoeOpenDataAdapter` with the registry before the scheduler's first tick can fire — the DI-boot-time point where the INGEST-07 denylist gate (D-06) structurally applies. Explicitly documented as the sole 13-04/13-05 extension point for future adapter registrations.
- `npx tsc --noEmit` clean; `src/tenders` slice 19 files/190 tests green; full API suite 23 files/253 tests green.

## Task Commits

Each task was committed atomically:

1. **Task 1a: RED — failing TenderDedupService spec** - `605ea4c` (test)
2. **Task 1b: GREEN — TenderDedupService three-tier resolver** - `1cd2fcf` (feat)
3. **Task 2: pollDueSources fan-out + catch-per-source + dedup hook** - `d453dbb` (feat)
4. **Task 3: Module wiring — SourceRegistry + TenderDedupService providers** - `1d4f9cf` (feat)

_Task 1 (tdd="true") produced two commits (test -> feat) per protocol; no refactor commit was needed — the first implementation passed all 8 tests cleanly._

## Files Created/Modified
- `apps/api/src/tenders/tender-dedup.service.ts` - `@Injectable() TenderDedupService`, three-tier `resolve()` resolver
- `apps/api/src/tenders/tender-dedup.service.spec.ts` - 8 unit tests (D-05 inert-proof, tier1/2/3 match, no-match create, composite-key upsert target, SCHEMA-02 preservation x2)
- `apps/api/src/tenders/tender-ingestion.service.ts` - `pollDueSources` rebuilt as `findMany` fan-out with catch-per-source + `dedup.resolve()` hand-off; constructor now takes `(prisma, registry, normalizer, matching, dedup)`
- `apps/api/src/tenders/tender-ingestion.service.spec.ts` - fake `registry`/`dedup` collaborators; existing suites migrated to the new constructor shape; 4 new tests (fan-out, adapter-missing skip, catch-per-source isolation, dedupActive gate x2)
- `apps/api/src/tenders/tenders.module.ts` - `SourceRegistry` + `TenderDedupService` providers; `onModuleInit` registers `DoeOpenDataAdapter`

## Decisions Made
- **Tier 2 (source:noticeId) is unconditional, not gated by `dedupActive`** — it's the same-source re-poll identity check (idempotency), orthogonal to cross-source fingerprint merging. Gating it would have broken idempotent re-polls of a single active source.
- **Fingerprint is always computed on create**, independent of `dedupActive` — so a DÖE-only tender is immediately fingerprint-matchable the instant a second source activates, with no further backfill needed (13-01's backfill already covers the pre-existing 2851 rows; this covers every NEW tender ingested going forward).
- **SCHEMA-02 mutable-field list copied verbatim** from the old `tender.upsert` UPDATE branch into `resolve()`'s match path, now gated by an explicit `contentHash !==` comparison (previously implicit in every upsert call) — behaviorally identical, with an added explicit "unchanged hash -> no update call" test that didn't exist before.
- **`pruneExpiredTenders()`/`matchDelta` run once per tick, not once per source** — `anyDayFetched` and `newTenderIds` are tick-scoped across the whole fan-out loop, preserving the pre-existing once-per-tick retention/matching cadence.

## Deviations from Plan

None — plan executed exactly as written. All four `must_haves.truths` are met and directly tested:
1. `dedupActive=false` inertness — tested (D1 above).
2. 2nd-source fingerprint match attaches TenderSource instead of creating a new Tender — tested (D2).
3. `pollDueSources` fans out via `findMany({isActive:true})` with catch-per-source — tested (D4).
4. SCHEMA-02 change detection preserved through the move into `resolve()` — tested (D3), with both the original ingestion-service spec suite kept green AND two new dedicated dedup-service tests.

## Issues Encountered

None. Local (non-worktree, `main` branch) execution; no migrations touched in this plan (13-01 already applied the schema). No Docker/DB interaction required — pure TypeScript service/wiring plan, verified entirely via Vitest on host.

## User Setup Required

None. No external service configuration, no migration, no environment changes.

## Next Phase Readiness
- `TenderDedupService.resolve()` and the `SourceRegistry`-driven fan-out are the exact seams 13-04 (NetServer adapter) and 13-05 (cosinex adapter) need: adding a new adapter is (a) implement `TenderSourceAdapter`, (b) add one `providers` entry + one `registry.register(...)` call in `tenders.module.ts`'s `onModuleInit`, (c) seed a `TenderSourcePollConfig` row for the new `sourceType`. No further changes to `pollDueSources` or `TenderDedupService` are needed — both are already source-count-agnostic.
- Cross-source dedup end-to-end (real DÖE + NetServer overlap producing ONE Tender with two TenderSource links) remains a **manual/live verification** per 13-VALIDATION.md — this plan's automated coverage proves the resolver and fan-out logic in isolation; the live confirmation needs a second active source (13-04) to observe.
- No blockers.

---
*Phase: 13-scraping-adapters-cross-source-dedup*
*Completed: 2026-07-23*

## Self-Check: PASSED

All created/modified files verified present on disk; all 4 task commit hashes (605ea4c, 1cd2fcf, d453dbb, 1d4f9cf) verified in git log.