From 1c523c039cc077f0e0ba02514ca9a25d645342d3 Mon Sep 17 00:00:00 2001 From: Schalli Date: Thu, 23 Jul 2026 08:53:33 +0200 Subject: [PATCH] docs(13-03): complete dedup-resolver-fan-out-wiring plan --- .planning/ROADMAP.md | 6 +- .planning/STATE.md | 17 +- .../13-03-SUMMARY.md | 180 ++++++++++++++++++ 3 files changed, 193 insertions(+), 10 deletions(-) create mode 100644 .planning/phases/13-scraping-adapters-cross-source-dedup/13-03-SUMMARY.md diff --git a/.planning/ROADMAP.md b/.planning/ROADMAP.md index aa2f934..4bf15fc 100644 --- a/.planning/ROADMAP.md +++ b/.planning/ROADMAP.md @@ -439,13 +439,13 @@ Plans: 3. A tender that appears via both DÖE and a scraping adapter shows up once in the results list (fuzzy fingerprint dedup on buyer+title+CPV+deadline+value), with links to all of its source portals -- dedup logic only activates once a second source is live 4. Attempting to register vergabe24 or aumass as a poll source is refused by the system itself (adapter registry denylist enforced in code), not just documented as forbidden -**Plans**: 2/6 plans executed +**Plans**: 3/6 plans executed Plans: - [x] 13-01-PLAN.md — TenderSource-Schema + Backfill + NULL-tolerante Fingerprint-Fn + SourceType-Union (SCHEMA-03 Datenschicht) - [x] 13-02-PLAN.md — SourceRegistry + harte Denylist-Gate (vergabe24/aumass) + Adapter-Interface-Generalisierung (INGEST-07) -- [ ] 13-03-PLAN.md — 3-Stufen-Dedup-Resolver + pollDueSources Fan-out (catch-per-source) + Modul-Wiring (SCHEMA-03) +- [x] 13-03-PLAN.md — 3-Stufen-Dedup-Resolver + pollDueSources Fan-out (catch-per-source) + Modul-Wiring (SCHEMA-03) - [ ] 13-04-PLAN.md — NetServer-Adapter (config-getrieben, 3 Portale) + Package-Legitimacy-Checkpoint (INGEST-02) - [ ] 13-05-PLAN.md — cosinex/DTVP-Adapter (separat, best-effort) (INGEST-03) - [ ] 13-06-PLAN.md — Read-Endpoint include sources[] + TenderDetail Multi-Source-Links (SCHEMA-03 Anzeige) @@ -486,5 +486,5 @@ Phases execute in numeric order: 1 -> 2 -> 3 -> 4 -> 5 -> 6 -> 7 -> 8 -> 9 -> 10 | 10. Ausschreibungs-Radar Foundation & DÖE Ingestion | 6/6 | Complete | 2026-07-21 | | 11. Filter Engine, Results UI & Saved Searches | 6/6 | In Progress| | | 12. Tender Notifications | 4/4 | In Progress| | -| 13. Scraping Adapters & Cross-Source Deduplication | 2/6 | In Progress| | +| 13. Scraping Adapters & Cross-Source Deduplication | 3/6 | In Progress| | | 14. RSS, Email-Alert Ingestion & Module Rollout | 0/TBD | Not started | - | diff --git a/.planning/STATE.md b/.planning/STATE.md index b2134c9..3ceea34 100644 --- a/.planning/STATE.md +++ b/.planning/STATE.md @@ -5,15 +5,15 @@ milestone_name: Ausschreibungs-Radar current_phase: 13 current_phase_name: scraping-adapters-cross-source-dedup status: executing -stopped_at: Completed 13-02-PLAN.md -last_updated: "2026-07-23T06:44:34.483Z" +stopped_at: Completed 13-03-PLAN.md +last_updated: "2026-07-23T06:53:28.370Z" last_activity: 2026-07-23 last_activity_desc: Phase 13 execution started progress: total_phases: 13 completed_phases: 11 total_plans: 62 - completed_plans: 57 + completed_plans: 58 --- # Project State @@ -28,11 +28,11 @@ See: .planning/PROJECT.md (updated 2026-07-17) ## Current Position Phase: 13 (scraping-adapters-cross-source-dedup) — EXECUTING -Plan: 3 of 6 +Plan: 4 of 6 Status: Ready to execute Last activity: 2026-07-23 — Phase 13 execution started -Progress: [█████████░] 92% +Progress: [█████████░] 94% ## Performance Metrics @@ -92,6 +92,7 @@ Progress: [█████████░] 92% | Phase 12 P04 | 12min | 3 tasks | 10 files | | Phase 13 P01 | 35min | 3 tasks | 6 files | | Phase 13 P02 | 20min | 2 tasks | 4 files | +| Phase 13 P03 | 45min | 3 tasks | 5 files | ## Accumulated Context @@ -196,6 +197,8 @@ Recent decisions affecting current work: - [Phase ?]: SCHEMA-03 fingerprint: title+buyer dominant, CPV division, value-bucket, deadline-day, sha256; dedupKey untouched, fingerprint additive - [Phase ?]: One-time TS backfill scripts run via compiled dist/ output (not raw .ts execution) to keep tsc --noEmit clean - [Phase ?]: SourceRegistry.register() throws DeniedPortalError for any portal in DENYLISTED_PORTALS (vergabe24, aumass), enforced at DI-registration time not just documented (INGEST-07) +- [Phase ?]: TenderDedupService tier-2 (source:noticeId) match is unconditional (not gated by dedupActive) — idempotent same-source re-poll must work even with a single active source +- [Phase ?]: Fingerprint always computed on tender.create regardless of dedupActive, so DÖE-only tenders become fingerprint-matchable the instant a 2nd source activates without further backfill ### Pending Todos @@ -233,7 +236,7 @@ Items acknowledged and carried forward from previous milestone close: ## Session Continuity -Last session: 2026-07-23T06:44:34.470Z -Stopped at: Completed 13-02-PLAN.md +Last session: 2026-07-23T06:53:22.726Z +Stopped at: Completed 13-03-PLAN.md Resume file: None Last activity: 2026-07-14 - Built LDAP per-user exclude/denylist filter (9d1323f), migration applied on live DB, verified via Playwright: sync deactivated 4 excluded service accounts (administrator/krbtgt/guest/dns-ldap), 2 real LDAP users stay active, 0 wrongly created diff --git a/.planning/phases/13-scraping-adapters-cross-source-dedup/13-03-SUMMARY.md b/.planning/phases/13-scraping-adapters-cross-source-dedup/13-03-SUMMARY.md new file mode 100644 index 0000000..bab924f --- /dev/null +++ b/.planning/phases/13-scraping-adapters-cross-source-dedup/13-03-SUMMARY.md @@ -0,0 +1,180 @@ +--- +phase: 13-scraping-adapters-cross-source-dedup +plan: 03 +subsystem: backend +tags: [nestjs, dedup, prisma, fan-out, error-isolation] + +requires: + - phase: 13-scraping-adapters-cross-source-dedup + plan: 01 + provides: tenderFingerprint(), TenderSource model, Tender.fingerprint, SourceType open union + - phase: 13-scraping-adapters-cross-source-dedup + plan: 02 + provides: SourceRegistry (get/activeAdapters), TenderSourceAdapter.portals[], DeniedPortalError denylist gate +provides: + - "TenderDedupService.resolve(n, {dedupActive}) — three-tier cross-source dedup resolver (OCID -> source:noticeId -> fingerprint), fingerprint tier hard-gated by dedupActive (D-05)" + - "pollDueSources fan-out over ALL active TenderSourcePollConfig rows (findMany, not findUnique) with catch-per-source error isolation" + - "SourceRegistry + TenderDedupService registered as TendersModule providers; DoeOpenDataAdapter registered with the registry at DI boot (onModuleInit)" +affects: [13-04-netserver-adapter, 13-05-cosinex-adapter, 13-06-multi-source-display] + +tech-stack: + added: [] + patterns: + - "Dedup resolver owns both the create-vs-attach decision AND the SCHEMA-02 mutable-field refresh — the old direct tender.upsert UPDATE branch moved into resolve()'s match path unchanged in semantics" + - "dedupActive computed once per tick from configs.length (activePortalCount >= 2) and threaded through every resolve() call in the fan-out loop" + - "catch-per-source: each config's day-cursor-gate/fetch/normalize/resolve block runs in its own try/catch inside the fan-out for-loop, nested inside the existing outer tick-level try/catch" + +key-files: + created: + - apps/api/src/tenders/tender-dedup.service.ts + - apps/api/src/tenders/tender-dedup.service.spec.ts + modified: + - apps/api/src/tenders/tender-ingestion.service.ts + - apps/api/src/tenders/tender-ingestion.service.spec.ts + - apps/api/src/tenders/tenders.module.ts + +key-decisions: + - "Tier 2 (source:noticeId via TenderSource lookup) is NOT gated by dedupActive — it is the same-source re-poll identity check, independent from cross-source merging, and must remain active even with a single source (idempotent re-poll)." + - "Fingerprint is always computed and stored on tender.create (n.fingerprint ?? tenderFingerprint(n)), regardless of dedupActive — a DÖE-only tender still gets a fingerprint so it becomes matchable the moment a 2nd source activates, without a further backfill." + - "SCHEMA-02 mutable-field list is copied verbatim from the pre-existing tender.upsert UPDATE block (title, buyerName, cpvCodes, cpvDivisions, region, plz, bundesland, deadlineAt, estimatedValue, procedureType, sourceUrl, contentHash) into resolve()'s match branch, gated by a contentHash comparison instead of always running on every upsert call — functionally identical outcome, explicit no-op test added." + - "pruneExpiredTenders() runs once per tick (not once per source) — anyDayFetched is now an across-all-sources flag, matching the pre-existing once-per-tick retention semantics." + +patterns-established: + - "Fake TenderDedupService.resolve() in tender-ingestion.service.spec.ts mirrors the real service's dedupKey-upsert contract (created vs. updated) without re-testing tier logic — keeps the fan-out spec focused on orchestration, not dedup internals (covered separately by tender-dedup.service.spec.ts)." + +requirements-completed: [SCHEMA-03] + +coverage: + - id: D1 + description: "dedupActive=false: two normalized records with the same fingerprint but different source:noticeId create TWO Tender rows (fingerprint tier skipped, D-05 inert-proof, Erfolgskriterium 3)" + requirement: SCHEMA-03 + verification: + - kind: unit + ref: "apps/api/src/tenders/tender-dedup.service.spec.ts — 'creates TWO Tender rows for records with the same fingerprint but different source:noticeId, when dedupActive=false'" + status: pass + human_judgment: false + - id: D2 + description: "dedupActive=true: OCID match, idempotent source:noticeId re-poll, and fingerprint match all attach an additional TenderSource to the existing Tender (created=false, D-03 merge); tenderSource.upsert targets the @@unique([sourcePortal, sourceNoticeId]) composite key" + requirement: SCHEMA-03 + verification: + - kind: unit + ref: "apps/api/src/tenders/tender-dedup.service.spec.ts — OCID-match/idempotent-reseen/fingerprint-match/no-match/composite-key tests (8/8 pass)" + status: pass + human_judgment: false + - id: D3 + description: "SCHEMA-02 change-detection preserved through resolve(): a matched Tender whose incoming contentHash differs gets its mutable fields (title, deadlineAt, contentHash, etc.) updated; an unchanged contentHash does NOT call tender.update — no Phase-10 regression" + requirement: SCHEMA-03 + verification: + - kind: unit + ref: "apps/api/src/tenders/tender-dedup.service.spec.ts — 'OCID match with a changed contentHash still updates...' + 'OCID match with an UNCHANGED contentHash does NOT call tender.update'" + status: pass + - kind: unit + ref: "apps/api/src/tenders/tender-ingestion.service.spec.ts — existing SCHEMA-02 re-poll suite (insert-once, update-in-place-on-changed-hash) kept green through the resolve() hand-off" + status: pass + human_judgment: false + - id: D4 + description: "pollDueSources fans out over ALL active TenderSourcePollConfig rows (findMany, not findFirst/findUnique) with catch-per-source error isolation — one broken/blocking source does not abort polling of the others, nor does it throw out of the tick" + requirement: SCHEMA-03 + verification: + - kind: unit + ref: "apps/api/src/tenders/tender-ingestion.service.spec.ts — 'polls every active config...', 'skips a config whose sourceType has no registered adapter, without throwing', 'a throwing source does not prevent another active source from being polled' (14/14 pass)" + status: pass + human_judgment: false + - id: D5 + description: "dedupActive is bound to activePortalCount >= 2 and threaded into every resolve() call; delta-only matchDelta boundary (D-07) preserved across the fan-out (only genuinely-created IDs across all sources)" + requirement: SCHEMA-03 + verification: + - kind: unit + ref: "apps/api/src/tenders/tender-ingestion.service.spec.ts — 'passes dedupActive=false...single config', 'passes dedupActive=true...two configs', existing D-07 delta-only suite kept green" + status: pass + human_judgment: false + - id: D6 + description: "SourceRegistry + TenderDedupService are TendersModule providers; DoeOpenDataAdapter is registered with the registry in onModuleInit (denylist gate applies at DI boot); DI graph resolves cleanly" + requirement: SCHEMA-03 + verification: + - kind: unit + ref: "cd apps/api && npx tsc --noEmit -p tsconfig.json (clean)" + status: pass + - kind: unit + ref: "cd apps/api && npx vitest run src/tenders (19 files, 190/190 pass)" + status: pass + - kind: unit + ref: "cd apps/api && npx vitest run (full suite, 23 files, 253/253 pass)" + status: pass + human_judgment: false + +duration: 45min +completed: 2026-07-23 +status: complete +--- + +# Phase 13 Plan 03: Dedup Resolver + Fan-out Wiring Summary + +**Three-tier `TenderDedupService.resolve()` (OCID -> source:noticeId -> fingerprint, fingerprint tier hard-gated by `activePortalCount >= 2`) now drives `pollDueSources`, which fans out over every active `TenderSourcePollConfig` with catch-per-source error isolation — replacing the DÖE-only `findUnique`/direct `tender.upsert`, while preserving SCHEMA-02 change detection byte-for-byte.** + +## Performance + +- **Duration:** 45 min +- **Started:** 2026-07-23T08:45:00Z +- **Completed:** 2026-07-23T09:30:00Z +- **Tasks:** 3 +- **Files modified:** 5 (2 created, 3 modified) + +## Accomplishments +- `TenderDedupService.resolve(n, {dedupActive})`: OCID -> source:noticeId -> fingerprint (D-04), fingerprint tier only reachable when `dedupActive` is true (D-05). Match -> `tenderSource.upsert` attaches an additional source to the existing Tender (`created: false`, D-03 merge) + SCHEMA-02 mutable-field refresh when `contentHash` differs. No match -> `tender.create` (with computed `fingerprint`) + `tenderSource.create` (`created: true`). Plain `PrismaService`, no `forTenant()`/RLS. TDD RED->GREEN, 8/8 tests green, including the D-05 inert-proof (fingerprint-colliding records with `dedupActive=false` stay two Tenders) and two dedicated SCHEMA-02-preservation tests. +- `pollDueSources` rebuilt on `findMany({ isActive: true })` fan-out (poll-once-fan-out-many, NOT findFirst/findUnique). Each config's day-cursor-gate/fetch/normalize/resolve block runs inside its own `try/catch` — a throwing source (portal down) no longer kills the tick or blocks other sources (D-01). `dedupActive = configs.length >= 2` computed once per tick and threaded through every `resolve()` call. `pruneExpiredTenders()` and `matchDelta` (D-07 delta-only boundary) run once across the whole tick, unchanged in intent. Existing SCHEMA-02/D-07/retention/day-cursor spec suites updated to the new `(prisma, registry, normalizer, matching, dedup)` constructor shape and kept green; four new fan-out/catch-per-source/dedupActive-gate tests added. 14/14 pass. +- `TendersModule`: `SourceRegistry` + `TenderDedupService` added as providers; `onModuleInit` registers `DoeOpenDataAdapter` with the registry before the scheduler's first tick can fire — the DI-boot-time point where the INGEST-07 denylist gate (D-06) structurally applies. Explicitly documented as the sole 13-04/13-05 extension point for future adapter registrations. +- `npx tsc --noEmit` clean; `src/tenders` slice 19 files/190 tests green; full API suite 23 files/253 tests green. + +## Task Commits + +Each task was committed atomically: + +1. **Task 1a: RED — failing TenderDedupService spec** - `605ea4c` (test) +2. **Task 1b: GREEN — TenderDedupService three-tier resolver** - `1cd2fcf` (feat) +3. **Task 2: pollDueSources fan-out + catch-per-source + dedup hook** - `d453dbb` (feat) +4. **Task 3: Module wiring — SourceRegistry + TenderDedupService providers** - `1d4f9cf` (feat) + +_Task 1 (tdd="true") produced two commits (test -> feat) per protocol; no refactor commit was needed — the first implementation passed all 8 tests cleanly._ + +## Files Created/Modified +- `apps/api/src/tenders/tender-dedup.service.ts` - `@Injectable() TenderDedupService`, three-tier `resolve()` resolver +- `apps/api/src/tenders/tender-dedup.service.spec.ts` - 8 unit tests (D-05 inert-proof, tier1/2/3 match, no-match create, composite-key upsert target, SCHEMA-02 preservation x2) +- `apps/api/src/tenders/tender-ingestion.service.ts` - `pollDueSources` rebuilt as `findMany` fan-out with catch-per-source + `dedup.resolve()` hand-off; constructor now takes `(prisma, registry, normalizer, matching, dedup)` +- `apps/api/src/tenders/tender-ingestion.service.spec.ts` - fake `registry`/`dedup` collaborators; existing suites migrated to the new constructor shape; 4 new tests (fan-out, adapter-missing skip, catch-per-source isolation, dedupActive gate x2) +- `apps/api/src/tenders/tenders.module.ts` - `SourceRegistry` + `TenderDedupService` providers; `onModuleInit` registers `DoeOpenDataAdapter` + +## Decisions Made +- **Tier 2 (source:noticeId) is unconditional, not gated by `dedupActive`** — it's the same-source re-poll identity check (idempotency), orthogonal to cross-source fingerprint merging. Gating it would have broken idempotent re-polls of a single active source. +- **Fingerprint is always computed on create**, independent of `dedupActive` — so a DÖE-only tender is immediately fingerprint-matchable the instant a second source activates, with no further backfill needed (13-01's backfill already covers the pre-existing 2851 rows; this covers every NEW tender ingested going forward). +- **SCHEMA-02 mutable-field list copied verbatim** from the old `tender.upsert` UPDATE branch into `resolve()`'s match path, now gated by an explicit `contentHash !==` comparison (previously implicit in every upsert call) — behaviorally identical, with an added explicit "unchanged hash -> no update call" test that didn't exist before. +- **`pruneExpiredTenders()`/`matchDelta` run once per tick, not once per source** — `anyDayFetched` and `newTenderIds` are tick-scoped across the whole fan-out loop, preserving the pre-existing once-per-tick retention/matching cadence. + +## Deviations from Plan + +None — plan executed exactly as written. All four `must_haves.truths` are met and directly tested: +1. `dedupActive=false` inertness — tested (D1 above). +2. 2nd-source fingerprint match attaches TenderSource instead of creating a new Tender — tested (D2). +3. `pollDueSources` fans out via `findMany({isActive:true})` with catch-per-source — tested (D4). +4. SCHEMA-02 change detection preserved through the move into `resolve()` — tested (D3), with both the original ingestion-service spec suite kept green AND two new dedicated dedup-service tests. + +## Issues Encountered + +None. Local (non-worktree, `main` branch) execution; no migrations touched in this plan (13-01 already applied the schema). No Docker/DB interaction required — pure TypeScript service/wiring plan, verified entirely via Vitest on host. + +## User Setup Required + +None. No external service configuration, no migration, no environment changes. + +## Next Phase Readiness +- `TenderDedupService.resolve()` and the `SourceRegistry`-driven fan-out are the exact seams 13-04 (NetServer adapter) and 13-05 (cosinex adapter) need: adding a new adapter is (a) implement `TenderSourceAdapter`, (b) add one `providers` entry + one `registry.register(...)` call in `tenders.module.ts`'s `onModuleInit`, (c) seed a `TenderSourcePollConfig` row for the new `sourceType`. No further changes to `pollDueSources` or `TenderDedupService` are needed — both are already source-count-agnostic. +- Cross-source dedup end-to-end (real DÖE + NetServer overlap producing ONE Tender with two TenderSource links) remains a **manual/live verification** per 13-VALIDATION.md — this plan's automated coverage proves the resolver and fan-out logic in isolation; the live confirmation needs a second active source (13-04) to observe. +- No blockers. + +--- +*Phase: 13-scraping-adapters-cross-source-dedup* +*Completed: 2026-07-23* + +## Self-Check: PASSED + +All created/modified files verified present on disk; all 4 task commit hashes (605ea4c, 1cd2fcf, d453dbb, 1d4f9cf) verified in git log.