docs(13-03): complete dedup-resolver-fan-out-wiring plan
This commit is contained in:
@@ -439,13 +439,13 @@ Plans:
|
||||
3. A tender that appears via both DÖE and a scraping adapter shows up once in the results list (fuzzy fingerprint dedup on buyer+title+CPV+deadline+value), with links to all of its source portals -- dedup logic only activates once a second source is live
|
||||
4. Attempting to register vergabe24 or aumass as a poll source is refused by the system itself (adapter registry denylist enforced in code), not just documented as forbidden
|
||||
|
||||
**Plans**: 2/6 plans executed
|
||||
**Plans**: 3/6 plans executed
|
||||
|
||||
Plans:
|
||||
|
||||
- [x] 13-01-PLAN.md — TenderSource-Schema + Backfill + NULL-tolerante Fingerprint-Fn + SourceType-Union (SCHEMA-03 Datenschicht)
|
||||
- [x] 13-02-PLAN.md — SourceRegistry + harte Denylist-Gate (vergabe24/aumass) + Adapter-Interface-Generalisierung (INGEST-07)
|
||||
- [ ] 13-03-PLAN.md — 3-Stufen-Dedup-Resolver + pollDueSources Fan-out (catch-per-source) + Modul-Wiring (SCHEMA-03)
|
||||
- [x] 13-03-PLAN.md — 3-Stufen-Dedup-Resolver + pollDueSources Fan-out (catch-per-source) + Modul-Wiring (SCHEMA-03)
|
||||
- [ ] 13-04-PLAN.md — NetServer-Adapter (config-getrieben, 3 Portale) + Package-Legitimacy-Checkpoint (INGEST-02)
|
||||
- [ ] 13-05-PLAN.md — cosinex/DTVP-Adapter (separat, best-effort) (INGEST-03)
|
||||
- [ ] 13-06-PLAN.md — Read-Endpoint include sources[] + TenderDetail Multi-Source-Links (SCHEMA-03 Anzeige)
|
||||
@@ -486,5 +486,5 @@ Phases execute in numeric order: 1 -> 2 -> 3 -> 4 -> 5 -> 6 -> 7 -> 8 -> 9 -> 10
|
||||
| 10. Ausschreibungs-Radar Foundation & DÖE Ingestion | 6/6 | Complete | 2026-07-21 |
|
||||
| 11. Filter Engine, Results UI & Saved Searches | 6/6 | In Progress| |
|
||||
| 12. Tender Notifications | 4/4 | In Progress| |
|
||||
| 13. Scraping Adapters & Cross-Source Deduplication | 2/6 | In Progress| |
|
||||
| 13. Scraping Adapters & Cross-Source Deduplication | 3/6 | In Progress| |
|
||||
| 14. RSS, Email-Alert Ingestion & Module Rollout | 0/TBD | Not started | - |
|
||||
|
||||
+10
-7
@@ -5,15 +5,15 @@ milestone_name: Ausschreibungs-Radar
|
||||
current_phase: 13
|
||||
current_phase_name: scraping-adapters-cross-source-dedup
|
||||
status: executing
|
||||
stopped_at: Completed 13-02-PLAN.md
|
||||
last_updated: "2026-07-23T06:44:34.483Z"
|
||||
stopped_at: Completed 13-03-PLAN.md
|
||||
last_updated: "2026-07-23T06:53:28.370Z"
|
||||
last_activity: 2026-07-23
|
||||
last_activity_desc: Phase 13 execution started
|
||||
progress:
|
||||
total_phases: 13
|
||||
completed_phases: 11
|
||||
total_plans: 62
|
||||
completed_plans: 57
|
||||
completed_plans: 58
|
||||
---
|
||||
|
||||
# Project State
|
||||
@@ -28,11 +28,11 @@ See: .planning/PROJECT.md (updated 2026-07-17)
|
||||
## Current Position
|
||||
|
||||
Phase: 13 (scraping-adapters-cross-source-dedup) — EXECUTING
|
||||
Plan: 3 of 6
|
||||
Plan: 4 of 6
|
||||
Status: Ready to execute
|
||||
Last activity: 2026-07-23 — Phase 13 execution started
|
||||
|
||||
Progress: [█████████░] 92%
|
||||
Progress: [█████████░] 94%
|
||||
|
||||
## Performance Metrics
|
||||
|
||||
@@ -92,6 +92,7 @@ Progress: [█████████░] 92%
|
||||
| Phase 12 P04 | 12min | 3 tasks | 10 files |
|
||||
| Phase 13 P01 | 35min | 3 tasks | 6 files |
|
||||
| Phase 13 P02 | 20min | 2 tasks | 4 files |
|
||||
| Phase 13 P03 | 45min | 3 tasks | 5 files |
|
||||
|
||||
## Accumulated Context
|
||||
|
||||
@@ -196,6 +197,8 @@ Recent decisions affecting current work:
|
||||
- [Phase ?]: SCHEMA-03 fingerprint: title+buyer dominant, CPV division, value-bucket, deadline-day, sha256; dedupKey untouched, fingerprint additive
|
||||
- [Phase ?]: One-time TS backfill scripts run via compiled dist/ output (not raw .ts execution) to keep tsc --noEmit clean
|
||||
- [Phase ?]: SourceRegistry.register() throws DeniedPortalError for any portal in DENYLISTED_PORTALS (vergabe24, aumass), enforced at DI-registration time not just documented (INGEST-07)
|
||||
- [Phase ?]: TenderDedupService tier-2 (source:noticeId) match is unconditional (not gated by dedupActive) — idempotent same-source re-poll must work even with a single active source
|
||||
- [Phase ?]: Fingerprint always computed on tender.create regardless of dedupActive, so DÖE-only tenders become fingerprint-matchable the instant a 2nd source activates without further backfill
|
||||
|
||||
### Pending Todos
|
||||
|
||||
@@ -233,7 +236,7 @@ Items acknowledged and carried forward from previous milestone close:
|
||||
|
||||
## Session Continuity
|
||||
|
||||
Last session: 2026-07-23T06:44:34.470Z
|
||||
Stopped at: Completed 13-02-PLAN.md
|
||||
Last session: 2026-07-23T06:53:22.726Z
|
||||
Stopped at: Completed 13-03-PLAN.md
|
||||
Resume file: None
|
||||
Last activity: 2026-07-14 - Built LDAP per-user exclude/denylist filter (9d1323f), migration applied on live DB, verified via Playwright: sync deactivated 4 excluded service accounts (administrator/krbtgt/guest/dns-ldap), 2 real LDAP users stay active, 0 wrongly created
|
||||
|
||||
@@ -0,0 +1,180 @@
|
||||
---
|
||||
phase: 13-scraping-adapters-cross-source-dedup
|
||||
plan: 03
|
||||
subsystem: backend
|
||||
tags: [nestjs, dedup, prisma, fan-out, error-isolation]
|
||||
|
||||
requires:
|
||||
- phase: 13-scraping-adapters-cross-source-dedup
|
||||
plan: 01
|
||||
provides: tenderFingerprint(), TenderSource model, Tender.fingerprint, SourceType open union
|
||||
- phase: 13-scraping-adapters-cross-source-dedup
|
||||
plan: 02
|
||||
provides: SourceRegistry (get/activeAdapters), TenderSourceAdapter.portals[], DeniedPortalError denylist gate
|
||||
provides:
|
||||
- "TenderDedupService.resolve(n, {dedupActive}) — three-tier cross-source dedup resolver (OCID -> source:noticeId -> fingerprint), fingerprint tier hard-gated by dedupActive (D-05)"
|
||||
- "pollDueSources fan-out over ALL active TenderSourcePollConfig rows (findMany, not findUnique) with catch-per-source error isolation"
|
||||
- "SourceRegistry + TenderDedupService registered as TendersModule providers; DoeOpenDataAdapter registered with the registry at DI boot (onModuleInit)"
|
||||
affects: [13-04-netserver-adapter, 13-05-cosinex-adapter, 13-06-multi-source-display]
|
||||
|
||||
tech-stack:
|
||||
added: []
|
||||
patterns:
|
||||
- "Dedup resolver owns both the create-vs-attach decision AND the SCHEMA-02 mutable-field refresh — the old direct tender.upsert UPDATE branch moved into resolve()'s match path unchanged in semantics"
|
||||
- "dedupActive computed once per tick from configs.length (activePortalCount >= 2) and threaded through every resolve() call in the fan-out loop"
|
||||
- "catch-per-source: each config's day-cursor-gate/fetch/normalize/resolve block runs in its own try/catch inside the fan-out for-loop, nested inside the existing outer tick-level try/catch"
|
||||
|
||||
key-files:
|
||||
created:
|
||||
- apps/api/src/tenders/tender-dedup.service.ts
|
||||
- apps/api/src/tenders/tender-dedup.service.spec.ts
|
||||
modified:
|
||||
- apps/api/src/tenders/tender-ingestion.service.ts
|
||||
- apps/api/src/tenders/tender-ingestion.service.spec.ts
|
||||
- apps/api/src/tenders/tenders.module.ts
|
||||
|
||||
key-decisions:
|
||||
- "Tier 2 (source:noticeId via TenderSource lookup) is NOT gated by dedupActive — it is the same-source re-poll identity check, independent from cross-source merging, and must remain active even with a single source (idempotent re-poll)."
|
||||
- "Fingerprint is always computed and stored on tender.create (n.fingerprint ?? tenderFingerprint(n)), regardless of dedupActive — a DÖE-only tender still gets a fingerprint so it becomes matchable the moment a 2nd source activates, without a further backfill."
|
||||
- "SCHEMA-02 mutable-field list is copied verbatim from the pre-existing tender.upsert UPDATE block (title, buyerName, cpvCodes, cpvDivisions, region, plz, bundesland, deadlineAt, estimatedValue, procedureType, sourceUrl, contentHash) into resolve()'s match branch, gated by a contentHash comparison instead of always running on every upsert call — functionally identical outcome, explicit no-op test added."
|
||||
- "pruneExpiredTenders() runs once per tick (not once per source) — anyDayFetched is now an across-all-sources flag, matching the pre-existing once-per-tick retention semantics."
|
||||
|
||||
patterns-established:
|
||||
- "Fake TenderDedupService.resolve() in tender-ingestion.service.spec.ts mirrors the real service's dedupKey-upsert contract (created vs. updated) without re-testing tier logic — keeps the fan-out spec focused on orchestration, not dedup internals (covered separately by tender-dedup.service.spec.ts)."
|
||||
|
||||
requirements-completed: [SCHEMA-03]
|
||||
|
||||
coverage:
|
||||
- id: D1
|
||||
description: "dedupActive=false: two normalized records with the same fingerprint but different source:noticeId create TWO Tender rows (fingerprint tier skipped, D-05 inert-proof, Erfolgskriterium 3)"
|
||||
requirement: SCHEMA-03
|
||||
verification:
|
||||
- kind: unit
|
||||
ref: "apps/api/src/tenders/tender-dedup.service.spec.ts — 'creates TWO Tender rows for records with the same fingerprint but different source:noticeId, when dedupActive=false'"
|
||||
status: pass
|
||||
human_judgment: false
|
||||
- id: D2
|
||||
description: "dedupActive=true: OCID match, idempotent source:noticeId re-poll, and fingerprint match all attach an additional TenderSource to the existing Tender (created=false, D-03 merge); tenderSource.upsert targets the @@unique([sourcePortal, sourceNoticeId]) composite key"
|
||||
requirement: SCHEMA-03
|
||||
verification:
|
||||
- kind: unit
|
||||
ref: "apps/api/src/tenders/tender-dedup.service.spec.ts — OCID-match/idempotent-reseen/fingerprint-match/no-match/composite-key tests (8/8 pass)"
|
||||
status: pass
|
||||
human_judgment: false
|
||||
- id: D3
|
||||
description: "SCHEMA-02 change-detection preserved through resolve(): a matched Tender whose incoming contentHash differs gets its mutable fields (title, deadlineAt, contentHash, etc.) updated; an unchanged contentHash does NOT call tender.update — no Phase-10 regression"
|
||||
requirement: SCHEMA-03
|
||||
verification:
|
||||
- kind: unit
|
||||
ref: "apps/api/src/tenders/tender-dedup.service.spec.ts — 'OCID match with a changed contentHash still updates...' + 'OCID match with an UNCHANGED contentHash does NOT call tender.update'"
|
||||
status: pass
|
||||
- kind: unit
|
||||
ref: "apps/api/src/tenders/tender-ingestion.service.spec.ts — existing SCHEMA-02 re-poll suite (insert-once, update-in-place-on-changed-hash) kept green through the resolve() hand-off"
|
||||
status: pass
|
||||
human_judgment: false
|
||||
- id: D4
|
||||
description: "pollDueSources fans out over ALL active TenderSourcePollConfig rows (findMany, not findFirst/findUnique) with catch-per-source error isolation — one broken/blocking source does not abort polling of the others, nor does it throw out of the tick"
|
||||
requirement: SCHEMA-03
|
||||
verification:
|
||||
- kind: unit
|
||||
ref: "apps/api/src/tenders/tender-ingestion.service.spec.ts — 'polls every active config...', 'skips a config whose sourceType has no registered adapter, without throwing', 'a throwing source does not prevent another active source from being polled' (14/14 pass)"
|
||||
status: pass
|
||||
human_judgment: false
|
||||
- id: D5
|
||||
description: "dedupActive is bound to activePortalCount >= 2 and threaded into every resolve() call; delta-only matchDelta boundary (D-07) preserved across the fan-out (only genuinely-created IDs across all sources)"
|
||||
requirement: SCHEMA-03
|
||||
verification:
|
||||
- kind: unit
|
||||
ref: "apps/api/src/tenders/tender-ingestion.service.spec.ts — 'passes dedupActive=false...single config', 'passes dedupActive=true...two configs', existing D-07 delta-only suite kept green"
|
||||
status: pass
|
||||
human_judgment: false
|
||||
- id: D6
|
||||
description: "SourceRegistry + TenderDedupService are TendersModule providers; DoeOpenDataAdapter is registered with the registry in onModuleInit (denylist gate applies at DI boot); DI graph resolves cleanly"
|
||||
requirement: SCHEMA-03
|
||||
verification:
|
||||
- kind: unit
|
||||
ref: "cd apps/api && npx tsc --noEmit -p tsconfig.json (clean)"
|
||||
status: pass
|
||||
- kind: unit
|
||||
ref: "cd apps/api && npx vitest run src/tenders (19 files, 190/190 pass)"
|
||||
status: pass
|
||||
- kind: unit
|
||||
ref: "cd apps/api && npx vitest run (full suite, 23 files, 253/253 pass)"
|
||||
status: pass
|
||||
human_judgment: false
|
||||
|
||||
duration: 45min
|
||||
completed: 2026-07-23
|
||||
status: complete
|
||||
---
|
||||
|
||||
# Phase 13 Plan 03: Dedup Resolver + Fan-out Wiring Summary
|
||||
|
||||
**Three-tier `TenderDedupService.resolve()` (OCID -> source:noticeId -> fingerprint, fingerprint tier hard-gated by `activePortalCount >= 2`) now drives `pollDueSources`, which fans out over every active `TenderSourcePollConfig` with catch-per-source error isolation — replacing the DÖE-only `findUnique`/direct `tender.upsert`, while preserving SCHEMA-02 change detection byte-for-byte.**
|
||||
|
||||
## Performance
|
||||
|
||||
- **Duration:** 45 min
|
||||
- **Started:** 2026-07-23T08:45:00Z
|
||||
- **Completed:** 2026-07-23T09:30:00Z
|
||||
- **Tasks:** 3
|
||||
- **Files modified:** 5 (2 created, 3 modified)
|
||||
|
||||
## Accomplishments
|
||||
- `TenderDedupService.resolve(n, {dedupActive})`: OCID -> source:noticeId -> fingerprint (D-04), fingerprint tier only reachable when `dedupActive` is true (D-05). Match -> `tenderSource.upsert` attaches an additional source to the existing Tender (`created: false`, D-03 merge) + SCHEMA-02 mutable-field refresh when `contentHash` differs. No match -> `tender.create` (with computed `fingerprint`) + `tenderSource.create` (`created: true`). Plain `PrismaService`, no `forTenant()`/RLS. TDD RED->GREEN, 8/8 tests green, including the D-05 inert-proof (fingerprint-colliding records with `dedupActive=false` stay two Tenders) and two dedicated SCHEMA-02-preservation tests.
|
||||
- `pollDueSources` rebuilt on `findMany({ isActive: true })` fan-out (poll-once-fan-out-many, NOT findFirst/findUnique). Each config's day-cursor-gate/fetch/normalize/resolve block runs inside its own `try/catch` — a throwing source (portal down) no longer kills the tick or blocks other sources (D-01). `dedupActive = configs.length >= 2` computed once per tick and threaded through every `resolve()` call. `pruneExpiredTenders()` and `matchDelta` (D-07 delta-only boundary) run once across the whole tick, unchanged in intent. Existing SCHEMA-02/D-07/retention/day-cursor spec suites updated to the new `(prisma, registry, normalizer, matching, dedup)` constructor shape and kept green; four new fan-out/catch-per-source/dedupActive-gate tests added. 14/14 pass.
|
||||
- `TendersModule`: `SourceRegistry` + `TenderDedupService` added as providers; `onModuleInit` registers `DoeOpenDataAdapter` with the registry before the scheduler's first tick can fire — the DI-boot-time point where the INGEST-07 denylist gate (D-06) structurally applies. Explicitly documented as the sole 13-04/13-05 extension point for future adapter registrations.
|
||||
- `npx tsc --noEmit` clean; `src/tenders` slice 19 files/190 tests green; full API suite 23 files/253 tests green.
|
||||
|
||||
## Task Commits
|
||||
|
||||
Each task was committed atomically:
|
||||
|
||||
1. **Task 1a: RED — failing TenderDedupService spec** - `605ea4c` (test)
|
||||
2. **Task 1b: GREEN — TenderDedupService three-tier resolver** - `1cd2fcf` (feat)
|
||||
3. **Task 2: pollDueSources fan-out + catch-per-source + dedup hook** - `d453dbb` (feat)
|
||||
4. **Task 3: Module wiring — SourceRegistry + TenderDedupService providers** - `1d4f9cf` (feat)
|
||||
|
||||
_Task 1 (tdd="true") produced two commits (test -> feat) per protocol; no refactor commit was needed — the first implementation passed all 8 tests cleanly._
|
||||
|
||||
## Files Created/Modified
|
||||
- `apps/api/src/tenders/tender-dedup.service.ts` - `@Injectable() TenderDedupService`, three-tier `resolve()` resolver
|
||||
- `apps/api/src/tenders/tender-dedup.service.spec.ts` - 8 unit tests (D-05 inert-proof, tier1/2/3 match, no-match create, composite-key upsert target, SCHEMA-02 preservation x2)
|
||||
- `apps/api/src/tenders/tender-ingestion.service.ts` - `pollDueSources` rebuilt as `findMany` fan-out with catch-per-source + `dedup.resolve()` hand-off; constructor now takes `(prisma, registry, normalizer, matching, dedup)`
|
||||
- `apps/api/src/tenders/tender-ingestion.service.spec.ts` - fake `registry`/`dedup` collaborators; existing suites migrated to the new constructor shape; 4 new tests (fan-out, adapter-missing skip, catch-per-source isolation, dedupActive gate x2)
|
||||
- `apps/api/src/tenders/tenders.module.ts` - `SourceRegistry` + `TenderDedupService` providers; `onModuleInit` registers `DoeOpenDataAdapter`
|
||||
|
||||
## Decisions Made
|
||||
- **Tier 2 (source:noticeId) is unconditional, not gated by `dedupActive`** — it's the same-source re-poll identity check (idempotency), orthogonal to cross-source fingerprint merging. Gating it would have broken idempotent re-polls of a single active source.
|
||||
- **Fingerprint is always computed on create**, independent of `dedupActive` — so a DÖE-only tender is immediately fingerprint-matchable the instant a second source activates, with no further backfill needed (13-01's backfill already covers the pre-existing 2851 rows; this covers every NEW tender ingested going forward).
|
||||
- **SCHEMA-02 mutable-field list copied verbatim** from the old `tender.upsert` UPDATE branch into `resolve()`'s match path, now gated by an explicit `contentHash !==` comparison (previously implicit in every upsert call) — behaviorally identical, with an added explicit "unchanged hash -> no update call" test that didn't exist before.
|
||||
- **`pruneExpiredTenders()`/`matchDelta` run once per tick, not once per source** — `anyDayFetched` and `newTenderIds` are tick-scoped across the whole fan-out loop, preserving the pre-existing once-per-tick retention/matching cadence.
|
||||
|
||||
## Deviations from Plan
|
||||
|
||||
None — plan executed exactly as written. All four `must_haves.truths` are met and directly tested:
|
||||
1. `dedupActive=false` inertness — tested (D1 above).
|
||||
2. 2nd-source fingerprint match attaches TenderSource instead of creating a new Tender — tested (D2).
|
||||
3. `pollDueSources` fans out via `findMany({isActive:true})` with catch-per-source — tested (D4).
|
||||
4. SCHEMA-02 change detection preserved through the move into `resolve()` — tested (D3), with both the original ingestion-service spec suite kept green AND two new dedicated dedup-service tests.
|
||||
|
||||
## Issues Encountered
|
||||
|
||||
None. Local (non-worktree, `main` branch) execution; no migrations touched in this plan (13-01 already applied the schema). No Docker/DB interaction required — pure TypeScript service/wiring plan, verified entirely via Vitest on host.
|
||||
|
||||
## User Setup Required
|
||||
|
||||
None. No external service configuration, no migration, no environment changes.
|
||||
|
||||
## Next Phase Readiness
|
||||
- `TenderDedupService.resolve()` and the `SourceRegistry`-driven fan-out are the exact seams 13-04 (NetServer adapter) and 13-05 (cosinex adapter) need: adding a new adapter is (a) implement `TenderSourceAdapter`, (b) add one `providers` entry + one `registry.register(...)` call in `tenders.module.ts`'s `onModuleInit`, (c) seed a `TenderSourcePollConfig` row for the new `sourceType`. No further changes to `pollDueSources` or `TenderDedupService` are needed — both are already source-count-agnostic.
|
||||
- Cross-source dedup end-to-end (real DÖE + NetServer overlap producing ONE Tender with two TenderSource links) remains a **manual/live verification** per 13-VALIDATION.md — this plan's automated coverage proves the resolver and fan-out logic in isolation; the live confirmation needs a second active source (13-04) to observe.
|
||||
- No blockers.
|
||||
|
||||
---
|
||||
*Phase: 13-scraping-adapters-cross-source-dedup*
|
||||
*Completed: 2026-07-23*
|
||||
|
||||
## Self-Check: PASSED
|
||||
|
||||
All created/modified files verified present on disk; all 4 task commit hashes (605ea4c, 1cd2fcf, d453dbb, 1d4f9cf) verified in git log.
|
||||
Reference in New Issue
Block a user