From a547a1d31d7fc88843c696dcdd05c190aa741691 Mon Sep 17 00:00:00 2001 From: Schalli Date: Thu, 23 Jul 2026 08:40:47 +0200 Subject: [PATCH] docs(13-01): complete fingerprint + TenderSource datenkern plan --- .planning/REQUIREMENTS.md | 4 +- .planning/ROADMAP.md | 7 +- .planning/STATE.md | 39 ++-- .../13-01-SUMMARY.md | 169 ++++++++++++++++++ 4 files changed, 196 insertions(+), 23 deletions(-) create mode 100644 .planning/phases/13-scraping-adapters-cross-source-dedup/13-01-SUMMARY.md diff --git a/.planning/REQUIREMENTS.md b/.planning/REQUIREMENTS.md index 4af3c93..61f48ff 100644 --- a/.planning/REQUIREMENTS.md +++ b/.planning/REQUIREMENTS.md @@ -22,7 +22,7 @@ - [x] **SCHEMA-01**: Alle Quellen werden in ein einheitliches, OCDS-orientiertes Ausschreibungs-Schema normalisiert (Titel, Auftraggeber, CPV-Codes, Region/PLZ, Frist, geschätzter Wert, Verfahrensart, Quell-URL, Rohdaten). Ausschreibungsdaten sind plattform-global, nicht mandantengebunden. - [x] **SCHEMA-02**: Das System erkennt Änderungen an bereits importierten Ausschreibungen (Fristverlängerung, Aufhebung) über einen Content-Hash und aktualisiert den Datensatz. -- [ ] **SCHEMA-03**: Dieselbe Ausschreibung aus mehreren Quellen wird zu einem Eintrag mit mehreren Quell-Links dedupliziert (Schlüssel: OCID → Quelle:NoticeId → Fuzzy-Fingerprint aus Auftraggeber+Titel+CPV+Frist+Wert). Dedup greift erst ab der zweiten aktiven Quelle. +- [x] **SCHEMA-03**: Dieselbe Ausschreibung aus mehreren Quellen wird zu einem Eintrag mit mehreren Quell-Links dedupliziert (Schlüssel: OCID → Quelle:NoticeId → Fuzzy-Fingerprint aus Auftraggeber+Titel+CPV+Frist+Wert). Dedup greift erst ab der zweiten aktiven Quelle. ### FILTER — Suche & Profile @@ -99,7 +99,7 @@ | INGEST-02 | Phase 13 | Pending | | INGEST-03 | Phase 13 | Pending | | INGEST-07 | Phase 13 | Pending | -| SCHEMA-03 | Phase 13 | Pending | +| SCHEMA-03 | Phase 13 | Complete | | INGEST-04 | Phase 14 | Pending | | INGEST-05 | Phase 14 | Pending | | CONFIG-02 | Phase 14 | Pending | diff --git a/.planning/ROADMAP.md b/.planning/ROADMAP.md index 0396df9..e3be6e4 100644 --- a/.planning/ROADMAP.md +++ b/.planning/ROADMAP.md @@ -439,10 +439,11 @@ Plans: 3. A tender that appears via both DÖE and a scraping adapter shows up once in the results list (fuzzy fingerprint dedup on buyer+title+CPV+deadline+value), with links to all of its source portals -- dedup logic only activates once a second source is live 4. Attempting to register vergabe24 or aumass as a poll source is refused by the system itself (adapter registry denylist enforced in code), not just documented as forbidden -**Plans**: 6 plans +**Plans**: 1/6 plans executed Plans: -- [ ] 13-01-PLAN.md — TenderSource-Schema + Backfill + NULL-tolerante Fingerprint-Fn + SourceType-Union (SCHEMA-03 Datenschicht) + +- [x] 13-01-PLAN.md — TenderSource-Schema + Backfill + NULL-tolerante Fingerprint-Fn + SourceType-Union (SCHEMA-03 Datenschicht) - [ ] 13-02-PLAN.md — SourceRegistry + harte Denylist-Gate (vergabe24/aumass) + Adapter-Interface-Generalisierung (INGEST-07) - [ ] 13-03-PLAN.md — 3-Stufen-Dedup-Resolver + pollDueSources Fan-out (catch-per-source) + Modul-Wiring (SCHEMA-03) - [ ] 13-04-PLAN.md — NetServer-Adapter (config-getrieben, 3 Portale) + Package-Legitimacy-Checkpoint (INGEST-02) @@ -485,5 +486,5 @@ Phases execute in numeric order: 1 -> 2 -> 3 -> 4 -> 5 -> 6 -> 7 -> 8 -> 9 -> 10 | 10. Ausschreibungs-Radar Foundation & DÖE Ingestion | 6/6 | Complete | 2026-07-21 | | 11. Filter Engine, Results UI & Saved Searches | 6/6 | In Progress| | | 12. Tender Notifications | 4/4 | In Progress| | -| 13. Scraping Adapters & Cross-Source Deduplication | 0/TBD | Not started | - | +| 13. Scraping Adapters & Cross-Source Deduplication | 1/6 | In Progress| | | 14. RSS, Email-Alert Ingestion & Module Rollout | 0/TBD | Not started | - | diff --git a/.planning/STATE.md b/.planning/STATE.md index 21a637b..fd30425 100644 --- a/.planning/STATE.md +++ b/.planning/STATE.md @@ -2,18 +2,18 @@ gsd_state_version: 1.0 milestone: v1.1 milestone_name: Ausschreibungs-Radar -current_phase: 12 -current_phase_name: tender-notifications -status: verified -stopped_at: Phase 12 verified — live email UAT passed (digest via Mailhog, no double-send) -last_updated: "2026-07-22T07:32:37.037Z" -last_activity: 2026-07-22 -last_activity_desc: Phase 12 execution started +current_phase: 13 +current_phase_name: scraping-adapters-cross-source-dedup +status: executing +stopped_at: Completed 13-01-PLAN.md +last_updated: "2026-07-23T06:40:39.176Z" +last_activity: 2026-07-23 +last_activity_desc: Phase 13 execution started progress: - total_phases: 12 + total_phases: 13 completed_phases: 11 - total_plans: 56 - completed_plans: 55 + total_plans: 62 + completed_plans: 56 --- # Project State @@ -23,16 +23,16 @@ progress: See: .planning/PROJECT.md (updated 2026-07-17) **Core value:** Eine zentrale Plattform, in der beliebige Workflow-Tools als Module lizenziert, aktiviert und genutzt werden koennen -- ohne zwischen verschiedenen Anwendungen wechseln zu muessen. -**Current focus:** Phase 12 — tender-notifications +**Current focus:** Phase 13 — scraping-adapters-cross-source-dedup ## Current Position -Phase: 12 (tender-notifications) — EXECUTING -Plan: 4 of 4 -Status: Phase complete — ready for verification -Last activity: 2026-07-22 — Phase 12 execution started +Phase: 13 (scraping-adapters-cross-source-dedup) — EXECUTING +Plan: 2 of 6 +Status: Ready to execute +Last activity: 2026-07-23 — Phase 13 execution started -Progress: [██████████] 98% +Progress: [█████████░] 90% ## Performance Metrics @@ -90,6 +90,7 @@ Progress: [██████████] 98% | Phase 12 P02 | 8min | 3 tasks | 5 files | | Phase 12 P03 | 15min | 2 tasks | 3 files | | Phase 12 P04 | 12min | 3 tasks | 10 files | +| Phase 13 P01 | 35min | 3 tasks | 6 files | ## Accumulated Context @@ -191,6 +192,8 @@ Recent decisions affecting current work: - [Phase ?]: TenderDigestScheduler.runDigest(now) takes an injectable clock parameter for testable Monday-only weekly-digest gating - [Phase ?]: Instant-Dispatch filtert die bereits geladenen savedSearches (kein zweiter Query); notifiedAt/channel='instant' nur nach Erfolg gestempelt — identisches Gate wie Digest (D-06) - [Phase ?]: Digest-interval selector inline in settings/page.tsx (already 'use client'); instantAlert toggle uses plain checkbox for chip-based SavedSearchBar UI +- [Phase ?]: SCHEMA-03 fingerprint: title+buyer dominant, CPV division, value-bucket, deadline-day, sha256; dedupKey untouched, fingerprint additive +- [Phase ?]: One-time TS backfill scripts run via compiled dist/ output (not raw .ts execution) to keep tsc --noEmit clean ### Pending Todos @@ -228,7 +231,7 @@ Items acknowledged and carried forward from previous milestone close: ## Session Continuity -Last session: 2026-07-22T07:32:37.021Z -Stopped at: Completed 12-04-PLAN.md +Last session: 2026-07-23T06:40:39.162Z +Stopped at: Completed 13-01-PLAN.md Resume file: None Last activity: 2026-07-14 - Built LDAP per-user exclude/denylist filter (9d1323f), migration applied on live DB, verified via Playwright: sync deactivated 4 excluded service accounts (administrator/krbtgt/guest/dns-ldap), 2 real LDAP users stay active, 0 wrongly created diff --git a/.planning/phases/13-scraping-adapters-cross-source-dedup/13-01-SUMMARY.md b/.planning/phases/13-scraping-adapters-cross-source-dedup/13-01-SUMMARY.md new file mode 100644 index 0000000..f9015d7 --- /dev/null +++ b/.planning/phases/13-scraping-adapters-cross-source-dedup/13-01-SUMMARY.md @@ -0,0 +1,169 @@ +--- +phase: 13-scraping-adapters-cross-source-dedup +plan: 01 +subsystem: database +tags: [prisma, postgresql, crypto, dedup, sha256] + +requires: + - phase: 10-doe-opendata-ingestion + provides: Tender model (sourcePortal, sourceNoticeId, ocid, dedupKey, cpvDivisions, contentHash), TenderNormalizerService, tender.types.ts contract +provides: + - "TenderSource Prisma model (1:n Tender, @@unique[sourcePortal, sourceNoticeId], onDelete Cascade)" + - "Tender.fingerprint nullable column + index, additive alongside the unchanged dedupKey" + - "pure tenderFingerprint() NULL-tolerant dedup-key function (title+buyer dominant, CPV division, value-bucket, deadline-day)" + - "2851 pre-existing DÖE tenders backfilled with one TenderSource row + a computed fingerprint each" + - "SourceType widened to open union ('doe-opendata' | 'ai-netserver' | 'cosinex-dtvp')" +affects: [13-02-source-registry, 13-03-dedup-resolver, 13-04-netserver-adapter, 13-05-cosinex-adapter, 13-06-multi-source-display] + +tech-stack: + added: [] + patterns: + - "Pure fingerprint function (no I/O, no Prisma import) computed once and reused by both the one-time backfill script and (in Plan 13-03) the live dedup resolver" + - "Two-step migration ordering for additive schema + data backfill: (1) table/column DDL, (2) SQL data copy, (3) unique constraint — avoids constraint violations from pre-existing data" + - "One-time TS backfill scripts run via the compiled dist/ output (standard extensionless imports), not as raw .ts via node's native type-stripping — keeps tsc --noEmit clean project-wide" + +key-files: + created: + - apps/api/src/tenders/tender-fingerprint.ts + - apps/api/src/tenders/tender-fingerprint.spec.ts + - apps/api/src/tenders/backfill-tender-source.ts + - apps/api/prisma/migrations/20260723120000_add_tender_source/migration.sql + modified: + - apps/api/prisma/schema.prisma + - apps/api/src/tenders/tender.types.ts + +key-decisions: + - "fingerprint canonical string is [buyer, title, cpvDivisionKey, deadlineKey, valueBucket].join('|') -> sha256 hex, exactly per 13-RESEARCH Pattern 4 — deterministic O(1) lookup, no similarity threshold" + - "dedupKey @unique on Tender is left untouched (SCHEMA-02 upsert target); fingerprint is a separate, additive, nullable column" + - "One-time backfill script uses standard (extensionless) TS imports and documents running the compiled dist/ output, rather than node's raw .ts execution — Node 24's native type-stripping treats ESM-syntax .ts files as ESM requiring explicit .ts/.js extensions, which tsc rejects under moduleResolution:node; compiling first keeps both runtime execution and tsc --noEmit clean" + +patterns-established: + - "Fuzzy-dedup fingerprint: pure function, NULL segments collapse to empty string rather than excluding the record, title always included as the collision guard" + +requirements-completed: [SCHEMA-03] + +coverage: + - id: D1 + description: "tenderFingerprint() is NULL-tolerant (title+buyer+CPV match with value/deadline both NULL), collision-resistant (different title -> different hash), umlaut-normalizing, order-independent on CPV divisions, magnitude-bucketed on value, and returns a stable sha256 hex string" + requirement: SCHEMA-03 + verification: + - kind: unit + ref: "apps/api/src/tenders/tender-fingerprint.spec.ts (8 tests, all pass)" + status: pass + human_judgment: false + - id: D2 + description: "TenderSource model (1:n, @@unique[sourcePortal, sourceNoticeId]) and Tender.fingerprint exist in schema.prisma; migration applied locally; every pre-existing Tender has exactly one TenderSource row backfilled from its current sourcePortal/sourceNoticeId/sourceUrl/ocid" + requirement: SCHEMA-03 + verification: + - kind: integration + ref: "npx prisma validate && npx prisma generate (clean)" + status: pass + - kind: manual_procedural + ref: "docker compose exec -T db psql -U tessera -d tessera -c 'SELECT count(*) FROM \"TenderSource\"' -> 2851 == SELECT count(*) FROM \"Tender\" -> 2851" + status: pass + human_judgment: false + - id: D3 + description: "Tender.fingerprint backfilled for all pre-existing rows using the Task-1 tenderFingerprint() function, with Decimal->number conversion for estimatedValue" + requirement: SCHEMA-03 + verification: + - kind: manual_procedural + ref: "backfill-tender-source.ts run via compiled dist/ output — console output 'fingerprint set on 2851/2851 tenders', re-verified idempotent on second run; SELECT count(*) FROM \"Tender\" WHERE fingerprint IS NOT NULL -> 2851" + status: pass + human_judgment: false + - id: D4 + description: "SourceType is an open union ('doe-opendata' | 'ai-netserver' | 'cosinex-dtvp') and NormalizedTenderFields carries an optional fingerprint field, without breaking the existing normalizer contract" + requirement: SCHEMA-03 + verification: + - kind: unit + ref: "cd apps/api && npx tsc --noEmit -p tsconfig.json (clean)" + status: pass + - kind: unit + ref: "pnpm --filter @tessera/api test (full suite, 234/234 pass)" + status: pass + human_judgment: false + +duration: 35min +completed: 2026-07-23 +status: complete +--- + +# Phase 13 Plan 01: Fingerprint + TenderSource Datenkern Summary + +**NULL-tolerante `tenderFingerprint()`-Funktion, additives `TenderSource`-1:n-Modell + `Tender.fingerprint`, und Backfill aller 2851 Bestands-DÖE-Tender — der dependency-freie SCHEMA-03-Datenkern für Cross-Source-Dedup.** + +## Performance + +- **Duration:** 35 min +- **Started:** 2026-07-23T08:33:00Z +- **Completed:** 2026-07-23T08:39:00Z +- **Tasks:** 3 +- **Files modified:** 6 (2 created new spec/source pairs collapsed to 4 created + 2 modified) + +## Accomplishments +- Pure, NULL-tolerante `tenderFingerprint()` (title+buyer dominant, CPV-Division, value-bucket, deadline-day, sha256) — TDD RED→GREEN, 8/8 Tests grün. +- `TenderSource`-Modell (1:n zu `Tender`, `@@unique([sourcePortal, sourceNoticeId])`, `onDelete: Cascade`) + additive nullable `Tender.fingerprint`-Spalte + Index, `dedupKey` unverändert. +- Handgeschriebene Migration `20260723120000_add_tender_source` in strikter 3-Schritt-Reihenfolge (DDL → Backfill-INSERT → Unique-Constraint), lokal gegen die `tessera`-Dev-DB angewendet. +- Alle 2851 Bestands-DÖE-Tender: je eine `TenderSource`-Zeile (verifiziert `count(*) = 2851 = 2851`) + berechneter `fingerprint` (verifiziert `2851/2851`, idempotent re-run bestätigt). +- `SourceType` zur offenen Union erweitert, `NormalizedTenderFields.fingerprint?` ergänzt — `tsc --noEmit` und volle API-Testsuite (234 Tests) grün. + +## Task Commits + +Each task was committed atomically: + +1. **Task 1a: RED — failing tenderFingerprint test** - `063ba5b` (test) +2. **Task 1b: GREEN — implement tenderFingerprint** - `c6cac69` (feat) +3. **Task 2: TenderSource model + migration + backfill** - `447fb74` (feat) +4. **Task 3: SourceType union + NormalizedTenderFields.fingerprint** - `c2a6021` (feat) + +_TDD task (Task 1) produced two commits (test → feat) per protocol; no refactor commit was needed._ + +## Files Created/Modified +- `apps/api/src/tenders/tender-fingerprint.ts` - pure NULL-tolerant sha256 dedup-fingerprint function +- `apps/api/src/tenders/tender-fingerprint.spec.ts` - 8 unit tests (NULL-tolerance, collision guard, umlauts, CPV order-independence, value bucketing, determinism) +- `apps/api/src/tenders/backfill-tender-source.ts` - one-time script computing `Tender.fingerprint` for all rows +- `apps/api/prisma/migrations/20260723120000_add_tender_source/migration.sql` - table+column DDL, TenderSource backfill INSERT, unique constraint +- `apps/api/prisma/schema.prisma` - `model TenderSource` + `Tender.fingerprint`/`sources`/`@@index([fingerprint])` +- `apps/api/src/tenders/tender.types.ts` - `SourceType` open union, `NormalizedTenderFields.fingerprint?` + +## Decisions Made +- **Fingerprint algorithm exactly per 13-RESEARCH.md Pattern 4** (title+buyer dominant, CPV division not full code, value order-of-magnitude bucket, deadline day-grain, sha256 hex) — deterministic O(1) lookup, no similarity-threshold matching (avoids O(n) scans against the 2851+ row backlog per ingest). +- **`dedupKey @unique` left untouched** — it remains the SCHEMA-02 DÖE upsert target; `fingerprint` is a fully additive, nullable column, no migration risk to existing ingestion. +- **Backfill script runs via compiled `dist/` output, not raw `.ts` via node's native type-stripping.** Node 24 treats a `.ts` file containing `import`/`export` syntax as ESM and requires explicit `.ts`/`.js` extensions for relative imports; but the project's `tsconfig.json` (`module: commonjs`, `moduleResolution: node`) rejects explicit `.ts` extensions in import specifiers (`TS5097`). Using standard extensionless imports (tsc-clean) and documenting `pnpm --filter @tessera/api build && node dist/tenders/backfill-tender-source.js` as the run command resolves both constraints — confirmed working end-to-end (2851/2851, idempotent re-run). + +## Deviations from Plan + +### Auto-fixed Issues + +**1. [Rule 3 - Blocking] Backfill script import style adjusted for tsc/runtime compatibility** +- **Found during:** Task 2 (backfill-tender-source.ts) +- **Issue:** The plan didn't specify how the one-time TS backfill script would actually be executed. An initial version imported `./tender-fingerprint.ts` with an explicit extension to satisfy Node 24's native TS execution — but that broke `npx tsc --noEmit` (Task 3's verification command) with `TS5097: An import path can only end with a '.ts' extension when 'allowImportingTsExtensions' is enabled`. +- **Fix:** Switched to standard extensionless imports (tsc-compatible) and ran the script via `nest build` + `node dist/tenders/backfill-tender-source.js` instead of raw `.ts` execution. Documented this invocation in the file's header comment. +- **Files modified:** apps/api/src/tenders/backfill-tender-source.ts +- **Verification:** `npx tsc --noEmit -p tsconfig.json` clean; compiled script re-run confirmed idempotent (`2851/2851` both times). +- **Committed in:** 447fb74 (Task 2 commit) + +--- + +**Total deviations:** 1 auto-fixed (1 blocking — script invocation mechanics) +**Impact on plan:** No scope creep; fix was purely about how the already-planned one-time script gets executed so it doesn't regress the project's typecheck verification. + +## Issues Encountered +- `apps/api/prisma/migrations/20260722180000_ldap_tls_reject_unauthorized` (from a prior phase) had not yet been applied to this local dev DB. `prisma migrate deploy` applied it together with the new `20260723120000_add_tender_source` migration in the same run — expected catch-up, not a regression caused by this plan. +- `DATABASE_URL` is not present in `apps/api/.env` on the host (only used inside the Docker network); per project MEMORY (`project_local_db_migrations`), it must be exported manually with the `db` container's ephemeral bridge IP (`docker inspect ... tessera-ctl-db-1`) for any host-side Prisma CLI invocation. Followed the documented workflow. + +## User Setup Required + +None - no external service configuration required. All changes applied to the local dev DB only (no test/prod server touched, per MEMORY constraints). + +## Next Phase Readiness +- The dependency-free SCHEMA-03 data core (D-01) is fully in place and tested: `TenderSource`, `Tender.fingerprint`, and `tenderFingerprint()` are ready for Plan 13-02 (Source-Registry + Denylist) and Plan 13-03 (3-tier dedup resolver) to build on directly. +- `SourceType`'s open union unblocks Plan 13-04 (NetServer adapter) and Plan 13-05 (cosinex adapter) without further type-contract changes. +- No blockers. `dedupKey`'s eventual consolidation with `fingerprint` remains explicitly deferred (13-RESEARCH Open Question 3) — not in scope for this milestone. + +--- +*Phase: 13-scraping-adapters-cross-source-dedup* +*Completed: 2026-07-23* + +## Self-Check: PASSED + +All created files verified present on disk; all 4 task commit hashes verified in git log.