docs(13-01): complete fingerprint + TenderSource datenkern plan
This commit is contained in:
@@ -22,7 +22,7 @@
|
||||
|
||||
- [x] **SCHEMA-01**: Alle Quellen werden in ein einheitliches, OCDS-orientiertes Ausschreibungs-Schema normalisiert (Titel, Auftraggeber, CPV-Codes, Region/PLZ, Frist, geschätzter Wert, Verfahrensart, Quell-URL, Rohdaten). Ausschreibungsdaten sind plattform-global, nicht mandantengebunden.
|
||||
- [x] **SCHEMA-02**: Das System erkennt Änderungen an bereits importierten Ausschreibungen (Fristverlängerung, Aufhebung) über einen Content-Hash und aktualisiert den Datensatz.
|
||||
- [ ] **SCHEMA-03**: Dieselbe Ausschreibung aus mehreren Quellen wird zu einem Eintrag mit mehreren Quell-Links dedupliziert (Schlüssel: OCID → Quelle:NoticeId → Fuzzy-Fingerprint aus Auftraggeber+Titel+CPV+Frist+Wert). Dedup greift erst ab der zweiten aktiven Quelle.
|
||||
- [x] **SCHEMA-03**: Dieselbe Ausschreibung aus mehreren Quellen wird zu einem Eintrag mit mehreren Quell-Links dedupliziert (Schlüssel: OCID → Quelle:NoticeId → Fuzzy-Fingerprint aus Auftraggeber+Titel+CPV+Frist+Wert). Dedup greift erst ab der zweiten aktiven Quelle.
|
||||
|
||||
### FILTER — Suche & Profile
|
||||
|
||||
@@ -99,7 +99,7 @@
|
||||
| INGEST-02 | Phase 13 | Pending |
|
||||
| INGEST-03 | Phase 13 | Pending |
|
||||
| INGEST-07 | Phase 13 | Pending |
|
||||
| SCHEMA-03 | Phase 13 | Pending |
|
||||
| SCHEMA-03 | Phase 13 | Complete |
|
||||
| INGEST-04 | Phase 14 | Pending |
|
||||
| INGEST-05 | Phase 14 | Pending |
|
||||
| CONFIG-02 | Phase 14 | Pending |
|
||||
|
||||
@@ -439,10 +439,11 @@ Plans:
|
||||
3. A tender that appears via both DÖE and a scraping adapter shows up once in the results list (fuzzy fingerprint dedup on buyer+title+CPV+deadline+value), with links to all of its source portals -- dedup logic only activates once a second source is live
|
||||
4. Attempting to register vergabe24 or aumass as a poll source is refused by the system itself (adapter registry denylist enforced in code), not just documented as forbidden
|
||||
|
||||
**Plans**: 6 plans
|
||||
**Plans**: 1/6 plans executed
|
||||
|
||||
Plans:
|
||||
- [ ] 13-01-PLAN.md — TenderSource-Schema + Backfill + NULL-tolerante Fingerprint-Fn + SourceType-Union (SCHEMA-03 Datenschicht)
|
||||
|
||||
- [x] 13-01-PLAN.md — TenderSource-Schema + Backfill + NULL-tolerante Fingerprint-Fn + SourceType-Union (SCHEMA-03 Datenschicht)
|
||||
- [ ] 13-02-PLAN.md — SourceRegistry + harte Denylist-Gate (vergabe24/aumass) + Adapter-Interface-Generalisierung (INGEST-07)
|
||||
- [ ] 13-03-PLAN.md — 3-Stufen-Dedup-Resolver + pollDueSources Fan-out (catch-per-source) + Modul-Wiring (SCHEMA-03)
|
||||
- [ ] 13-04-PLAN.md — NetServer-Adapter (config-getrieben, 3 Portale) + Package-Legitimacy-Checkpoint (INGEST-02)
|
||||
@@ -485,5 +486,5 @@ Phases execute in numeric order: 1 -> 2 -> 3 -> 4 -> 5 -> 6 -> 7 -> 8 -> 9 -> 10
|
||||
| 10. Ausschreibungs-Radar Foundation & DÖE Ingestion | 6/6 | Complete | 2026-07-21 |
|
||||
| 11. Filter Engine, Results UI & Saved Searches | 6/6 | In Progress| |
|
||||
| 12. Tender Notifications | 4/4 | In Progress| |
|
||||
| 13. Scraping Adapters & Cross-Source Deduplication | 0/TBD | Not started | - |
|
||||
| 13. Scraping Adapters & Cross-Source Deduplication | 1/6 | In Progress| |
|
||||
| 14. RSS, Email-Alert Ingestion & Module Rollout | 0/TBD | Not started | - |
|
||||
|
||||
+21
-18
@@ -2,18 +2,18 @@
|
||||
gsd_state_version: 1.0
|
||||
milestone: v1.1
|
||||
milestone_name: Ausschreibungs-Radar
|
||||
current_phase: 12
|
||||
current_phase_name: tender-notifications
|
||||
status: verified
|
||||
stopped_at: Phase 12 verified — live email UAT passed (digest via Mailhog, no double-send)
|
||||
last_updated: "2026-07-22T07:32:37.037Z"
|
||||
last_activity: 2026-07-22
|
||||
last_activity_desc: Phase 12 execution started
|
||||
current_phase: 13
|
||||
current_phase_name: scraping-adapters-cross-source-dedup
|
||||
status: executing
|
||||
stopped_at: Completed 13-01-PLAN.md
|
||||
last_updated: "2026-07-23T06:40:39.176Z"
|
||||
last_activity: 2026-07-23
|
||||
last_activity_desc: Phase 13 execution started
|
||||
progress:
|
||||
total_phases: 12
|
||||
total_phases: 13
|
||||
completed_phases: 11
|
||||
total_plans: 56
|
||||
completed_plans: 55
|
||||
total_plans: 62
|
||||
completed_plans: 56
|
||||
---
|
||||
|
||||
# Project State
|
||||
@@ -23,16 +23,16 @@ progress:
|
||||
See: .planning/PROJECT.md (updated 2026-07-17)
|
||||
|
||||
**Core value:** Eine zentrale Plattform, in der beliebige Workflow-Tools als Module lizenziert, aktiviert und genutzt werden koennen -- ohne zwischen verschiedenen Anwendungen wechseln zu muessen.
|
||||
**Current focus:** Phase 12 — tender-notifications
|
||||
**Current focus:** Phase 13 — scraping-adapters-cross-source-dedup
|
||||
|
||||
## Current Position
|
||||
|
||||
Phase: 12 (tender-notifications) — EXECUTING
|
||||
Plan: 4 of 4
|
||||
Status: Phase complete — ready for verification
|
||||
Last activity: 2026-07-22 — Phase 12 execution started
|
||||
Phase: 13 (scraping-adapters-cross-source-dedup) — EXECUTING
|
||||
Plan: 2 of 6
|
||||
Status: Ready to execute
|
||||
Last activity: 2026-07-23 — Phase 13 execution started
|
||||
|
||||
Progress: [██████████] 98%
|
||||
Progress: [█████████░] 90%
|
||||
|
||||
## Performance Metrics
|
||||
|
||||
@@ -90,6 +90,7 @@ Progress: [██████████] 98%
|
||||
| Phase 12 P02 | 8min | 3 tasks | 5 files |
|
||||
| Phase 12 P03 | 15min | 2 tasks | 3 files |
|
||||
| Phase 12 P04 | 12min | 3 tasks | 10 files |
|
||||
| Phase 13 P01 | 35min | 3 tasks | 6 files |
|
||||
|
||||
## Accumulated Context
|
||||
|
||||
@@ -191,6 +192,8 @@ Recent decisions affecting current work:
|
||||
- [Phase ?]: TenderDigestScheduler.runDigest(now) takes an injectable clock parameter for testable Monday-only weekly-digest gating
|
||||
- [Phase ?]: Instant-Dispatch filtert die bereits geladenen savedSearches (kein zweiter Query); notifiedAt/channel='instant' nur nach Erfolg gestempelt — identisches Gate wie Digest (D-06)
|
||||
- [Phase ?]: Digest-interval selector inline in settings/page.tsx (already 'use client'); instantAlert toggle uses plain checkbox for chip-based SavedSearchBar UI
|
||||
- [Phase ?]: SCHEMA-03 fingerprint: title+buyer dominant, CPV division, value-bucket, deadline-day, sha256; dedupKey untouched, fingerprint additive
|
||||
- [Phase ?]: One-time TS backfill scripts run via compiled dist/ output (not raw .ts execution) to keep tsc --noEmit clean
|
||||
|
||||
### Pending Todos
|
||||
|
||||
@@ -228,7 +231,7 @@ Items acknowledged and carried forward from previous milestone close:
|
||||
|
||||
## Session Continuity
|
||||
|
||||
Last session: 2026-07-22T07:32:37.021Z
|
||||
Stopped at: Completed 12-04-PLAN.md
|
||||
Last session: 2026-07-23T06:40:39.162Z
|
||||
Stopped at: Completed 13-01-PLAN.md
|
||||
Resume file: None
|
||||
Last activity: 2026-07-14 - Built LDAP per-user exclude/denylist filter (9d1323f), migration applied on live DB, verified via Playwright: sync deactivated 4 excluded service accounts (administrator/krbtgt/guest/dns-ldap), 2 real LDAP users stay active, 0 wrongly created
|
||||
|
||||
@@ -0,0 +1,169 @@
|
||||
---
|
||||
phase: 13-scraping-adapters-cross-source-dedup
|
||||
plan: 01
|
||||
subsystem: database
|
||||
tags: [prisma, postgresql, crypto, dedup, sha256]
|
||||
|
||||
requires:
|
||||
- phase: 10-doe-opendata-ingestion
|
||||
provides: Tender model (sourcePortal, sourceNoticeId, ocid, dedupKey, cpvDivisions, contentHash), TenderNormalizerService, tender.types.ts contract
|
||||
provides:
|
||||
- "TenderSource Prisma model (1:n Tender, @@unique[sourcePortal, sourceNoticeId], onDelete Cascade)"
|
||||
- "Tender.fingerprint nullable column + index, additive alongside the unchanged dedupKey"
|
||||
- "pure tenderFingerprint() NULL-tolerant dedup-key function (title+buyer dominant, CPV division, value-bucket, deadline-day)"
|
||||
- "2851 pre-existing DÖE tenders backfilled with one TenderSource row + a computed fingerprint each"
|
||||
- "SourceType widened to open union ('doe-opendata' | 'ai-netserver' | 'cosinex-dtvp')"
|
||||
affects: [13-02-source-registry, 13-03-dedup-resolver, 13-04-netserver-adapter, 13-05-cosinex-adapter, 13-06-multi-source-display]
|
||||
|
||||
tech-stack:
|
||||
added: []
|
||||
patterns:
|
||||
- "Pure fingerprint function (no I/O, no Prisma import) computed once and reused by both the one-time backfill script and (in Plan 13-03) the live dedup resolver"
|
||||
- "Two-step migration ordering for additive schema + data backfill: (1) table/column DDL, (2) SQL data copy, (3) unique constraint — avoids constraint violations from pre-existing data"
|
||||
- "One-time TS backfill scripts run via the compiled dist/ output (standard extensionless imports), not as raw .ts via node's native type-stripping — keeps tsc --noEmit clean project-wide"
|
||||
|
||||
key-files:
|
||||
created:
|
||||
- apps/api/src/tenders/tender-fingerprint.ts
|
||||
- apps/api/src/tenders/tender-fingerprint.spec.ts
|
||||
- apps/api/src/tenders/backfill-tender-source.ts
|
||||
- apps/api/prisma/migrations/20260723120000_add_tender_source/migration.sql
|
||||
modified:
|
||||
- apps/api/prisma/schema.prisma
|
||||
- apps/api/src/tenders/tender.types.ts
|
||||
|
||||
key-decisions:
|
||||
- "fingerprint canonical string is [buyer, title, cpvDivisionKey, deadlineKey, valueBucket].join('|') -> sha256 hex, exactly per 13-RESEARCH Pattern 4 — deterministic O(1) lookup, no similarity threshold"
|
||||
- "dedupKey @unique on Tender is left untouched (SCHEMA-02 upsert target); fingerprint is a separate, additive, nullable column"
|
||||
- "One-time backfill script uses standard (extensionless) TS imports and documents running the compiled dist/ output, rather than node's raw .ts execution — Node 24's native type-stripping treats ESM-syntax .ts files as ESM requiring explicit .ts/.js extensions, which tsc rejects under moduleResolution:node; compiling first keeps both runtime execution and tsc --noEmit clean"
|
||||
|
||||
patterns-established:
|
||||
- "Fuzzy-dedup fingerprint: pure function, NULL segments collapse to empty string rather than excluding the record, title always included as the collision guard"
|
||||
|
||||
requirements-completed: [SCHEMA-03]
|
||||
|
||||
coverage:
|
||||
- id: D1
|
||||
description: "tenderFingerprint() is NULL-tolerant (title+buyer+CPV match with value/deadline both NULL), collision-resistant (different title -> different hash), umlaut-normalizing, order-independent on CPV divisions, magnitude-bucketed on value, and returns a stable sha256 hex string"
|
||||
requirement: SCHEMA-03
|
||||
verification:
|
||||
- kind: unit
|
||||
ref: "apps/api/src/tenders/tender-fingerprint.spec.ts (8 tests, all pass)"
|
||||
status: pass
|
||||
human_judgment: false
|
||||
- id: D2
|
||||
description: "TenderSource model (1:n, @@unique[sourcePortal, sourceNoticeId]) and Tender.fingerprint exist in schema.prisma; migration applied locally; every pre-existing Tender has exactly one TenderSource row backfilled from its current sourcePortal/sourceNoticeId/sourceUrl/ocid"
|
||||
requirement: SCHEMA-03
|
||||
verification:
|
||||
- kind: integration
|
||||
ref: "npx prisma validate && npx prisma generate (clean)"
|
||||
status: pass
|
||||
- kind: manual_procedural
|
||||
ref: "docker compose exec -T db psql -U tessera -d tessera -c 'SELECT count(*) FROM \"TenderSource\"' -> 2851 == SELECT count(*) FROM \"Tender\" -> 2851"
|
||||
status: pass
|
||||
human_judgment: false
|
||||
- id: D3
|
||||
description: "Tender.fingerprint backfilled for all pre-existing rows using the Task-1 tenderFingerprint() function, with Decimal->number conversion for estimatedValue"
|
||||
requirement: SCHEMA-03
|
||||
verification:
|
||||
- kind: manual_procedural
|
||||
ref: "backfill-tender-source.ts run via compiled dist/ output — console output 'fingerprint set on 2851/2851 tenders', re-verified idempotent on second run; SELECT count(*) FROM \"Tender\" WHERE fingerprint IS NOT NULL -> 2851"
|
||||
status: pass
|
||||
human_judgment: false
|
||||
- id: D4
|
||||
description: "SourceType is an open union ('doe-opendata' | 'ai-netserver' | 'cosinex-dtvp') and NormalizedTenderFields carries an optional fingerprint field, without breaking the existing normalizer contract"
|
||||
requirement: SCHEMA-03
|
||||
verification:
|
||||
- kind: unit
|
||||
ref: "cd apps/api && npx tsc --noEmit -p tsconfig.json (clean)"
|
||||
status: pass
|
||||
- kind: unit
|
||||
ref: "pnpm --filter @tessera/api test (full suite, 234/234 pass)"
|
||||
status: pass
|
||||
human_judgment: false
|
||||
|
||||
duration: 35min
|
||||
completed: 2026-07-23
|
||||
status: complete
|
||||
---
|
||||
|
||||
# Phase 13 Plan 01: Fingerprint + TenderSource Datenkern Summary
|
||||
|
||||
**NULL-tolerante `tenderFingerprint()`-Funktion, additives `TenderSource`-1:n-Modell + `Tender.fingerprint`, und Backfill aller 2851 Bestands-DÖE-Tender — der dependency-freie SCHEMA-03-Datenkern für Cross-Source-Dedup.**
|
||||
|
||||
## Performance
|
||||
|
||||
- **Duration:** 35 min
|
||||
- **Started:** 2026-07-23T08:33:00Z
|
||||
- **Completed:** 2026-07-23T08:39:00Z
|
||||
- **Tasks:** 3
|
||||
- **Files modified:** 6 (2 created new spec/source pairs collapsed to 4 created + 2 modified)
|
||||
|
||||
## Accomplishments
|
||||
- Pure, NULL-tolerante `tenderFingerprint()` (title+buyer dominant, CPV-Division, value-bucket, deadline-day, sha256) — TDD RED→GREEN, 8/8 Tests grün.
|
||||
- `TenderSource`-Modell (1:n zu `Tender`, `@@unique([sourcePortal, sourceNoticeId])`, `onDelete: Cascade`) + additive nullable `Tender.fingerprint`-Spalte + Index, `dedupKey` unverändert.
|
||||
- Handgeschriebene Migration `20260723120000_add_tender_source` in strikter 3-Schritt-Reihenfolge (DDL → Backfill-INSERT → Unique-Constraint), lokal gegen die `tessera`-Dev-DB angewendet.
|
||||
- Alle 2851 Bestands-DÖE-Tender: je eine `TenderSource`-Zeile (verifiziert `count(*) = 2851 = 2851`) + berechneter `fingerprint` (verifiziert `2851/2851`, idempotent re-run bestätigt).
|
||||
- `SourceType` zur offenen Union erweitert, `NormalizedTenderFields.fingerprint?` ergänzt — `tsc --noEmit` und volle API-Testsuite (234 Tests) grün.
|
||||
|
||||
## Task Commits
|
||||
|
||||
Each task was committed atomically:
|
||||
|
||||
1. **Task 1a: RED — failing tenderFingerprint test** - `063ba5b` (test)
|
||||
2. **Task 1b: GREEN — implement tenderFingerprint** - `c6cac69` (feat)
|
||||
3. **Task 2: TenderSource model + migration + backfill** - `447fb74` (feat)
|
||||
4. **Task 3: SourceType union + NormalizedTenderFields.fingerprint** - `c2a6021` (feat)
|
||||
|
||||
_TDD task (Task 1) produced two commits (test → feat) per protocol; no refactor commit was needed._
|
||||
|
||||
## Files Created/Modified
|
||||
- `apps/api/src/tenders/tender-fingerprint.ts` - pure NULL-tolerant sha256 dedup-fingerprint function
|
||||
- `apps/api/src/tenders/tender-fingerprint.spec.ts` - 8 unit tests (NULL-tolerance, collision guard, umlauts, CPV order-independence, value bucketing, determinism)
|
||||
- `apps/api/src/tenders/backfill-tender-source.ts` - one-time script computing `Tender.fingerprint` for all rows
|
||||
- `apps/api/prisma/migrations/20260723120000_add_tender_source/migration.sql` - table+column DDL, TenderSource backfill INSERT, unique constraint
|
||||
- `apps/api/prisma/schema.prisma` - `model TenderSource` + `Tender.fingerprint`/`sources`/`@@index([fingerprint])`
|
||||
- `apps/api/src/tenders/tender.types.ts` - `SourceType` open union, `NormalizedTenderFields.fingerprint?`
|
||||
|
||||
## Decisions Made
|
||||
- **Fingerprint algorithm exactly per 13-RESEARCH.md Pattern 4** (title+buyer dominant, CPV division not full code, value order-of-magnitude bucket, deadline day-grain, sha256 hex) — deterministic O(1) lookup, no similarity-threshold matching (avoids O(n) scans against the 2851+ row backlog per ingest).
|
||||
- **`dedupKey @unique` left untouched** — it remains the SCHEMA-02 DÖE upsert target; `fingerprint` is a fully additive, nullable column, no migration risk to existing ingestion.
|
||||
- **Backfill script runs via compiled `dist/` output, not raw `.ts` via node's native type-stripping.** Node 24 treats a `.ts` file containing `import`/`export` syntax as ESM and requires explicit `.ts`/`.js` extensions for relative imports; but the project's `tsconfig.json` (`module: commonjs`, `moduleResolution: node`) rejects explicit `.ts` extensions in import specifiers (`TS5097`). Using standard extensionless imports (tsc-clean) and documenting `pnpm --filter @tessera/api build && node dist/tenders/backfill-tender-source.js` as the run command resolves both constraints — confirmed working end-to-end (2851/2851, idempotent re-run).
|
||||
|
||||
## Deviations from Plan
|
||||
|
||||
### Auto-fixed Issues
|
||||
|
||||
**1. [Rule 3 - Blocking] Backfill script import style adjusted for tsc/runtime compatibility**
|
||||
- **Found during:** Task 2 (backfill-tender-source.ts)
|
||||
- **Issue:** The plan didn't specify how the one-time TS backfill script would actually be executed. An initial version imported `./tender-fingerprint.ts` with an explicit extension to satisfy Node 24's native TS execution — but that broke `npx tsc --noEmit` (Task 3's verification command) with `TS5097: An import path can only end with a '.ts' extension when 'allowImportingTsExtensions' is enabled`.
|
||||
- **Fix:** Switched to standard extensionless imports (tsc-compatible) and ran the script via `nest build` + `node dist/tenders/backfill-tender-source.js` instead of raw `.ts` execution. Documented this invocation in the file's header comment.
|
||||
- **Files modified:** apps/api/src/tenders/backfill-tender-source.ts
|
||||
- **Verification:** `npx tsc --noEmit -p tsconfig.json` clean; compiled script re-run confirmed idempotent (`2851/2851` both times).
|
||||
- **Committed in:** 447fb74 (Task 2 commit)
|
||||
|
||||
---
|
||||
|
||||
**Total deviations:** 1 auto-fixed (1 blocking — script invocation mechanics)
|
||||
**Impact on plan:** No scope creep; fix was purely about how the already-planned one-time script gets executed so it doesn't regress the project's typecheck verification.
|
||||
|
||||
## Issues Encountered
|
||||
- `apps/api/prisma/migrations/20260722180000_ldap_tls_reject_unauthorized` (from a prior phase) had not yet been applied to this local dev DB. `prisma migrate deploy` applied it together with the new `20260723120000_add_tender_source` migration in the same run — expected catch-up, not a regression caused by this plan.
|
||||
- `DATABASE_URL` is not present in `apps/api/.env` on the host (only used inside the Docker network); per project MEMORY (`project_local_db_migrations`), it must be exported manually with the `db` container's ephemeral bridge IP (`docker inspect ... tessera-ctl-db-1`) for any host-side Prisma CLI invocation. Followed the documented workflow.
|
||||
|
||||
## User Setup Required
|
||||
|
||||
None - no external service configuration required. All changes applied to the local dev DB only (no test/prod server touched, per MEMORY constraints).
|
||||
|
||||
## Next Phase Readiness
|
||||
- The dependency-free SCHEMA-03 data core (D-01) is fully in place and tested: `TenderSource`, `Tender.fingerprint`, and `tenderFingerprint()` are ready for Plan 13-02 (Source-Registry + Denylist) and Plan 13-03 (3-tier dedup resolver) to build on directly.
|
||||
- `SourceType`'s open union unblocks Plan 13-04 (NetServer adapter) and Plan 13-05 (cosinex adapter) without further type-contract changes.
|
||||
- No blockers. `dedupKey`'s eventual consolidation with `fingerprint` remains explicitly deferred (13-RESEARCH Open Question 3) — not in scope for this milestone.
|
||||
|
||||
---
|
||||
*Phase: 13-scraping-adapters-cross-source-dedup*
|
||||
*Completed: 2026-07-23*
|
||||
|
||||
## Self-Check: PASSED
|
||||
|
||||
All created files verified present on disk; all 4 task commit hashes verified in git log.
|
||||
Reference in New Issue
Block a user