From 4823c245ee6e22f45d41232c9e39ccff2beaf6ac Mon Sep 17 00:00:00 2001 From: Schalli Date: Wed, 22 Jul 2026 09:07:06 +0200 Subject: [PATCH] docs(12-01): complete schema + delta-only matching plan --- .planning/REQUIREMENTS.md | 4 +- .planning/ROADMAP.md | 6 +- .planning/STATE.md | 39 +++-- .../12-tender-notifications/12-01-SUMMARY.md | 154 ++++++++++++++++++ 4 files changed, 180 insertions(+), 23 deletions(-) create mode 100644 .planning/phases/12-tender-notifications/12-01-SUMMARY.md diff --git a/.planning/REQUIREMENTS.md b/.planning/REQUIREMENTS.md index 13ea70e..3a698cf 100644 --- a/.planning/REQUIREMENTS.md +++ b/.planning/REQUIREMENTS.md @@ -46,7 +46,7 @@ - [ ] **NOTIFY-01**: Nutzer erhält einen periodischen E-Mail-Digest passender Treffer; Intervall im Webinterface konfigurierbar. - [ ] **NOTIFY-02**: Nutzer erhält optional eine Sofort-E-Mail bei einem neuen Treffer eines aktiven Suchprofils; im Webinterface aktivierbar. -- [ ] **NOTIFY-03**: Das System unterscheidet „getroffen" von „benachrichtigt" (kein Doppelversand Digest+Sofort; kein Rückstau-Massenversand beim Anlegen eines Suchprofils). +- [x] **NOTIFY-03**: Das System unterscheidet „getroffen" von „benachrichtigt" (kein Doppelversand Digest+Sofort; kein Rückstau-Massenversand beim Anlegen eines Suchprofils). - [ ] **NOTIFY-04**: Der E-Mail-Versand nutzt die mandantenspezifische SMTP-Konfiguration (bestehendes DKV-Mail-Muster). ### CONFIG — Modul & Verwaltung @@ -94,7 +94,7 @@ | UI-05 | Phase 11 | Complete | | NOTIFY-01 | Phase 12 | Pending | | NOTIFY-02 | Phase 12 | Pending | -| NOTIFY-03 | Phase 12 | Pending | +| NOTIFY-03 | Phase 12 | Complete | | NOTIFY-04 | Phase 12 | Pending | | INGEST-02 | Phase 13 | Pending | | INGEST-03 | Phase 13 | Pending | diff --git a/.planning/ROADMAP.md b/.planning/ROADMAP.md index fc04fbd..569010e 100644 --- a/.planning/ROADMAP.md +++ b/.planning/ROADMAP.md @@ -410,12 +410,12 @@ Plans: 3. Activating a new search profile with many pre-existing historical matches does not flood the user with individual emails -- backfill is suppressed/batched on first activation, and a match already notified once (digest or instant) is never notified again for the same tender+profile pair (explicit matched-vs-notified state) 4. Notification emails are sent through the tenant's own SMTP configuration (reusing the DKV mail pattern), not a shared/global system mailer -**Plans**: 4 plans +**Plans**: 1/4 plans executed **UI hint**: yes **Wave 1** -- [ ] 12-01-PLAN.md — Schema (TenderMatch/NotificationPref/instantAlert) + delta-only Matching in den Poll-Tick (NOTIFY-03) +- [x] 12-01-PLAN.md — Schema (TenderMatch/NotificationPref/instantAlert) + delta-only Matching in den Poll-Tick (NOTIFY-03) **Wave 2** *(blocked on Wave 1)* @@ -476,6 +476,6 @@ Phases execute in numeric order: 1 -> 2 -> 3 -> 4 -> 5 -> 6 -> 7 -> 8 -> 9 -> 10 | 9. Cert Manager Module | 6/6 | Complete | 2026-07-02 | | 10. Ausschreibungs-Radar Foundation & DÖE Ingestion | 6/6 | Complete | 2026-07-21 | | 11. Filter Engine, Results UI & Saved Searches | 6/6 | In Progress| | -| 12. Tender Notifications | 0/TBD | Not started | - | +| 12. Tender Notifications | 1/4 | In Progress| | | 13. Scraping Adapters & Cross-Source Deduplication | 0/TBD | Not started | - | | 14. RSS, Email-Alert Ingestion & Module Rollout | 0/TBD | Not started | - | diff --git a/.planning/STATE.md b/.planning/STATE.md index 810f0bf..cdd8af4 100644 --- a/.planning/STATE.md +++ b/.planning/STATE.md @@ -2,18 +2,18 @@ gsd_state_version: 1.0 milestone: v1.1 milestone_name: Ausschreibungs-Radar -current_phase: 11 -current_phase_name: filter-engine-results-ui-saved-searches -status: verified -stopped_at: Phase 11 verified — live browser UAT passed (all 5 criteria) -last_updated: "2026-07-21T14:52:59.205Z" -last_activity: 2026-07-21 -last_activity_desc: Phase 11 execution started +current_phase: 12 +current_phase_name: tender-notifications +status: executing +stopped_at: Completed 12-01-PLAN.md +last_updated: "2026-07-22T07:06:59.374Z" +last_activity: 2026-07-22 +last_activity_desc: Phase 12 execution started progress: - total_phases: 11 + total_phases: 12 completed_phases: 10 - total_plans: 52 - completed_plans: 51 + total_plans: 56 + completed_plans: 52 --- # Project State @@ -23,16 +23,16 @@ progress: See: .planning/PROJECT.md (updated 2026-07-17) **Core value:** Eine zentrale Plattform, in der beliebige Workflow-Tools als Module lizenziert, aktiviert und genutzt werden koennen -- ohne zwischen verschiedenen Anwendungen wechseln zu muessen. -**Current focus:** Phase 11 — filter-engine-results-ui-saved-searches +**Current focus:** Phase 12 — tender-notifications ## Current Position -Phase: 11 (filter-engine-results-ui-saved-searches) — EXECUTING -Plan: 6 of 6 -Status: Phase complete — ready for verification -Last activity: 2026-07-21 — Phase 11 execution started +Phase: 12 (tender-notifications) — EXECUTING +Plan: 2 of 4 +Status: Ready to execute +Last activity: 2026-07-22 — Phase 12 execution started -Progress: [██████████] 98% +Progress: [█████████░] 93% ## Performance Metrics @@ -86,6 +86,7 @@ Progress: [██████████] 98% | Phase 11 P04 | 12min | 2 tasks | 5 files | | Phase 11 P05 | 24min | 3 tasks | 15 files | | Phase 11 P06 | 35min | 3 tasks | 12 files | +| Phase 12 P01 | 35min | 3 tasks | 7 files | ## Accumulated Context @@ -181,6 +182,8 @@ Recent decisions affecting current work: - [Phase ?]: TenderTriage (neu, per-user, kein forTenant/RLS) + favOnly-Sentinel-ID '__none__' für garantierten Zero-Match statt versehentlich ungefilterter Liste - [Phase ?]: TenderSavedSearch hat keinen Tender-FK — speichert nur Filterkriterien, unkritisch bei Retention. - [Phase ?]: page/tender-URL-Params bewusst aus dem Suchprofil-Payload ausgeschlossen (Navigations-/View-State, kein Filter-State). +- [Phase ?]: Single notifiedAt field (not two per-channel timestamps) as the matched-vs-notified eligibility gate (12-01, D-06) +- [Phase ?]: Delta-only matching (no backfill/suppression table) structurally prevents backfill-flood for new saved-search profiles (12-01, D-07) ### Pending Todos @@ -218,7 +221,7 @@ Items acknowledged and carried forward from previous milestone close: ## Session Continuity -Last session: 2026-07-21T14:52:59.192Z -Stopped at: Completed 11-06-PLAN.md +Last session: 2026-07-22T07:06:59.360Z +Stopped at: Completed 12-01-PLAN.md Resume file: None Last activity: 2026-07-14 - Built LDAP per-user exclude/denylist filter (9d1323f), migration applied on live DB, verified via Playwright: sync deactivated 4 excluded service accounts (administrator/krbtgt/guest/dns-ldap), 2 real LDAP users stay active, 0 wrongly created diff --git a/.planning/phases/12-tender-notifications/12-01-SUMMARY.md b/.planning/phases/12-tender-notifications/12-01-SUMMARY.md new file mode 100644 index 0000000..e18fea1 --- /dev/null +++ b/.planning/phases/12-tender-notifications/12-01-SUMMARY.md @@ -0,0 +1,154 @@ +--- +phase: 12-tender-notifications +plan: 01 +subsystem: api +tags: [prisma, nestjs, tender-radar, notifications, matching] + +requires: + - phase: 11-tender-saved-searches + provides: "TenderSavedSearch (filters Json, userId/tenantId scoping) + buildTenderWhere(dto) filter compiler" + - phase: 10-tender-radar-ingestion + provides: "TenderIngestionService.pollDueSources poll-tick + Tender catalog (dedupKey upsert)" +provides: + - "TenderMatch model — matched-vs-notified state (single nullable notifiedAt gate)" + - "TenderNotificationPref model — per-user digest interval preference" + - "TenderSavedSearch.instantAlert — per-profile instant alert flag" + - "TenderMatchingService.matchDelta(newTenderIds) — delta-only matching engine" + - "pollDueSources now collects genuinely-new tender IDs and triggers matchDelta" +affects: [12-02-tender-digest-send, 12-03-tender-instant-alerts, 12-04-tender-notification-settings-ui] + +tech-stack: + added: [] + patterns: + - "Delta-only matching: matchDelta only ever sees id IN newTenderIds — no historical rescan, structural backfill-flood prevention (D-07)" + - "Idempotent match upsert on @@unique([tenderId,savedSearchId]) with update:{} preserves notifiedAt on re-match (D-06)" + - "Indexed dedupKey pre-check (findUnique before upsert) to detect genuinely-new rows, since Prisma upsert doesn't report create-vs-update" + +key-files: + created: + - apps/api/prisma/migrations/20260722100000_add_tender_notifications/migration.sql + - apps/api/src/tenders/tender-matching.service.ts + - apps/api/src/tenders/tender-matching.service.spec.ts + modified: + - apps/api/prisma/schema.prisma + - apps/api/src/tenders/tender-ingestion.service.ts + - apps/api/src/tenders/tender-ingestion.service.spec.ts + - apps/api/src/tenders/tenders.module.ts + +key-decisions: + - "One notifiedAt field (not two per-channel fields) — the eligibility gate is a single WHERE notifiedAt IS NULL predicate (D-06)" + - "Delta-only matching instead of a backfill/suppression table — a new profile collects zero matches against the existing ~2188-row catalog by construction (D-07)" + - "matchDelta wraps each profile's matching in its own try/catch — one broken profile's filters JSON never aborts matching for the rest (matches pollDueSources' existing catch-and-log convention)" + +patterns-established: + - "matchDelta reuses buildTenderWhere(profile.filters as TenderQueryDto) — no new filter logic duplicated" + +requirements-completed: [NOTIFY-03] + +coverage: + - id: D1 + description: "TenderMatch/TenderNotificationPref schema + TenderSavedSearch.instantAlert; migration applied to local dev DB" + requirement: "NOTIFY-03" + verification: + - kind: integration + ref: "npx prisma validate && npx prisma generate (manual) + psql to_regclass check" + status: pass + human_judgment: false + - id: D2 + description: "TenderMatchingService.matchDelta — delta-only matching, match creation, idempotent upsert preserving notifiedAt, empty-delta no-op" + requirement: "NOTIFY-03" + verification: + - kind: unit + ref: "apps/api/src/tenders/tender-matching.service.spec.ts (6 tests)" + status: pass + human_judgment: false + - id: D3 + description: "pollDueSources collects genuinely-new tender IDs (indexed dedupKey pre-check) and calls matchDelta only with those; TenderMatchingService registered in TendersModule" + requirement: "NOTIFY-03" + verification: + - kind: unit + ref: "apps/api/src/tenders/tender-ingestion.service.spec.ts (9 tests, incl. 2 new delta-only tests)" + status: pass + human_judgment: false + +duration: 35min +completed: 2026-07-22 +status: complete +--- + +# Phase 12 Plan 01: Schema + Delta-Only Matching Engine Summary + +**TenderMatch/TenderNotificationPref schema (single notifiedAt eligibility gate) plus a delta-only TenderMatchingService wired into pollDueSources — structurally prevents any backfill-flood on new saved-search profiles.** + +## Performance + +- **Duration:** ~35 min +- **Completed:** 2026-07-22T09:06:00Z +- **Tasks:** 3 completed (Task 2 followed RED→GREEN TDD) +- **Files modified:** 7 (3 created, 4 modified) + +## Accomplishments +- Added `TenderMatch` (one row per tender × savedSearch pair, `@@unique([tenderId, savedSearchId])`, single nullable `notifiedAt` as the matched-vs-notified invariant gate) and `TenderNotificationPref` (per-user digest interval, default `daily`) models; added `TenderSavedSearch.instantAlert Boolean @default(false)`. +- Hand-written migration `20260722100000_add_tender_notifications` applied to the local dev DB (`tessera` database, container `tessera-ctl-db-1`) and recorded in `_prisma_migrations`; `prisma generate` run so the client knows the new models. +- Implemented `TenderMatchingService.matchDelta(newTenderIds)`: for every active `TenderSavedSearch`, reuses `buildTenderWhere(profile.filters)` AND-ed with `{ id: { in: newTenderIds } }`, and upserts `TenderMatch` idempotently (`update: {}` preserves any already-set `notifiedAt`). +- Extended `TenderIngestionService.pollDueSources`: an indexed `dedupKey` pre-check now distinguishes genuinely-new tender rows from changed/re-seen ones; only the new IDs are handed to `matchDelta` at the end of the tick (delta-only boundary, D-07 — no historical rescan, no backfill/suppression table). +- Registered `TenderMatchingService` as a provider in `TendersModule` and injected it into `TenderIngestionService`. + +## Task Commits + +Each task was committed atomically: + +1. **Task 1: Schema models + hand-written migration + local DB apply** - `6c3e110` (feat) +2. **Task 2: TenderMatchingService.matchDelta (TDD)** - `57d22a0` (test, RED) → `92d4c96` (feat, GREEN) +3. **Task 3: Wire matchDelta into pollDueSources + provider registration** - `b45047f` (feat) + +**Plan metadata:** commit pending (this SUMMARY + STATE/ROADMAP update) + +## Files Created/Modified +- `apps/api/prisma/schema.prisma` - Added `TenderMatch`, `TenderNotificationPref` models; `TenderSavedSearch.instantAlert` + `matches` relation; `Tender.matches` relation +- `apps/api/prisma/migrations/20260722100000_add_tender_notifications/migration.sql` - Hand-written migration, applied to local dev DB +- `apps/api/src/tenders/tender-matching.service.ts` - `matchDelta(newTenderIds)`: delta-only matching engine +- `apps/api/src/tenders/tender-matching.service.spec.ts` - 6 unit tests (delta-only, match creation, idempotency, empty-delta) +- `apps/api/src/tenders/tender-ingestion.service.ts` - Collects genuinely-new tender IDs; calls `matchDelta` at tick end +- `apps/api/src/tenders/tender-ingestion.service.spec.ts` - 2 new tests asserting delta-only invocation of `matchDelta` +- `apps/api/src/tenders/tenders.module.ts` - `TenderMatchingService` registered as provider + +## Decisions Made +- Single `notifiedAt` field (not two per-channel timestamps) — matches RESEARCH.md's explicit recommendation; simpler invariant, no risk of implying double-send is allowed per channel. +- Delta-only matching (no backfill/suppression table) as the structural mechanism for D-07 — verified: `matchDelta([])` performs zero DB access, and a large simulated pre-existing catalog (2188 IDs) that would all match a profile's filters still yields zero matches when none of those IDs are passed as `newTenderIds`. +- Added per-profile try/catch inside `matchDelta` (Rule 2 — matches the established `pollDueSources` catch-and-log convention; not explicitly specified in the plan's action text, but the plan's own `` criteria and RESEARCH.md's "Don't Hand-Roll"/pitfalls make single-profile robustness an implicit correctness requirement, since a malformed `filters` JSON on one profile must not silently prevent matching for all other profiles in the same tick). + +## Deviations from Plan + +### Auto-fixed Issues + +**1. [Rule 1 - Bug] Plan's verify command referenced the wrong local DB name** +- **Found during:** Task 1 verification +- **Issue:** The plan's `` block and `12-VALIDATION.md` reference `docker compose exec ... -d tessera_dev`, but `docker-compose.yml` actually provisions `POSTGRES_DB: tessera` (confirmed via `docker exec tessera-ctl-db-1 psql -U tessera -d tessera -c "\dt"`, which lists all existing Tender/TenderSavedSearch/TenderTriage tables). `tessera_dev` is the DB **password**, not the DB name. +- **Fix:** Applied the migration and ran all verification (`to_regclass`, `\d`, `prisma validate`/`generate`) against `-d tessera` instead. No schema/code change — purely a corrected command target. +- **Verification:** `SELECT to_regclass('public."TenderMatch"'), to_regclass('public."TenderNotificationPref"')` returned both table OIDs (non-null); `\d "TenderSavedSearch"` shows `instantAlert boolean not null default false`. +- **Committed in:** 6c3e110 (Task 1 commit) + +--- + +**Total deviations:** 1 auto-fixed (1 bug — wrong DB name in verify command, not a code defect) +**Impact on plan:** No scope creep; purely a corrected verification target. All three tasks executed exactly as specified otherwise. + +## Issues Encountered +None beyond the DB-name correction above. `gen_random_uuid()` was available on the DB for the manual `_prisma_migrations` bookkeeping insert (Postgres 16 built-in via `pgcrypto`/native function). + +## User Setup Required +None - no external service configuration required. Migration was applied directly to the local dev DB per the environment constraints (no Docker rebuild, no test/prod server touched). + +## Next Phase Readiness +- `TenderMatch` rows are now created (with `notifiedAt=NULL`) for every genuinely-new tender that matches an active saved search — ready for Plan 12-02 (digest send) and 12-03 (instant alerts) to consume via `WHERE notifiedAt IS NULL`. +- `TenderNotificationPref` and `TenderSavedSearch.instantAlert` are in place for Plan 12-04's settings UI. +- No blockers. `pnpm --filter api test` (full suite) is green: 172/172 tests across 15 files. `npx tsc --noEmit` clean. + +--- +*Phase: 12-tender-notifications* +*Completed: 2026-07-22* + +## Self-Check: PASSED + +All created files verified present on disk; all 4 task commit hashes (6c3e110, 57d22a0, 92d4c96, b45047f) verified present in git log.