diff --git a/.planning/REQUIREMENTS.md b/.planning/REQUIREMENTS.md index 61f48ff..304362d 100644 --- a/.planning/REQUIREMENTS.md +++ b/.planning/REQUIREMENTS.md @@ -16,7 +16,7 @@ - [ ] **INGEST-04**: Das System importiert Ausschreibungen aus RSS-Feeds (subreport-elvis, service.bund.de). - [ ] **INGEST-05**: Das System liest Portal-Benachrichtigungs-E-Mails aus einem konfigurierten Postfach ein (nutzt bestehende DKV-Inbox-Infrastruktur) und extrahiert daraus Ausschreibungen. - [x] **INGEST-06**: Jede Quelle wird einmal zentral pro Zeitplan abgefragt (poll-once-fan-out-many), Intervall pro Quelle im Admin-Bereich konfigurierbar; das Ergebnis wird an alle passenden Mandanten-Suchprofile verteilt. -- [ ] **INGEST-07**: vergabe24 und aumass sind als harte Denylist hinterlegt und können nicht als automatische Scraping-Quelle registriert werden (AGB-Verbot). +- [x] **INGEST-07**: vergabe24 und aumass sind als harte Denylist hinterlegt und können nicht als automatische Scraping-Quelle registriert werden (AGB-Verbot). ### SCHEMA — Normalisierung & Deduplizierung @@ -98,7 +98,7 @@ | NOTIFY-04 | Phase 12 | Complete | | INGEST-02 | Phase 13 | Pending | | INGEST-03 | Phase 13 | Pending | -| INGEST-07 | Phase 13 | Pending | +| INGEST-07 | Phase 13 | Complete | | SCHEMA-03 | Phase 13 | Complete | | INGEST-04 | Phase 14 | Pending | | INGEST-05 | Phase 14 | Pending | diff --git a/.planning/ROADMAP.md b/.planning/ROADMAP.md index e3be6e4..aa2f934 100644 --- a/.planning/ROADMAP.md +++ b/.planning/ROADMAP.md @@ -439,12 +439,12 @@ Plans: 3. A tender that appears via both DÖE and a scraping adapter shows up once in the results list (fuzzy fingerprint dedup on buyer+title+CPV+deadline+value), with links to all of its source portals -- dedup logic only activates once a second source is live 4. Attempting to register vergabe24 or aumass as a poll source is refused by the system itself (adapter registry denylist enforced in code), not just documented as forbidden -**Plans**: 1/6 plans executed +**Plans**: 2/6 plans executed Plans: - [x] 13-01-PLAN.md — TenderSource-Schema + Backfill + NULL-tolerante Fingerprint-Fn + SourceType-Union (SCHEMA-03 Datenschicht) -- [ ] 13-02-PLAN.md — SourceRegistry + harte Denylist-Gate (vergabe24/aumass) + Adapter-Interface-Generalisierung (INGEST-07) +- [x] 13-02-PLAN.md — SourceRegistry + harte Denylist-Gate (vergabe24/aumass) + Adapter-Interface-Generalisierung (INGEST-07) - [ ] 13-03-PLAN.md — 3-Stufen-Dedup-Resolver + pollDueSources Fan-out (catch-per-source) + Modul-Wiring (SCHEMA-03) - [ ] 13-04-PLAN.md — NetServer-Adapter (config-getrieben, 3 Portale) + Package-Legitimacy-Checkpoint (INGEST-02) - [ ] 13-05-PLAN.md — cosinex/DTVP-Adapter (separat, best-effort) (INGEST-03) @@ -486,5 +486,5 @@ Phases execute in numeric order: 1 -> 2 -> 3 -> 4 -> 5 -> 6 -> 7 -> 8 -> 9 -> 10 | 10. Ausschreibungs-Radar Foundation & DÖE Ingestion | 6/6 | Complete | 2026-07-21 | | 11. Filter Engine, Results UI & Saved Searches | 6/6 | In Progress| | | 12. Tender Notifications | 4/4 | In Progress| | -| 13. Scraping Adapters & Cross-Source Deduplication | 1/6 | In Progress| | +| 13. Scraping Adapters & Cross-Source Deduplication | 2/6 | In Progress| | | 14. RSS, Email-Alert Ingestion & Module Rollout | 0/TBD | Not started | - | diff --git a/.planning/STATE.md b/.planning/STATE.md index fd30425..b2134c9 100644 --- a/.planning/STATE.md +++ b/.planning/STATE.md @@ -5,15 +5,15 @@ milestone_name: Ausschreibungs-Radar current_phase: 13 current_phase_name: scraping-adapters-cross-source-dedup status: executing -stopped_at: Completed 13-01-PLAN.md -last_updated: "2026-07-23T06:40:39.176Z" +stopped_at: Completed 13-02-PLAN.md +last_updated: "2026-07-23T06:44:34.483Z" last_activity: 2026-07-23 last_activity_desc: Phase 13 execution started progress: total_phases: 13 completed_phases: 11 total_plans: 62 - completed_plans: 56 + completed_plans: 57 --- # Project State @@ -28,11 +28,11 @@ See: .planning/PROJECT.md (updated 2026-07-17) ## Current Position Phase: 13 (scraping-adapters-cross-source-dedup) — EXECUTING -Plan: 2 of 6 +Plan: 3 of 6 Status: Ready to execute Last activity: 2026-07-23 — Phase 13 execution started -Progress: [█████████░] 90% +Progress: [█████████░] 92% ## Performance Metrics @@ -91,6 +91,7 @@ Progress: [█████████░] 90% | Phase 12 P03 | 15min | 2 tasks | 3 files | | Phase 12 P04 | 12min | 3 tasks | 10 files | | Phase 13 P01 | 35min | 3 tasks | 6 files | +| Phase 13 P02 | 20min | 2 tasks | 4 files | ## Accumulated Context @@ -194,6 +195,7 @@ Recent decisions affecting current work: - [Phase ?]: Digest-interval selector inline in settings/page.tsx (already 'use client'); instantAlert toggle uses plain checkbox for chip-based SavedSearchBar UI - [Phase ?]: SCHEMA-03 fingerprint: title+buyer dominant, CPV division, value-bucket, deadline-day, sha256; dedupKey untouched, fingerprint additive - [Phase ?]: One-time TS backfill scripts run via compiled dist/ output (not raw .ts execution) to keep tsc --noEmit clean +- [Phase ?]: SourceRegistry.register() throws DeniedPortalError for any portal in DENYLISTED_PORTALS (vergabe24, aumass), enforced at DI-registration time not just documented (INGEST-07) ### Pending Todos @@ -231,7 +233,7 @@ Items acknowledged and carried forward from previous milestone close: ## Session Continuity -Last session: 2026-07-23T06:40:39.162Z -Stopped at: Completed 13-01-PLAN.md +Last session: 2026-07-23T06:44:34.470Z +Stopped at: Completed 13-02-PLAN.md Resume file: None Last activity: 2026-07-14 - Built LDAP per-user exclude/denylist filter (9d1323f), migration applied on live DB, verified via Playwright: sync deactivated 4 excluded service accounts (administrator/krbtgt/guest/dns-ldap), 2 real LDAP users stay active, 0 wrongly created diff --git a/.planning/phases/13-scraping-adapters-cross-source-dedup/13-02-SUMMARY.md b/.planning/phases/13-scraping-adapters-cross-source-dedup/13-02-SUMMARY.md new file mode 100644 index 0000000..805c7b2 --- /dev/null +++ b/.planning/phases/13-scraping-adapters-cross-source-dedup/13-02-SUMMARY.md @@ -0,0 +1,137 @@ +--- +phase: 13-scraping-adapters-cross-source-dedup +plan: 02 +subsystem: backend +tags: [nestjs, di, security-gate, adapter-pattern] + +requires: + - phase: 13-scraping-adapters-cross-source-dedup + plan: 01 + provides: SourceType open union, TenderSourceAdapter interface (Phase 10 foundation) +provides: + - "TenderSourceAdapter.portals: readonly string[] — adapters declare all portals they serve" + - "SourceRegistry (Injectable) mapping SourceType -> TenderSourceAdapter, with get()/activeAdapters()" + - "DeniedPortalError + DENYLISTED_PORTALS = ['vergabe24', 'aumass'] — hard code-level registration refusal" +affects: [13-03-dedup-resolver-fan-out, 13-04-netserver-adapter, 13-05-cosinex-adapter] + +tech-stack: + added: [] + patterns: + - "Denylist enforced as a code exception at DI-registration time (SourceRegistry.register), not documentation-only — checked per-portal, so a mixed portals array is rejected wholesale if any single entry is denylisted" + +key-files: + created: + - apps/api/src/tenders/source-registry.ts + - apps/api/src/tenders/source-registry.spec.ts + modified: + - apps/api/src/tenders/adapters/tender-source-adapter.interface.ts + - apps/api/src/tenders/adapters/doe-opendata.adapter.ts + +key-decisions: + - "portals is a readonly string[] on the adapter interface (not a single sourcePortal) exactly per 13-RESEARCH Pattern 1/2 — lets one NetServer adapter (Plan 13-04) serve tender24/lhs-vpbw/vergabe.landbw while the registry still gates each portal individually" + - "Denylist check iterates ALL adapter.portals before registering any — one denylisted portal in a mixed array rejects the entire adapter, no partial registration (T-13-02-02)" + - "Fake adapter stub used in the spec (no real scraping/HTTP dependency) — keeps the registry test suite dependency-free per D-01" + +patterns-established: + - "SourceRegistry.get()/activeAdapters() is the seam Plan 13-03's pollDueSources fan-out will use instead of a hardwired single-adapter injection" + +requirements-completed: [INGEST-07] + +coverage: + - id: D1 + description: "Registering an adapter whose portals include 'vergabe24' or 'aumass' throws DeniedPortalError as a code-level exception (not documentation-only), including when the denylisted portal is mixed into an otherwise-legitimate portals array" + requirement: INGEST-07 + verification: + - kind: unit + ref: "apps/api/src/tenders/source-registry.spec.ts — 'throws DeniedPortalError when registering an adapter serving vergabe24', 'throws DeniedPortalError when registering an adapter serving aumass', 'rejects a mixed portals array wholesale when one entry is denylisted' (6/6 tests pass)" + status: pass + human_judgment: false + - id: D2 + description: "Legitimate adapters register and are retrievable via get(sourceType)/activeAdapters(); get() of an unregistered sourceType returns undefined without throwing" + requirement: INGEST-07 + verification: + - kind: unit + ref: "apps/api/src/tenders/source-registry.spec.ts — 'registers a legitimate adapter...', 'activeAdapters() returns all registered adapters', 'get() returns undefined for an unregistered sourceType (no throw)'" + status: pass + human_judgment: false + - id: D3 + description: "TenderSourceAdapter interface generalized with portals[]; DoeOpenDataAdapter declares portals = ['doe-opendata'] without behavior change; project-wide typecheck and full existing tenders test slice remain green" + requirement: INGEST-07 + verification: + - kind: unit + ref: "cd apps/api && npx tsc --noEmit -p tsconfig.json (clean)" + status: pass + - kind: unit + ref: "cd apps/api && npx vitest run src/tenders (18 files, 177/177 tests pass, includes unchanged doe-opendata.adapter.spec.ts 6/6)" + status: pass + human_judgment: false + +duration: 20min +completed: 2026-07-23 +status: complete +--- + +# Phase 13 Plan 02: SourceRegistry + Denylist Gate Summary + +**Injectable `SourceRegistry` that throws `DeniedPortalError` in code — not just documentation — the instant an adapter declares `vergabe24` or `aumass` among its `portals`, plus the generalized `TenderSourceAdapter.portals[]` contract that lets one adapter serve multiple portals.** + +## Performance + +- **Duration:** 20 min +- **Started:** 2026-07-23T08:41:00Z +- **Completed:** 2026-07-23T08:44:30Z +- **Tasks:** 2 +- **Files modified:** 4 (2 created, 2 modified) + +## Accomplishments +- `TenderSourceAdapter` interface gained `readonly portals: readonly string[]`; `DoeOpenDataAdapter` declares `portals = ['doe-opendata'] as const` — additive, no behavior change, existing spec (6/6) still green. +- New `SourceRegistry` (`@Injectable`): `register()` iterates every `adapter.portals` entry and throws `DeniedPortalError` (German AGB message) if any match `DENYLISTED_PORTALS = ['vergabe24', 'aumass']`; `get(sourceType)` returns `undefined` (not a throw) for unregistered types; `activeAdapters()` returns all registered adapters for Plan 13-03's fan-out scheduler. +- **Denylist refusal proof (Erfolgskriterium 4):** `source-registry.spec.ts` — RED-first TDD — proves `register()` throws for `portals: ['vergabe24']`, throws for `portals: ['aumass']`, and rejects a **mixed** array `['tender24', 'aumass']` wholesale (one denylisted portal is enough — no partial registration, T-13-02-02). Legitimate registration + `get()`/`activeAdapters()` also covered. 6/6 tests pass. +- `npx tsc --noEmit` clean and full `src/tenders` slice (18 files, 177/177 tests) green — no Prisma/scraping import introduced. + +## Task Commits + +Each task was committed atomically: + +1. **Task 1: Adapter interface generalized with portals[]** - `1b11ada` (feat) +2. **Task 2a: RED — failing SourceRegistry denylist-gate spec** - `78b17ef` (test) +3. **Task 2b: GREEN — SourceRegistry implementation** - `0fa9567` (feat) + +_TDD task (Task 2) produced two commits (test → feat) per protocol; no refactor commit was needed — the first implementation passed all 6 tests cleanly._ + +## Files Created/Modified +- `apps/api/src/tenders/adapters/tender-source-adapter.interface.ts` - added `portals: readonly string[]` field + doc explaining the denylist-gate hook +- `apps/api/src/tenders/adapters/doe-opendata.adapter.ts` - declares `readonly portals = ['doe-opendata'] as const` +- `apps/api/src/tenders/source-registry.ts` - `DENYLISTED_PORTALS`, `DeniedPortalError`, `@Injectable() SourceRegistry` (register/get/activeAdapters) +- `apps/api/src/tenders/source-registry.spec.ts` - 6 unit tests (denylist refusal x2, mixed-array refusal, legitimate register/get, activeAdapters, unregistered get) + +## Decisions Made +- **`portals` as `readonly string[]` on the interface, exactly per 13-RESEARCH Pattern 1/2** — decouples "one adapter" from "one portal" so the future NetServer adapter (Plan 13-04) can serve `tender24`/`lhs-vpbw`/`vergabe.landbw` from one class while the registry still checks each portal individually. +- **Denylist check iterates the full `portals` array before any mutation of the adapters Map** — a mixed portals array containing even one denylisted entry is rejected entirely; the registry never partially registers an adapter. +- **Fake adapter stub in the spec** (no `doe-opendata.adapter.ts` reuse, no HTTP/Prisma) — keeps `source-registry.spec.ts` fully dependency-free, consistent with D-01 ("robust core first, independent of live scrapability"). + +## Deviations from Plan + +None - plan executed exactly as written. No Rule 1-4 auto-fixes were needed; the plan's design (Pattern 1 + Pattern 2 from 13-RESEARCH.md) was followed verbatim. + +## Issues Encountered + +None. + +## User Setup Required + +None - no external service configuration, no migration, no environment changes. Pure TypeScript/DI addition. + +## Next Phase Readiness +- `SourceRegistry` is ready to be wired into the Nest module (Plan 13-03 Task 3: register `DoeOpenDataAdapter` at boot) and consumed by `pollDueSources`'s fan-out (`activeAdapters()`/`get()`), replacing the current hardwired single-adapter injection. +- The `portals[]` contract is the exact shape Plan 13-04's config-driven `NetServerAdapter` (3 portals) and Plan 13-05's `CosinexAdapter` will implement — no further interface changes needed. +- INGEST-07 is now structurally satisfied at the registry layer; Plan 13-04/13-05 adapters will never be able to accidentally register `vergabe24`/`aumass` since the gate lives in `register()`, not in adapter-author discipline. +- No blockers. + +--- +*Phase: 13-scraping-adapters-cross-source-dedup* +*Completed: 2026-07-23* + +## Self-Check: PASSED + +All created/modified files verified present on disk; all 3 task commit hashes verified in git log.