docs(13): create phase plan — scraping adapters + cross-source dedup
6 plans (INGEST-02/03/07, SCHEMA-03) + Nyquist validation. Core (registry, fingerprint, dedup, TenderSource, backfill) lands first and is green independent of live scraping (D-01). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,104 @@
|
||||
---
|
||||
phase: 13
|
||||
slug: scraping-adapters-cross-source-dedup
|
||||
status: draft
|
||||
nyquist_compliant: true
|
||||
wave_0_complete: false
|
||||
created: 2026-07-23
|
||||
---
|
||||
|
||||
# Phase 13 — Validation Strategy
|
||||
|
||||
> Per-phase validation contract for feedback sampling during execution.
|
||||
> Extracted from 13-RESEARCH.md § Validation Architecture (Test framework, Requirements→Test map, Sampling rate, Wave-0 gaps).
|
||||
|
||||
---
|
||||
|
||||
## Test Infrastructure
|
||||
|
||||
| Property | Value |
|
||||
|----------|-------|
|
||||
| **Framework** | Vitest (API 3.x, Web 4.x — beide vorhanden, kein Install) |
|
||||
| **Config file** | `apps/api/vitest.config.ts` + `apps/web` vitest (bestehend — nicht von dieser Phase geaendert) |
|
||||
| **Quick run (API slice)** | `pnpm --filter @tessera/api test -- <spec>` |
|
||||
| **Quick run (API tenders)** | `pnpm --filter @tessera/api test -- src/tenders` |
|
||||
| **Quick run (Web slice)** | `pnpm --filter web test -- <spec>` |
|
||||
| **Full suite** | `pnpm test` (turbo) bzw. `pnpm --filter @tessera/api test && pnpm --filter web test` |
|
||||
| **Estimated runtime** | ~10-15 s (scoped) |
|
||||
|
||||
Ein neues Package: `node-html-parser` (oder `cheerio`) — NUR nach blockierendem Package-Legitimacy-Checkpoint (Plan 13-04). Der Kern (Registry, Fingerprint, Dedup, TenderSource, Backfill) braucht KEIN Scraping-Package und ist unabhaengig davon voll testbar (D-01). `prisma`/`crypto`/`fetch` bereits vorhanden.
|
||||
|
||||
---
|
||||
|
||||
## Sampling Rate
|
||||
|
||||
- **Nach jedem Task-Commit:** betroffene Slice-Spec (`pnpm --filter @tessera/api test -- <spec>`).
|
||||
- **Nach jeder Plan-Wave:** `pnpm --filter @tessera/api test && pnpm --filter web test`.
|
||||
- **Vor `/gsd-verify-work`:** `pnpm test` gruen + Backfill-Migration lokal gegen die Dev-DB verifiziert.
|
||||
- **Max Feedback-Latenz:** ~15 s (scoped).
|
||||
|
||||
---
|
||||
|
||||
## Per-Task Verification Map
|
||||
|
||||
| Task ID | Plan | Wave | Requirement | Threat Ref | Secure Behavior | Test Type | Automated Command | File Exists | Status |
|
||||
|---------|------|------|-------------|------------|-----------------|-----------|-------------------|-------------|--------|
|
||||
| 13-01-01 | 01 | 1 | SCHEMA-03 | T-13-01-01/03 | TenderSource (1:n, @@unique[sourcePortal,sourceNoticeId]) + Tender.fingerprint additiv; 2-Schritt-Migration + Backfill lokal | integration | `cd apps/api && npx prisma validate && npx prisma generate` (+ psql-Count, manual) | ⚠️ migration | ⬜ pending |
|
||||
| 13-01-02 | 01 | 1 | SCHEMA-03 | — | tenderFingerprint pure, NULL-tolerant (Wert/Frist NULL -> Match), kollisionsarm (Titel im Hash), sha256 | unit | `pnpm --filter @tessera/api test -- tender-fingerprint` | ❌ W0 | ⬜ pending |
|
||||
| 13-01-03 | 01 | 1 | SCHEMA-03 | — | SourceType-Union offen + NormalizedTenderFields.fingerprint; typecheck | typecheck | `cd apps/api && npx tsc --noEmit -p tsconfig.json` | n/a | ⬜ pending |
|
||||
| 13-02-01 | 02 | 1 | INGEST-07 | — | Adapter-Interface um portals[] generalisiert; DoeOpenDataAdapter deklariert portals | typecheck | `cd apps/api && npx tsc --noEmit -p tsconfig.json` | n/a | ⬜ pending |
|
||||
| 13-02-02 | 02 | 1 | INGEST-07 | T-13-02-01/02 | SourceRegistry.register wirft DeniedPortalError fuer vergabe24 UND aumass (Erfolgskriterium 4); legitime Registrierung/get funktionieren | unit | `pnpm --filter @tessera/api test -- source-registry` | ❌ W0 | ⬜ pending |
|
||||
| 13-03-01 | 03 | 2 | SCHEMA-03 | T-13-03-02/03/04 | 3-Stufen-Resolver OCID→source:noticeId→fingerprint; inert bei 1 Quelle (D-05), Merge->TenderSource anhaengen ab 2 (D-03); plain Prisma (kein RLS) | unit | `pnpm --filter @tessera/api test -- tender-dedup` | ❌ W0 | ⬜ pending |
|
||||
| 13-03-02 | 03 | 2 | SCHEMA-03 | T-13-03-01/02 | pollDueSources fan-out (findMany isActive), catch-per-source, dedupActive=activePortalCount>=2, Delta-Boundary erhalten | unit | `pnpm --filter @tessera/api test -- tender-ingestion.service` | ⚠️ extend | ⬜ pending |
|
||||
| 13-03-03 | 03 | 2 | SCHEMA-03 | — | Modul-Wiring: SourceRegistry + TenderDedupService Provider + DoeOpenDataAdapter bei Boot registriert; DI aufloesbar | typecheck+suite | `cd apps/api && npx tsc --noEmit && pnpm --filter @tessera/api test -- src/tenders` | n/a | ⬜ pending |
|
||||
| 13-06-01 | 06 | 2 | SCHEMA-03 | T-13-06-03 | getTender include sources[]; Route-Order unveraendert (:id zuletzt) | integration | `pnpm --filter @tessera/api test -- tenders.controller` | ⚠️ extend | ⬜ pending |
|
||||
| 13-06-02 | 06 | 2 | SCHEMA-03 | T-13-06-01/02 | TenderDetail rendert alle Quell-Links (Text-escaped, rel=noopener), Fallback bei fehlendem sources | component | `cd apps/web && npx tsc --noEmit && pnpm --filter web test -- TenderDetail` | ⚠️ extend | ⬜ pending |
|
||||
| 13-04-CP | 04 | 3 | INGEST-02 | T-13-04-SC | Package-Legitimacy-Checkpoint (blocking-human) VOR Install node-html-parser/cheerio | manual-gate | Human verify via `gsd-tools query package-legitimacy check --ecosystem npm node-html-parser cheerio` | n/a | ⬜ pending |
|
||||
| 13-04-01 | 04 | 3 | INGEST-02 | T-13-04-01/02/03 | NetServer-Adapter config-getrieben (3 Portale), parst <table>-Fixture, []-Fallback, SSRF-sichere Konstanten, 15s-Timeout | unit (Fixture) | `pnpm --filter @tessera/api test -- netserver.adapter` | ❌ W0 | ⬜ pending |
|
||||
| 13-04-02 | 04 | 3 | INGEST-02 | — | NetServerAdapter Provider + bei Boot registriert (nicht denylisted); DI aufloesbar | typecheck+suite | `cd apps/api && npx tsc --noEmit && pnpm --filter @tessera/api test -- src/tenders` | n/a | ⬜ pending |
|
||||
| 13-05-01 | 05 | 4 | INGEST-03 | T-13-05-01/02/03 | cosinex-Adapter getrennt (nicht NetServer), parst Satellite-Fixture ODER dokumentiert needs-JS -> []; []-Fallback, SSRF-sicher | unit (Fixture) | `pnpm --filter @tessera/api test -- cosinex.adapter` | ❌ W0 | ⬜ pending |
|
||||
| 13-05-02 | 05 | 4 | INGEST-03 | — | CosinexAdapter Provider + bei Boot registriert; DI aufloesbar | typecheck+suite | `cd apps/api && npx tsc --noEmit && pnpm --filter @tessera/api test -- src/tenders` | n/a | ⬜ pending |
|
||||
|
||||
*Status: ⬜ pending · ✅ green · ❌ red · ⚠️ flaky*
|
||||
|
||||
---
|
||||
|
||||
## Wave 0 Requirements
|
||||
|
||||
- [ ] `apps/api/src/tenders/tender-fingerprint.spec.ts` — SCHEMA-03 NULL-Toleranz (Wert+Frist NULL -> gleicher Fingerprint), Kollisions-Fall (anderer Titel -> anderer Hash), Umlaut-/CPV-Division-Normalisierung, valueBucket, deterministischer sha256 (Plan 13-01 Task 2, RED-first).
|
||||
- [ ] `apps/api/src/tenders/source-registry.spec.ts` — INGEST-07/Erfolgskriterium 4: register() wirft DeniedPortalError fuer vergabe24 UND aumass; gemischtes portals-Array abgelehnt; legitime Registrierung + get/activeAdapters (Plan 13-02 Task 2, RED-first).
|
||||
- [ ] `apps/api/src/tenders/tender-dedup.service.spec.ts` — D-04/D-05: inert bei dedupActive=false (2 Tender), OCID-/noticeId-/Fingerprint-Match haengt TenderSource an (created=false), kein Match -> create (created=true); Prisma gemockt (Plan 13-03 Task 1, RED-first).
|
||||
- [ ] `apps/api/src/tenders/adapters/netserver.adapter.spec.ts` + `__fixtures__/netserver-search.html` — INGEST-02: parst live-gecapturte <table>-Fixture -> RawTenderRecord[] mit korrektem sourcePortal je Zeile; []-Fallback bei kaputtem HTML; Fetch gemockt (Plan 13-04 Task 1).
|
||||
- [ ] `apps/api/src/tenders/adapters/cosinex.adapter.spec.ts` + `__fixtures__/cosinex-search.html` — INGEST-03: parst Satellite-Fixture ODER dokumentierter needs-JS-Leer-Fall; []-Fallback; Fetch gemockt (Plan 13-05 Task 1).
|
||||
- [ ] (extend) `apps/api/src/tenders/tender-ingestion.service.spec.ts` — fan-out ueber mehrere aktive Configs, catch-per-source (Quelle A wirft -> B laeuft), dedupActive-Gate, matchDelta nur mit created-IDs (Plan 13-03 Task 2).
|
||||
- [ ] (extend) `apps/api/src/tenders/tenders.controller.spec.ts` — getTender liefert sources[] mit; NotFound bleibt (Plan 13-06 Task 1).
|
||||
- [ ] (extend) `apps/web/.../components/TenderDetail.test.tsx` — mehrere Quell-Links bei sources[]; Fallback bei fehlendem sources (Plan 13-06 Task 2).
|
||||
|
||||
Vitest-Framework existiert bereits (api + web) — keine Framework-Installation. HTML-Fixtures live speichern fuer stabile Adapter-Tests.
|
||||
|
||||
---
|
||||
|
||||
## Manual-Only Verifications
|
||||
|
||||
| Behavior | Requirement | Why Manual | Test Instructions |
|
||||
|----------|-------------|------------|-------------------|
|
||||
| Migration `20260723120000_add_tender_source` + Backfill lokal angewendet | SCHEMA-03 | DB hat keinen Host-Port; lokale Anwendung via `docker compose exec -T db psql -U tessera -d tessera_dev` (MEMORY); kein Docker-Deploy Testserver durch Claude | `SELECT to_regclass('public."TenderSource"')` ≠ NULL; `SELECT count(*) FROM "TenderSource"` == `SELECT count(*) FROM "Tender"` (~2851); `SELECT count(*) FROM "Tender" WHERE fingerprint IS NOT NULL` == Bestand; `prisma migrate status` gruen |
|
||||
| Live-Poll NetServer (tender24) liefert reale Records | INGEST-02 | Live-Portal-HTML nicht im Unit-Test beobachtbar (Fixture-getestet); Selektor-Vollstaendigkeit nur live pruefbar | Optionaler Aktivierungs-/Poll-Lauf gegen `tender24.de/NetServer/PublicationSearchControllerServlet` -> >0 RawTenderRecord ODER "needs-JS/blocked, deferred" im SUMMARY dokumentiert (D-01) |
|
||||
| Live-Fetch cosinex (dtvp welcome.do) parsebar | INGEST-03 | JS-Rendering-Grad nur live feststellbar (Open Question 1); kein Headless-Browser im Stack | Server-gerenderte Trefferliste parsebar -> Records; sonst "needs-JS, deferred" dokumentiert, Adapter-Skelett + Test bleiben gruen |
|
||||
| Cross-Source-Dedup end-to-end (EIN Eintrag, alle Links) | SCHEMA-03 / Erfolgskriterium 3 | Echter Merge braucht 2 aktive Quellen mit ueberlappender Ausschreibung; Unit deckt Logik, finale Bestaetigung am realen Lauf | DÖE + NetServer aktiv, eine Ausschreibung in beiden -> erscheint EINMAL in der Trefferliste; Detailansicht listet beide Quell-Links; Tender-Anzahl steigt nicht doppelt |
|
||||
| Dedup-Inertheit mit nur DÖE (kein faelschlicher Merge) | SCHEMA-03 / Erfolgskriterium 3 / D-05 | Beobachtung am realen Bestand (~2851 Tender) | Nur DÖE aktiv -> Tender-Anzahl bleibt stabil nach Poll (kein Fingerprint-Merge, activePortalCount < 2) |
|
||||
| Denylist-Refusal strukturell | INGEST-07 / Erfolgskriterium 4 | Automatisiert durch 13-02-02; finale Bestaetigung, dass kein Boot-Pfad vergabe24/aumass registriert | Kein Poll-/Registrierungs-Pfad fuer vergabe24/aumass existiert; register-Versuch wirft (Test gruen) |
|
||||
|
||||
---
|
||||
|
||||
## Validation Sign-Off
|
||||
|
||||
- [x] Alle Tasks haben `<automated>` verify oder eine dokumentierte Wave-0-/Manual-Gate-Abhaengigkeit
|
||||
- [x] Sampling-Kontinuitaet: keine 3 aufeinanderfolgenden Tasks ohne automated verify
|
||||
- [x] Wave 0 deckt alle MISSING-Referenzen (Spec-Scaffolds; RED-first fuer pure-logic Units)
|
||||
- [x] Keine watch-mode-Flags (alle nutzen `vitest run` via `pnpm --filter … test`)
|
||||
- [x] Feedback-Latenz < 15 s (scoped runs)
|
||||
- [x] `nyquist_compliant: true` in Frontmatter gesetzt
|
||||
- [x] Package-Legitimacy-Gate (node-html-parser/cheerio) als blocking-human Checkpoint VOR Install (13-04)
|
||||
|
||||
**Approval:** draft — pending execution (wave_0_complete: false)
|
||||
Reference in New Issue
Block a user