docs(13): create phase plan — scraping adapters + cross-source dedup
6 plans (INGEST-02/03/07, SCHEMA-03) + Nyquist validation. Core (registry, fingerprint, dedup, TenderSource, backfill) lands first and is green independent of live scraping (D-01). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -439,7 +439,15 @@ Plans:
|
|||||||
3. A tender that appears via both DÖE and a scraping adapter shows up once in the results list (fuzzy fingerprint dedup on buyer+title+CPV+deadline+value), with links to all of its source portals -- dedup logic only activates once a second source is live
|
3. A tender that appears via both DÖE and a scraping adapter shows up once in the results list (fuzzy fingerprint dedup on buyer+title+CPV+deadline+value), with links to all of its source portals -- dedup logic only activates once a second source is live
|
||||||
4. Attempting to register vergabe24 or aumass as a poll source is refused by the system itself (adapter registry denylist enforced in code), not just documented as forbidden
|
4. Attempting to register vergabe24 or aumass as a poll source is refused by the system itself (adapter registry denylist enforced in code), not just documented as forbidden
|
||||||
|
|
||||||
**Plans**: TBD
|
**Plans**: 6 plans
|
||||||
|
|
||||||
|
Plans:
|
||||||
|
- [ ] 13-01-PLAN.md — TenderSource-Schema + Backfill + NULL-tolerante Fingerprint-Fn + SourceType-Union (SCHEMA-03 Datenschicht)
|
||||||
|
- [ ] 13-02-PLAN.md — SourceRegistry + harte Denylist-Gate (vergabe24/aumass) + Adapter-Interface-Generalisierung (INGEST-07)
|
||||||
|
- [ ] 13-03-PLAN.md — 3-Stufen-Dedup-Resolver + pollDueSources Fan-out (catch-per-source) + Modul-Wiring (SCHEMA-03)
|
||||||
|
- [ ] 13-04-PLAN.md — NetServer-Adapter (config-getrieben, 3 Portale) + Package-Legitimacy-Checkpoint (INGEST-02)
|
||||||
|
- [ ] 13-05-PLAN.md — cosinex/DTVP-Adapter (separat, best-effort) (INGEST-03)
|
||||||
|
- [ ] 13-06-PLAN.md — Read-Endpoint include sources[] + TenderDetail Multi-Source-Links (SCHEMA-03 Anzeige)
|
||||||
|
|
||||||
### Phase 14: RSS, Email-Alert Ingestion & Module Rollout
|
### Phase 14: RSS, Email-Alert Ingestion & Module Rollout
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,134 @@
|
|||||||
|
---
|
||||||
|
phase: 13-scraping-adapters-cross-source-dedup
|
||||||
|
plan: 01
|
||||||
|
type: execute
|
||||||
|
wave: 1
|
||||||
|
depends_on: []
|
||||||
|
files_modified:
|
||||||
|
- apps/api/prisma/schema.prisma
|
||||||
|
- apps/api/prisma/migrations
|
||||||
|
- apps/api/src/tenders/tender.types.ts
|
||||||
|
- apps/api/src/tenders/tender-fingerprint.ts
|
||||||
|
- apps/api/src/tenders/tender-fingerprint.spec.ts
|
||||||
|
autonomous: true
|
||||||
|
requirements: [SCHEMA-03]
|
||||||
|
must_haves:
|
||||||
|
truths:
|
||||||
|
- "Jeder der ~2851 Bestands-DÖE-Tender besitzt genau eine TenderSource-Zeile aus seinen sourcePortal/sourceNoticeId/sourceUrl/ocid."
|
||||||
|
- "Zwei Records mit gleichem Titel+Auftraggeber+CPV-Division ergeben denselben Fingerprint, auch wenn Wert und Frist NULL sind."
|
||||||
|
- "SourceType ist eine offene Union ('doe-opendata' | 'ai-netserver' | 'cosinex-dtvp')."
|
||||||
|
artifacts:
|
||||||
|
- apps/api/src/tenders/tender-fingerprint.ts
|
||||||
|
- apps/api/prisma/schema.prisma (model TenderSource, Tender.fingerprint, Tender.sources)
|
||||||
|
key_links:
|
||||||
|
- "TenderSource.@@unique([sourcePortal, sourceNoticeId]) — eine Quell-Notiz gehoert zu genau einem Tender."
|
||||||
|
- "Tender.fingerprint (nullable, @@index) — SCHEMA-03 Fuzzy-Dedup-Lookup."
|
||||||
|
---
|
||||||
|
|
||||||
|
<objective>
|
||||||
|
Legt die SCHEMA-03-Datengrundlage: neues 1:n-Modell `TenderSource` (D-03), eine additive nullable `fingerprint`-Spalte auf `Tender`, Backfill der ~2851 Bestands-DÖE-Tender (je eine TenderSource-Zeile + berechneter Fingerprint), und die pure NULL-tolerante Fingerprint-Funktion (SCHEMA-03, D-04). Erweitert `SourceType` zur offenen Union, damit die spaeteren Adapter (13-04/05) und die Registry (13-02) darauf aufbauen.
|
||||||
|
|
||||||
|
Purpose: Der Kern muss ohne jede Scraping-Abhaengigkeit voll getestet gruen sein (D-01). Dieser Plan ist genau dieser dependency-freie Datenkern.
|
||||||
|
Output: schema.prisma (TenderSource + fingerprint), lokal angewendete Migration + Backfill, tender-fingerprint.ts (+ spec), erweiterte SourceType-Union.
|
||||||
|
</objective>
|
||||||
|
|
||||||
|
<execution_context>
|
||||||
|
@$HOME/.claude/gsd-core/workflows/execute-plan.md
|
||||||
|
@$HOME/.claude/gsd-core/templates/summary.md
|
||||||
|
</execution_context>
|
||||||
|
|
||||||
|
<context>
|
||||||
|
@.planning/PROJECT.md
|
||||||
|
@.planning/ROADMAP.md
|
||||||
|
@.planning/STATE.md
|
||||||
|
@.planning/phases/13-scraping-adapters-cross-source-dedup/13-CONTEXT.md
|
||||||
|
@.planning/phases/13-scraping-adapters-cross-source-dedup/13-RESEARCH.md
|
||||||
|
@apps/api/prisma/schema.prisma
|
||||||
|
@apps/api/src/tenders/tender.types.ts
|
||||||
|
@apps/api/src/tenders/tender-normalizer.service.ts
|
||||||
|
</context>
|
||||||
|
|
||||||
|
<tasks>
|
||||||
|
|
||||||
|
<task type="auto">
|
||||||
|
<name>Task 1: TenderSource-Modell + fingerprint-Spalte + Migration + Backfill</name>
|
||||||
|
<files>apps/api/prisma/schema.prisma, apps/api/prisma/migrations/, apps/api/src/tenders/backfill-tender-source.ts</files>
|
||||||
|
<action>
|
||||||
|
Ergaenze in `schema.prisma` (per SCHEMA-03 / D-03, Referenz-Snippet in 13-RESEARCH.md "TenderSource-Modell (Prisma)"): neues `model TenderSource` mit id (uuid), tenderId, sourcePortal, sourceNoticeId, ocid (nullable), sourceUrl (nullable), createdAt, Relation `tender Tender @relation(fields:[tenderId], references:[id], onDelete: Cascade)`, `@@unique([sourcePortal, sourceNoticeId])` und `@@index([tenderId])`. Ergaenze in `model Tender`: `fingerprint String?` (nullable, additiv), `sources TenderSource[]`, `@@index([fingerprint])`. `dedupKey @unique` bleibt UNVERAENDERT (SCHEMA-02-Upsert-Target, D-01 — kein Drop in dieser Phase).
|
||||||
|
|
||||||
|
Erzeuge eine handgeschriebene Migration (Verzeichnis `apps/api/prisma/migrations/20260723120000_add_tender_source/migration.sql`) in ZWEI logischen Schritten in korrekter Reihenfolge (Pitfall 5): (a) CREATE TABLE "TenderSource" + ADD COLUMN "fingerprint" auf "Tender" + Indizes + Unique-Constraint; (b) Daten-Backfill `INSERT INTO "TenderSource" (...) SELECT gen_random_uuid(), t.id, t."sourcePortal", t."sourceNoticeId", t.ocid, t."sourceUrl", now() FROM "Tender" t;` (SQL-Skizze in 13-RESEARCH.md "Backfill-Migration"). Der Unique-Constraint wird NACH dem Backfill-Insert wirksam bzw. das Insert erzeugt keine Duplikate (DÖE-noticeIds sind live eindeutig).
|
||||||
|
|
||||||
|
Der `fingerprint`-Backfill der Bestands-Tender lebt NICHT in reinem SQL (Umlaut-/CPV-Normalisierung lebt im Code): schreibe ein einmaliges TS-Backfill-Script `backfill-tender-source.ts`, das alle Tender laedt, `tenderFingerprint(...)` (Task 2) aus title/buyerName/cpvDivisions/deadlineAt/estimatedValue berechnet und `Tender.fingerprint` per updateMany/Schleife setzt. estimatedValue ist `Decimal?` — in number/null konvertieren, bevor es an valueBucket geht.
|
||||||
|
|
||||||
|
Wende Migration lokal an (KEIN Docker-Deploy Testserver, MEMORY): `docker compose exec -T db psql -U tessera -d tessera_dev -f -` bzw. `pnpm --filter @tessera/api exec prisma migrate deploy` gegen die lokale DB; danach `pnpm --filter @tessera/api exec prisma generate`. DB-Name lokal verifizieren (tessera_dev laut Phase-12-Vorbild; Research nennt `tessera`) — vor dem Anwenden `\l` pruefen.
|
||||||
|
</action>
|
||||||
|
<verify>
|
||||||
|
<automated>cd apps/api && npx prisma validate && npx prisma generate</automated>
|
||||||
|
</verify>
|
||||||
|
<done>`model TenderSource` + `Tender.fingerprint` + `Tender.sources` existieren; `prisma validate` gruen; Migration lokal angewendet; `SELECT count(*) FROM "TenderSource"` == count der Tender (manuelle DB-Pruefung, siehe 13-VALIDATION.md).</done>
|
||||||
|
</task>
|
||||||
|
|
||||||
|
<task type="auto" tdd="true">
|
||||||
|
<name>Task 2: Pure NULL-tolerante Fingerprint-Funktion (SCHEMA-03, D-04)</name>
|
||||||
|
<files>apps/api/src/tenders/tender-fingerprint.ts, apps/api/src/tenders/tender-fingerprint.spec.ts</files>
|
||||||
|
<behavior>
|
||||||
|
- Gleicher Titel + Auftraggeber + CPV-Division, beide Wert=NULL und Frist=NULL -> gleicher Fingerprint (NULL-Toleranz, 92.4% NULL Wert / 15.7% NULL Frist live).
|
||||||
|
- Unterschiedlicher Titel (nach normText) -> unterschiedlicher Fingerprint (Kollisions-Schutz, Pitfall 1).
|
||||||
|
- Umlaut-Normalisierung: "Muenchen" und "München" ergeben denselben normText-Beitrag.
|
||||||
|
- CPV-Divisions reihenfolge-unabhaengig (sortiert), dedupliziert.
|
||||||
|
- valueBucket: NULL -> "", sonst Groessenordnung (log10-Floor) — 12000 und 15000 landen im selben Bucket.
|
||||||
|
- Rueckgabe ist stabiler sha256-Hex-String (deterministisch ueber Laeufe).
|
||||||
|
</behavior>
|
||||||
|
<action>
|
||||||
|
Implementiere `tender-fingerprint.ts` als pure Funktion `tenderFingerprint(f: { buyerName: string|null; title: string; cpvDivisions: string[]; deadlineAt: Date|null; estimatedValue: number|null }): string` exakt nach 13-RESEARCH.md Pattern 4: interne Helfer `normText` (lowercase, ae/oe/ue/ss, non-alnum->space, trim), `cpvDivisionKey` (Set+sort+join), `valueBucket` (NULL->"", sonst `String(Math.floor(Math.log10(Math.max(v,1))))`), `deadlineKey` (Datum auf Tageskorn oder ""). Kanonischer String `[buyer, title, cpvDivKey, deadlineKey, valueBucket].join('|')` -> `createHash('sha256').update(...).digest('hex')` (Node crypto, kein Selbstbau, V6). KEIN Aehnlichkeits-Threshold (deterministisch, O(1)-Lookup). Schreibe die Tests RED-first gemaess behavior-Block, dann Implementierung bis gruen.
|
||||||
|
</action>
|
||||||
|
<verify>
|
||||||
|
<automated>pnpm --filter @tessera/api test -- tender-fingerprint</automated>
|
||||||
|
</verify>
|
||||||
|
<done>Alle behavior-Faelle gruen; Funktion ist pure (kein Prisma/IO-Import); NULL-Toleranz + Kollisions-Fall abgedeckt.</done>
|
||||||
|
</task>
|
||||||
|
|
||||||
|
<task type="auto">
|
||||||
|
<name>Task 3: SourceType-Union erweitern + NormalizedTenderFields.fingerprint</name>
|
||||||
|
<files>apps/api/src/tenders/tender.types.ts</files>
|
||||||
|
<action>
|
||||||
|
Erweitere in `tender.types.ts` `SourceType` von `'doe-opendata'` zur offenen Union `'doe-opendata' | 'ai-netserver' | 'cosinex-dtvp'` (13-RESEARCH.md Pattern 1). Ergaenze `NormalizedTenderFields` um `fingerprint: string` (der in 13-03 vom Dedup-Resolver gesetzte SCHEMA-03-Schluessel) — Feld additiv, damit der Normalizer/Resolver ihn fuellen kann. Passe den Normalizer NICHT hier an (das macht 13-03), aber stelle sicher, dass der Typ-Zusatz `npx tsc --noEmit` nicht bricht (Feld ggf. vorerst optional `fingerprint?: string` falls der Normalizer es noch nicht setzt — dokumentiere die Wahl im Kommentar). Kommentar aktualisieren: Union-Erweiterung ist die einzige breaking-Aenderung, lokal begrenzt.
|
||||||
|
</action>
|
||||||
|
<verify>
|
||||||
|
<automated>cd apps/api && npx tsc --noEmit -p tsconfig.json</automated>
|
||||||
|
</verify>
|
||||||
|
<done>SourceType-Union enthaelt die drei Slugs; NormalizedTenderFields kennt fingerprint; `tsc --noEmit` gruen im API-Package.</done>
|
||||||
|
</task>
|
||||||
|
|
||||||
|
</tasks>
|
||||||
|
|
||||||
|
<threat_model>
|
||||||
|
## Trust Boundaries
|
||||||
|
|
||||||
|
| Boundary | Description |
|
||||||
|
|----------|-------------|
|
||||||
|
| Migration -> DB | Schema-/Datenaenderung an Produktions-naher lokaler DB; Reihenfolge-Fehler kann Backfill brechen. |
|
||||||
|
|
||||||
|
## STRIDE Threat Register
|
||||||
|
|
||||||
|
| Threat ID | Category | Component | Severity | Disposition | Mitigation Plan |
|
||||||
|
|-----------|----------|-----------|----------|-------------|-----------------|
|
||||||
|
| T-13-01-01 | Tampering | Backfill-Migration | medium | mitigate | 2-Schritt-Reihenfolge (Tabelle+Spalten vor Constraint/Backfill, Pitfall 5); lokal angewendet, kein Testserver-Deploy. |
|
||||||
|
| T-13-01-02 | Denial of Service | fingerprint-Backfill ueber 2851 Rows | low | accept | Einmaliges Script, Batch/Schleife; kein Laufzeit-Pfad. |
|
||||||
|
| T-13-01-03 | Info Disclosure | Decimal->number Konvertierung estimatedValue | low | mitigate | NULL bleibt NULL (valueBucket NULL-tolerant); keine Praezisionsannahme im Hash (nur Groessenordnung). |
|
||||||
|
</threat_model>
|
||||||
|
|
||||||
|
<verification>
|
||||||
|
- `prisma validate` + `prisma generate` gruen.
|
||||||
|
- `pnpm --filter @tessera/api test -- tender-fingerprint` gruen.
|
||||||
|
- `npx tsc --noEmit` gruen.
|
||||||
|
- Manuell (13-VALIDATION.md): `SELECT count(*) FROM "TenderSource"` == `SELECT count(*) FROM "Tender"`; `SELECT count(*) FROM "Tender" WHERE fingerprint IS NOT NULL` == Bestand.
|
||||||
|
</verification>
|
||||||
|
|
||||||
|
<success_criteria>
|
||||||
|
TenderSource-Tabelle existiert und ist 1:1 mit Bestands-Tendern backfilled; fingerprint-Spalte existiert + backfilled; pure Fingerprint-Fn NULL-tolerant + kollisionsarm getestet; SourceType-Union offen. TrAegt SCHEMA-03 (Datenschicht).
|
||||||
|
</success_criteria>
|
||||||
|
|
||||||
|
<output>
|
||||||
|
Create `.planning/phases/13-scraping-adapters-cross-source-dedup/13-01-SUMMARY.md` when done
|
||||||
|
</output>
|
||||||
@@ -0,0 +1,110 @@
|
|||||||
|
---
|
||||||
|
phase: 13-scraping-adapters-cross-source-dedup
|
||||||
|
plan: 02
|
||||||
|
type: execute
|
||||||
|
wave: 1
|
||||||
|
depends_on: []
|
||||||
|
files_modified:
|
||||||
|
- apps/api/src/tenders/source-registry.ts
|
||||||
|
- apps/api/src/tenders/source-registry.spec.ts
|
||||||
|
- apps/api/src/tenders/adapters/tender-source-adapter.interface.ts
|
||||||
|
autonomous: true
|
||||||
|
requirements: [INGEST-07]
|
||||||
|
must_haves:
|
||||||
|
truths:
|
||||||
|
- "Der Versuch, einen Adapter mit portal 'vergabe24' oder 'aumass' zu registrieren, wirft eine DeniedPortalError-Exception im Code (nicht nur Doku)."
|
||||||
|
- "Ein Test beweist die Ablehnung (Erfolgskriterium 4)."
|
||||||
|
- "Ein Adapter deklariert seine bedienten Portale ueber ein readonly portals-Array (NetServer: 3)."
|
||||||
|
artifacts:
|
||||||
|
- apps/api/src/tenders/source-registry.ts
|
||||||
|
- apps/api/src/tenders/adapters/tender-source-adapter.interface.ts
|
||||||
|
key_links:
|
||||||
|
- "SourceRegistry.register() -> assertNotDenied ueber alle adapter.portals bei DI-Boot."
|
||||||
|
---
|
||||||
|
|
||||||
|
<objective>
|
||||||
|
Baut das INGEST-07-Rueckgrat: eine `SourceRegistry` mit hartem Denylist-Gate, die vergabe24/aumass strukturell im Code ablehnt (D-06 / Erfolgskriterium 4), plus die Generalisierung des Adapter-Vertrags (`portals`-Array), damit EIN NetServer-Adapter mehrere Portale bedienen kann. Voll getestet, ohne jede Scraping-Abhaengigkeit (D-01).
|
||||||
|
|
||||||
|
Purpose: Compliance-/Sicherheitsgrenze gehoert testbar in den DI-Layer, bevor irgendein Poll laeuft.
|
||||||
|
Output: source-registry.ts (Registry + DeniedPortalError), source-registry.spec.ts (Refusal-Beweis), generalisiertes TenderSourceAdapter-Interface.
|
||||||
|
</objective>
|
||||||
|
|
||||||
|
<execution_context>
|
||||||
|
@$HOME/.claude/gsd-core/workflows/execute-plan.md
|
||||||
|
@$HOME/.claude/gsd-core/templates/summary.md
|
||||||
|
</execution_context>
|
||||||
|
|
||||||
|
<context>
|
||||||
|
@.planning/PROJECT.md
|
||||||
|
@.planning/ROADMAP.md
|
||||||
|
@.planning/STATE.md
|
||||||
|
@.planning/phases/13-scraping-adapters-cross-source-dedup/13-CONTEXT.md
|
||||||
|
@.planning/phases/13-scraping-adapters-cross-source-dedup/13-RESEARCH.md
|
||||||
|
@apps/api/src/tenders/adapters/tender-source-adapter.interface.ts
|
||||||
|
@apps/api/src/tenders/adapters/doe-opendata.adapter.ts
|
||||||
|
</context>
|
||||||
|
|
||||||
|
<tasks>
|
||||||
|
|
||||||
|
<task type="auto">
|
||||||
|
<name>Task 1: Adapter-Interface generalisieren (portals-Array)</name>
|
||||||
|
<files>apps/api/src/tenders/adapters/tender-source-adapter.interface.ts</files>
|
||||||
|
<action>
|
||||||
|
Erweitere `TenderSourceAdapter` (13-RESEARCH.md Pattern 1) um `readonly portals: readonly string[]` — die Liste der von diesem Adapter bedienten Quell-Portale (DÖE: `['doe-opendata']`; NetServer spaeter: `['tender24','lhs-vpbw','vergabe.landbw']`). `sourceType` und `fetchTenders(dayCursor)` bleiben unveraendert. Aktualisiere den bestehenden `DoeOpenDataAdapter`, sodass er `readonly portals = ['doe-opendata'] as const` deklariert (minimaler additiver Edit, keine Verhaltensaenderung). Kommentar: `portals` entkoppelt "ein Adapter" von "ein Portal" (config-driven multi-portal, INGEST-02).
|
||||||
|
</action>
|
||||||
|
<verify>
|
||||||
|
<automated>cd apps/api && npx tsc --noEmit -p tsconfig.json</automated>
|
||||||
|
</verify>
|
||||||
|
<done>Interface hat `portals`; DoeOpenDataAdapter deklariert es; `tsc --noEmit` gruen; bestehende doe-Adapter-Spec bleibt gruen.</done>
|
||||||
|
</task>
|
||||||
|
|
||||||
|
<task type="auto" tdd="true">
|
||||||
|
<name>Task 2: SourceRegistry + Denylist-Gate + Refusal-Test (INGEST-07 / D-06)</name>
|
||||||
|
<files>apps/api/src/tenders/source-registry.ts, apps/api/src/tenders/source-registry.spec.ts</files>
|
||||||
|
<behavior>
|
||||||
|
- register() eines Fake-Adapters mit portals ['vergabe24'] wirft DeniedPortalError.
|
||||||
|
- register() mit portals ['aumass'] wirft DeniedPortalError.
|
||||||
|
- register() eines legitimen Adapters (portals ['tender24']) fuegt ihn hinzu; get(sourceType) liefert ihn zurueck.
|
||||||
|
- activeAdapters() liefert alle registrierten Adapter.
|
||||||
|
- get() eines nicht registrierten sourceType liefert undefined (kein Throw).
|
||||||
|
- Ein gemischter Adapter (portals ['tender24','aumass']) wird komplett abgelehnt (ein verbotenes Portal genuegt).
|
||||||
|
</behavior>
|
||||||
|
<action>
|
||||||
|
Implementiere `source-registry.ts` nach 13-RESEARCH.md Pattern 2: exportiere `DENYLISTED_PORTALS = ['vergabe24','aumass'] as const`, Klasse `DeniedPortalError extends Error` (deutsche Meldung: AGB-seitig gesperrt), und `@Injectable() class SourceRegistry` mit `private adapters = new Map<SourceType, TenderSourceAdapter>()`, `register(adapter)` (iteriert `adapter.portals`, wirft DeniedPortalError bei Treffer in DENYLISTED_PORTALS, sonst set), `get(type)` und `activeAdapters()`. Der Denylist-Vergleich ist der Sicherheitskern (D-06) — harte Code-Exception, NIE nur Doku. Schreibe die Spec RED-first exakt nach behavior-Block; sie ist der Beweis fuer Erfolgskriterium 4. Verwende einen Fake-Adapter-Stub im Test (kein echtes Scraping). Beachte die comment-text-Disziplin: kein negativer grep-Gate auf 'vergabe24'/'aumass' im Datei-Body ausserhalb der Konstante.
|
||||||
|
</action>
|
||||||
|
<verify>
|
||||||
|
<automated>pnpm --filter @tessera/api test -- source-registry</automated>
|
||||||
|
</verify>
|
||||||
|
<done>register() wirft DeniedPortalError fuer vergabe24 UND aumass (Test gruen); legitime Registrierung + get()/activeAdapters() funktionieren; kein Prisma/Scraping-Import.</done>
|
||||||
|
</task>
|
||||||
|
|
||||||
|
</tasks>
|
||||||
|
|
||||||
|
<threat_model>
|
||||||
|
## Trust Boundaries
|
||||||
|
|
||||||
|
| Boundary | Description |
|
||||||
|
|----------|-------------|
|
||||||
|
| Adapter-Registrierung -> Poll-System | Ein verbotenes Portal darf nie in den aktiven Poll-Pfad gelangen (AGB/Compliance). |
|
||||||
|
|
||||||
|
## STRIDE Threat Register
|
||||||
|
|
||||||
|
| Threat ID | Category | Component | Severity | Disposition | Mitigation Plan |
|
||||||
|
|-----------|----------|-----------|----------|-------------|-----------------|
|
||||||
|
| T-13-02-01 | Elevation of Privilege | Registrierung eines Denylist-Portals | high | mitigate | Harte DeniedPortalError-Exception in register() bei DI-Boot; Test beweist Refusal (D-06, INGEST-07). |
|
||||||
|
| T-13-02-02 | Tampering | Umgehung des Gates via gemischtem portals-Array | high | mitigate | Iteration ueber ALLE portals; ein verbotenes Portal lehnt den ganzen Adapter ab. |
|
||||||
|
| T-13-02-SC | Tampering | keine neue Dependency in diesem Plan | low | accept | Kern nutzt nur bestehende Deps; node-html-parser erst in 13-04 hinter Checkpoint. |
|
||||||
|
</threat_model>
|
||||||
|
|
||||||
|
<verification>
|
||||||
|
- `pnpm --filter @tessera/api test -- source-registry` gruen (Refusal beweist Erfolgskriterium 4).
|
||||||
|
- `npx tsc --noEmit` gruen.
|
||||||
|
</verification>
|
||||||
|
|
||||||
|
<success_criteria>
|
||||||
|
SourceRegistry lehnt vergabe24/aumass im Code ab (getestet); Adapter-Vertrag um portals generalisiert. Erfuellt INGEST-07 strukturell.
|
||||||
|
</success_criteria>
|
||||||
|
|
||||||
|
<output>
|
||||||
|
Create `.planning/phases/13-scraping-adapters-cross-source-dedup/13-02-SUMMARY.md` when done
|
||||||
|
</output>
|
||||||
@@ -0,0 +1,134 @@
|
|||||||
|
---
|
||||||
|
phase: 13-scraping-adapters-cross-source-dedup
|
||||||
|
plan: 03
|
||||||
|
type: execute
|
||||||
|
wave: 2
|
||||||
|
depends_on: ["13-01", "13-02"]
|
||||||
|
files_modified:
|
||||||
|
- apps/api/src/tenders/tender-dedup.service.ts
|
||||||
|
- apps/api/src/tenders/tender-dedup.service.spec.ts
|
||||||
|
- apps/api/src/tenders/tender-ingestion.service.ts
|
||||||
|
- apps/api/src/tenders/tender-ingestion.service.spec.ts
|
||||||
|
- apps/api/src/tenders/tenders.module.ts
|
||||||
|
autonomous: true
|
||||||
|
requirements: [SCHEMA-03]
|
||||||
|
must_haves:
|
||||||
|
truths:
|
||||||
|
- "Mit nur DÖE aktiv (activePortalCount < 2) laeuft die Fingerprint-Stufe NIE — zwei Tender mit gleichem Fingerprint bleiben zwei Tender (D-05, Erfolgskriterium 3, inert)."
|
||||||
|
- "Ab der 2. aktiven Quelle haengt ein Fingerprint-Match eine zusaetzliche TenderSource an den bestehenden Tender an, statt einen neuen Tender zu erzeugen (D-03/D-04)."
|
||||||
|
- "pollDueSources faechert ueber ALLE aktiven Configs (findMany isActive), mit catch-per-source (eine kaputte Quelle killt den Tick nicht, D-01)."
|
||||||
|
artifacts:
|
||||||
|
- apps/api/src/tenders/tender-dedup.service.ts
|
||||||
|
key_links:
|
||||||
|
- "pollDueSources -> SourceRegistry.get(config.sourceType) -> adapter -> normalize -> TenderDedupService.resolve(dedupActive = activePortalCount >= 2)."
|
||||||
|
- "TenderDedupService: OCID -> source:noticeId -> fingerprint; Match -> tenderSource.upsert; kein Match -> tender.create + tenderSource.create."
|
||||||
|
---
|
||||||
|
|
||||||
|
<objective>
|
||||||
|
Verdrahtet SCHEMA-03 in den Ingestion-Pfad: der dreistufige Dedup-Resolver (OCID -> Quelle:NoticeId -> Fingerprint, D-04) und der Fan-out von `pollDueSources` (poll-once-fan-out-many ueber die Registry, NICHT findUnique/findFirst) mit catch-per-source-Fehlerisolation (D-01). Das Fingerprint-Gate greift hart erst ab `activePortalCount >= 2` (D-05) — mit nur DÖE bleibt das bestehende Verhalten exakt unveraendert (Erfolgskriterium 3, inert).
|
||||||
|
|
||||||
|
Purpose: Der Merge-Kern, der aus "dieselbe Ausschreibung ueber mehrere Quellen" EINEN Tender mit mehreren TenderSource macht.
|
||||||
|
Output: tender-dedup.service.ts (+ spec), erweitertes pollDueSources (+ spec), Modul-Wiring von SourceRegistry + Adaptern + DedupService.
|
||||||
|
</objective>
|
||||||
|
|
||||||
|
<execution_context>
|
||||||
|
@$HOME/.claude/gsd-core/workflows/execute-plan.md
|
||||||
|
@$HOME/.claude/gsd-core/templates/summary.md
|
||||||
|
</execution_context>
|
||||||
|
|
||||||
|
<context>
|
||||||
|
@.planning/PROJECT.md
|
||||||
|
@.planning/ROADMAP.md
|
||||||
|
@.planning/STATE.md
|
||||||
|
@.planning/phases/13-scraping-adapters-cross-source-dedup/13-CONTEXT.md
|
||||||
|
@.planning/phases/13-scraping-adapters-cross-source-dedup/13-RESEARCH.md
|
||||||
|
@apps/api/src/tenders/tender-ingestion.service.ts
|
||||||
|
@apps/api/src/tenders/tenders.module.ts
|
||||||
|
@apps/api/src/tenders/tender-normalizer.service.ts
|
||||||
|
@apps/api/src/tenders/source-registry.ts
|
||||||
|
@apps/api/src/tenders/tender-fingerprint.ts
|
||||||
|
</context>
|
||||||
|
|
||||||
|
<tasks>
|
||||||
|
|
||||||
|
<task type="auto" tdd="true">
|
||||||
|
<name>Task 1: Dreistufiger Dedup-Resolver (D-04/D-05)</name>
|
||||||
|
<files>apps/api/src/tenders/tender-dedup.service.ts, apps/api/src/tenders/tender-dedup.service.spec.ts</files>
|
||||||
|
<behavior>
|
||||||
|
- dedupActive=false (nur DÖE): zwei normalisierte Records mit gleichem Fingerprint aber verschiedenen source:noticeIds erzeugen ZWEI Tender (Fingerprint-Stufe uebersprungen, D-05).
|
||||||
|
- dedupActive=true, OCID-Match: haengt TenderSource an bestehenden Tender, created=false.
|
||||||
|
- dedupActive=true, kein OCID aber source:noticeId bereits vorhanden (idempotenter Re-Poll): upsert der TenderSource, kein neuer Tender.
|
||||||
|
- dedupActive=true, kein OCID/noticeId-Match aber Fingerprint-Match: haengt zusaetzliche TenderSource an, created=false (D-03 Merge).
|
||||||
|
- Kein Match: tender.create (inkl. fingerprint) + tenderSource.create, created=true.
|
||||||
|
- tenderSource.upsert nutzt @@unique([sourcePortal, sourceNoticeId]) als where-Target.
|
||||||
|
</behavior>
|
||||||
|
<action>
|
||||||
|
Implementiere `tender-dedup.service.ts` (`@Injectable`) nach 13-RESEARCH.md Pattern 5: Methode `resolve(n: NormalizedTenderFields, opts: { dedupActive: boolean }): Promise<{ tenderId: string; created: boolean }>`. Reihenfolge: (1) OCID-Match `prisma.tender.findFirst({ where: { ocid } })` falls ocid vorhanden; (2) `prisma.tenderSource.findUnique({ where: { sourcePortal_sourceNoticeId: {...} } })` -> zugehoerigen Tender; (3) NUR wenn `opts.dedupActive` und noch kein Match: `prisma.tender.findFirst({ where: { fingerprint: tenderFingerprint(n) } })`. Bei Match: `tenderSource.upsert` (create: an match.id haengen, update: sourceUrl) und `created:false`. Ohne Match: `tender.create({ data: { ...felder, fingerprint: tenderFingerprint(n) } })` + `tenderSource.create`, `created:true`. Der `dedupActive`-Gate ist der D-05-Kern — Stufe 3 wird bei nur einer Quelle NIE erreicht.
|
||||||
|
|
||||||
|
WICHTIG (T-10-09 / Phase-10-Muster): plain PrismaService, KEIN forTenant()/RLS — Tender/TenderSource sind plattform-global. Schreibe die Spec RED-first mit gemocktem PrismaService (kein echtes IO) nach behavior-Block; der inert-Fall (dedupActive=false -> zwei Tender) ist der D-05-Beweis.
|
||||||
|
</action>
|
||||||
|
<verify>
|
||||||
|
<automated>pnpm --filter @tessera/api test -- tender-dedup</automated>
|
||||||
|
</verify>
|
||||||
|
<done>Alle behavior-Faelle gruen; inert bei einer Quelle, Merge ab zwei Quellen; nutzt tenderFingerprint aus 13-01.</done>
|
||||||
|
</task>
|
||||||
|
|
||||||
|
<task type="auto">
|
||||||
|
<name>Task 2: pollDueSources Fan-out + catch-per-source + Dedup-Hook</name>
|
||||||
|
<files>apps/api/src/tenders/tender-ingestion.service.ts, apps/api/src/tenders/tender-ingestion.service.spec.ts</files>
|
||||||
|
<action>
|
||||||
|
Baue `pollDueSources` von `findUnique({ sourceType:'doe-opendata' })` auf Fan-out um (13-RESEARCH.md Pattern 3): `const configs = await prisma.tenderSourcePollConfig.findMany({ where: { isActive: true } })`; `activePortalCount = configs.length`; `dedupActive = activePortalCount >= 2` (D-05). Iteriere die Configs: `const adapter = registry.get(config.sourceType as SourceType); if (!adapter) continue;` — je Config in `try { ... } catch (err) { this.logger.error(...) }` (catch-per-source, D-01: eine kaputte Quelle killt die anderen nicht). Der bestehende Day-Cursor-Gate (`nextDayToFetch`) + fetch + normalize bleiben PRO Quelle erhalten; ersetze den direkten `tender.upsert`-Block durch `dedup.resolve(normalized, { dedupActive })` und sammle `created`-Tender-IDs (Delta-Boundary D-07 bleibt: nur genuin neue IDs an `matching.matchDelta`). lastIngestedDay-Advance bleibt pro Config (Cursor je sourceType). Injiziere `SourceRegistry` und `TenderDedupService` in den Constructor; entferne die direkte `DoeOpenDataAdapter`-Nutzung im Poll-Pfad zugunsten `registry.get('doe-opendata')`.
|
||||||
|
|
||||||
|
Erweitere die bestehende `tender-ingestion.service.spec.ts`: (a) fan-out ueber mehrere aktive Configs; (b) catch-per-source (Quelle A wirft -> Quelle B laeuft trotzdem, Tick wirft nicht); (c) dedupActive-Gate (1 Config -> resolve mit dedupActive=false; 2 Configs -> dedupActive=true); (d) matchDelta bekommt nur created-IDs. Registry/Dedup im Test mocken.
|
||||||
|
</action>
|
||||||
|
<verify>
|
||||||
|
<automated>pnpm --filter @tessera/api test -- tender-ingestion.service</automated>
|
||||||
|
</verify>
|
||||||
|
<done>pollDueSources faechert ueber alle aktiven Configs; catch-per-source getestet; dedupActive an activePortalCount>=2 gebunden; Delta-Boundary erhalten; Spec gruen.</done>
|
||||||
|
</task>
|
||||||
|
|
||||||
|
<task type="auto">
|
||||||
|
<name>Task 3: Modul-Wiring — SourceRegistry + Adapter-Registrierung + DedupService</name>
|
||||||
|
<files>apps/api/src/tenders/tenders.module.ts</files>
|
||||||
|
<action>
|
||||||
|
Registriere in `tenders.module.ts` die neuen Provider `SourceRegistry` und `TenderDedupService`. Verdrahte die Adapter-Registrierung bei DI-Boot: der `DoeOpenDataAdapter` (und in 13-04/05 die neuen Adapter) werden nach Instanziierung ueber `SourceRegistry.register(adapter)` eingetragen — nutze ein `OnModuleInit`-Hook oder einen Factory-Provider, der die vorhandenen Adapter-Provider einsammelt und registriert (so greift auch das Denylist-Gate strukturell bei Boot, D-06). Halte die Registrierungsstelle so, dass 13-04/13-05 ihren Adapter additiv ergaenzen koennen (dieser Plan ist der einzige Schreiber von tenders.module.ts in Wave 2; 13-04/05 folgen serialisiert in spaeteren Waves). DI-Graph muss aufloesbar bleiben.
|
||||||
|
</action>
|
||||||
|
<verify>
|
||||||
|
<automated>cd apps/api && npx tsc --noEmit -p tsconfig.json && pnpm --filter @tessera/api test -- src/tenders</automated>
|
||||||
|
</verify>
|
||||||
|
<done>SourceRegistry + TenderDedupService sind Provider; DoeOpenDataAdapter wird bei Boot registriert; DI-Graph aufloesbar; tenders-Suite gruen.</done>
|
||||||
|
</task>
|
||||||
|
|
||||||
|
</tasks>
|
||||||
|
|
||||||
|
<threat_model>
|
||||||
|
## Trust Boundaries
|
||||||
|
|
||||||
|
| Boundary | Description |
|
||||||
|
|----------|-------------|
|
||||||
|
| Adapter-Fetch -> Ingestion-Tick | Ein blockendes/kaputtes Portal darf den globalen Tick nicht killen (D-01). |
|
||||||
|
| Normalisierte Records -> Dedup/DB | Falsches Gate koennte Bestand faelschlich mergen (D-05). |
|
||||||
|
|
||||||
|
## STRIDE Threat Register
|
||||||
|
|
||||||
|
| Threat ID | Category | Component | Severity | Disposition | Mitigation Plan |
|
||||||
|
|-----------|----------|-----------|----------|-------------|-----------------|
|
||||||
|
| T-13-03-01 | Denial of Service | ein blockendes Portal im Fan-out | high | mitigate | catch-per-source: try/catch je Config, Tick wirft nie (D-01, Pattern 3). |
|
||||||
|
| T-13-03-02 | Tampering | Dedup laeuft versehentlich mit 1 Quelle | high | mitigate | dedupActive = activePortalCount >= 2, explizit getestet (inert-Fall, D-05/Pitfall 3). |
|
||||||
|
| T-13-03-03 | Info Disclosure | Tenant-Leak durch RLS-Extension auf globalen Tabellen | medium | mitigate | plain PrismaService, kein forTenant() (T-10-09-Muster). |
|
||||||
|
| T-13-03-04 | Tampering | Fingerprint-Kollision merged unaehnliche Tender | medium | accept | Titel im Hash (Pitfall 1); bei Kollision Granularitaet erhoehen, kein Threshold. |
|
||||||
|
</threat_model>
|
||||||
|
|
||||||
|
<verification>
|
||||||
|
- `pnpm --filter @tessera/api test -- tender-dedup` gruen (inert + merge).
|
||||||
|
- `pnpm --filter @tessera/api test -- tender-ingestion.service` gruen (fan-out + catch-per-source + gate).
|
||||||
|
- `npx tsc --noEmit` + `pnpm --filter @tessera/api test -- src/tenders` gruen (DI-Graph).
|
||||||
|
</verification>
|
||||||
|
|
||||||
|
<success_criteria>
|
||||||
|
Fan-out ueber alle aktiven Quellen mit Fehlerisolation; dreistufiger Dedup ab 2. Quelle haengt TenderSource an statt neuen Tender; inert mit nur DÖE. TrAegt SCHEMA-03 (Wiring) + Erfolgskriterium 3 (Dedup-Logik).
|
||||||
|
</success_criteria>
|
||||||
|
|
||||||
|
<output>
|
||||||
|
Create `.planning/phases/13-scraping-adapters-cross-source-dedup/13-03-SUMMARY.md` when done
|
||||||
|
</output>
|
||||||
@@ -0,0 +1,124 @@
|
|||||||
|
---
|
||||||
|
phase: 13-scraping-adapters-cross-source-dedup
|
||||||
|
plan: 04
|
||||||
|
type: execute
|
||||||
|
wave: 3
|
||||||
|
depends_on: ["13-02", "13-03"]
|
||||||
|
files_modified:
|
||||||
|
- apps/api/package.json
|
||||||
|
- apps/api/src/tenders/adapters/netserver.adapter.ts
|
||||||
|
- apps/api/src/tenders/adapters/netserver.adapter.spec.ts
|
||||||
|
- apps/api/src/tenders/__fixtures__/netserver-search.html
|
||||||
|
- apps/api/src/tenders/tenders.module.ts
|
||||||
|
autonomous: false
|
||||||
|
requirements: [INGEST-02]
|
||||||
|
user_setup: []
|
||||||
|
must_haves:
|
||||||
|
truths:
|
||||||
|
- "EIN config-getriebener NetServer-Adapter bedient tender24, lhs-vpbw und vergabe.landbw (portals-Array, Base-URL je Portal)."
|
||||||
|
- "Der Adapter parst die NetServer-<table>-Trefferliste zu RawTenderRecord[] und setzt sourcePortal je Zeile korrekt."
|
||||||
|
- "Bei Parse-Totalausfall liefert der Adapter [] statt zu werfen (D-01, catch-per-source im Ingestion-Tick faengt den Rest)."
|
||||||
|
artifacts:
|
||||||
|
- apps/api/src/tenders/adapters/netserver.adapter.ts
|
||||||
|
- apps/api/src/tenders/__fixtures__/netserver-search.html
|
||||||
|
key_links:
|
||||||
|
- "SourceRegistry.register(netServerAdapter) bei Boot; pollDueSources fan-out ruft fetchTenders je aktiver Config."
|
||||||
|
---
|
||||||
|
|
||||||
|
<objective>
|
||||||
|
Baut den NetServer-Adapter (INGEST-02 / Erfolgskriterium 1): EIN config-getriebener HTML-Adapter fuer die drei AI-AG-Vergabe@Net-Portale (tender24, lhs-vpbw, vergabe.landbw), der die oeffentliche `PublicationSearchControllerServlet`-Trefferliste (`<table>`, live verifiziert HTTP 200/161 KB, keine Auth) parst. Selektoren best-effort gefuellt (D-01); die Adapter-Grenze kapselt Fragilitaet — Test laeuft gegen eine live-gecapturte HTML-Fixture, unabhaengig von Live-Erreichbarkeit. Zuerst ein blockierender Package-Legitimacy-Checkpoint vor dem HTML-Parser-Install.
|
||||||
|
|
||||||
|
Purpose: Zweite reale Quelle neben DÖE — erst dadurch wird der Dedup-Kern (13-03) live wirksam (activePortalCount >= 2).
|
||||||
|
Output: node-html-parser (gated), netserver.adapter.ts (+ spec + fixture), Modul-Registrierung.
|
||||||
|
</objective>
|
||||||
|
|
||||||
|
<execution_context>
|
||||||
|
@$HOME/.claude/gsd-core/workflows/execute-plan.md
|
||||||
|
@$HOME/.claude/gsd-core/templates/summary.md
|
||||||
|
</execution_context>
|
||||||
|
|
||||||
|
<context>
|
||||||
|
@.planning/PROJECT.md
|
||||||
|
@.planning/ROADMAP.md
|
||||||
|
@.planning/STATE.md
|
||||||
|
@.planning/phases/13-scraping-adapters-cross-source-dedup/13-CONTEXT.md
|
||||||
|
@.planning/phases/13-scraping-adapters-cross-source-dedup/13-RESEARCH.md
|
||||||
|
@apps/api/src/tenders/adapters/doe-opendata.adapter.ts
|
||||||
|
@apps/api/src/tenders/adapters/tender-source-adapter.interface.ts
|
||||||
|
@apps/api/src/tenders/source-registry.ts
|
||||||
|
</context>
|
||||||
|
|
||||||
|
<tasks>
|
||||||
|
|
||||||
|
<task type="checkpoint:human-verify" gate="blocking-human">
|
||||||
|
<name>Task 1: Checkpoint — Package-Legitimacy-Gate HTML-Parser (vor Install)</name>
|
||||||
|
<action>Blockierender Package-Legitimacy-Checkpoint VOR jedem Install (13-RESEARCH.md § Package Legitimacy Audit). node-html-parser ist [ASSUMED] aus Trainingswissen, nicht autoritativ — Registry/Alter/Downloads/Repo pruefen, Wahl (node-html-parser oder cheerio) bestaetigen, erst dann installieren. Nie auto-approve.</action>
|
||||||
|
<what-built>Package-Legitimacy-Gate fuer den HTML-Parser VOR jedem Install (13-RESEARCH.md § Package Legitimacy Audit; node-html-parser ist [ASSUMED] aus Trainingswissen, nicht autoritativ).</what-built>
|
||||||
|
<how-to-verify>
|
||||||
|
1. Fuehre `gsd-tools query package-legitimacy check --ecosystem npm node-html-parser cheerio` aus (bzw. `npm view node-html-parser` + `npm view cheerio`): Alter, Wochendownloads, Repo pruefen (github.com/taoqf/node-html-parser).
|
||||||
|
2. Entscheide: `node-html-parser` (leichtgewichtig, CSS-Selektoren, empfohlen) ODER `cheerio` (vollere jQuery-API, weit verbreitet). Beide erfuellen den Zweck.
|
||||||
|
3. Bestaetige die Wahl; erst dann installiert der Executor `pnpm --filter @tessera/api add <gewaehltes-package>`.
|
||||||
|
</how-to-verify>
|
||||||
|
<resume-signal>Tippe "node-html-parser", "cheerio", oder beschreibe Bedenken.</resume-signal>
|
||||||
|
</task>
|
||||||
|
|
||||||
|
<task type="auto">
|
||||||
|
<name>Task 2: NetServer-Adapter (config-getrieben, 3 Portale) + HTML-Fixture + Spec</name>
|
||||||
|
<files>apps/api/src/tenders/adapters/netserver.adapter.ts, apps/api/src/tenders/adapters/netserver.adapter.spec.ts, apps/api/src/tenders/__fixtures__/netserver-search.html, apps/api/package.json</files>
|
||||||
|
<action>
|
||||||
|
Installiere zuerst das im Checkpoint bestaetigte Package. Implementiere `netserver.adapter.ts` (`@Injectable`, implements TenderSourceAdapter, 13-RESEARCH.md Pattern 1): `sourceType = 'ai-netserver'`, `portals = ['tender24','lhs-vpbw','vergabe.landbw'] as const`, plus interne `NETSERVER_PORTALS`-Config-Map mit exakter Base-URL je Portal (tender24: `https://www.tender24.de`; lhs-vpbw: `https://lhs-vpbw.vmstart.de`; vergabe.landbw: `https://vergabe.landbw.de` — Base-URL MUSS direkt auf `/NetServer` zeigen, meta-refresh-Stub vermeiden, Pitfall 4). `fetchTenders(dayCursor)` iteriert die Portale, ruft je `${baseUrl}/NetServer/PublicationSearchControllerServlet?function=SearchPublications` via native `fetch` + `AbortController` 15s-Timeout (DoeOpenData-Muster, kein axios), `redirect:'follow'`, parst die `<table>`-Trefferzeilen mit dem HTML-Parser zu `RawTenderRecord[]`, setzt `sourcePortal` je Zeile korrekt, extrahiert Zeilen-Anchor-href als `sourceUrl` und eine stabile ID daraus als `sourceNoticeId` (Open Question 2). Fehlertoleranz (Pitfall 2 / D-01): try/catch pro Zeile (`logger.warn`, Zeile ueberspringen), `[]` bei Totalausfall statt Throw; Pagination best-effort (erste Seite genuegt fuer MVP-Nachweis).
|
||||||
|
|
||||||
|
Speichere eine live-gecapturte NetServer-Trefferliste als `__fixtures__/netserver-search.html` (stabile Testgrundlage). Schreibe `netserver.adapter.spec.ts`: Adapter parst die Fixture -> N RawTenderRecord[] mit korrektem sourcePortal/sourceUrl/sourceNoticeId; leeres/kaputtes HTML -> []; Fetch wird gemockt (kein Live-Netz im Test). SSRF-Schutz (V5): Base-URLs sind hardcodierte Konstanten, NIE aus Input interpoliert (T-10-06).
|
||||||
|
</action>
|
||||||
|
<verify>
|
||||||
|
<automated>pnpm --filter @tessera/api test -- netserver.adapter</automated>
|
||||||
|
</verify>
|
||||||
|
<done>Adapter parst die Fixture zu RawTenderRecord[] mit korrektem sourcePortal je Zeile; []-Fallback bei kaputtem HTML; Spec gruen. Falls ein Portal live nicht parsebar: als "needs-JS/blocked, deferred" im SUMMARY dokumentieren (D-01), Adapter-Skelett + Test bleiben gruen.</done>
|
||||||
|
</task>
|
||||||
|
|
||||||
|
<task type="auto">
|
||||||
|
<name>Task 3: NetServer-Adapter im Modul registrieren</name>
|
||||||
|
<files>apps/api/src/tenders/tenders.module.ts</files>
|
||||||
|
<action>
|
||||||
|
Ergaenze `NetServerAdapter` als Provider in `tenders.module.ts` und haenge ihn additiv in die in 13-03 gebaute Boot-Registrierungsstelle (`SourceRegistry.register(netServerAdapter)`). Portale tender24/lhs-vpbw/vergabe.landbw sind NICHT auf der Denylist -> Registrierung erlaubt. Lege optional eine `TenderSourcePollConfig`-Row fuer `ai-netserver` (isActive default false, kein Zwang zur Aktivierung in P13 — Aktivierung ist Admin/Seed). DI-Graph muss aufloesbar bleiben; dieser Plan ist der einzige Wave-3-Schreiber von tenders.module.ts (13-05 folgt serialisiert in Wave 4).
|
||||||
|
</action>
|
||||||
|
<verify>
|
||||||
|
<automated>cd apps/api && npx tsc --noEmit -p tsconfig.json && pnpm --filter @tessera/api test -- src/tenders</automated>
|
||||||
|
</verify>
|
||||||
|
<done>NetServerAdapter ist Provider + bei Boot registriert; DI aufloesbar; tenders-Suite gruen.</done>
|
||||||
|
</task>
|
||||||
|
|
||||||
|
</tasks>
|
||||||
|
|
||||||
|
<threat_model>
|
||||||
|
## Trust Boundaries
|
||||||
|
|
||||||
|
| Boundary | Description |
|
||||||
|
|----------|-------------|
|
||||||
|
| NetServer-Portal (Internet) -> Adapter | Extern gefetchtes HTML ist untrusted (V5). |
|
||||||
|
| Package-Install -> Build | Neue Dependency node-html-parser/cheerio (Supply-Chain). |
|
||||||
|
|
||||||
|
## STRIDE Threat Register
|
||||||
|
|
||||||
|
| Threat ID | Category | Component | Severity | Disposition | Mitigation Plan |
|
||||||
|
|-----------|----------|-----------|----------|-------------|-----------------|
|
||||||
|
| T-13-04-SC | Tampering | node-html-parser/cheerio Install | high | mitigate | Blocking-human Package-Legitimacy-Checkpoint VOR Install (nie auto-approve); [ASSUMED] bis Gate-Lauf. |
|
||||||
|
| T-13-04-01 | Tampering (SSRF) | Portal-Base-URL | high | mitigate | Hardcodierte Konstanten je Portal, NIE aus Input interpoliert (T-10-06). |
|
||||||
|
| T-13-04-02 | Denial of Service | malformed HTML / haengender Fetch | high | mitigate | AbortController-15s-Timeout, try/catch pro Zeile + []-Fallback, catch-per-source im Tick (D-01). |
|
||||||
|
| T-13-04-03 | Info Disclosure | Roh-HTML an Frontend | medium | mitigate | Parser-Ausgabe als Text coercen (Titel/Buyer), kein Roh-HTML weiterreichen (V5). |
|
||||||
|
</threat_model>
|
||||||
|
|
||||||
|
<verification>
|
||||||
|
- Checkpoint bestaetigt (Package-Wahl), erst dann Install.
|
||||||
|
- `pnpm --filter @tessera/api test -- netserver.adapter` gruen (gegen Fixture).
|
||||||
|
- `npx tsc --noEmit` + `pnpm --filter @tessera/api test -- src/tenders` gruen.
|
||||||
|
- Manuell (13-VALIDATION.md): optionaler Live-Poll gegen tender24 liefert >0 Records ODER dokumentierter "needs-JS/blocked"-Vermerk.
|
||||||
|
</verification>
|
||||||
|
|
||||||
|
<success_criteria>
|
||||||
|
EIN config-getriebener Adapter bedient die 3 NetServer-Portale, parst die Trefferliste, ist fehlertolerant. Erfuellt INGEST-02 / Erfolgskriterium 1.
|
||||||
|
</success_criteria>
|
||||||
|
|
||||||
|
<output>
|
||||||
|
Create `.planning/phases/13-scraping-adapters-cross-source-dedup/13-04-SUMMARY.md` when done
|
||||||
|
</output>
|
||||||
@@ -0,0 +1,107 @@
|
|||||||
|
---
|
||||||
|
phase: 13-scraping-adapters-cross-source-dedup
|
||||||
|
plan: 05
|
||||||
|
type: execute
|
||||||
|
wave: 4
|
||||||
|
depends_on: ["13-04"]
|
||||||
|
files_modified:
|
||||||
|
- apps/api/src/tenders/adapters/cosinex.adapter.ts
|
||||||
|
- apps/api/src/tenders/adapters/cosinex.adapter.spec.ts
|
||||||
|
- apps/api/src/tenders/__fixtures__/cosinex-search.html
|
||||||
|
- apps/api/src/tenders/tenders.module.ts
|
||||||
|
autonomous: true
|
||||||
|
requirements: [INGEST-03]
|
||||||
|
must_haves:
|
||||||
|
truths:
|
||||||
|
- "Ein separater cosinex/DTVP-Adapter (getrennt vom NetServer-Adapter, cosinex-HTML != NetServer) parst die oeffentliche Satellite-Trefferliste zu RawTenderRecord[]."
|
||||||
|
- "Bei JS-abhaengiger/nicht-parsebarer Trefferliste liefert der Adapter [] und der Fall ist als 'needs-JS, deferred' dokumentiert (D-01) — Adapter-Grenze + Test bleiben gruen."
|
||||||
|
- "sourceType = 'cosinex-dtvp', sourcePortal je Record korrekt gesetzt."
|
||||||
|
artifacts:
|
||||||
|
- apps/api/src/tenders/adapters/cosinex.adapter.ts
|
||||||
|
- apps/api/src/tenders/__fixtures__/cosinex-search.html
|
||||||
|
key_links:
|
||||||
|
- "SourceRegistry.register(cosinexAdapter) bei Boot; getrennter Adapter, kein NetServer-Reuse."
|
||||||
|
---
|
||||||
|
|
||||||
|
<objective>
|
||||||
|
Baut den cosinex/DTVP-Adapter (INGEST-03 / Erfolgskriterium 2): ein SEPARATER HTML-Adapter fuer den cosinex-Vergabemarktplatz (`www.dtvp.de/Satellite/company/welcome.do`, live HTTP 200/40 KB, oeffentlich) — cosinex-HTML unterscheidet sich strukturell vom NetServer, daher kein geteilter Adapter. Selektoren best-effort (D-01): falls die Trefferliste server-seitig gerendert ist, werden die Selektoren gefuellt; ist sie JS-abhaengig, wird der Adapter als "needs-JS, deferred" dokumentiert, liefert `[]`, und Adapter-Skelett + Test bleiben trotzdem gruen (Open Question 1, kein Headless-Browser im Stack).
|
||||||
|
|
||||||
|
Purpose: Dritte Quelle; erfuellt INGEST-03 und liefert einen weiteren realen Merge-Kandidaten fuer den Dedup-Kern.
|
||||||
|
Output: cosinex.adapter.ts (+ spec + fixture), Modul-Registrierung.
|
||||||
|
</objective>
|
||||||
|
|
||||||
|
<execution_context>
|
||||||
|
@$HOME/.claude/gsd-core/workflows/execute-plan.md
|
||||||
|
@$HOME/.claude/gsd-core/templates/summary.md
|
||||||
|
</execution_context>
|
||||||
|
|
||||||
|
<context>
|
||||||
|
@.planning/PROJECT.md
|
||||||
|
@.planning/ROADMAP.md
|
||||||
|
@.planning/STATE.md
|
||||||
|
@.planning/phases/13-scraping-adapters-cross-source-dedup/13-CONTEXT.md
|
||||||
|
@.planning/phases/13-scraping-adapters-cross-source-dedup/13-RESEARCH.md
|
||||||
|
@apps/api/src/tenders/adapters/netserver.adapter.ts
|
||||||
|
@apps/api/src/tenders/adapters/tender-source-adapter.interface.ts
|
||||||
|
@apps/api/src/tenders/source-registry.ts
|
||||||
|
</context>
|
||||||
|
|
||||||
|
<tasks>
|
||||||
|
|
||||||
|
<task type="auto">
|
||||||
|
<name>Task 1: cosinex/DTVP-Adapter + HTML-Fixture + Spec</name>
|
||||||
|
<files>apps/api/src/tenders/adapters/cosinex.adapter.ts, apps/api/src/tenders/adapters/cosinex.adapter.spec.ts, apps/api/src/tenders/__fixtures__/cosinex-search.html</files>
|
||||||
|
<action>
|
||||||
|
Implementiere `cosinex.adapter.ts` (`@Injectable`, implements TenderSourceAdapter) als EIGENEN Adapter (nicht NetServer wiederverwenden): `sourceType = 'cosinex-dtvp'`, `portals = ['cosinex-dtvp'] as const`, Base-URL `https://www.dtvp.de`. `fetchTenders(dayCursor)` fetcht die oeffentliche Satellite-Trefferliste (`Satellite/company/welcome.do` bzw. die konkrete Suchseite) via native `fetch` + AbortController 15s (DoeOpenData/NetServer-Muster, kein axios), parst mit dem in 13-04 bereits installierten HTML-Parser die server-gerenderten Trefferzeilen zu `RawTenderRecord[]`, setzt sourcePortal='cosinex-dtvp', extrahiert sourceUrl + stabile sourceNoticeId je Zeile. Fehlertoleranz identisch zu NetServer (Pitfall 2 / D-01): try/catch pro Zeile, `[]` bei Totalausfall, kein Throw.
|
||||||
|
|
||||||
|
Speichere eine live-gecapturte cosinex-Trefferliste als `__fixtures__/cosinex-search.html`. Schreibe `cosinex.adapter.spec.ts`: Adapter parst die Fixture -> RawTenderRecord[] mit sourceType/sourcePortal/sourceUrl/sourceNoticeId; kaputtes/leeres HTML -> []; Fetch gemockt. Falls die Live-Trefferliste JS-abhaengig ist (Open Question 1): Test gegen den server-gerenderten Teil oder gegen einen dokumentierten Leer-Fall; im SUMMARY als "needs-JS, deferred" vermerken. SSRF (V5): Base-URL hardcodiert.
|
||||||
|
</action>
|
||||||
|
<verify>
|
||||||
|
<automated>pnpm --filter @tessera/api test -- cosinex.adapter</automated>
|
||||||
|
</verify>
|
||||||
|
<done>cosinex-Adapter parst die Fixture (oder liefert dokumentiert [] bei JS-Abhaengigkeit); []-Fallback bei kaputtem HTML; Spec gruen; getrennt vom NetServer-Adapter.</done>
|
||||||
|
</task>
|
||||||
|
|
||||||
|
<task type="auto">
|
||||||
|
<name>Task 2: cosinex-Adapter im Modul registrieren</name>
|
||||||
|
<files>apps/api/src/tenders/tenders.module.ts</files>
|
||||||
|
<action>
|
||||||
|
Ergaenze `CosinexAdapter` als Provider in `tenders.module.ts` und registriere ihn additiv in der Boot-Registrierungsstelle (`SourceRegistry.register(cosinexAdapter)`). Portal cosinex-dtvp ist nicht auf der Denylist. Optional `TenderSourcePollConfig`-Row fuer `cosinex-dtvp` (isActive default false). Dieser Plan ist der einzige Wave-4-Schreiber von tenders.module.ts (serialisiert nach 13-04). DI-Graph aufloesbar.
|
||||||
|
</action>
|
||||||
|
<verify>
|
||||||
|
<automated>cd apps/api && npx tsc --noEmit -p tsconfig.json && pnpm --filter @tessera/api test -- src/tenders</automated>
|
||||||
|
</verify>
|
||||||
|
<done>CosinexAdapter ist Provider + bei Boot registriert; DI aufloesbar; tenders-Suite gruen.</done>
|
||||||
|
</task>
|
||||||
|
|
||||||
|
</tasks>
|
||||||
|
|
||||||
|
<threat_model>
|
||||||
|
## Trust Boundaries
|
||||||
|
|
||||||
|
| Boundary | Description |
|
||||||
|
|----------|-------------|
|
||||||
|
| cosinex/DTVP-Portal (Internet) -> Adapter | Extern gefetchtes HTML ist untrusted (V5). |
|
||||||
|
|
||||||
|
## STRIDE Threat Register
|
||||||
|
|
||||||
|
| Threat ID | Category | Component | Severity | Disposition | Mitigation Plan |
|
||||||
|
|-----------|----------|-----------|----------|-------------|-----------------|
|
||||||
|
| T-13-05-01 | Tampering (SSRF) | cosinex Base-URL | high | mitigate | Hardcodierte Konstante, nie aus Input interpoliert (T-10-06). |
|
||||||
|
| T-13-05-02 | Denial of Service | JS-lastiges/malformed HTML, haengender Fetch | high | mitigate | AbortController-15s, try/catch pro Zeile, []-Fallback, catch-per-source (D-01). |
|
||||||
|
| T-13-05-03 | Info Disclosure | Roh-HTML an Frontend | medium | mitigate | Parser-Ausgabe als Text coercen (V5). |
|
||||||
|
</threat_model>
|
||||||
|
|
||||||
|
<verification>
|
||||||
|
- `pnpm --filter @tessera/api test -- cosinex.adapter` gruen (Fixture oder dokumentierter Leer-Fall).
|
||||||
|
- `npx tsc --noEmit` + `pnpm --filter @tessera/api test -- src/tenders` gruen.
|
||||||
|
- Manuell (13-VALIDATION.md): optionaler Live-Fetch dtvp welcome.do liefert parsebares HTML ODER "needs-JS"-Vermerk dokumentiert.
|
||||||
|
</verification>
|
||||||
|
|
||||||
|
<success_criteria>
|
||||||
|
Separater cosinex/DTVP-Adapter parst die Trefferliste (best-effort, D-01) und ist registriert. Erfuellt INGEST-03 / Erfolgskriterium 2.
|
||||||
|
</success_criteria>
|
||||||
|
|
||||||
|
<output>
|
||||||
|
Create `.planning/phases/13-scraping-adapters-cross-source-dedup/13-05-SUMMARY.md` when done
|
||||||
|
</output>
|
||||||
@@ -0,0 +1,105 @@
|
|||||||
|
---
|
||||||
|
phase: 13-scraping-adapters-cross-source-dedup
|
||||||
|
plan: 06
|
||||||
|
type: execute
|
||||||
|
wave: 2
|
||||||
|
depends_on: ["13-01"]
|
||||||
|
files_modified:
|
||||||
|
- apps/api/src/tenders/tenders.controller.ts
|
||||||
|
- apps/api/src/tenders/tenders.controller.spec.ts
|
||||||
|
- apps/web/src/lib/tender-radar-api.ts
|
||||||
|
- apps/web/src/app/(portal)/modules/tender-radar/components/TenderDetail.tsx
|
||||||
|
- apps/web/src/app/(portal)/modules/tender-radar/components/TenderDetail.test.tsx
|
||||||
|
autonomous: true
|
||||||
|
requirements: [SCHEMA-03]
|
||||||
|
must_haves:
|
||||||
|
truths:
|
||||||
|
- "GET /modules/tender-radar/:id liefert tender.sources[] (sourcePortal, sourceUrl, sourceNoticeId) via Prisma include mit."
|
||||||
|
- "TenderDetail rendert die Liste ALLER Quell-Links eines deduplizierten Tenders (Erfolgskriterium 3, Anzeigeseite)."
|
||||||
|
- "Fallback: fehlt sources[], wird auf den bestehenden Einzel-sourceUrl-Block zurueckgefallen."
|
||||||
|
artifacts:
|
||||||
|
- apps/web/src/app/(portal)/modules/tender-radar/components/TenderDetail.tsx
|
||||||
|
key_links:
|
||||||
|
- "getTender include: { sources } -> API-Client-Typ Tender.sources[] -> TenderDetail Multi-Source-Liste."
|
||||||
|
---
|
||||||
|
|
||||||
|
<objective>
|
||||||
|
Macht die Multi-Source-Deduplizierung sichtbar (D-03, Erfolgskriterium 3 Anzeigeteil): der Read-Endpoint `getTender` liefert die `TenderSource`-Liste via Prisma-`include` mit, und `TenderDetail` rendert alle Quell-Portale mit Links statt eines einzelnen `sourceUrl`. Hardcodiertes Deutsch (i18n = Phase 14, NICHT hier).
|
||||||
|
|
||||||
|
Purpose: Ein deduplizierter Tender erscheint EINMAL in der Trefferliste und zeigt in der Detailansicht Links zu allen seinen Quell-Portalen.
|
||||||
|
Output: getTender mit sources-include (+ spec), API-Client-Typ + TenderDetail-Liste (+ component test).
|
||||||
|
</objective>
|
||||||
|
|
||||||
|
<execution_context>
|
||||||
|
@$HOME/.claude/gsd-core/workflows/execute-plan.md
|
||||||
|
@$HOME/.claude/gsd-core/templates/summary.md
|
||||||
|
</execution_context>
|
||||||
|
|
||||||
|
<context>
|
||||||
|
@.planning/PROJECT.md
|
||||||
|
@.planning/ROADMAP.md
|
||||||
|
@.planning/STATE.md
|
||||||
|
@.planning/phases/13-scraping-adapters-cross-source-dedup/13-CONTEXT.md
|
||||||
|
@.planning/phases/13-scraping-adapters-cross-source-dedup/13-RESEARCH.md
|
||||||
|
@apps/api/src/tenders/tenders.controller.ts
|
||||||
|
@apps/web/src/lib/tender-radar-api.ts
|
||||||
|
@apps/web/src/app/(portal)/modules/tender-radar/components/TenderDetail.tsx
|
||||||
|
</context>
|
||||||
|
|
||||||
|
<tasks>
|
||||||
|
|
||||||
|
<task type="auto">
|
||||||
|
<name>Task 1: getTender liefert sources[] via include (Read-Surface)</name>
|
||||||
|
<files>apps/api/src/tenders/tenders.controller.ts, apps/api/src/tenders/tenders.controller.spec.ts</files>
|
||||||
|
<action>
|
||||||
|
Erweitere `getTender` (aktuell `prisma.tender.findUnique({ where: { id } })`) um `include: { sources: { select: { sourcePortal: true, sourceUrl: true, sourceNoticeId: true } } }` (13-RESEARCH.md "Read-Endpoint mit Quell-Links"). Gating bleibt `@UseModule('tender-radar')`, global (kein Tenant-Scoping, T-10-13). ACHTUNG Route-Order-Pitfall (MEMORY / bestehender Kommentar): der `@Get(':id')` bleibt NACH allen statischen Routen — nichts an der Reihenfolge aendern, nur den findUnique-Aufruf. Erweitere `tenders.controller.spec.ts`: getTender liefert das sources-Array mit (gemockter Prisma-Return mit sources[]); NotFound-Fall bleibt.
|
||||||
|
</action>
|
||||||
|
<verify>
|
||||||
|
<automated>pnpm --filter @tessera/api test -- tenders.controller</automated>
|
||||||
|
</verify>
|
||||||
|
<done>getTender include sources; Response enthaelt sources[]; Route-Order unveraendert; Spec gruen.</done>
|
||||||
|
</task>
|
||||||
|
|
||||||
|
<task type="auto">
|
||||||
|
<name>Task 2: API-Client-Typ + TenderDetail Multi-Source-Liste</name>
|
||||||
|
<files>apps/web/src/lib/tender-radar-api.ts, apps/web/src/app/(portal)/modules/tender-radar/components/TenderDetail.tsx, apps/web/src/app/(portal)/modules/tender-radar/components/TenderDetail.test.tsx</files>
|
||||||
|
<action>
|
||||||
|
Ergaenze im API-Client-Typ (`tender-radar-api.ts`) das `Tender` um `sources?: { sourcePortal: string; sourceUrl: string | null; sourceNoticeId: string }[]`. In `TenderDetail.tsx` ersetze den einzelnen sourceUrl-Block durch eine Liste (13-RESEARCH.md "TenderDetail: Multi-Source-Liste"): wenn `tender.sources?.length`, rendere `<ul>` mit einem `<a>` je Quelle (`target="_blank" rel="noopener noreferrer"` — Sicherheits-Muster, kein dangerouslySetInnerHTML, React escaped Text), Label ueber eine kleine `portalLabel(sourcePortal)`-Map (hardcodiertes Deutsch: z.B. 'doe-opendata'->'DÖE', 'tender24'->'tender24', 'cosinex-dtvp'->'DTVP'). Fallback: fehlt/leer sources[], bestehender Einzel-sourceUrl-Block. Erweitere `TenderDetail.test.tsx`: rendert mehrere Quell-Links bei sources[]; Fallback bei fehlendem sources.
|
||||||
|
</action>
|
||||||
|
<verify>
|
||||||
|
<automated>cd apps/web && npx tsc --noEmit && pnpm --filter web test -- TenderDetail</automated>
|
||||||
|
</verify>
|
||||||
|
<done>TenderDetail zeigt Liste aller Quell-Links; Fallback bei fehlendem sources; tsc + component test gruen.</done>
|
||||||
|
</task>
|
||||||
|
|
||||||
|
</tasks>
|
||||||
|
|
||||||
|
<threat_model>
|
||||||
|
## Trust Boundaries
|
||||||
|
|
||||||
|
| Boundary | Description |
|
||||||
|
|----------|-------------|
|
||||||
|
| Portal-HTML (Titel/Buyer/URL) -> Frontend | Extern gescrapte Strings landen in der Detailansicht (untrusted). |
|
||||||
|
|
||||||
|
## STRIDE Threat Register
|
||||||
|
|
||||||
|
| Threat ID | Category | Component | Severity | Disposition | Mitigation Plan |
|
||||||
|
|-----------|----------|-----------|----------|-------------|-----------------|
|
||||||
|
| T-13-06-01 | Tampering (Stored XSS) | Portal-Strings in TenderDetail | medium | mitigate | React rendert als Text (default-escaped), kein dangerouslySetInnerHTML (V5). |
|
||||||
|
| T-13-06-02 | Tampering | externer Quell-Link | low | mitigate | rel="noopener noreferrer" + target="_blank" (bestehendes Muster). |
|
||||||
|
| T-13-06-03 | Info Disclosure | Tenant-Leak ueber Read-Endpoint | low | mitigate | @UseModule('tender-radar'), Tender global, kein Tenant-Scoping-Wechsel. |
|
||||||
|
</threat_model>
|
||||||
|
|
||||||
|
<verification>
|
||||||
|
- `pnpm --filter @tessera/api test -- tenders.controller` gruen (sources im Response).
|
||||||
|
- `cd apps/web && npx tsc --noEmit && pnpm --filter web test -- TenderDetail` gruen.
|
||||||
|
- Manuell (13-VALIDATION.md): Detailansicht eines Tenders mit >1 TenderSource zeigt mehrere Links.
|
||||||
|
</verification>
|
||||||
|
|
||||||
|
<success_criteria>
|
||||||
|
Read-Endpoint liefert sources[]; Detailansicht rendert alle Quell-Links; Fallback fuer Alt-Tender. TrAegt Erfolgskriterium 3 (Anzeige aller Quell-Links).
|
||||||
|
</success_criteria>
|
||||||
|
|
||||||
|
<output>
|
||||||
|
Create `.planning/phases/13-scraping-adapters-cross-source-dedup/13-06-SUMMARY.md` when done
|
||||||
|
</output>
|
||||||
@@ -0,0 +1,104 @@
|
|||||||
|
---
|
||||||
|
phase: 13
|
||||||
|
slug: scraping-adapters-cross-source-dedup
|
||||||
|
status: draft
|
||||||
|
nyquist_compliant: true
|
||||||
|
wave_0_complete: false
|
||||||
|
created: 2026-07-23
|
||||||
|
---
|
||||||
|
|
||||||
|
# Phase 13 — Validation Strategy
|
||||||
|
|
||||||
|
> Per-phase validation contract for feedback sampling during execution.
|
||||||
|
> Extracted from 13-RESEARCH.md § Validation Architecture (Test framework, Requirements→Test map, Sampling rate, Wave-0 gaps).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Test Infrastructure
|
||||||
|
|
||||||
|
| Property | Value |
|
||||||
|
|----------|-------|
|
||||||
|
| **Framework** | Vitest (API 3.x, Web 4.x — beide vorhanden, kein Install) |
|
||||||
|
| **Config file** | `apps/api/vitest.config.ts` + `apps/web` vitest (bestehend — nicht von dieser Phase geaendert) |
|
||||||
|
| **Quick run (API slice)** | `pnpm --filter @tessera/api test -- <spec>` |
|
||||||
|
| **Quick run (API tenders)** | `pnpm --filter @tessera/api test -- src/tenders` |
|
||||||
|
| **Quick run (Web slice)** | `pnpm --filter web test -- <spec>` |
|
||||||
|
| **Full suite** | `pnpm test` (turbo) bzw. `pnpm --filter @tessera/api test && pnpm --filter web test` |
|
||||||
|
| **Estimated runtime** | ~10-15 s (scoped) |
|
||||||
|
|
||||||
|
Ein neues Package: `node-html-parser` (oder `cheerio`) — NUR nach blockierendem Package-Legitimacy-Checkpoint (Plan 13-04). Der Kern (Registry, Fingerprint, Dedup, TenderSource, Backfill) braucht KEIN Scraping-Package und ist unabhaengig davon voll testbar (D-01). `prisma`/`crypto`/`fetch` bereits vorhanden.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Sampling Rate
|
||||||
|
|
||||||
|
- **Nach jedem Task-Commit:** betroffene Slice-Spec (`pnpm --filter @tessera/api test -- <spec>`).
|
||||||
|
- **Nach jeder Plan-Wave:** `pnpm --filter @tessera/api test && pnpm --filter web test`.
|
||||||
|
- **Vor `/gsd-verify-work`:** `pnpm test` gruen + Backfill-Migration lokal gegen die Dev-DB verifiziert.
|
||||||
|
- **Max Feedback-Latenz:** ~15 s (scoped).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Per-Task Verification Map
|
||||||
|
|
||||||
|
| Task ID | Plan | Wave | Requirement | Threat Ref | Secure Behavior | Test Type | Automated Command | File Exists | Status |
|
||||||
|
|---------|------|------|-------------|------------|-----------------|-----------|-------------------|-------------|--------|
|
||||||
|
| 13-01-01 | 01 | 1 | SCHEMA-03 | T-13-01-01/03 | TenderSource (1:n, @@unique[sourcePortal,sourceNoticeId]) + Tender.fingerprint additiv; 2-Schritt-Migration + Backfill lokal | integration | `cd apps/api && npx prisma validate && npx prisma generate` (+ psql-Count, manual) | ⚠️ migration | ⬜ pending |
|
||||||
|
| 13-01-02 | 01 | 1 | SCHEMA-03 | — | tenderFingerprint pure, NULL-tolerant (Wert/Frist NULL -> Match), kollisionsarm (Titel im Hash), sha256 | unit | `pnpm --filter @tessera/api test -- tender-fingerprint` | ❌ W0 | ⬜ pending |
|
||||||
|
| 13-01-03 | 01 | 1 | SCHEMA-03 | — | SourceType-Union offen + NormalizedTenderFields.fingerprint; typecheck | typecheck | `cd apps/api && npx tsc --noEmit -p tsconfig.json` | n/a | ⬜ pending |
|
||||||
|
| 13-02-01 | 02 | 1 | INGEST-07 | — | Adapter-Interface um portals[] generalisiert; DoeOpenDataAdapter deklariert portals | typecheck | `cd apps/api && npx tsc --noEmit -p tsconfig.json` | n/a | ⬜ pending |
|
||||||
|
| 13-02-02 | 02 | 1 | INGEST-07 | T-13-02-01/02 | SourceRegistry.register wirft DeniedPortalError fuer vergabe24 UND aumass (Erfolgskriterium 4); legitime Registrierung/get funktionieren | unit | `pnpm --filter @tessera/api test -- source-registry` | ❌ W0 | ⬜ pending |
|
||||||
|
| 13-03-01 | 03 | 2 | SCHEMA-03 | T-13-03-02/03/04 | 3-Stufen-Resolver OCID→source:noticeId→fingerprint; inert bei 1 Quelle (D-05), Merge->TenderSource anhaengen ab 2 (D-03); plain Prisma (kein RLS) | unit | `pnpm --filter @tessera/api test -- tender-dedup` | ❌ W0 | ⬜ pending |
|
||||||
|
| 13-03-02 | 03 | 2 | SCHEMA-03 | T-13-03-01/02 | pollDueSources fan-out (findMany isActive), catch-per-source, dedupActive=activePortalCount>=2, Delta-Boundary erhalten | unit | `pnpm --filter @tessera/api test -- tender-ingestion.service` | ⚠️ extend | ⬜ pending |
|
||||||
|
| 13-03-03 | 03 | 2 | SCHEMA-03 | — | Modul-Wiring: SourceRegistry + TenderDedupService Provider + DoeOpenDataAdapter bei Boot registriert; DI aufloesbar | typecheck+suite | `cd apps/api && npx tsc --noEmit && pnpm --filter @tessera/api test -- src/tenders` | n/a | ⬜ pending |
|
||||||
|
| 13-06-01 | 06 | 2 | SCHEMA-03 | T-13-06-03 | getTender include sources[]; Route-Order unveraendert (:id zuletzt) | integration | `pnpm --filter @tessera/api test -- tenders.controller` | ⚠️ extend | ⬜ pending |
|
||||||
|
| 13-06-02 | 06 | 2 | SCHEMA-03 | T-13-06-01/02 | TenderDetail rendert alle Quell-Links (Text-escaped, rel=noopener), Fallback bei fehlendem sources | component | `cd apps/web && npx tsc --noEmit && pnpm --filter web test -- TenderDetail` | ⚠️ extend | ⬜ pending |
|
||||||
|
| 13-04-CP | 04 | 3 | INGEST-02 | T-13-04-SC | Package-Legitimacy-Checkpoint (blocking-human) VOR Install node-html-parser/cheerio | manual-gate | Human verify via `gsd-tools query package-legitimacy check --ecosystem npm node-html-parser cheerio` | n/a | ⬜ pending |
|
||||||
|
| 13-04-01 | 04 | 3 | INGEST-02 | T-13-04-01/02/03 | NetServer-Adapter config-getrieben (3 Portale), parst <table>-Fixture, []-Fallback, SSRF-sichere Konstanten, 15s-Timeout | unit (Fixture) | `pnpm --filter @tessera/api test -- netserver.adapter` | ❌ W0 | ⬜ pending |
|
||||||
|
| 13-04-02 | 04 | 3 | INGEST-02 | — | NetServerAdapter Provider + bei Boot registriert (nicht denylisted); DI aufloesbar | typecheck+suite | `cd apps/api && npx tsc --noEmit && pnpm --filter @tessera/api test -- src/tenders` | n/a | ⬜ pending |
|
||||||
|
| 13-05-01 | 05 | 4 | INGEST-03 | T-13-05-01/02/03 | cosinex-Adapter getrennt (nicht NetServer), parst Satellite-Fixture ODER dokumentiert needs-JS -> []; []-Fallback, SSRF-sicher | unit (Fixture) | `pnpm --filter @tessera/api test -- cosinex.adapter` | ❌ W0 | ⬜ pending |
|
||||||
|
| 13-05-02 | 05 | 4 | INGEST-03 | — | CosinexAdapter Provider + bei Boot registriert; DI aufloesbar | typecheck+suite | `cd apps/api && npx tsc --noEmit && pnpm --filter @tessera/api test -- src/tenders` | n/a | ⬜ pending |
|
||||||
|
|
||||||
|
*Status: ⬜ pending · ✅ green · ❌ red · ⚠️ flaky*
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Wave 0 Requirements
|
||||||
|
|
||||||
|
- [ ] `apps/api/src/tenders/tender-fingerprint.spec.ts` — SCHEMA-03 NULL-Toleranz (Wert+Frist NULL -> gleicher Fingerprint), Kollisions-Fall (anderer Titel -> anderer Hash), Umlaut-/CPV-Division-Normalisierung, valueBucket, deterministischer sha256 (Plan 13-01 Task 2, RED-first).
|
||||||
|
- [ ] `apps/api/src/tenders/source-registry.spec.ts` — INGEST-07/Erfolgskriterium 4: register() wirft DeniedPortalError fuer vergabe24 UND aumass; gemischtes portals-Array abgelehnt; legitime Registrierung + get/activeAdapters (Plan 13-02 Task 2, RED-first).
|
||||||
|
- [ ] `apps/api/src/tenders/tender-dedup.service.spec.ts` — D-04/D-05: inert bei dedupActive=false (2 Tender), OCID-/noticeId-/Fingerprint-Match haengt TenderSource an (created=false), kein Match -> create (created=true); Prisma gemockt (Plan 13-03 Task 1, RED-first).
|
||||||
|
- [ ] `apps/api/src/tenders/adapters/netserver.adapter.spec.ts` + `__fixtures__/netserver-search.html` — INGEST-02: parst live-gecapturte <table>-Fixture -> RawTenderRecord[] mit korrektem sourcePortal je Zeile; []-Fallback bei kaputtem HTML; Fetch gemockt (Plan 13-04 Task 1).
|
||||||
|
- [ ] `apps/api/src/tenders/adapters/cosinex.adapter.spec.ts` + `__fixtures__/cosinex-search.html` — INGEST-03: parst Satellite-Fixture ODER dokumentierter needs-JS-Leer-Fall; []-Fallback; Fetch gemockt (Plan 13-05 Task 1).
|
||||||
|
- [ ] (extend) `apps/api/src/tenders/tender-ingestion.service.spec.ts` — fan-out ueber mehrere aktive Configs, catch-per-source (Quelle A wirft -> B laeuft), dedupActive-Gate, matchDelta nur mit created-IDs (Plan 13-03 Task 2).
|
||||||
|
- [ ] (extend) `apps/api/src/tenders/tenders.controller.spec.ts` — getTender liefert sources[] mit; NotFound bleibt (Plan 13-06 Task 1).
|
||||||
|
- [ ] (extend) `apps/web/.../components/TenderDetail.test.tsx` — mehrere Quell-Links bei sources[]; Fallback bei fehlendem sources (Plan 13-06 Task 2).
|
||||||
|
|
||||||
|
Vitest-Framework existiert bereits (api + web) — keine Framework-Installation. HTML-Fixtures live speichern fuer stabile Adapter-Tests.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Manual-Only Verifications
|
||||||
|
|
||||||
|
| Behavior | Requirement | Why Manual | Test Instructions |
|
||||||
|
|----------|-------------|------------|-------------------|
|
||||||
|
| Migration `20260723120000_add_tender_source` + Backfill lokal angewendet | SCHEMA-03 | DB hat keinen Host-Port; lokale Anwendung via `docker compose exec -T db psql -U tessera -d tessera_dev` (MEMORY); kein Docker-Deploy Testserver durch Claude | `SELECT to_regclass('public."TenderSource"')` ≠ NULL; `SELECT count(*) FROM "TenderSource"` == `SELECT count(*) FROM "Tender"` (~2851); `SELECT count(*) FROM "Tender" WHERE fingerprint IS NOT NULL` == Bestand; `prisma migrate status` gruen |
|
||||||
|
| Live-Poll NetServer (tender24) liefert reale Records | INGEST-02 | Live-Portal-HTML nicht im Unit-Test beobachtbar (Fixture-getestet); Selektor-Vollstaendigkeit nur live pruefbar | Optionaler Aktivierungs-/Poll-Lauf gegen `tender24.de/NetServer/PublicationSearchControllerServlet` -> >0 RawTenderRecord ODER "needs-JS/blocked, deferred" im SUMMARY dokumentiert (D-01) |
|
||||||
|
| Live-Fetch cosinex (dtvp welcome.do) parsebar | INGEST-03 | JS-Rendering-Grad nur live feststellbar (Open Question 1); kein Headless-Browser im Stack | Server-gerenderte Trefferliste parsebar -> Records; sonst "needs-JS, deferred" dokumentiert, Adapter-Skelett + Test bleiben gruen |
|
||||||
|
| Cross-Source-Dedup end-to-end (EIN Eintrag, alle Links) | SCHEMA-03 / Erfolgskriterium 3 | Echter Merge braucht 2 aktive Quellen mit ueberlappender Ausschreibung; Unit deckt Logik, finale Bestaetigung am realen Lauf | DÖE + NetServer aktiv, eine Ausschreibung in beiden -> erscheint EINMAL in der Trefferliste; Detailansicht listet beide Quell-Links; Tender-Anzahl steigt nicht doppelt |
|
||||||
|
| Dedup-Inertheit mit nur DÖE (kein faelschlicher Merge) | SCHEMA-03 / Erfolgskriterium 3 / D-05 | Beobachtung am realen Bestand (~2851 Tender) | Nur DÖE aktiv -> Tender-Anzahl bleibt stabil nach Poll (kein Fingerprint-Merge, activePortalCount < 2) |
|
||||||
|
| Denylist-Refusal strukturell | INGEST-07 / Erfolgskriterium 4 | Automatisiert durch 13-02-02; finale Bestaetigung, dass kein Boot-Pfad vergabe24/aumass registriert | Kein Poll-/Registrierungs-Pfad fuer vergabe24/aumass existiert; register-Versuch wirft (Test gruen) |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Validation Sign-Off
|
||||||
|
|
||||||
|
- [x] Alle Tasks haben `<automated>` verify oder eine dokumentierte Wave-0-/Manual-Gate-Abhaengigkeit
|
||||||
|
- [x] Sampling-Kontinuitaet: keine 3 aufeinanderfolgenden Tasks ohne automated verify
|
||||||
|
- [x] Wave 0 deckt alle MISSING-Referenzen (Spec-Scaffolds; RED-first fuer pure-logic Units)
|
||||||
|
- [x] Keine watch-mode-Flags (alle nutzen `vitest run` via `pnpm --filter … test`)
|
||||||
|
- [x] Feedback-Latenz < 15 s (scoped runs)
|
||||||
|
- [x] `nyquist_compliant: true` in Frontmatter gesetzt
|
||||||
|
- [x] Package-Legitimacy-Gate (node-html-parser/cheerio) als blocking-human Checkpoint VOR Install (13-04)
|
||||||
|
|
||||||
|
**Approval:** draft — pending execution (wave_0_complete: false)
|
||||||
Reference in New Issue
Block a user