docs(10-01): complete tender-radar foundation plan
This commit is contained in:
@@ -20,7 +20,7 @@
|
||||
|
||||
### SCHEMA — Normalisierung & Deduplizierung
|
||||
|
||||
- [ ] **SCHEMA-01**: Alle Quellen werden in ein einheitliches, OCDS-orientiertes Ausschreibungs-Schema normalisiert (Titel, Auftraggeber, CPV-Codes, Region/PLZ, Frist, geschätzter Wert, Verfahrensart, Quell-URL, Rohdaten). Ausschreibungsdaten sind plattform-global, nicht mandantengebunden.
|
||||
- [x] **SCHEMA-01**: Alle Quellen werden in ein einheitliches, OCDS-orientiertes Ausschreibungs-Schema normalisiert (Titel, Auftraggeber, CPV-Codes, Region/PLZ, Frist, geschätzter Wert, Verfahrensart, Quell-URL, Rohdaten). Ausschreibungsdaten sind plattform-global, nicht mandantengebunden.
|
||||
- [ ] **SCHEMA-02**: Das System erkennt Änderungen an bereits importierten Ausschreibungen (Fristverlängerung, Aufhebung) über einen Content-Hash und aktualisiert den Datensatz.
|
||||
- [ ] **SCHEMA-03**: Dieselbe Ausschreibung aus mehreren Quellen wird zu einem Eintrag mit mehreren Quell-Links dedupliziert (Schlüssel: OCID → Quelle:NoticeId → Fuzzy-Fingerprint aus Auftraggeber+Titel+CPV+Frist+Wert). Dedup greift erst ab der zweiten aktiven Quelle.
|
||||
|
||||
@@ -79,7 +79,7 @@
|
||||
| CONFIG-01 | Phase 10 | Pending |
|
||||
| INGEST-01 | Phase 10 | Pending |
|
||||
| INGEST-06 | Phase 10 | Pending |
|
||||
| SCHEMA-01 | Phase 10 | Pending |
|
||||
| SCHEMA-01 | Phase 10 | Complete |
|
||||
| SCHEMA-02 | Phase 10 | Pending |
|
||||
| FILTER-01 | Phase 11 | Pending |
|
||||
| FILTER-02 | Phase 11 | Pending |
|
||||
|
||||
@@ -344,11 +344,11 @@ Plans:
|
||||
4. DÖE is polled once on a shared, admin-configurable interval regardless of how many tenants have the module active -- never once per tenant (poll-once-fan-out-many, not the DKV single-tenant `findFirst()` pattern)
|
||||
5. Activating the module for a second tenant does not duplicate ingestion, re-trigger a redundant DÖE poll, or interfere with the first tenant's data
|
||||
|
||||
**Plans**: 6 plans
|
||||
**Plans**: 1/6 plans executed
|
||||
|
||||
**Wave 1**
|
||||
|
||||
- [ ] 10-01-PLAN.md — Foundation: install fast-xml-parser/adm-zip/csv-parse (adm-zip supply-chain checkpoint), add global Tender + TenderSourcePollConfig models, [BLOCKING] migration (SCHEMA-01)
|
||||
- [x] 10-01-PLAN.md — Foundation: install fast-xml-parser/adm-zip/csv-parse (adm-zip supply-chain checkpoint), add global Tender + TenderSourcePollConfig models, [BLOCKING] migration (SCHEMA-01)
|
||||
|
||||
**Wave 2** *(blocked on Wave 1 completion)*
|
||||
|
||||
@@ -453,7 +453,7 @@ Phases execute in numeric order: 1 -> 2 -> 3 -> 4 -> 5 -> 6 -> 7 -> 8 -> 9 -> 10
|
||||
| 7. DKV Fleet Module | 6/6 | Complete | 2026-06-27 |
|
||||
| 8. Dashboard Widgets Vollimplementierung | 4/4 | Complete | 2026-07-01 |
|
||||
| 9. Cert Manager Module | 6/6 | Complete | 2026-07-02 |
|
||||
| 10. Ausschreibungs-Radar Foundation & DÖE Ingestion | 0/6 | Not started | - |
|
||||
| 10. Ausschreibungs-Radar Foundation & DÖE Ingestion | 1/6 | In Progress| |
|
||||
| 11. Filter Engine, Results UI & Saved Searches | 0/TBD | Not started | - |
|
||||
| 12. Tender Notifications | 0/TBD | Not started | - |
|
||||
| 13. Scraping Adapters & Cross-Source Deduplication | 0/TBD | Not started | - |
|
||||
|
||||
+9
-7
@@ -5,15 +5,15 @@ milestone_name: Ausschreibungs-Radar
|
||||
current_phase: 10
|
||||
current_phase_name: ausschreibungs-radar-foundation-d-e-ingestion
|
||||
status: executing
|
||||
stopped_at: Phase 10 context gathered
|
||||
last_updated: "2026-07-21T08:24:17.585Z"
|
||||
stopped_at: Completed 10-01-PLAN.md
|
||||
last_updated: "2026-07-21T08:39:50.924Z"
|
||||
last_activity: 2026-07-21
|
||||
last_activity_desc: Phase 10 execution started
|
||||
progress:
|
||||
total_phases: 14
|
||||
completed_phases: 8
|
||||
total_plans: 46
|
||||
completed_plans: 39
|
||||
completed_plans: 40
|
||||
percent: 57
|
||||
---
|
||||
|
||||
@@ -29,8 +29,8 @@ See: .planning/PROJECT.md (updated 2026-07-17)
|
||||
## Current Position
|
||||
|
||||
Phase: 10 (ausschreibungs-radar-foundation-d-e-ingestion) — EXECUTING
|
||||
Plan: 1 of 6
|
||||
Status: Executing Phase 10
|
||||
Plan: 2 of 6
|
||||
Status: Ready to execute
|
||||
Last activity: 2026-07-21 — Phase 10 execution started
|
||||
|
||||
Progress: [░░░░░░░░░░] 0%
|
||||
@@ -71,6 +71,7 @@ Progress: [░░░░░░░░░░] 0%
|
||||
| Phase 09-cert-manager-module P04 | 4 | 3 tasks | 5 files |
|
||||
| Phase 09-cert-manager-module P05 | 8 | 3 tasks | 6 files |
|
||||
| Phase 09 P06 | 7 | 3 tasks | 7 files |
|
||||
| Phase 10 P01 | 15min | 3 tasks | 4 files |
|
||||
|
||||
## Accumulated Context
|
||||
|
||||
@@ -138,6 +139,7 @@ Recent decisions affecting current work:
|
||||
- [Phase ?]: 09-03
|
||||
- [Phase ?]: splitCerts PEM path reuses parsePemChain; P7B sniffs first bytes
|
||||
- [Phase ?]: splitCerts format crt comparison removed — detectFormat returns pem|der|pfx|p7b only
|
||||
- [Phase 10]: Tender ist plattform-global (D-03): kein tenantId, keine RLS/forTenant() — Verhindert versehentliches Ausblenden globaler Daten fuer einen zweiten Mandanten
|
||||
|
||||
### Pending Todos
|
||||
|
||||
@@ -175,7 +177,7 @@ Items acknowledged and carried forward from previous milestone close:
|
||||
|
||||
## Session Continuity
|
||||
|
||||
Last session: 2026-07-17T11:49:35.920Z
|
||||
Stopped at: Phase 10 context gathered
|
||||
Last session: 2026-07-21T08:39:50.913Z
|
||||
Stopped at: Completed 10-01-PLAN.md
|
||||
Resume file: .planning/phases/10-ausschreibungs-radar-foundation-d-e-ingestion/10-CONTEXT.md
|
||||
Last activity: 2026-07-14 - Built LDAP per-user exclude/denylist filter (9d1323f), migration applied on live DB, verified via Playwright: sync deactivated 4 excluded service accounts (administrator/krbtgt/guest/dns-ldap), 2 real LDAP users stay active, 0 wrongly created
|
||||
|
||||
@@ -0,0 +1,153 @@
|
||||
---
|
||||
phase: 10-ausschreibungs-radar-foundation-d-e-ingestion
|
||||
plan: 01
|
||||
subsystem: database
|
||||
tags: [prisma, postgres, tender-radar, ingestion, adm-zip, fast-xml-parser, csv-parse, multi-tenant]
|
||||
|
||||
# Dependency graph
|
||||
requires:
|
||||
- phase: 09 (module-registry / dkv patterns)
|
||||
provides: Prisma schema conventions, handwritten-migration workflow, tenant-extension (forTenant) reference
|
||||
provides:
|
||||
- Global (tenant-agnostic) Tender table with OCDS-oriented nullable deadline/value fields (SCHEMA-01)
|
||||
- Singleton TenderSourcePollConfig model (sourceType @unique) as INGEST-06 foundation
|
||||
- fast-xml-parser, adm-zip, csv-parse installed in @tessera/api
|
||||
- Applied migration + regenerated Prisma client types for Tender / TenderSourcePollConfig
|
||||
affects: [10-02 module registration, 10-03 adapter/normalizer, 10-04 ingestion/scheduler, 10-05 controller/DTOs, phase 11 saved searches]
|
||||
|
||||
# Tech tracking
|
||||
tech-stack:
|
||||
added: [fast-xml-parser@5, adm-zip@0.6, csv-parse@7]
|
||||
patterns:
|
||||
- "Global RLS-exempt reference table (no tenantId, no forTenant()) for platform-wide data (D-03)"
|
||||
- "Singleton-per-source config via fixed-slug @unique column (mirrors DkvModuleConfig.tenantId @unique)"
|
||||
- "Handwritten timestamped migration with IF NOT EXISTS guards (repo convention)"
|
||||
|
||||
key-files:
|
||||
created:
|
||||
- apps/api/prisma/migrations/20260721120000_add_tender_radar/migration.sql
|
||||
modified:
|
||||
- apps/api/prisma/schema.prisma
|
||||
- apps/api/package.json
|
||||
|
||||
key-decisions:
|
||||
- "Tender carries NO tenantId column and is NOT wrapped by forTenant()/RLS — platform-global data (D-03)"
|
||||
- "deadlineAt / estimatedValue are nullable by mandate (RESEARCH Pattern 4), not optional hardening"
|
||||
- "TenderSourcePollConfig.lastIngestedDay is a day-cursor (DateTime?), not a poll timestamp (RESEARCH Pattern 1)"
|
||||
- "sourceType @unique makes the singleton-per-source intent explicit and enforceable at the DB level"
|
||||
|
||||
patterns-established:
|
||||
- "Global reference table: no tenantId, no @@index([tenantId]), plain PrismaService client (no RLS wrap)"
|
||||
- "Singleton config row keyed by fixed slug via @unique"
|
||||
|
||||
requirements-completed: [SCHEMA-01]
|
||||
|
||||
coverage:
|
||||
- id: D1
|
||||
description: "Global Tender table exists with OCDS-oriented nullable deadline/value fields and no tenantId (SCHEMA-01, D-03)"
|
||||
requirement: "SCHEMA-01"
|
||||
verification:
|
||||
- kind: automated_ui
|
||||
ref: "grep -c 'model Tender' schema.prisma >=1; awk Tender-block | grep -c tenantId == 0"
|
||||
status: pass
|
||||
- kind: integration
|
||||
ref: "prisma migrate status -> 'Database schema is up to date!'; prisma.tender.count() == 0"
|
||||
status: pass
|
||||
human_judgment: false
|
||||
- id: D2
|
||||
description: "Singleton TenderSourcePollConfig model (sourceType @unique) present and live (INGEST-06 foundation)"
|
||||
verification:
|
||||
- kind: integration
|
||||
ref: "prisma.tenderSourcePollConfig.count() == 0 without error; grep -c 'model TenderSourcePollConfig' == 1"
|
||||
status: pass
|
||||
human_judgment: false
|
||||
- id: D3
|
||||
description: "fast-xml-parser, adm-zip, csv-parse installed and resolvable in @tessera/api"
|
||||
verification:
|
||||
- kind: integration
|
||||
ref: "node -e require('adm-zip');require('fast-xml-parser');require('csv-parse/sync') -> packages OK"
|
||||
status: pass
|
||||
human_judgment: false
|
||||
|
||||
# Metrics
|
||||
duration: ~15min (aktive Ausführung, exkl. Checkpoint-Wartezeit)
|
||||
completed: 2026-07-21
|
||||
status: complete
|
||||
---
|
||||
|
||||
# Phase 10 Plan 01: Ausschreibungs-Radar Foundation & Dependencies Summary
|
||||
|
||||
**Global (tenant-agnostic) `Tender`-Referenztabelle + singleton `TenderSourcePollConfig` in Prisma/Postgres, drei Ingestion-Pakete (fast-xml-parser, adm-zip, csv-parse) installiert, Migration angewendet — Datenfundament für das gesamte DÖE-Ingestion-Modul.**
|
||||
|
||||
## Performance
|
||||
|
||||
- **Duration:** ~15 min aktive Ausführung (exkl. Checkpoint- und Infrastruktur-Wartezeit)
|
||||
- **Started:** 2026-07-21
|
||||
- **Completed:** 2026-07-21
|
||||
- **Tasks:** 3 (1 Checkpoint + 2 auto)
|
||||
- **Files modified:** 4 (schema.prisma, package.json, pnpm-lock.yaml, migration.sql)
|
||||
|
||||
## Accomplishments
|
||||
- Globale `Tender`-Tabelle (OCDS-orientiert, nullable `deadlineAt`/`estimatedValue`) ohne `tenantId` — D-03 Invariante im Schema verankert (SCHEMA-01).
|
||||
- Singleton `TenderSourcePollConfig` mit `sourceType @unique` als Fundament für die geteilte, mandantensichere Poll-Konfiguration (INGEST-06).
|
||||
- Drei Ingestion-Bibliotheken installiert und auflösbar (fast-xml-parser, adm-zip, csv-parse) — adm-zip nach blockierendem Supply-Chain-Checkpoint vom User freigegeben.
|
||||
- Handgeschriebene, timestamped Migration angewendet; Prisma-Client-Typen regeneriert; DB-Schema up to date.
|
||||
|
||||
## Task Commits
|
||||
|
||||
1. **Task 1: adm-zip package legitimacy verification** — kein Commit (blocking-human Checkpoint, vom User "approved": cthackers/adm-zip, MIT, seit 2012, Mio. DLs/Woche)
|
||||
2. **Task 2: Install packages + Tender/TenderSourcePollConfig models** — `713b1eb` (feat)
|
||||
3. **Task 3: [BLOCKING] Apply tender-radar migration** — kein separater Code-Commit; Migrations-SQL ist Teil von `713b1eb`, Anwendung auf die laufende DB durch den Coordinator (Stack lokal gestartet)
|
||||
|
||||
**Plan metadata:** siehe finalen `docs(10-01)`-Commit.
|
||||
|
||||
## Files Created/Modified
|
||||
- `apps/api/prisma/schema.prisma` — Modelle `Tender` (global, kein tenantId) + `TenderSourcePollConfig` (singleton) ergänzt
|
||||
- `apps/api/prisma/migrations/20260721120000_add_tender_radar/migration.sql` — handgeschriebene additive Migration (2 Tabellen, 4 Indexe, 2 Unique-Indexe)
|
||||
- `apps/api/package.json` — fast-xml-parser, adm-zip, csv-parse als Dependencies
|
||||
- `pnpm-lock.yaml` — Lockfile-Aktualisierung
|
||||
|
||||
## Decisions Made
|
||||
- **Tender ist plattform-global (D-03):** kein `tenantId`, kein `@@index([tenantId])`, keine RLS/`forTenant()`-Umhüllung. Verhindert, dass ein späterer `where: { tenantId }`-Reflex globale Daten für einen zweiten Mandanten unsichtbar macht (Threat T-10-01, mitigiert).
|
||||
- **Nullable deadline/value verpflichtend:** `deadlineAt` und `estimatedValue` sind laut RESEARCH Pattern 4 häufig leer — nullable ist Korrektheitsanforderung, kein optionales Hardening.
|
||||
- **Day-cursor statt Timestamp:** `TenderSourcePollConfig.lastIngestedDay` ist ein Tages-Cursor (`DateTime?`), passend zum DÖE-Daily-Batch-ZIP-Modell (RESEARCH Pattern 1).
|
||||
- **Singleton via `sourceType @unique`:** macht die "eine Konfig pro Quelle"-Absicht auf DB-Ebene explizit und erzwingbar.
|
||||
|
||||
## Deviations from Plan
|
||||
|
||||
### Auto-fixed Issues
|
||||
|
||||
**1. [Rule 1 - Bug] Kommentar-Wortlaut verletzte die tenantId-Assertion**
|
||||
- **Found during:** Task 2 (Schema-Verifikation)
|
||||
- **Issue:** Der erläuternde Kommentar im Tender-Modell enthielt das Token `tenantId` ("Deliberately NO tenantId field ..."), wodurch die Acceptance-Assertion `grep -c "tenantId"` im Tender-Block `1` statt `0` lieferte — ein False-Positive gegen die D-03-Invariante.
|
||||
- **Fix:** Kommentar umformuliert zu "NO tenant column and NO tenant index — this is global data (D-03)"; Aussage bleibt erhalten, Token entfernt.
|
||||
- **Files modified:** apps/api/prisma/schema.prisma
|
||||
- **Verification:** `awk Tender-block | grep -c "tenantId"` == 0; `tsc --noEmit` exit 0.
|
||||
- **Committed in:** 713b1eb (Task-2-Commit)
|
||||
|
||||
---
|
||||
|
||||
**Total deviations:** 1 auto-fixed (1 Bug)
|
||||
**Impact on plan:** Reine Kommentar-Korrektur zur Erfüllung der Design-Invarianten-Assertion. Kein Scope-Creep, keine Verhaltensänderung.
|
||||
|
||||
## Issues Encountered
|
||||
- **Task 3 blockiert durch offline DB-Stack:** Der Tessera-Docker-Stack (`tessera-ctl-db-1/-api-1/-web-1`) war beendet, die Postgres-DB liegt auf einem `internal: true`-Netzwerk ohne Host-Port-Mapping, und `DATABASE_URL` war vom Host nicht zugänglich. Als blockierendes Infrastruktur-Gate an den Coordinator/User zurückgegeben. **Auflösung:** Coordinator startete den lokalen Stack und wandte `prisma migrate deploy` an. Verifikation (auch eigenständig via `DATABASE_URL=...@172.19.0.2:5432` gegengeprüft): `prisma migrate status` → "Database schema is up to date!", `Tender`-Count = 0, `TenderSourcePollConfig`-Count = 0 — beide Tabellen live, keine Fehler.
|
||||
|
||||
## User Setup Required
|
||||
None — keine externe Service-Konfiguration nötig. (Hinweis: Migrations werden in diesem Repo bei laufendem Stack angewendet; `up`/`rebuild` liegt beim User.)
|
||||
|
||||
## Next Phase Readiness
|
||||
- Datenfundament steht: `Tender` + `TenderSourcePollConfig` live, Prisma-Client-Typen generiert, Pakete installiert.
|
||||
- Plan 10-02 (TendersModule + seedTendersModule) kann direkt aufsetzen.
|
||||
- Keine offenen Blocker.
|
||||
|
||||
## Self-Check: PASSED
|
||||
|
||||
- FOUND: apps/api/prisma/schema.prisma
|
||||
- FOUND: apps/api/prisma/migrations/20260721120000_add_tender_radar/migration.sql
|
||||
- FOUND: .planning/phases/10-ausschreibungs-radar-foundation-d-e-ingestion/10-01-SUMMARY.md
|
||||
- FOUND commit: 713b1eb
|
||||
|
||||
---
|
||||
*Phase: 10-ausschreibungs-radar-foundation-d-e-ingestion*
|
||||
*Completed: 2026-07-21*
|
||||
Reference in New Issue
Block a user