docs(10): create phase plan (5 plans, DÖE ingestion foundation)
This commit is contained in:
+24
-2
@@ -344,7 +344,29 @@ Plans:
|
||||
4. DÖE is polled once on a shared, admin-configurable interval regardless of how many tenants have the module active -- never once per tenant (poll-once-fan-out-many, not the DKV single-tenant `findFirst()` pattern)
|
||||
5. Activating the module for a second tenant does not duplicate ingestion, re-trigger a redundant DÖE poll, or interfere with the first tenant's data
|
||||
|
||||
**Plans**: TBD
|
||||
**Plans**: 5 plans
|
||||
|
||||
**Wave 1**
|
||||
|
||||
- [ ] 10-01-PLAN.md — Foundation: install fast-xml-parser/adm-zip/csv-parse (adm-zip supply-chain checkpoint), add global Tender + TenderSourcePollConfig models, [BLOCKING] migration (SCHEMA-01)
|
||||
|
||||
**Wave 2** *(blocked on Wave 1 completion)*
|
||||
|
||||
- [ ] 10-02-PLAN.md — Marketplace registration slice: self-seed tender-radar module, module-loader whitelist entry, placeholder page, singleton doe-opendata poll config (CONFIG-01)
|
||||
|
||||
**Wave 3** *(blocked on Wave 2 completion)*
|
||||
|
||||
- [ ] 10-03-PLAN.md — DÖE adapter + normalizer (TDD): fetch/extract/parse day-export ZIP, D-02 open-tender filter, eForms-primary normalize with dedupKey + contentHash (INGEST-01, SCHEMA-01)
|
||||
|
||||
**Wave 4** *(blocked on Wave 3 completion)*
|
||||
|
||||
- [ ] 10-04-PLAN.md — Ingestion + shared scheduler: day-cursor gate, upsert change-detection, single global cron (poll-once-fan-out-many), 90-day retention, two-tenant safety test (SCHEMA-02, INGEST-06)
|
||||
|
||||
**Wave 5** *(blocked on Wave 4 completion)*
|
||||
|
||||
- [ ] 10-05-PLAN.md — Controller + admin source-config: global ModuleGuard-gated read, admin-configurable poll interval applied live to scheduler (INGEST-06)
|
||||
|
||||
**UI hint**: no (backend-foundation phase; results UI is Phase 11)
|
||||
|
||||
### Phase 11: Filter Engine, Results UI & Saved Searches
|
||||
|
||||
@@ -427,7 +449,7 @@ Phases execute in numeric order: 1 -> 2 -> 3 -> 4 -> 5 -> 6 -> 7 -> 8 -> 9 -> 10
|
||||
| 7. DKV Fleet Module | 6/6 | Complete | 2026-06-27 |
|
||||
| 8. Dashboard Widgets Vollimplementierung | 4/4 | Complete | 2026-07-01 |
|
||||
| 9. Cert Manager Module | 6/6 | Complete | 2026-07-02 |
|
||||
| 10. Ausschreibungs-Radar Foundation & DÖE Ingestion | 0/TBD | Not started | - |
|
||||
| 10. Ausschreibungs-Radar Foundation & DÖE Ingestion | 0/5 | Not started | - |
|
||||
| 11. Filter Engine, Results UI & Saved Searches | 0/TBD | Not started | - |
|
||||
| 12. Tender Notifications | 0/TBD | Not started | - |
|
||||
| 13. Scraping Adapters & Cross-Source Deduplication | 0/TBD | Not started | - |
|
||||
|
||||
@@ -0,0 +1,167 @@
|
||||
---
|
||||
phase: 10-ausschreibungs-radar-foundation-d-e-ingestion
|
||||
plan: 01
|
||||
type: execute
|
||||
wave: 1
|
||||
depends_on: []
|
||||
files_modified:
|
||||
- apps/api/package.json
|
||||
- apps/api/prisma/schema.prisma
|
||||
- apps/api/prisma/migrations/20260721120000_add_tender_radar/migration.sql
|
||||
autonomous: false
|
||||
requirements: [SCHEMA-01]
|
||||
must_haves:
|
||||
truths:
|
||||
- "Global (tenant-agnostic) Tender table exists in Postgres with OCDS-oriented nullable deadline/value fields (SCHEMA-01)"
|
||||
- "Singleton TenderSourcePollConfig row model exists to drive the shared poll (INGEST-06 foundation)"
|
||||
- "fast-xml-parser, adm-zip and csv-parse are installed in @tessera/api"
|
||||
artifacts:
|
||||
- apps/api/prisma/schema.prisma
|
||||
- apps/api/prisma/migrations/20260721120000_add_tender_radar/migration.sql
|
||||
key_links:
|
||||
- "Tender.dedupKey @unique is the upsert target for SCHEMA-02 change detection"
|
||||
- "Tender has NO tenantId column — RLS/forTenant deliberately does not apply (global data, per D-03)"
|
||||
---
|
||||
|
||||
<objective>
|
||||
Lay the data + dependency foundation for the Ausschreibungs-Radar module: install the three ingestion libraries (with a supply-chain checkpoint for the [SUS] `adm-zip` package), add the platform-global `Tender` and singleton `TenderSourcePollConfig` Prisma models, and apply the migration to the live database.
|
||||
|
||||
## Phase Goal (user story)
|
||||
**As a** Tessera-Administrator, **I want to** das Ausschreibungs-Radar-Modul im Marketplace aktivieren und automatisch normalisierte deutsche DÖE-Ausschreibungen auf einem gemeinsamen, mandantensicheren Zeitplan erfassen lassen, **so that** alle Mandanten aktuelle, bietbare Vergaben durchsuchen koennen, ohne dass pro Mandant redundant gepollt wird.
|
||||
|
||||
Purpose: Every later slice (module registration, adapter, normalizer, ingestion, scheduler) depends on these tables and packages existing. The global-not-tenant-scoped shape of `Tender` (D-03) is the single highest-leverage decision and is locked in here.
|
||||
Output: Migrated `Tender` + `TenderSourcePollConfig` tables, generated Prisma client types, three installed packages.
|
||||
</objective>
|
||||
|
||||
<execution_context>
|
||||
@$HOME/.claude/gsd-core/workflows/execute-plan.md
|
||||
@$HOME/.claude/gsd-core/templates/summary.md
|
||||
</execution_context>
|
||||
|
||||
<context>
|
||||
@.planning/PROJECT.md
|
||||
@.planning/ROADMAP.md
|
||||
@.planning/STATE.md
|
||||
@.planning/phases/10-ausschreibungs-radar-foundation-d-e-ingestion/10-RESEARCH.md
|
||||
@.planning/phases/10-ausschreibungs-radar-foundation-d-e-ingestion/10-PATTERNS.md
|
||||
@.planning/phases/10-ausschreibungs-radar-foundation-d-e-ingestion/10-CONTEXT.md
|
||||
@apps/api/prisma/schema.prisma
|
||||
@apps/api/prisma/migrations/20260714090000_add_ldap_user_exclude_list/migration.sql
|
||||
</context>
|
||||
|
||||
## Artifacts This Phase Produces (whole-phase reference)
|
||||
|
||||
Prisma models: `Tender`, `TenderSourcePollConfig` (Plan 01).
|
||||
NestJS classes: `TendersModule`, `seedTendersModule()` (Plan 02); `DoeOpenDataAdapter`, `TenderSourceAdapter` interface, `TenderNormalizerService` (Plan 03); `TenderIngestionService`, `TenderSchedulerService` (Plan 04); `TendersController`, `SourceConfigDto`, `TenderQueryDto` (Plan 05).
|
||||
Types: `RawTenderRecord`, `NormalizedTenderFields`, `SourceType` in `tenders/tender.types.ts` (Plan 03).
|
||||
Scheduler methods: `TenderSchedulerService.setInterval(intervalMin)`, `stopJob()`; `TenderIngestionService.pollDueSources()`, `pruneExpiredTenders()` (Plan 04).
|
||||
Config keys / slugs: module slug `tender-radar`; `TenderSourcePollConfig.sourceType = 'doe-opendata'` (singleton).
|
||||
New file paths: `apps/api/src/tenders/**`, `apps/web/src/app/(portal)/modules/tender-radar/page.tsx`, `apps/web/src/lib/module-loader.ts` (+`tender-radar` entry), fixture ZIPs under `apps/api/src/tenders/__fixtures__/`.
|
||||
|
||||
<tasks>
|
||||
|
||||
<task type="checkpoint:human-verify" gate="blocking-human">
|
||||
<name>Task 1: adm-zip package legitimacy verification</name>
|
||||
<what-built>Nothing yet — this gate precedes the `pnpm add adm-zip` step. `adm-zip` was tagged [SUS]/[ASSUMED] in 10-RESEARCH.md Package Legitimacy Audit (flagged `too-new`, manually assessed a false positive, but package-name provenance is training-knowledge so it requires a human spot-check per the package-legitimacy protocol).</what-built>
|
||||
<action>Pause before installing `adm-zip`. Present the legitimacy evidence below to the human and wait for explicit approval (or a named alternative) before Task 2 runs `pnpm add`. Do not auto-approve — this is a blocking supply-chain gate (T-10-SC).</action>
|
||||
<how-to-verify>
|
||||
1. Open https://www.npmjs.com/package/adm-zip — confirm weekly downloads in the millions, last publish is a patch release (not a brand-new package), and repository points to github.com/cthackers/adm-zip.
|
||||
2. Open https://github.com/cthackers/adm-zip — confirm it is an established, maintained repo (stars, issue history, MIT license).
|
||||
3. Optionally check the Socket.dev / Snyk score for adm-zip@0.6.0.
|
||||
4. Confirm `fast-xml-parser` and `csv-parse` are already approved in 10-RESEARCH.md (carried over from STACK.md — no re-flag needed).
|
||||
</how-to-verify>
|
||||
<acceptance_criteria>
|
||||
- Human types "approved" after confirming adm-zip is a legitimate, established package, OR names an alternative (e.g. `unzipper`) if not.
|
||||
- This checkpoint is NOT auto-approvable regardless of workflow.auto_advance (supply-chain gate).
|
||||
</acceptance_criteria>
|
||||
<resume-signal>Type "approved" to proceed with the install, or name a replacement ZIP library.</resume-signal>
|
||||
</task>
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 2: Install ingestion packages + add Tender/TenderSourcePollConfig models</name>
|
||||
<read_first>
|
||||
- apps/api/prisma/schema.prisma (DkvModuleConfig model at ~line 178 for field/convention reference; datasource + generator at top)
|
||||
- apps/api/prisma/migrations/20260714090000_add_ldap_user_exclude_list/migration.sql (handwritten migration style: IF NOT EXISTS guards, timestamped folder)
|
||||
- .planning/phases/10-ausschreibungs-radar-foundation-d-e-ingestion/10-PATTERNS.md (schema.prisma section — exact model field list)
|
||||
</read_first>
|
||||
<files>apps/api/package.json, apps/api/prisma/schema.prisma, apps/api/prisma/migrations/20260721120000_add_tender_radar/migration.sql</files>
|
||||
<action>
|
||||
Run `pnpm --filter @tessera/api add fast-xml-parser adm-zip csv-parse` (only after Task 1 approval).
|
||||
|
||||
Add two models to `apps/api/prisma/schema.prisma` following the 10-PATTERNS.md schema section exactly:
|
||||
|
||||
`Tender` — global reference table, deliberately NO `tenantId` and NO `@@index([tenantId])` (D-03: platform-global data). Fields: `id String @id @default(uuid())`, `sourcePortal String`, `sourceNoticeId String`, `ocid String?`, `dedupKey String @unique`, `title String`, `buyerName String?`, `cpvCodes String[] @default([])`, `region String?`, `plz String?`, `bundesland String?`, `deadlineAt DateTime?` (frequently null per RESEARCH Pattern 4 — nullable is mandatory, not optional hardening), `estimatedValue Decimal? @db.Decimal(14,2)`, `procedureType String?`, `status String @default("active")` (values `active` | `expired`; supports D-05 retention marking), `sourceUrl String?`, `contentHash String` (SCHEMA-02 change detection), `rawPayload Json?` (debugging/re-normalization), `publishedAt DateTime`, `createdAt DateTime @default(now())`, `updatedAt DateTime @updatedAt`. Indexes: `@@index([status])`, `@@index([deadlineAt])`, `@@index([publishedAt])`.
|
||||
|
||||
`TenderSourcePollConfig` — singleton-per-source admin config (INGEST-06 foundation). Fields: `id String @id @default(uuid())`, `sourceType String @unique` (fixed slug, `doe-opendata` — the `@unique` makes the singleton intent explicit, mirroring DkvModuleConfig.tenantId @unique), `pollIntervalMin Int @default(60)` (D-04 default hourly), `isActive Boolean @default(false)`, `lastIngestedDay DateTime?` (day-cursor, NOT a timestamp — RESEARCH Pattern 1), `createdAt DateTime @default(now())`, `updatedAt DateTime @updatedAt`.
|
||||
|
||||
Write the handwritten migration at `apps/api/prisma/migrations/20260721120000_add_tender_radar/migration.sql` mirroring the ldap-user-exclude-list migration style: `CREATE TABLE IF NOT EXISTS "Tender" (...)` and `CREATE TABLE IF NOT EXISTS "TenderSourcePollConfig" (...)` with matching column types, plus `CREATE UNIQUE INDEX IF NOT EXISTS` on `Tender.dedupKey` and `TenderSourcePollConfig.sourceType`, and the three `Tender` indexes.
|
||||
|
||||
Run `pnpm --filter @tessera/api exec prisma generate` so the Prisma client TS types for `Tender`/`TenderSourcePollConfig` are available to downstream service tasks.
|
||||
</action>
|
||||
<verify>
|
||||
<automated>cd apps/api && grep -c "model Tender" prisma/schema.prisma && grep -c "model TenderSourcePollConfig" prisma/schema.prisma && test -f prisma/migrations/20260721120000_add_tender_radar/migration.sql && pnpm exec tsc --noEmit -p tsconfig.json 2>&1 | tail -5</automated>
|
||||
</verify>
|
||||
<acceptance_criteria>
|
||||
- `grep -c "model Tender"` and `grep -c "model TenderSourcePollConfig"` in schema.prisma each return >= 1.
|
||||
- Migration SQL file exists at the timestamped path.
|
||||
- `Tender` has no `tenantId` column (assert: `grep -c "tenantId" ` within the Tender model block returns 0).
|
||||
- `prisma generate` succeeds and `import { Tender } from '@prisma/client'` type-checks.
|
||||
</acceptance_criteria>
|
||||
<done>Both models present with correct fields, migration SQL written, Prisma client regenerated, packages installed.</done>
|
||||
</task>
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 3: [BLOCKING] Apply the tender-radar migration to the database</name>
|
||||
<read_first>
|
||||
- apps/api/prisma/migrations/migration_lock.toml (confirms postgresql provider + migrate workflow)
|
||||
- apps/api/prisma/migrations/20260721120000_add_tender_radar/migration.sql (the migration written in Task 2)
|
||||
</read_first>
|
||||
<files>apps/api/prisma/migrations/20260721120000_add_tender_radar/migration.sql</files>
|
||||
<action>
|
||||
[BLOCKING] Apply the handwritten migration to the live dev database so the Prisma client's compile-time types are backed by real tables (build/type checks pass WITHOUT this step because types come from the generated client, not the DB — this creates a false-positive verification state, so this task is mandatory before any ingestion/persistence verification).
|
||||
|
||||
Run `pnpm --filter @tessera/api exec prisma migrate deploy` (non-interactive; applies the committed handwritten migration). If `migrate deploy` reports drift or a non-empty schema conflict that requires interactive resolution, fall back to `pnpm --filter @tessera/api exec prisma db push` (schema is additive — two new tables, no destructive change expected). Then confirm the tables exist.
|
||||
</action>
|
||||
<verify>
|
||||
<automated>cd apps/api && pnpm exec prisma migrate status 2>&1 | tail -8</automated>
|
||||
</verify>
|
||||
<acceptance_criteria>
|
||||
- `prisma migrate status` reports the database schema is up to date (no pending migrations), OR a follow-up `prisma db push` reports "already in sync".
|
||||
- Tables `Tender` and `TenderSourcePollConfig` exist in the database (verifiable via `prisma migrate status` clean state or a `prisma.tender.count()` call returning 0 without error).
|
||||
</acceptance_criteria>
|
||||
<done>Migration applied; both tables live in the database; no pending migrations.</done>
|
||||
</task>
|
||||
|
||||
</tasks>
|
||||
|
||||
<threat_model>
|
||||
## Trust Boundaries
|
||||
|
||||
| Boundary | Description |
|
||||
|----------|-------------|
|
||||
| npm registry → build | Third-party package code (`adm-zip`) enters the trusted build via `pnpm add` |
|
||||
| ORM → Postgres | New global tables added; tenant-isolation design decision is encoded in schema shape |
|
||||
|
||||
## STRIDE Threat Register
|
||||
|
||||
| Threat ID | Category | Component | Severity | Disposition | Mitigation Plan |
|
||||
|-----------|----------|-----------|----------|-------------|-----------------|
|
||||
| T-10-SC | Tampering | `pnpm add adm-zip` (supply chain) | high | mitigate | Task 1 blocking-human checkpoint verifies adm-zip on npmjs.com + github before install; fast-xml-parser/csv-parse pre-approved in RESEARCH |
|
||||
| T-10-01 | Information Disclosure | `Tender` table schema | high | mitigate | `Tender` carries NO `tenantId` by design (D-03) — it is platform-global reference data; per-tenant scoping lives one layer up (Phase 11 saved searches). Encoding this now prevents a later blanket `where: { tenantId }` habit from silently hiding global data from a second tenant |
|
||||
| T-10-02 | Denial of Service | `Decimal`/`String[]` columns | low | accept | Standard Postgres column types; ingestion-side size guards handled in Plan 03 (zip-bomb) — no DB-level risk introduced by the schema itself |
|
||||
</threat_model>
|
||||
|
||||
<verification>
|
||||
- `prisma migrate status` clean; both tables present.
|
||||
- `Tender` model has no `tenantId` field (design invariant for D-03 / multi-tenant global data).
|
||||
- Three packages resolvable: `pnpm --filter @tessera/api exec node -e "require('adm-zip');require('fast-xml-parser');require('csv-parse/sync')"`.
|
||||
</verification>
|
||||
|
||||
<success_criteria>
|
||||
- SCHEMA-01 data foundation: normalized OCDS-oriented `Tender` table exists with nullable deadline/value.
|
||||
- Packages installed; migration applied; client regenerated.
|
||||
</success_criteria>
|
||||
|
||||
<output>
|
||||
Create `.planning/phases/10-ausschreibungs-radar-foundation-d-e-ingestion/10-01-SUMMARY.md` when done.
|
||||
</output>
|
||||
@@ -0,0 +1,162 @@
|
||||
---
|
||||
phase: 10-ausschreibungs-radar-foundation-d-e-ingestion
|
||||
plan: 02
|
||||
type: execute
|
||||
wave: 2
|
||||
depends_on: ["10-01"]
|
||||
files_modified:
|
||||
- apps/api/src/tenders/tenders.module.ts
|
||||
- apps/api/src/tenders/tenders.seed.ts
|
||||
- apps/api/src/tenders/tenders.seed.spec.ts
|
||||
- apps/api/src/app.module.ts
|
||||
- apps/web/src/lib/module-loader.ts
|
||||
- apps/web/src/app/(portal)/modules/tender-radar/page.tsx
|
||||
autonomous: true
|
||||
requirements: [CONFIG-01]
|
||||
must_haves:
|
||||
truths:
|
||||
- "Admin can find 'Ausschreibungs-Radar' in the marketplace and activate it for a tenant, same as DKV/Cert-Manager (CONFIG-01)"
|
||||
- "Activating the module for a tenant opens the module page (no 404) because the module-loader whitelist entry exists"
|
||||
- "The singleton 'doe-opendata' TenderSourcePollConfig row is seeded so the shared poll is admin-drivable"
|
||||
artifacts:
|
||||
- apps/api/src/tenders/tenders.module.ts
|
||||
- apps/api/src/tenders/tenders.seed.ts
|
||||
- apps/web/src/lib/module-loader.ts
|
||||
- apps/web/src/app/(portal)/modules/tender-radar/page.tsx
|
||||
key_links:
|
||||
- "TendersModule.onModuleInit() calls seedTendersModule() → ModuleRegistryService.seedModule({ slug: 'tender-radar', isSystem: true })"
|
||||
- "module-loader.ts MODULE_REGISTRY['tender-radar'] entry is MANDATORY — page 404s without it even when activated"
|
||||
---
|
||||
|
||||
<objective>
|
||||
Deliver the first user-facing vertical slice: register the Ausschreibungs-Radar module in the marketplace (`isSystem: true`, admin-activatable per tenant) and make its portal page load when activated. Seed the singleton `doe-opendata` poll-config row so later plans have a config to drive.
|
||||
|
||||
## Phase Goal (user story)
|
||||
**As a** Tessera-Administrator, **I want to** das Ausschreibungs-Radar-Modul im Marketplace aktivieren, **so that** ich Zugang zum Ausschreibungs-Katalog fuer einen Mandanten freischalten kann — genau wie beim DKV-Fleet- und Cert-Manager-Modul (CONFIG-01).
|
||||
|
||||
Purpose: After this plan a real admin can DO something new: find the module in the marketplace, activate it for a tenant, and open its page. This is the CONFIG-01 slice end-to-end (registry seed → marketplace → activation → lazy-loaded page).
|
||||
Output: Self-seeding `TendersModule`, module-loader whitelist entry, minimal placeholder page, seeded singleton poll config.
|
||||
</objective>
|
||||
|
||||
<execution_context>
|
||||
@$HOME/.claude/gsd-core/workflows/execute-plan.md
|
||||
@$HOME/.claude/gsd-core/templates/summary.md
|
||||
</execution_context>
|
||||
|
||||
<context>
|
||||
@.planning/PROJECT.md
|
||||
@.planning/ROADMAP.md
|
||||
@.planning/STATE.md
|
||||
@.planning/phases/10-ausschreibungs-radar-foundation-d-e-ingestion/10-PATTERNS.md
|
||||
@apps/api/src/dkv/dkv.module.ts
|
||||
@apps/api/src/cert-manager/cert-manager.seed.ts
|
||||
@apps/api/src/module-registry/module-registry.service.ts
|
||||
@apps/web/src/lib/module-loader.ts
|
||||
</context>
|
||||
|
||||
<tasks>
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 1: TendersModule skeleton + registry self-seed + singleton poll config</name>
|
||||
<read_first>
|
||||
- apps/api/src/dkv/dkv.module.ts (exact OnModuleInit self-seed pattern to copy)
|
||||
- apps/api/src/cert-manager/cert-manager.seed.ts (preferred seed analog — isSystem:true + "admin must activate via Marketplace" phrasing matches CONFIG-01)
|
||||
- apps/api/src/module-registry/module-registry.service.ts (seedModule upsert-by-slug signature, lines ~158-187)
|
||||
- apps/api/src/app.module.ts (imports array — add TendersModule alongside DkvModule/CertManagerModule)
|
||||
</read_first>
|
||||
<files>apps/api/src/tenders/tenders.module.ts, apps/api/src/tenders/tenders.seed.ts, apps/api/src/app.module.ts</files>
|
||||
<action>
|
||||
Create `tenders.seed.ts` exporting `seedTendersModule(moduleRegistryService)` that calls `moduleRegistryService.seedModule({ slug: 'tender-radar', name: 'Ausschreibungs-Radar', version: '1.0.0', category: 'procurement', description: { de: 'Deutsche Ausschreibungen automatisch erfassen und durchsuchen', en: 'Automatically track and search German public tenders' }, isSystem: true })` — mirror `cert-manager.seed.ts` verbatim in structure (CONFIG-01). Confirm `procurement` is acceptable as a new category value (seedModule does not constrain category); if the marketplace UI needs a known category, reuse an existing one and note it in the summary.
|
||||
|
||||
Create `tenders.module.ts` implementing `OnModuleInit` exactly like `DkvModule`: `imports: [ModuleRegistryModule]`, `controllers: []` (added Plan 05), `providers: []` (services added Plans 03-04), and an `onModuleInit()` that calls `seedTendersModule(this.moduleRegistryService)` inside try/catch with a Logger. Also, in `onModuleInit()`, upsert the singleton poll config: `prisma.tenderSourcePollConfig.upsert({ where: { sourceType: 'doe-opendata' }, update: {}, create: { sourceType: 'doe-opendata', pollIntervalMin: 60, isActive: true } })` — inject `PrismaService` (PrismaModule is global). isActive defaults true so the platform-global DÖE poll (D-04 hourly) runs regardless of tenant activation; the day-cursor gate (Plan 04) makes this idempotent. Do NOT use `forTenant()` — this config and the Tender table are global (D-03).
|
||||
|
||||
Add `TendersModule` to the `imports` array in `app.module.ts`.
|
||||
</action>
|
||||
<verify>
|
||||
<automated>cd apps/api && pnpm exec tsc --noEmit -p tsconfig.json 2>&1 | tail -5 && grep -c "TendersModule" src/app.module.ts</automated>
|
||||
</verify>
|
||||
<acceptance_criteria>
|
||||
- `tenders.seed.ts` seeds slug `tender-radar` with `isSystem: true`.
|
||||
- `TendersModule` is imported in `app.module.ts` (`grep -c "TendersModule" src/app.module.ts` >= 2 — import line + array entry).
|
||||
- Singleton `doe-opendata` config upserted on init (idempotent — re-boot does not create a second row; guaranteed by `sourceType @unique`).
|
||||
- `tsc --noEmit` passes.
|
||||
</acceptance_criteria>
|
||||
<done>Module self-seeds into the registry on boot and seeds the singleton poll config; app compiles.</done>
|
||||
</task>
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 2: Web module-loader whitelist entry + minimal portal page</name>
|
||||
<read_first>
|
||||
- apps/web/src/lib/module-loader.ts (MODULE_REGISTRY object — copy the 'cert-manager' entry shape)
|
||||
- apps/web/src/app/(portal)/modules/cert-manager/page.tsx (page.tsx conventions: 'use client', useTranslations, default export)
|
||||
</read_first>
|
||||
<files>apps/web/src/lib/module-loader.ts, apps/web/src/app/(portal)/modules/tender-radar/page.tsx</files>
|
||||
<action>
|
||||
Add a `'tender-radar'` entry to `MODULE_REGISTRY` in `module-loader.ts` pointing at `dynamic(() => import('@/app/(portal)/modules/tender-radar/page'), { ssr: false })` — identical shape to the existing `'cert-manager'`/`'dkv-fleet'` entries. This whitelist entry is MANDATORY for CONFIG-01: the page 404s even when the module is activated if the slug is not whitelisted (security-motivated whitelist, module-loader header comment).
|
||||
|
||||
Create a minimal `tender-radar/page.tsx` client component (default export, `'use client'`) that renders the module title and a short "Ausschreibungen werden erfasst" placeholder. Real results UI is Phase 11 — this is the thinnest page that proves the activation → page-load path. Use a hardcoded German string here (full i18n is CONFIG-03, Phase 14); note this as an intentional MVP stub in the summary.
|
||||
</action>
|
||||
<verify>
|
||||
<automated>cd apps/web && grep -c "tender-radar" src/lib/module-loader.ts && test -f "src/app/(portal)/modules/tender-radar/page.tsx" && pnpm exec tsc --noEmit 2>&1 | tail -5</automated>
|
||||
</verify>
|
||||
<acceptance_criteria>
|
||||
- `MODULE_REGISTRY['tender-radar']` entry present (`grep -c "tender-radar" module-loader.ts` >= 1).
|
||||
- `tender-radar/page.tsx` exists and default-exports a component.
|
||||
- Web `tsc --noEmit` passes.
|
||||
</acceptance_criteria>
|
||||
<done>Activated module opens its page instead of 404; whitelist entry in place.</done>
|
||||
</task>
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 3: Seed registration test (Wave 0 coverage for CONFIG-01)</name>
|
||||
<read_first>
|
||||
- apps/api/src/cert-manager/cert-manager.service.spec.ts (existing vitest spec style in this repo)
|
||||
- apps/api/src/tenders/tenders.seed.ts (unit under test, from Task 1)
|
||||
</read_first>
|
||||
<files>apps/api/src/tenders/tenders.seed.spec.ts</files>
|
||||
<action>
|
||||
Create `tenders.seed.spec.ts` (Vitest). Provide a mock `ModuleRegistryService` with a spied `seedModule`. Assert that `seedTendersModule(mock)` calls `seedModule` once with `slug: 'tender-radar'`, `isSystem: true`, and a description object containing both `de` and `en` keys. This is the Wave 0 automated proof for CONFIG-01 registration.
|
||||
</action>
|
||||
<verify>
|
||||
<automated>cd apps/api && pnpm test -- tenders.seed 2>&1 | tail -15</automated>
|
||||
</verify>
|
||||
<acceptance_criteria>
|
||||
- Test asserts `seedModule` called with `slug: 'tender-radar'` and `isSystem: true`.
|
||||
- `pnpm --filter @tessera/api test -- tenders.seed` passes.
|
||||
</acceptance_criteria>
|
||||
<done>Registration behaviour has automated coverage; green.</done>
|
||||
</task>
|
||||
|
||||
</tasks>
|
||||
|
||||
<threat_model>
|
||||
## Trust Boundaries
|
||||
|
||||
| Boundary | Description |
|
||||
|----------|-------------|
|
||||
| tenant → module activation | A tenant's activation must gate only visibility, not create a second data copy or leak another tenant's config |
|
||||
| URL slug → dynamic import | Only whitelisted slugs may resolve to a component (module-loader security whitelist) |
|
||||
|
||||
## STRIDE Threat Register
|
||||
|
||||
| Threat ID | Category | Component | Severity | Disposition | Mitigation Plan |
|
||||
|-----------|----------|-----------|----------|-------------|-----------------|
|
||||
| T-10-03 | Elevation of Privilege | `module-loader.ts` dynamic import | medium | mitigate | Only the fixed `'tender-radar'` slug is whitelisted; arbitrary URL slugs return null (existing whitelist mechanism, unchanged) |
|
||||
| T-10-04 | Information Disclosure | per-tenant module activation | high | mitigate | Activation uses the existing `ModuleRegistryService` / `TenantModuleActivation` mechanism unchanged; the singleton `doe-opendata` config is global (one row, `sourceType @unique`) so a 2nd tenant activation cannot create or read a per-tenant duplicate. No `forTenant()` applied to the global config |
|
||||
| T-10-05 | Tampering | registry self-seed on boot | low | accept | `seedModule` is an idempotent upsert-by-slug; repeated boots do not duplicate the module row |
|
||||
</threat_model>
|
||||
|
||||
<verification>
|
||||
- Module appears in the registry with slug `tender-radar` after boot (seed test green).
|
||||
- Activating the module for a tenant opens `/modules/tender-radar` (whitelist entry present).
|
||||
- Singleton poll config seeded exactly once (idempotent upsert).
|
||||
- `pnpm --filter @tessera/api test` green; both apps `tsc --noEmit` clean.
|
||||
</verification>
|
||||
|
||||
<success_criteria>
|
||||
- CONFIG-01: admin can find and activate Ausschreibungs-Radar in the marketplace, same as DKV/Cert-Manager, and open its page.
|
||||
</success_criteria>
|
||||
|
||||
<output>
|
||||
Create `.planning/phases/10-ausschreibungs-radar-foundation-d-e-ingestion/10-02-SUMMARY.md` when done.
|
||||
</output>
|
||||
@@ -0,0 +1,185 @@
|
||||
---
|
||||
phase: 10-ausschreibungs-radar-foundation-d-e-ingestion
|
||||
plan: 03
|
||||
type: tdd
|
||||
wave: 3
|
||||
depends_on: ["10-01", "10-02"]
|
||||
files_modified:
|
||||
- apps/api/src/tenders/__fixtures__/doe-eforms-sample.zip
|
||||
- apps/api/src/tenders/__fixtures__/doe-ocds-sample.zip
|
||||
- apps/api/src/tenders/tender.types.ts
|
||||
- apps/api/src/tenders/adapters/tender-source-adapter.interface.ts
|
||||
- apps/api/src/tenders/adapters/doe-opendata.adapter.ts
|
||||
- apps/api/src/tenders/adapters/doe-opendata.adapter.spec.ts
|
||||
- apps/api/src/tenders/tender-normalizer.service.ts
|
||||
- apps/api/src/tenders/tender-normalizer.service.spec.ts
|
||||
- apps/api/src/tenders/tenders.module.ts
|
||||
autonomous: true
|
||||
requirements: [INGEST-01, SCHEMA-01]
|
||||
must_haves:
|
||||
truths:
|
||||
- "DoeOpenDataAdapter fetches a day's DÖE export ZIP, extracts it, and parses eForms-DE XML (primary) + OCDS JSON (ocid) into RawTenderRecord[] (INGEST-01)"
|
||||
- "Only open tenders survive the D-02 filter: tag=['tender'] included; award/planning/untagged-with-awards excluded"
|
||||
- "TenderNormalizer maps a real eForms+OCDS notice pair into Tender fields with nullable deadline/value, a stable dedupKey (ocid → sourcePortal:noticeId), and a contentHash (SCHEMA-01)"
|
||||
artifacts:
|
||||
- apps/api/src/tenders/adapters/doe-opendata.adapter.ts
|
||||
- apps/api/src/tenders/tender-normalizer.service.ts
|
||||
- apps/api/src/tenders/__fixtures__/doe-eforms-sample.zip
|
||||
key_links:
|
||||
- "Adapter uses native fetch + AbortController timeout (icon-discovery idiom); HTTP 400 = expected no-op (today/future pubDay), 5xx/network throws"
|
||||
- "dedupKey (ocid, else sourcePortal:sourceNoticeId) is the @unique upsert target consumed by Plan 04 ingestion"
|
||||
---
|
||||
|
||||
<objective>
|
||||
Build the DÖE source adapter and the normalizer — the parse+map core of INGEST-01/SCHEMA-01 — test-first against REAL captured fixture ZIPs (not synthetic data), per this repo's real-fixture precedent (DKV PDF parser).
|
||||
|
||||
## Phase Goal (user story)
|
||||
**As a** Tessera-Administrator, **I want to** dass echte DÖE-Ausschreibungen aus dem Tages-Export korrekt gelesen und in ein einheitliches Schema normalisiert werden, **so that** nur offene, bietbare Vergaben (D-02) mit stabilen Dedup-Schluesseln im globalen Katalog landen.
|
||||
|
||||
Purpose: This is the structurally hardest part of the phase — correctly interpreting the DÖE day-batch ZIP shape, choosing eForms-DE XML as the primary structured-field source (deadline/value that OCDS drops), and applying the tag-presence D-02 filter. Both units are pure/deterministic → ideal for TDD.
|
||||
Output: `DoeOpenDataAdapter`, `TenderNormalizerService`, the adapter interface, shared types, and committed real fixtures.
|
||||
</objective>
|
||||
|
||||
<execution_context>
|
||||
@$HOME/.claude/gsd-core/workflows/execute-plan.md
|
||||
@$HOME/.claude/gsd-core/templates/summary.md
|
||||
</execution_context>
|
||||
|
||||
<context>
|
||||
@.planning/PROJECT.md
|
||||
@.planning/ROADMAP.md
|
||||
@.planning/phases/10-ausschreibungs-radar-foundation-d-e-ingestion/10-RESEARCH.md
|
||||
@.planning/phases/10-ausschreibungs-radar-foundation-d-e-ingestion/10-PATTERNS.md
|
||||
@apps/api/src/favorites/icon-discovery.service.ts
|
||||
@apps/api/src/dkv/dkv-parser.service.ts
|
||||
</context>
|
||||
|
||||
<tasks>
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 1: Wave 0 — capture real DÖE fixtures + shared types + adapter interface + failing specs</name>
|
||||
<read_first>
|
||||
- .planning/phases/10-ausschreibungs-radar-foundation-d-e-ingestion/10-RESEARCH.md (DÖE API Live Findings: endpoint, pubDay boundary rules, tag distribution, eForms-vs-OCDS field cross-check)
|
||||
- apps/api/src/dkv/dkv.types.ts (types file conventions)
|
||||
- .planning/research/ARCHITECTURE.md (Pattern 1: RawTenderRecord / TenderSourceAdapter interface sketch)
|
||||
</read_first>
|
||||
<files>apps/api/src/tenders/__fixtures__/doe-eforms-sample.zip, apps/api/src/tenders/__fixtures__/doe-ocds-sample.zip, apps/api/src/tenders/tender.types.ts, apps/api/src/tenders/adapters/tender-source-adapter.interface.ts, apps/api/src/tenders/adapters/doe-opendata.adapter.spec.ts, apps/api/src/tenders/tender-normalizer.service.spec.ts</files>
|
||||
<action>
|
||||
Capture REAL fixtures: compute a pubDay at least 2 calendar days in the past (Europe/Berlin), then download `https://oeffentlichevergabe.de/api/notice-exports?pubDay={day}&format=eforms.zip` and `...&format=ocds.zip`. Trim each archive to ~5-10 representative notices spanning the tag classes RESEARCH observed (`tender`, `award`, `planning`, and at least one untagged-with-awards) and save the trimmed ZIPs as `doe-eforms-sample.zip` / `doe-ocds-sample.zip` under `__fixtures__/`. Keep them small (do not commit the full 105-file archive). If the host is unreachable from the execution environment, construct minimal fixtures faithfully reproducing the real element shapes documented in RESEARCH (eForms `TenderSubmissionDeadlinePeriod/EndDate`, OCDS `releases[].tag`, `ocid` `ocds-mnwr74-...`) and note the substitution in the summary.
|
||||
|
||||
Define `tender.types.ts`: `SourceType` (`'doe-opendata'` for this phase), `RawTenderRecord` (sourceType, sourcePortal, sourceNoticeId, ocid?, sourceUrl, fetchedAt, eformsPayload/ocdsPayload or a unified payload, publishedAt), and `NormalizedTenderFields` (the Tender column subset the normalizer emits, plus dedupKey + contentHash).
|
||||
|
||||
Define `tender-source-adapter.interface.ts`: `TenderSourceAdapter` with `readonly sourceType: SourceType` and `fetchTenders(dayCursor: string): Promise<RawTenderRecord[]>` (day-cursor string `YYYY-MM-DD`, NOT a `since: Date` — RESEARCH Pattern 1 corrects ARCHITECTURE.md's signature).
|
||||
|
||||
Write FAILING specs first (RED): `doe-opendata.adapter.spec.ts` loads the fixture ZIPs and asserts (a) N raw records parsed, (b) the D-02 filter keeps only `tag=['tender']` and drops award/planning/untagged-with-awards (assert exact filtered count), (c) HTTP 400 fetch path returns `[]` (no-op). `tender-normalizer.service.spec.ts` asserts a known eForms+OCDS notice pair maps to the expected Tender fields including a non-null deadline recovered from eForms XML where OCDS is null, nullable value handled, and dedupKey = the notice's `ocid`.
|
||||
</action>
|
||||
<verify>
|
||||
<automated>cd apps/api && test -f src/tenders/__fixtures__/doe-eforms-sample.zip && pnpm test -- doe-opendata.adapter tender-normalizer 2>&1 | tail -20</automated>
|
||||
</verify>
|
||||
<acceptance_criteria>
|
||||
- Fixture ZIPs exist under `__fixtures__/`.
|
||||
- `tender.types.ts` and `tender-source-adapter.interface.ts` compile.
|
||||
- Both spec files exist and FAIL for the right reason (implementation not yet written) — RED state confirmed.
|
||||
</acceptance_criteria>
|
||||
<done>Real fixtures captured, types + interface defined, failing specs committed (test(...) commit).</done>
|
||||
</task>
|
||||
|
||||
<task type="auto" tdd="true">
|
||||
<name>Task 2: DoeOpenDataAdapter — fetch + adm-zip extract + parse + D-02 filter (GREEN)</name>
|
||||
<read_first>
|
||||
- apps/api/src/favorites/icon-discovery.service.ts (native fetch + AbortController timeout idiom, lines ~254-287)
|
||||
- .planning/phases/10-ausschreibungs-radar-foundation-d-e-ingestion/10-PATTERNS.md (doe-opendata.adapter section: fetchDoeDay shape, 400=no-op)
|
||||
- .planning/phases/10-ausschreibungs-radar-foundation-d-e-ingestion/10-RESEARCH.md (Security Domain: zip-bomb ceiling; D-02 isOpenTenderNotice)
|
||||
- apps/api/src/tenders/adapters/doe-opendata.adapter.spec.ts (the RED spec from Task 1)
|
||||
</read_first>
|
||||
<behavior>
|
||||
- Test: fetching a valid past pubDay returns RawTenderRecord[] parsed from the fixture ZIP.
|
||||
- Test: a pubDay yielding HTTP 400 (today/future) returns [] without throwing.
|
||||
- Test: D-02 filter keeps tag=['tender'], excludes award/planning/untagged — assert exact retained count for the fixture.
|
||||
- Test: total uncompressed ZIP size over the ceiling (e.g. 50MB) is rejected before extraction (zip-bomb guard).
|
||||
</behavior>
|
||||
<files>apps/api/src/tenders/adapters/doe-opendata.adapter.ts, apps/api/src/tenders/tenders.module.ts</files>
|
||||
<action>
|
||||
Implement `DoeOpenDataAdapter` (Injectable) with `sourceType = 'doe-opendata'` and `fetchTenders(dayCursor)`:
|
||||
- Fetch `eforms.zip` and `ocds.zip` for the given `dayCursor` via native `fetch` wrapped in an `AbortController` + 15s `setTimeout` (icon-discovery idiom — no axios). Treat HTTP 400 as an expected no-op signal (return `[]`); any other `!res.ok` throws.
|
||||
- Extract with `adm-zip`: `new AdmZip(buffer).getEntries()`. BEFORE reading entry buffers, sum `entry.header.size` across entries and reject if it exceeds a sane ceiling (~50MB) — decompression-bomb defence-in-depth (RESEARCH Security Domain). Zip-slip is not a write risk here (in-memory only, never writing entries to disk) — do not write extracted entries to the filesystem.
|
||||
- Parse each eForms-DE XML entry with `fast-xml-parser` (primary structured fields); parse the paired OCDS JSON entries with `JSON.parse` for `ocid` and buyer/party names. Pair eForms↔OCDS by notice id.
|
||||
- Apply the D-02 open-tender filter on the OCDS release tag: keep when `release.tag` includes `'tender'`; exclude award, planning, and untagged-with-awards notices (positive tag-presence match per D-02 / RESEARCH Pattern 2 — never "absence of award = open"). Ingest is whole-Germany with NO region/CPV pre-filtering (D-03).
|
||||
- Return `RawTenderRecord[]`.
|
||||
Register `DoeOpenDataAdapter` in `TendersModule.providers`.
|
||||
</action>
|
||||
<verify>
|
||||
<automated>cd apps/api && pnpm test -- doe-opendata.adapter 2>&1 | tail -15</automated>
|
||||
</verify>
|
||||
<acceptance_criteria>
|
||||
- All `doe-opendata.adapter.spec.ts` tests pass (GREEN).
|
||||
- D-02 retained-count assertion matches the fixture's `tender`-tagged subset exactly.
|
||||
- Zip-bomb ceiling test passes (oversized archive rejected pre-extraction).
|
||||
- Uses native fetch (assert no axios import).
|
||||
</acceptance_criteria>
|
||||
<done>Adapter parses fixtures into filtered RawTenderRecord[]; spec green; feat(...) commit.</done>
|
||||
</task>
|
||||
|
||||
<task type="auto" tdd="true">
|
||||
<name>Task 3: TenderNormalizerService — fields + dedupKey + contentHash (GREEN)</name>
|
||||
<read_first>
|
||||
- apps/api/src/dkv/dkv-parser.service.ts (transform-service shape: one public entry point + private helpers, no I/O)
|
||||
- .planning/phases/10-ausschreibungs-radar-foundation-d-e-ingestion/10-PATTERNS.md (tender-normalizer.service section)
|
||||
- .planning/phases/10-ausschreibungs-radar-foundation-d-e-ingestion/10-RESEARCH.md (Pattern 3 eForms-primary; Pattern 4 nullable deadline/value; dedup key priority)
|
||||
- apps/api/src/tenders/tender-normalizer.service.spec.ts (the RED spec from Task 1)
|
||||
</read_first>
|
||||
<behavior>
|
||||
- Test: eForms `TenderSubmissionDeadlinePeriod/EndDate` populates `deadlineAt` even when OCDS deadline is null.
|
||||
- Test: missing deadline/value normalize to null (not thrown, not zero).
|
||||
- Test: dedupKey = ocid when present; falls back to `${sourcePortal}:${sourceNoticeId}` when ocid absent.
|
||||
- Test: contentHash is a stable sha256 over title+deadline+value+status; changing the deadline changes the hash.
|
||||
</behavior>
|
||||
<files>apps/api/src/tenders/tender-normalizer.service.ts, apps/api/src/tenders/tenders.module.ts</files>
|
||||
<action>
|
||||
Implement `TenderNormalizerService` (Injectable, pure — no I/O) with `normalize(raw: RawTenderRecord): NormalizedTenderFields`. Map eForms-DE XML as the PRIMARY source for `deadlineAt`, `estimatedValue`, `procedureType` (Pattern 3 — reverses ARCHITECTURE.md's OCDS-first guidance based on live cross-check); use OCDS for `ocid`, `buyerName`, party names. Extract `cpvCodes`, `region`/`plz`/`bundesland`, `title`, `sourceUrl`, `publishedAt`. Keep `deadlineAt`/`estimatedValue` nullable (Pattern 4 — null is the common case for Unterschwelle). Compute `dedupKey` priority-ordered: `ocid` → `${sourcePortal}:${sourceNoticeId}` (fuzzy fingerprint is Phase 13, out of scope here). Compute `contentHash = sha256(title + deadlineAt + estimatedValue + status)` (SCHEMA-02 hook). Set initial `status = 'active'`. Register the service in `TendersModule.providers`.
|
||||
</action>
|
||||
<verify>
|
||||
<automated>cd apps/api && pnpm test -- tender-normalizer 2>&1 | tail -15</automated>
|
||||
</verify>
|
||||
<acceptance_criteria>
|
||||
- All `tender-normalizer.service.spec.ts` tests pass (GREEN).
|
||||
- Deadline recovered from eForms XML where OCDS is null (the key SCHEMA-01 assertion).
|
||||
- dedupKey falls back correctly when ocid absent; contentHash changes when deadline changes.
|
||||
</acceptance_criteria>
|
||||
<done>Normalizer maps notice pairs to Tender fields deterministically; spec green; feat(...) commit.</done>
|
||||
</task>
|
||||
|
||||
</tasks>
|
||||
|
||||
<threat_model>
|
||||
## Trust Boundaries
|
||||
|
||||
| Boundary | Description |
|
||||
|----------|-------------|
|
||||
| DÖE host → API process | Outbound HTTP to a fixed external government host; untrusted archive + XML/JSON content enters the Node process |
|
||||
| ZIP archive → extraction | Compressed third-party payload decompressed in-memory |
|
||||
|
||||
## STRIDE Threat Register
|
||||
|
||||
| Threat ID | Category | Component | Severity | Disposition | Mitigation Plan |
|
||||
|-----------|----------|-----------|----------|-------------|-----------------|
|
||||
| T-10-06 | Server-Side Request Forgery | `DoeOpenDataAdapter` outbound fetch | high | mitigate | URL host is a hardcoded constant (`oeffentlichevergabe.de`); only the internally-computed `pubDay` cursor is interpolated (never user input). Native fetch + AbortController 15s timeout bounds the request (icon-discovery idiom). No admin-supplied URL flows into the fetch |
|
||||
| T-10-07 | Denial of Service | `adm-zip` extraction | high | mitigate | Sum `entry.header.size` and reject archives over a ~50MB uncompressed ceiling BEFORE reading entry buffers (decompression-bomb guard). Entries are read in-memory only — never written to disk, so zip-slip path traversal is structurally absent |
|
||||
| T-10-08 | Tampering / Injection | parsed eForms/OCDS text fields | medium | mitigate | All ingested text (buyer names, titles, descriptions) is external untrusted input; store raw, do NOT assume "safe" because it came from a government API. Escaping happens at render time (Phase 11 UI) — the normalizer must not emit HTML |
|
||||
</threat_model>
|
||||
|
||||
<verification>
|
||||
- `pnpm --filter @tessera/api test -- doe-opendata.adapter tender-normalizer` all green.
|
||||
- D-02 filter proven by exact retained-count assertion on real fixtures.
|
||||
- eForms-primary deadline recovery proven (deadline present where OCDS is null).
|
||||
- No axios import; zip-bomb ceiling enforced.
|
||||
</verification>
|
||||
|
||||
<success_criteria>
|
||||
- INGEST-01: DÖE day-export ZIP is fetched, extracted, and parsed into filtered open-tender raw records.
|
||||
- SCHEMA-01: raw records normalize into the unified Tender field shape with stable dedupKey + contentHash.
|
||||
</success_criteria>
|
||||
|
||||
<output>
|
||||
Create `.planning/phases/10-ausschreibungs-radar-foundation-d-e-ingestion/10-03-SUMMARY.md` when done.
|
||||
</output>
|
||||
@@ -0,0 +1,173 @@
|
||||
---
|
||||
phase: 10-ausschreibungs-radar-foundation-d-e-ingestion
|
||||
plan: 04
|
||||
type: execute
|
||||
wave: 4
|
||||
depends_on: ["10-01", "10-02", "10-03"]
|
||||
files_modified:
|
||||
- apps/api/src/tenders/tender-ingestion.service.ts
|
||||
- apps/api/src/tenders/tender-ingestion.service.spec.ts
|
||||
- apps/api/src/tenders/tender-scheduler.service.ts
|
||||
- apps/api/src/tenders/tender-scheduler.service.spec.ts
|
||||
- apps/api/src/tenders/tenders.module.ts
|
||||
autonomous: true
|
||||
requirements: [SCHEMA-02, INGEST-06]
|
||||
must_haves:
|
||||
truths:
|
||||
- "A changed DÖE notice (same ocid/dedupKey, new contentHash) updates the existing Tender row instead of creating a duplicate (SCHEMA-02)"
|
||||
- "DÖE is polled once on a shared global schedule regardless of tenant count — poll-once-fan-out-many, never per-tenant (INGEST-06)"
|
||||
- "Activating the module for a 2nd tenant triggers zero additional DÖE HTTP calls, zero additional cron jobs, and zero additional Tender rows (Success Criteria 4 & 5)"
|
||||
- "The poll tick is day-cursor gated: no upstream fetch when dayCursor >= today Europe/Berlin (D-01 from-now, no historical backfill)"
|
||||
- "Tenders past their deadline are marked expired and pruned after 90 days; deadline-less rows are never auto-expired (D-05)"
|
||||
artifacts:
|
||||
- apps/api/src/tenders/tender-ingestion.service.ts
|
||||
- apps/api/src/tenders/tender-scheduler.service.ts
|
||||
key_links:
|
||||
- "Scheduler cron tick → TenderIngestionService.pollDueSources(); day-cursor gate lives in the ingestion service, not the scheduler"
|
||||
- "prisma.tender.upsert({ where: { dedupKey } }) is the SCHEMA-02 change-detection seam; plain PrismaService (no forTenant) on the global table"
|
||||
---
|
||||
|
||||
<objective>
|
||||
Wire the ingestion orchestration and the shared scheduler — the poll-once-fan-out-many heart of the phase (INGEST-06) and the content-hash change detection (SCHEMA-02) — and prove the two-tenant safety property that is the phase's headline acceptance criterion.
|
||||
|
||||
## Phase Goal (user story)
|
||||
**As a** Tessera-Administrator, **I want to** dass DÖE genau einmal plattformweit auf einem admin-konfigurierbaren Intervall abgefragt wird — egal wie viele Mandanten das Modul aktiviert haben, **so that** kein redundanter Poll und keine doppelte Ingestion pro Mandant entsteht (INGEST-06, Success Criteria 4 & 5).
|
||||
|
||||
Purpose: This plan turns "parse a fixture" (Plan 03) into "the platform continuously ingests real DÖE data safely." The multi-tenant scheduler is where the DKV `findFirst()` single-tenant anti-pattern must NOT be copied; the DÖE config is a genuine platform-wide singleton (RESEARCH Pitfall D).
|
||||
Output: `TenderIngestionService`, `TenderSchedulerService`, and the two-tenant integration test.
|
||||
</objective>
|
||||
|
||||
<execution_context>
|
||||
@$HOME/.claude/gsd-core/workflows/execute-plan.md
|
||||
@$HOME/.claude/gsd-core/templates/summary.md
|
||||
</execution_context>
|
||||
|
||||
<context>
|
||||
@.planning/PROJECT.md
|
||||
@.planning/ROADMAP.md
|
||||
@.planning/phases/10-ausschreibungs-radar-foundation-d-e-ingestion/10-RESEARCH.md
|
||||
@.planning/phases/10-ausschreibungs-radar-foundation-d-e-ingestion/10-PATTERNS.md
|
||||
@apps/api/src/dkv/dkv-scheduler.service.ts
|
||||
@apps/api/src/prisma/prisma-tenant.extension.ts
|
||||
</context>
|
||||
|
||||
<tasks>
|
||||
|
||||
<task type="auto" tdd="true">
|
||||
<name>Task 1: TenderIngestionService — day-cursor gate + upsert + change-detect + retention</name>
|
||||
<read_first>
|
||||
- .planning/phases/10-ausschreibungs-radar-foundation-d-e-ingestion/10-PATTERNS.md (tender-ingestion.service section: pollDueSources shape, findUnique on sourceType, NO forTenant)
|
||||
- .planning/phases/10-ausschreibungs-radar-foundation-d-e-ingestion/10-RESEARCH.md (Pattern 1 day-cursor gate + nextDayToFetch; Pitfall A check-vs-fetch; Pattern 4 D-05 null-deadline handling)
|
||||
- apps/api/src/dkv/dkv.service.ts (orchestration shape: fetch → normalize → persist → log; catch-and-log)
|
||||
- apps/api/src/prisma/prisma-tenant.extension.ts (forTenant — the extension to AVOID here)
|
||||
</read_first>
|
||||
<behavior>
|
||||
- Test: when dayCursor >= today (Europe/Berlin), pollDueSources() makes NO adapter call and returns (no-op tick, expected — Pitfall A).
|
||||
- Test: a fresh notice is inserted; re-ingesting the identical notice (same dedupKey, same contentHash) does NOT create a duplicate and does NOT change field data.
|
||||
- Test: re-ingesting a changed notice (same dedupKey, new contentHash — e.g. extended deadline) UPDATES the existing row (SCHEMA-02).
|
||||
- Test: after a successful fetch, lastIngestedDay advances by one day; catch-up loops from lastIngestedDay+1 up to today-1.
|
||||
- Test: pruneExpiredTenders() marks status='expired' for rows with deadlineAt < now, deletes rows expired+deadlineAt older than 90 days, and NEVER touches rows with deadlineAt = null (D-05).
|
||||
</behavior>
|
||||
<files>apps/api/src/tenders/tender-ingestion.service.ts, apps/api/src/tenders/tender-ingestion.service.spec.ts, apps/api/src/tenders/tenders.module.ts</files>
|
||||
<action>
|
||||
Write RED spec first, then implement `TenderIngestionService` (Injectable) using the plain global `PrismaService` — do NOT call `forTenant()` on `Tender`/`TenderSourcePollConfig` (global RLS-exempt tables, D-03; wrapping them in the tenant RLS extension would make a 2nd tenant's session silently filter out platform data).
|
||||
|
||||
`pollDueSources()`: load the singleton config via `prisma.tenderSourcePollConfig.findUnique({ where: { sourceType: 'doe-opendata' } })` (fixed-slug lookup makes the singleton intent explicit — NOT a per-tenant `findFirst()`). If not `isActive`, return. Compute `nextDay = nextDayToFetch(config.lastIngestedDay)` (RESEARCH day-cursor snippet: first eligible day is activation day per D-01 "from now"; returns null if nextDay is not strictly before Berlin-today). If null, log at debug level ("no new DÖE day yet") and return — this no-op is expected, NOT an error (Pitfall A). Otherwise loop the day-cursor from nextDay forward while still `< today`: for each day call `doeAdapter.fetchTenders(day)`, `normalizer.normalize()` each record, and `prisma.tender.upsert({ where: { dedupKey }, update: {...fields, contentHash}, create: {...} })`. Track newly-created / contentHash-changed ids (the delta — consumed by Phase 11 matching, not this phase). After each day, `update` config.lastIngestedDay. Between successive day fetches insert a ~1-2s polite delay (RESEARCH Open Question 1). Never throw out of the tick — catch-and-log per RESEARCH/DKV pattern.
|
||||
|
||||
`pruneExpiredTenders()` (D-05 retention): mark `status='expired'` where `deadlineAt < now` AND `status='active'`; delete where `status='expired'` AND `deadlineAt < now-90d`. Rows with `deadlineAt IS NULL` are excluded from both operations (Pattern 4 recommendation (a) — a wrongly-deleted no-deadline tender is unrecoverable). Call `pruneExpiredTenders()` once per successful tick.
|
||||
</action>
|
||||
<verify>
|
||||
<automated>cd apps/api && pnpm test -- tender-ingestion 2>&1 | tail -20</automated>
|
||||
</verify>
|
||||
<acceptance_criteria>
|
||||
- Day-cursor no-op test green (no adapter call when nothing new).
|
||||
- SCHEMA-02: changed notice updates in place, identical notice does not duplicate — both green.
|
||||
- D-05: expired-marking + 90-day prune skips null-deadline rows — green.
|
||||
- Service uses plain PrismaService (assert no `forTenant` call in the file).
|
||||
</acceptance_criteria>
|
||||
<done>Ingestion orchestration with change detection + retention; spec green.</done>
|
||||
</task>
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 2: TenderSchedulerService — single global cron (poll-once-fan-out-many)</name>
|
||||
<read_first>
|
||||
- apps/api/src/dkv/dkv-scheduler.service.ts (REUSE the CronJob require()-resolution + SchedulerRegistry addCronJob/deleteCronJob mechanics; do NOT copy activeTenantId / per-tenant framing)
|
||||
- .planning/phases/10-ausschreibungs-radar-foundation-d-e-ingestion/10-PATTERNS.md (tender-scheduler.service section + ANTI-PATTERN note)
|
||||
- .planning/phases/10-ausschreibungs-radar-foundation-d-e-ingestion/10-RESEARCH.md (Pitfall D: the one legitimate singleton findUnique; two-tenant verification framing)
|
||||
</read_first>
|
||||
<files>apps/api/src/tenders/tender-scheduler.service.ts, apps/api/src/tenders/tenders.module.ts</files>
|
||||
<action>
|
||||
Implement `TenderSchedulerService` (Injectable, OnModuleInit) reusing DKV's cron mechanics: the `require('cron').CronJob` resolution workaround, and a `setInterval(intervalMin)` that removes any existing job then `schedulerRegistry.addCronJob(JOB_NAME, job)` with `JOB_NAME = 'tender-doe-poll'`. Cron expression computed exactly as DKV (`*/${min} * * * *` under 60, else `0 */${hours} * * *`). The tick calls `tenderIngestionService.pollDueSources().catch(logErr)`.
|
||||
|
||||
CRITICAL deviations from DKV (poll-once-fan-out-many): the scheduler has NO `activeTenantId` field and `setInterval` takes NO tenant argument — there is exactly ONE global cron job for the whole platform (D-04 default hourly interval). `onModuleInit()` loads the singleton config via `findUnique({ where: { sourceType: 'doe-opendata' } })` (NOT `findFirst()`); if `isActive`, call `setInterval(config.pollIntervalMin)`, else register nothing. Provide `stopJob()` (stop+delete the single job). The day-cursor gate is NOT in the scheduler — the scheduler only sets cron-tick frequency; whether an HTTP call happens is decided inside `pollDueSources()` (Pitfall A separation). Register `TenderSchedulerService` in `TendersModule.providers`.
|
||||
</action>
|
||||
<verify>
|
||||
<automated>cd apps/api && pnpm exec tsc --noEmit -p tsconfig.json 2>&1 | tail -5 && grep -c "activeTenantId" src/tenders/tender-scheduler.service.ts</automated>
|
||||
</verify>
|
||||
<acceptance_criteria>
|
||||
- Single named cron job `tender-doe-poll`; `setInterval` has no tenant parameter.
|
||||
- No `activeTenantId` field (`grep -c "activeTenantId"` returns 0 — proves the per-tenant anti-pattern was not copied).
|
||||
- `onModuleInit` uses `findUnique` on `sourceType`, not `findFirst`.
|
||||
- `tsc --noEmit` passes.
|
||||
</acceptance_criteria>
|
||||
<done>One global cron drives the shared DÖE poll; no per-tenant scheduling dimension.</done>
|
||||
</task>
|
||||
|
||||
<task type="auto" tdd="true">
|
||||
<name>Task 3: Two-tenant safety integration test (Success Criteria 4 & 5)</name>
|
||||
<read_first>
|
||||
- .planning/phases/10-ausschreibungs-radar-foundation-d-e-ingestion/10-RESEARCH.md (Pitfall D "Verification for this phase's two-tenant acceptance criterion" — assert ABSENCE of tenant-scaled behavior)
|
||||
- apps/api/src/tenders/tender-scheduler.service.spec.ts (create)
|
||||
- apps/api/src/module-registry/module-registry.service.ts (isModuleActive / activation mechanism the test drives)
|
||||
</read_first>
|
||||
<files>apps/api/src/tenders/tender-scheduler.service.spec.ts</files>
|
||||
<action>
|
||||
Write an integration test proving the poll-once-fan-out-many property. Spy on the DÖE adapter's `fetchTenders` (or on the underlying `fetch`) and on `schedulerRegistry.addCronJob`. Simulate activating the `tender-radar` module for tenant A, run/inspect scheduler init, capture counts. Then activate the module for a SECOND tenant B and assert: zero ADDITIONAL DÖE adapter/HTTP calls, zero additional cron jobs registered (still exactly one `tender-doe-poll`), and zero additional `Tender` rows created purely as a result of the 2nd activation. The assertion is the ABSENCE of tenant-count-scaled behavior (there are no per-tenant DÖE configs to iterate) — not correct per-tenant iteration. This is the phase's headline acceptance criterion.
|
||||
</action>
|
||||
<verify>
|
||||
<automated>cd apps/api && pnpm test -- tender-scheduler 2>&1 | tail -20</automated>
|
||||
</verify>
|
||||
<acceptance_criteria>
|
||||
- Test asserts: after a 2nd tenant activation → additional DÖE fetch calls == 0, additional cron jobs == 0, additional Tender rows == 0.
|
||||
- `pnpm --filter @tessera/api test -- tender-scheduler` green.
|
||||
</acceptance_criteria>
|
||||
<done>Two-tenant safety proven by an automated integration test.</done>
|
||||
</task>
|
||||
|
||||
</tasks>
|
||||
|
||||
<threat_model>
|
||||
## Trust Boundaries
|
||||
|
||||
| Boundary | Description |
|
||||
|----------|-------------|
|
||||
| background job → global DB | Scheduled ingestion writes to the platform-global Tender table with no tenant context |
|
||||
| tenant activation → scheduler | A tenant activating the module must not alter platform-wide poll behavior |
|
||||
|
||||
## STRIDE Threat Register
|
||||
|
||||
| Threat ID | Category | Component | Severity | Disposition | Mitigation Plan |
|
||||
|-----------|----------|-----------|----------|-------------|-----------------|
|
||||
| T-10-09 | Information Disclosure | `TenderIngestionService` DB access | high | mitigate | Uses plain `PrismaService` (no `forTenant()`) on the global Tender/config tables by design; wrapping global tables in tenant RLS would silently hide platform data from a 2nd tenant. Verified by the no-forTenant assertion + two-tenant integration test |
|
||||
| T-10-10 | Elevation of Privilege | scheduler per-tenant scaling | high | mitigate | Exactly one global cron job; `setInterval` takes no tenant arg; singleton config via `findUnique` on a fixed slug. Two-tenant integration test asserts zero additional jobs/calls/rows on a 2nd activation (Success Criteria 4 & 5) |
|
||||
| T-10-11 | Tampering | outbound `pubDay` URL | medium | mitigate | The day-cursor is internally computed from `lastIngestedDay` (never user-supplied); `nextDayToFetch` bounds it to strictly-past days; no admin "fetch date X" feature added in this phase |
|
||||
| T-10-12 | Denial of Service | catch-up loop hammering DÖE | low | mitigate | ~1-2s polite delay between successive day fetches during catch-up (RESEARCH Open Question 1); day-cursor gate prevents re-fetching the same day repeatedly |
|
||||
</threat_model>
|
||||
|
||||
<verification>
|
||||
- `pnpm --filter @tessera/api test -- tender-ingestion tender-scheduler` all green.
|
||||
- SCHEMA-02 change detection proven (update-not-duplicate on contentHash change).
|
||||
- INGEST-06 poll-once proven (single global cron, two-tenant test).
|
||||
- D-01 day-cursor no-op, D-05 retention with null-deadline skip proven.
|
||||
- No `forTenant` / no `activeTenantId` in the tender services (grep gates green).
|
||||
</verification>
|
||||
|
||||
<success_criteria>
|
||||
- SCHEMA-02: a changed DÖE notice updates its existing record instead of duplicating.
|
||||
- INGEST-06: DÖE polled once on a shared admin-configurable interval regardless of tenant count.
|
||||
- Success Criteria 4 & 5: 2nd-tenant activation triggers no redundant poll / no duplicate ingestion.
|
||||
</success_criteria>
|
||||
|
||||
<output>
|
||||
Create `.planning/phases/10-ausschreibungs-radar-foundation-d-e-ingestion/10-04-SUMMARY.md` when done.
|
||||
</output>
|
||||
@@ -0,0 +1,165 @@
|
||||
---
|
||||
phase: 10-ausschreibungs-radar-foundation-d-e-ingestion
|
||||
plan: 05
|
||||
type: execute
|
||||
wave: 5
|
||||
depends_on: ["10-01", "10-02", "10-03", "10-04"]
|
||||
files_modified:
|
||||
- apps/api/src/tenders/dto/source-config.dto.ts
|
||||
- apps/api/src/tenders/dto/tender-query.dto.ts
|
||||
- apps/api/src/tenders/tenders.controller.ts
|
||||
- apps/api/src/tenders/tenders.controller.spec.ts
|
||||
- apps/api/src/tenders/tenders.module.ts
|
||||
autonomous: true
|
||||
requirements: [INGEST-06]
|
||||
must_haves:
|
||||
truths:
|
||||
- "Admin can read and update the shared DÖE poll interval / active state; saving pushes the change into the live scheduler without restart (INGEST-06 admin-configurable)"
|
||||
- "GET /tenders list + detail is gated by module activation (ModuleGuard), NOT by tenantId row-filtering — the global catalog is visible to any tenant with the module active"
|
||||
- "Admin source-config routes require ADMIN/SUPER_ADMIN roles"
|
||||
artifacts:
|
||||
- apps/api/src/tenders/tenders.controller.ts
|
||||
- apps/api/src/tenders/dto/source-config.dto.ts
|
||||
- apps/api/src/tenders/dto/tender-query.dto.ts
|
||||
key_links:
|
||||
- "PUT source-config → TenderSchedulerService.setInterval()/stopJob() applies the change live (DKV saveConfig pattern, minus the tenantId arg)"
|
||||
- "GET /tenders uses @UseModule('tender-radar') ModuleGuard, never a where:{tenantId} filter"
|
||||
---
|
||||
|
||||
<objective>
|
||||
Round out INGEST-06: expose the admin-configurable shared poll interval and a read endpoint over the global tender catalog. This is the boundary where "tenant-gated" and "tenant-scoped" genuinely differ — the read must be gated by module activation but NOT row-filtered by tenant.
|
||||
|
||||
## Phase Goal (user story)
|
||||
**As a** Tessera-Administrator, **I want to** das DÖE-Poll-Intervall im Admin-Bereich einstellen und die erfassten Ausschreibungen ueber eine API abrufen, **so that** ich den gemeinsamen Zeitplan steuern kann und Phase 11 eine Trefferliste darauf aufbauen kann (INGEST-06).
|
||||
|
||||
Purpose: Completes the admin-configurable half of INGEST-06 and gives Phase 11 a read surface. The controller intentionally diverges from DKV: list/detail are platform-global reads gated only by module licensing.
|
||||
Output: `TendersController`, `SourceConfigDto`, `TenderQueryDto`, controller test.
|
||||
</objective>
|
||||
|
||||
<execution_context>
|
||||
@$HOME/.claude/gsd-core/workflows/execute-plan.md
|
||||
@$HOME/.claude/gsd-core/templates/summary.md
|
||||
</execution_context>
|
||||
|
||||
<context>
|
||||
@.planning/PROJECT.md
|
||||
@.planning/ROADMAP.md
|
||||
@.planning/phases/10-ausschreibungs-radar-foundation-d-e-ingestion/10-PATTERNS.md
|
||||
@.planning/phases/10-ausschreibungs-radar-foundation-d-e-ingestion/10-RESEARCH.md
|
||||
@apps/api/src/dkv/dkv.controller.ts
|
||||
@apps/api/src/cert-manager/cert-manager.controller.ts
|
||||
@apps/api/src/dkv/dto/dkv-config.dto.ts
|
||||
@apps/api/src/dkv/dto/dkv-history.dto.ts
|
||||
</context>
|
||||
|
||||
<tasks>
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 1: SourceConfigDto + TenderQueryDto</name>
|
||||
<read_first>
|
||||
- apps/api/src/dkv/dto/dkv-config.dto.ts (class-validator conventions: @IsOptional + @Min/@Max, DoS floor)
|
||||
- apps/api/src/dkv/dto/dkv-history.dto.ts (pagination DTO: page/limit + @Type(() => Number))
|
||||
- .planning/phases/10-ausschreibungs-radar-foundation-d-e-ingestion/10-PATTERNS.md (dto sections)
|
||||
</read_first>
|
||||
<files>apps/api/src/tenders/dto/source-config.dto.ts, apps/api/src/tenders/dto/tender-query.dto.ts</files>
|
||||
<action>
|
||||
`SourceConfigDto`: `pollIntervalMin?` (`@IsOptional @IsInt @Min(5) @Max(1440)` — same DoS floor as DkvConfigDto) and `isActive?` (`@IsOptional @IsBoolean`). Document in a comment that `pollIntervalMin` is the cron-tick frequency (D-04 default 60), decoupled from the day-granularity DÖE fetch (day-cursor gated in the ingestion service).
|
||||
|
||||
`TenderQueryDto`: pagination `page?`/`limit?` copied verbatim from `dkv-history.dto.ts` (`@Type(() => Number)` coercion, `@Min(1)`, `limit @Max(100)`), plus an optional `status?` filter (`@IsOptional @IsIn(['active','expired'])`) defaulting to active-only at the query layer. No region/CPV filters here — rich filtering is Phase 11 (FILTER-*).
|
||||
</action>
|
||||
<verify>
|
||||
<automated>cd apps/api && pnpm exec tsc --noEmit -p tsconfig.json 2>&1 | tail -5</automated>
|
||||
</verify>
|
||||
<acceptance_criteria>
|
||||
- Both DTOs compile with class-validator decorators; numeric bounds present.
|
||||
- `SourceConfigDto.pollIntervalMin` has `@Min(5)`.
|
||||
</acceptance_criteria>
|
||||
<done>DTOs defined following the DKV validation conventions.</done>
|
||||
</task>
|
||||
|
||||
<task type="auto">
|
||||
<name>Task 2: TendersController — global read (ModuleGuard) + admin source-config (Roles) + live scheduler apply</name>
|
||||
<read_first>
|
||||
- apps/api/src/cert-manager/cert-manager.controller.ts (@UseModule('...') ModuleGuard usage — the module-activation gate for global reads)
|
||||
- apps/api/src/dkv/dkv.controller.ts (per-handler @Roles(ADMIN, SUPER_ADMIN) on config routes; saveConfig → scheduler.setInterval/stopJob pattern, lines ~76-90)
|
||||
- .planning/phases/10-ausschreibungs-radar-foundation-d-e-ingestion/10-PATTERNS.md (tenders.controller section — divergence from DKV)
|
||||
</read_first>
|
||||
<files>apps/api/src/tenders/tenders.controller.ts, apps/api/src/tenders/tenders.module.ts</files>
|
||||
<action>
|
||||
Create `TendersController` (`@Controller('modules/tender-radar')`). Read routes are GLOBAL (V4 divergence): `GET /` (list, paginated via `TenderQueryDto`) and `GET /:id` (detail) query the global `Tender` table via plain `PrismaService` with `where` built ONLY from the query DTO (status/pagination) — never a `where: { tenantId }` filter. Gate these reads with `@UseModule('tender-radar')` (ModuleGuard) so only tenants with the module active can read, without row-scoping the global catalog (RESEARCH V4: this is the one controller where tenant-gated differs from tenant-scoped). Return 404 via `NotFoundException` for a missing id.
|
||||
|
||||
Admin source-config routes: `GET /source-config` and `PUT /source-config`, each decorated `@Roles(Role.ADMIN, Role.SUPER_ADMIN)` (per-handler, like DkvController). `GET` returns the singleton `doe-opendata` config. `PUT` (body `SourceConfigDto`) upserts the singleton config, then applies it live to the scheduler: if `isActive && pollIntervalMin` then call `tenderScheduler.setInterval(pollIntervalMin)` (NO tenant arg — divergence from DKV); if `isActive === false` then call `tenderScheduler.stopJob()`. This satisfies INGEST-06 "admin-configurable interval, applied without restart".
|
||||
|
||||
Register `TendersController` in `TendersModule.controllers` and confirm `TenderSchedulerService`/`TenderIngestionService` are in providers so DI resolves.
|
||||
</action>
|
||||
<verify>
|
||||
<automated>cd apps/api && pnpm exec tsc --noEmit -p tsconfig.json 2>&1 | tail -5 && grep -c "UseModule" src/tenders/tenders.controller.ts</automated>
|
||||
</verify>
|
||||
<acceptance_criteria>
|
||||
- `GET /` and `GET /:id` gated by `@UseModule('tender-radar')`; no `where: { tenantId }` in the controller (assert `grep -c "tenantId" tenders.controller.ts` returns 0 for read routes — the global catalog is not row-scoped).
|
||||
- Both `source-config` routes carry `@Roles(Role.ADMIN, Role.SUPER_ADMIN)`.
|
||||
- `PUT /source-config` calls `tenderScheduler.setInterval`/`stopJob` (no tenant arg).
|
||||
- `tsc --noEmit` passes.
|
||||
</acceptance_criteria>
|
||||
<done>Global read + admin config wired; scheduler updates live on save.</done>
|
||||
</task>
|
||||
|
||||
<task type="auto" tdd="true">
|
||||
<name>Task 3: Controller test — global read not tenant-scoped + admin config applies to scheduler</name>
|
||||
<read_first>
|
||||
- apps/api/src/cert-manager/cert-manager.service.spec.ts (vitest spec style)
|
||||
- apps/api/src/tenders/tenders.controller.ts (unit under test)
|
||||
</read_first>
|
||||
<behavior>
|
||||
- Test: GET list returns global tenders with no tenantId filter passed to prisma (assert the prisma.tender.findMany where has no tenantId key).
|
||||
- Test: PUT /source-config with isActive=true + pollIntervalMin=30 calls scheduler.setInterval(30) with a single argument.
|
||||
- Test: PUT /source-config with isActive=false calls scheduler.stopJob().
|
||||
</behavior>
|
||||
<files>apps/api/src/tenders/tenders.controller.spec.ts</files>
|
||||
<action>
|
||||
Write a Vitest spec mocking `PrismaService` and `TenderSchedulerService`. Assert the read path never adds a `tenantId` to the prisma `where`, and that saving source-config drives `setInterval(intervalMin)` / `stopJob()` exactly as the DKV analog does (minus the tenant argument). This is the automated proof for the INGEST-06 admin-config surface and the tenant-gated-not-scoped invariant.
|
||||
</action>
|
||||
<verify>
|
||||
<automated>cd apps/api && pnpm test -- tenders.controller 2>&1 | tail -15</automated>
|
||||
</verify>
|
||||
<acceptance_criteria>
|
||||
- Read-path test asserts no `tenantId` in the prisma `where`.
|
||||
- Config-save tests assert `setInterval(n)` (single arg) and `stopJob()` are invoked.
|
||||
- `pnpm --filter @tessera/api test -- tenders.controller` green.
|
||||
</acceptance_criteria>
|
||||
<done>Controller behaviour has automated coverage; green.</done>
|
||||
</task>
|
||||
|
||||
</tasks>
|
||||
|
||||
<threat_model>
|
||||
## Trust Boundaries
|
||||
|
||||
| Boundary | Description |
|
||||
|----------|-------------|
|
||||
| client → GET /tenders | Any authenticated tenant user with the module active reads the global catalog |
|
||||
| admin → PUT /source-config | Privileged mutation of the platform-wide poll schedule |
|
||||
|
||||
## STRIDE Threat Register
|
||||
|
||||
| Threat ID | Category | Component | Severity | Disposition | Mitigation Plan |
|
||||
|-----------|----------|-----------|----------|-------------|-----------------|
|
||||
| T-10-13 | Elevation of Privilege | `PUT /source-config` | high | mitigate | Both source-config routes require `@Roles(Role.ADMIN, Role.SUPER_ADMIN)` (per-handler, like DkvController); global JwtAuthGuard + RolesGuard enforce it |
|
||||
| T-10-14 | Broken Access Control | `GET /tenders` gating | high | mitigate | Reads gated by `@UseModule('tender-radar')` ModuleGuard (tenant must have the module active) — deliberately NOT row-scoped by tenantId, since the catalog is platform-global (V4: tenant-gated ≠ tenant-scoped). Verified by the no-tenantId controller test |
|
||||
| T-10-15 | Input Validation | `TenderQueryDto` / `SourceConfigDto` | medium | mitigate | class-validator bounds on pagination (`limit @Max(100)`) and interval (`@Min(5) @Max(1440)`) mitigate DoS via oversized queries / runaway poll frequency |
|
||||
</threat_model>
|
||||
|
||||
<verification>
|
||||
- `pnpm --filter @tessera/api test -- tenders.controller` green.
|
||||
- GET reads gated by ModuleGuard, not tenantId-filtered (grep + test).
|
||||
- Admin source-config Roles-guarded; save applies to the live scheduler.
|
||||
- `tsc --noEmit` clean for both apps.
|
||||
</verification>
|
||||
|
||||
<success_criteria>
|
||||
- INGEST-06: admin-configurable shared poll interval, applied to the live scheduler without restart, plus a global read surface for the ingested catalog.
|
||||
</success_criteria>
|
||||
|
||||
<output>
|
||||
Create `.planning/phases/10-ausschreibungs-radar-foundation-d-e-ingestion/10-05-SUMMARY.md` when done.
|
||||
</output>
|
||||
+39
-24
@@ -2,7 +2,7 @@
|
||||
phase: 10
|
||||
slug: ausschreibungs-radar-foundation-d-e-ingestion
|
||||
status: draft
|
||||
nyquist_compliant: false
|
||||
nyquist_compliant: true
|
||||
wave_0_complete: false
|
||||
created: 2026-07-21
|
||||
---
|
||||
@@ -17,20 +17,20 @@ created: 2026-07-21
|
||||
|
||||
| Property | Value |
|
||||
|----------|-------|
|
||||
| **Framework** | {pytest 7.x / jest 29.x / vitest / go test / other} |
|
||||
| **Config file** | {path or "none — Wave 0 installs"} |
|
||||
| **Quick run command** | `{quick command}` |
|
||||
| **Full suite command** | `{full command}` |
|
||||
| **Estimated runtime** | ~10 seconds |
|
||||
| **Framework** | Vitest 3.x (existing, `apps/api/vitest.config.ts`) |
|
||||
| **Config file** | `apps/api/vitest.config.ts` (existing — not modified by this phase) |
|
||||
| **Quick run command** | `pnpm --filter @tessera/api test -- tenders` |
|
||||
| **Full suite command** | `pnpm --filter @tessera/api test` |
|
||||
| **Estimated runtime** | ~10 seconds (scoped) |
|
||||
|
||||
---
|
||||
|
||||
## Sampling Rate
|
||||
|
||||
- **After every task commit:** Run `{quick run command}`
|
||||
- **After every plan wave:** Run `{full suite command}`
|
||||
- **After every task commit:** Run `pnpm --filter @tessera/api test -- tenders`
|
||||
- **After every plan wave:** Run `pnpm --filter @tessera/api test`
|
||||
- **Before `/gsd-verify-work`:** Full suite must be green
|
||||
- **Max feedback latency:** 10 seconds
|
||||
- **Max feedback latency:** ~10 seconds (scoped run)
|
||||
|
||||
---
|
||||
|
||||
@@ -38,7 +38,21 @@ created: 2026-07-21
|
||||
|
||||
| Task ID | Plan | Wave | Requirement | Threat Ref | Secure Behavior | Test Type | Automated Command | File Exists | Status |
|
||||
|---------|------|------|-------------|------------|-----------------|-----------|-------------------|-------------|--------|
|
||||
| 10-01-01 | 01 | 1 | REQ-{XX} | T-10-01 / — | {expected secure behavior or "N/A"} | unit | `{command}` | ✅ / ❌ W0 | ⬜ pending |
|
||||
| 10-01-01 | 01 | 1 | SCHEMA-01 | T-10-SC | adm-zip verified before install (supply chain) | manual-gate | checkpoint:human-verify (blocking-human) | n/a | ⬜ pending |
|
||||
| 10-01-02 | 01 | 1 | SCHEMA-01 | T-10-01 | Tender has no tenantId (global data) | build | `grep -c "model Tender" schema.prisma`; `tsc --noEmit` | ❌ W0 | ⬜ pending |
|
||||
| 10-01-03 | 01 | 1 | SCHEMA-01 | T-10-01 | [BLOCKING] migration applied to DB | integration | `pnpm --filter @tessera/api exec prisma migrate status` | ❌ W0 | ⬜ pending |
|
||||
| 10-02-01 | 02 | 2 | CONFIG-01 | T-10-05 | idempotent registry seed + singleton config | build | `tsc --noEmit`; `grep -c "TendersModule" app.module.ts` | ❌ W0 | ⬜ pending |
|
||||
| 10-02-02 | 02 | 2 | CONFIG-01 | T-10-03 | whitelisted slug only | build | `grep -c "tender-radar" module-loader.ts`; web `tsc --noEmit` | ❌ W0 | ⬜ pending |
|
||||
| 10-02-03 | 02 | 2 | CONFIG-01 | T-10-04 | seed asserts slug + isSystem | unit | `pnpm --filter @tessera/api test -- tenders.seed` | ❌ W0 | ⬜ pending |
|
||||
| 10-03-01 | 03 | 3 | INGEST-01/SCHEMA-01 | T-10-06/07 | real fixtures + failing specs (RED) | unit | `pnpm --filter @tessera/api test -- doe-opendata.adapter tender-normalizer` | ❌ W0 | ⬜ pending |
|
||||
| 10-03-02 | 03 | 3 | INGEST-01 | T-10-06/07 | fixed host fetch + zip-bomb ceiling + D-02 filter | unit | `pnpm --filter @tessera/api test -- doe-opendata.adapter` | ❌ W0 | ⬜ pending |
|
||||
| 10-03-03 | 03 | 3 | SCHEMA-01 | T-10-08 | eForms-primary deadline, nullable value, dedupKey/hash | unit | `pnpm --filter @tessera/api test -- tender-normalizer` | ❌ W0 | ⬜ pending |
|
||||
| 10-04-01 | 04 | 4 | SCHEMA-02 | T-10-09/11 | day-cursor no-op; update-not-duplicate; D-05 prune skips null | integration | `pnpm --filter @tessera/api test -- tender-ingestion` | ❌ W0 | ⬜ pending |
|
||||
| 10-04-02 | 04 | 4 | INGEST-06 | T-10-10 | single global cron, no activeTenantId | build | `tsc --noEmit`; `grep -c "activeTenantId" tender-scheduler.service.ts` == 0 | ❌ W0 | ⬜ pending |
|
||||
| 10-04-03 | 04 | 4 | INGEST-06 | T-10-10 | 2nd-tenant activation → 0 extra calls/jobs/rows | integration | `pnpm --filter @tessera/api test -- tender-scheduler` | ❌ W0 | ⬜ pending |
|
||||
| 10-05-01 | 05 | 5 | INGEST-06 | T-10-15 | DTO bounds (interval floor, limit cap) | build | `tsc --noEmit` | ❌ W0 | ⬜ pending |
|
||||
| 10-05-02 | 05 | 5 | INGEST-06 | T-10-13/14 | ModuleGuard read (not tenant-scoped) + admin Roles | build | `tsc --noEmit`; `grep -c "UseModule" tenders.controller.ts` | ❌ W0 | ⬜ pending |
|
||||
| 10-05-03 | 05 | 5 | INGEST-06 | T-10-14 | read where has no tenantId; save drives scheduler | unit | `pnpm --filter @tessera/api test -- tenders.controller` | ❌ W0 | ⬜ pending |
|
||||
|
||||
*Status: ⬜ pending · ✅ green · ❌ red · ⚠️ flaky*
|
||||
|
||||
@@ -46,11 +60,13 @@ created: 2026-07-21
|
||||
|
||||
## Wave 0 Requirements
|
||||
|
||||
- [ ] `{tests/test_file.py}` — stubs for REQ-{XX}
|
||||
- [ ] `{tests/conftest.py}` — shared fixtures
|
||||
- [ ] `{framework install}` — if no framework detected
|
||||
- [ ] `apps/api/src/tenders/__fixtures__/doe-eforms-sample.zip` + `doe-ocds-sample.zip` — real captured, trimmed fixtures (Plan 03 Task 1)
|
||||
- [ ] `apps/api/src/tenders/tenders.seed.spec.ts` (Plan 02 Task 3)
|
||||
- [ ] `apps/api/src/tenders/adapters/doe-opendata.adapter.spec.ts`, `tender-normalizer.service.spec.ts` (Plan 03 Task 1 — RED first)
|
||||
- [ ] `apps/api/src/tenders/tender-ingestion.service.spec.ts`, `tender-scheduler.service.spec.ts` (Plan 04)
|
||||
- [ ] `apps/api/src/tenders/tenders.controller.spec.ts` (Plan 05 Task 3)
|
||||
|
||||
*If none: "Existing infrastructure covers all phase requirements."*
|
||||
Vitest framework already exists — no framework install needed.
|
||||
|
||||
---
|
||||
|
||||
@@ -58,19 +74,18 @@ created: 2026-07-21
|
||||
|
||||
| Behavior | Requirement | Why Manual | Test Instructions |
|
||||
|----------|-------------|------------|-------------------|
|
||||
| {behavior} | REQ-{XX} | {reason} | {steps} |
|
||||
|
||||
*If none: "All phase behaviors have automated verification."*
|
||||
| adm-zip package legitimacy | (supply chain / T-10-SC) | Human judgment on package provenance ([ASSUMED]) | Plan 01 Task 1 checkpoint: verify on npmjs.com + github.com/cthackers/adm-zip |
|
||||
| Marketplace activation opens module page | CONFIG-01 | Full portal round-trip (marketplace → activate → page render) is a browser flow | Activate Ausschreibungs-Radar for a tenant, open `/modules/tender-radar`, confirm no 404 |
|
||||
|
||||
---
|
||||
|
||||
## Validation Sign-Off
|
||||
|
||||
- [ ] All tasks have `<automated>` verify or Wave 0 dependencies
|
||||
- [ ] Sampling continuity: no 3 consecutive tasks without automated verify
|
||||
- [ ] Wave 0 covers all MISSING references
|
||||
- [ ] No watch-mode flags
|
||||
- [ ] Feedback latency < 10s
|
||||
- [ ] `nyquist_compliant: true` set in frontmatter
|
||||
- [x] All tasks have `<automated>` verify or a documented Wave 0 / manual-gate dependency
|
||||
- [x] Sampling continuity: no 3 consecutive tasks without automated verify
|
||||
- [x] Wave 0 covers all MISSING references (fixtures + spec scaffolds)
|
||||
- [x] No watch-mode flags (all use `vitest run` via `pnpm test`)
|
||||
- [x] Feedback latency < 10s (scoped runs)
|
||||
- [x] `nyquist_compliant: true` set in frontmatter
|
||||
|
||||
**Approval:** {pending / approved YYYY-MM-DD}
|
||||
**Approval:** approved 2026-07-21
|
||||
|
||||
Reference in New Issue
Block a user