Commit Graph

605 Commits

Author SHA1 Message Date
schalli f9591dc491 feat(11-03): cpvDivisions column — normalizer + backfill migration
Adds Tender.cpvDivisions String[] (@@index Gin) derived at ingestion time
via cpv-catalog.ts's divisionOf() — replaces exact-match cpvCodes
comparison with a typesafe, GIN-indexable hasSome target (FILTER-03,
Pitfall 2). Backfill migration 20260721150000_tender_cpv_divisions_backfill
applied locally: 1612/1671 rows populated across all observed divisions
(741 rows carry division '45' — Bauarbeiten); idempotent (second run:
UPDATE 0, ADD COLUMN IF NOT EXISTS / CREATE INDEX IF NOT EXISTS both skip
cleanly). Applied via docker exec psql + `prisma migrate resolve
--applied` + `prisma generate` against the local dev DB only — no
Docker deploy on the test server.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-21 16:11:40 +02:00
schalli 7651ab6813 feat(11-03): implement CPV-Divisions-Katalog + normalizeCpv (GREEN)
Static 45-entry EU-CPV division catalog with German labels plus
normalizeCpv/divisionOf/cpvMatchesDivisions helpers. Two-sided
normalization to the leading 2-digit division handles all three live-DB
formats ("45", "45000000", "45000000-7") without an exact-match
Pitfall-2 bug. No full EU CPV vocabulary shipped (D-03, Open Question 2
RESOLVED) — no new npm dependency, static data file only.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-21 16:08:47 +02:00
schalli a64f98463c test(11-03): add failing spec for CPV-Divisions-Katalog + normalizeCpv
RED-first (TDD): normalizeCpv/divisionOf/cpvMatchesDivisions across the
three inconsistent live-DB CPV formats ("45", "45000000", "45000000-7")
plus a CPV_DIVISIONS catalog shape check — module does not exist yet.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-21 16:08:07 +02:00
schalli a4c02116c1 docs(11-02): complete region/plz/bundesland filter plan 2026-07-21 16:05:25 +02:00
schalli 69e25298c9 feat(11-02): wire Region/PLZ/Bundesland filter into query builder + FilterPanel
TenderQueryDto gains validated plz (@MaxLength(5)), region, and
bundesland fields. buildTenderWhere adds three conditional AND-branches:
plz startsWith, bundesland exact-match against the now-backfilled indexed
column (Pitfall 1), and region startsWith (usable independent of the
bundesland column). FilterPanel gets a PLZ input and a 16-Land Bundesland
dropdown (mirrors NUTS1_BUNDESLAND — web/api are separate packages) that
write plz/bundesland into the URL searchParams; ResultsList already
forwards the full URLSearchParams to listTenders(), so no additional
fetch wiring was needed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-21 16:03:21 +02:00
schalli e4db602597 feat(11-02): normalizer derives bundesland + backfill migration for existing rows
Replace the Phase-10-deferred `bundesland = null` assignment with
bundeslandFromRegion(region) so new ingests are Bundesland-filterable
immediately. Add @@index([bundesland]) for filter performance. New
handwritten migration 20260721140000_tender_bundesland_backfill backfills
the ~1671 pre-existing rows (idempotent UPDATE, only where bundesland IS
NULL AND region IS NOT NULL) — applied locally via
`docker exec tessera-ctl-db-1 psql`, resolved as applied in
_prisma_migrations, and prisma generate re-run.

Verified on the local dev DB: 933/1671 rows now have bundesland set
across all 16 Länder (738 remain NULL where region itself is NULL).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-21 16:01:32 +02:00
schalli d12e1c9aa9 feat(11-02): implement NUTS-1 to Bundesland derivation
bundeslandFromRegion() derives one of the 16 Bundesland names from a
region's NUTS-1 (3-char) prefix; nutsPrefixFor() reverses a Bundesland
name back to its prefix for the query builder. Null-safe throughout —
7/7 tests green, validated against real region samples from the live DB.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-21 15:59:35 +02:00
schalli 3dec35f498 test(11-02): add failing spec for NUTS-1 to Bundesland derivation
RED: bundeslandFromRegion/nutsPrefixFor do not exist yet. Locks the
derivation (16 NUTS-1 prefixes + null-safety + real DB region samples)
that makes the Bundesland filter return real hits (Pitfall 1, FILTER-02).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-21 15:59:14 +02:00
schalli 3e064e0124 docs(11-01): complete query-engine + results-ui + coverage-banner plan 2026-07-21 15:57:25 +02:00
schalli ee2c40863e feat(11-01): replace tender-radar placeholder with real results UI (GREEN)
Replaces the Phase 10 module stub with a Master-Container page.tsx
(Suspense-wrapped for useSearchParams, Next 16 App Router) rendering
CoverageBanner + FilterPanel + ResultsList.

- ResultsList: URL-param-driven fetch via listTenders(), sortable
  Frist/Wert/Veröffentlicht column headers, "keine Wertangabe" for
  estimatedValue=null rows (D-05), pagination.
- FilterPanel: Freitext (q), Sortierung, "nur noch offene" toggle
  (openOnly, default on), Abgabefrist von/bis date inputs — param names
  match the TenderQueryDto field names 1:1 for the Plan 11-06
  Saved-Search serialization contract.
- CoverageBanner: German coverage hint shown while GET /coverage
  reports only the doe-opendata source (D-12, UI-05).

All strings hardcoded German (i18n = Phase 14). Plain fetch throughout,
no TanStack Query (not installed, per RESEARCH Open Question 4).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-21 15:54:51 +02:00
schalli b55bb55c0e test(11-01): add failing ResultsList spec + extend tender-radar-api client (RED)
Extends tender-radar-api.ts with the Tender type, listTenders(params)
and fetchCoverage() (plain fetch, credentials:'include', matching the
established fetchSourceConfig pattern — TanStack Query is not installed).

Adds the RED-first ResultsList.test.tsx: render items with
title/buyerName/deadline/value, "keine Wertangabe" for null estimatedValue
(D-05), and an empty-state message. ResultsList.tsx does not exist yet.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-21 15:53:04 +02:00
schalli 49622ba022 feat(11-01): wire listTenders through the query builder + add GET /coverage
listTenders now delegates where/orderBy composition to
tender-query.builder.ts (buildTenderWhere/buildOrderBy) instead of the
fixed status/publishedAt clause; pagination bounds unchanged (T-10-15).

New GET /modules/tender-radar/coverage handler returns active-tender
counts grouped by sourcePortal (D-12, UI-05 coverage banner data
source). Declared before @Get(':id') — same static-route-before-:id
convention as source-config (Pitfall 5, Phase-10 regression guard).

Extends tenders.controller.spec.ts: sort whitelist pass-through,
unknown-sort fallback, pagination skip/take, coverage response shape,
and a declaration-order regression test for getCoverage.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-21 15:50:23 +02:00
schalli fed5ecbc43 feat(11-01): implement tender-query.builder (GREEN)
buildTenderWhere: conditional Prisma where-builder covering keyword
(D-01), openOnly deadline default + explicit deadline range (D-04),
and NULL-graceful value filter (D-05, Kern-Test: value filter never
eliminates estimatedValue=null rows). buildOrderBy: sort whitelist
(deadline/value/published) defaulting to publishedAt desc (UI-01,
T-11-01 — no dynamic orderBy keys from user input).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-21 15:48:56 +02:00
schalli 426a1ea3b8 test(11-01): add failing tender-query.builder spec (RED)
Extends TenderQueryDto with validated filter/sort params (q, openOnly,
deadlineFrom/To, valueMin/Max, includeNullValue, sort) and adds the
RED-first spec for the not-yet-implemented tender-query.builder.ts:
NULL-graceful value filter (D-05), openOnly deadline default (D-04),
explicit deadline range, and sort whitelist (UI-01).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-21 15:47:40 +02:00
schalli 34b8df28d5 docs(11): validation strategy + state update — plans PASS, ready to execute
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-21 15:28:02 +02:00
schalli d7b8e9bf5f docs(11): revise phase plan per checker BLOCK — VALIDATION.md, FILTER-04 date range, serialized deps
- Add 11-VALIDATION.md (Nyquist Dimension 8: per-task test map, sampling, Wave-0 gaps)
- Implement FILTER-04 deadlineFrom/deadlineTo (DTO + builder branch + FilterPanel) in 11-01
- Serialize depends_on into a linear chain (11-04<-03, 11-05<-04, 11-06<-05) to prevent parallel shared-file/migration corruption
- Mark 11-RESEARCH Open Questions RESOLVED
- Fix 11-05 Task 2 <files> dto/ path typo

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-21 15:26:18 +02:00
schalli 4036991acd docs(11): create phase plan — 6 vertical-slice plans (filter engine, results UI, saved searches)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-21 15:17:48 +02:00
schalli 930a8d8f79 docs(11): phase research — live-DB findings, query patterns, new models
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-21 15:07:41 +02:00
schalli a4a661f829 docs(11): phase context from user requirements — filters, triage, saved searches
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-21 14:57:09 +02:00
schalli 48d1246043 fix(10): resolve GET /source-config 404 shadowed by :id route
The admin source-config settings form failed to load with "Failed to
fetch tender-radar source config". Network trace showed
GET /modules/tender-radar/source-config returning 404.

Root cause: NestJS RouterExplorer maps routes in method-declaration
order. `@Get(':id')` was declared before `@Get('source-config')`, so
the param route captured "source-config" as an id and shadowed the
static handler (401 unauthenticated, 404 past the guard — no Tender
with id "source-config").

Fix: declare `@Get('source-config')` before `@Get(':id')`. Add a
declaration-order regression test — unit tests call controller methods
directly, bypass routing, and could never catch route shadowing.

Verified live: settings form now loads real config, interval save
persists and live-re-registers the scheduler (INGEST-06). Phase 10
verification raised human_needed -> passed after full browser UAT.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-21 14:52:17 +02:00
schalli eb10bd3116 wip: pause phase 10 (code complete, UAT+rebuild pending) 2026-07-21 11:36:31 +02:00
schalli a3cef389df docs(10): phase verification — 5/5 must-haves verified, UAT pending docker rebuild
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-21 11:34:07 +02:00
schalli 4f3c6cf5e9 docs(10-06): complete tender-radar admin settings UI plan 2026-07-21 11:27:31 +02:00
schalli 8e3dfda64d test(10-06): SourceConfigForm component test — fetch, save, bounds
- fetch-on-mount populates pollIntervalMin input
- Speichern calls saveSourceConfig with edited interval + isActive
- interval below 5 or above 1440 rejected client-side, no save call
- automated proof for the INGEST-06 admin-UI slice
2026-07-21 11:25:43 +02:00
schalli 4360bc0c6b feat(10-06): tender-radar-api client + admin source-config settings form
- tender-radar-api.ts: fetchSourceConfig/saveSourceConfig hitting GET/PUT
  /modules/tender-radar/source-config (Plan 05 endpoint)
- SourceConfigForm: load-on-mount, numeric interval (5-1440 min, mirrors
  backend SourceConfigDto bounds) + isActive toggle, read-only sourceType/
  lastIngestedDay display
- settings/page.tsx: standard App Router route rendering the form,
  no module-loader whitelist change needed
2026-07-21 11:24:32 +02:00
schalli 6d6302294e docs(10-05): complete tenders controller plan 2026-07-21 11:21:52 +02:00
schalli eaff1d86bb test(10-05): controller spec — global read not tenant-scoped, admin config applies to scheduler
- GET / findMany where clause asserted to have no tenantId key
- GET /:id returns tender / throws NotFoundException for missing id
- PUT /source-config isActive=true+pollIntervalMin=30 -> setInterval(30) single-arg
- PUT /source-config isActive=false -> stopJob()
2026-07-21 11:18:35 +02:00
schalli 7b9b6b8c1c feat(10-05): wire TendersController — global read + admin source-config
- GET / and GET /🆔 paginated global Tender catalog, @UseModule('tender-radar')-gated, never row-scoped by tenant id
- GET/PUT /source-config: @Roles(ADMIN, SUPER_ADMIN)-guarded singleton doe-opendata config
- PUT /source-config live-applies pollIntervalMin/isActive to TenderSchedulerService (setInterval/stopJob, no tenant arg) — INGEST-06
- Registered TendersController in TendersModule.controllers
2026-07-21 11:17:13 +02:00
schalli f6e636c37d feat(10-05): add SourceConfigDto and TenderQueryDto
- SourceConfigDto: pollIntervalMin (@Min(5) @Max(1440)), isActive
- TenderQueryDto: page/limit pagination (copied from DkvHistoryQueryDto) + status filter
2026-07-21 11:15:36 +02:00
schalli 9227cc9e78 docs(10-04): complete ingestion orchestration & multi-tenant-safe scheduler plan 2026-07-21 11:13:39 +02:00
schalli 444c68b8ec test(10-04): add two-tenant safety integration test (Success Criteria 4 & 5)
Proves the phase's headline acceptance criterion: activating tender-radar
for a 2nd tenant triggers zero additional DÖE calls, zero additional cron
jobs (still exactly one 'tender-doe-poll'), and zero additional Tender rows.
Drives the real (unmocked) ModuleRegistryService against a fake prisma to
exercise the genuine activateForTenant() call path.

Passes immediately because Task 2's TenderSchedulerService already
implements the poll-once-fan-out-many invariant correctly — this test
locks in and regression-proofs that already-correct architecture rather
than driving new production code (documented in SUMMARY under TDD Gate
Compliance).
2026-07-21 11:11:09 +02:00
schalli 0ba74108db feat(10-04): implement TenderSchedulerService — single global cron (poll-once-fan-out-many)
- One named cron job 'tender-doe-poll' for the whole platform; setInterval()
  takes no tenant argument (INGEST-06) — reuses DkvSchedulerService's
  CronJob require()-resolution + SchedulerRegistry mechanics, drops the
  per-tenant activeTenantId framing entirely
- onModuleInit() loads the singleton doe-opendata config via findUnique on
  the fixed sourceType slug, never findFirst (Pitfall D)
- Day-cursor gate stays inside TenderIngestionService.pollDueSources() —
  this scheduler only controls cron-tick frequency (Pitfall A separation)
- Registered in TendersModule.providers; ScheduleModule already global via
  AppModule, no re-registration needed
2026-07-21 11:10:21 +02:00
schalli 6fb734bf3c feat(10-04): implement TenderIngestionService — day-cursor gate, upsert change-detect, D-05 retention
- pollDueSources(): singleton doe-opendata config via findUnique (fixed slug,
  not findFirst); day-cursor gate (nextDayToFetch) no-ops when nothing new
  (Pitfall A); catch-up loop from lastIngestedDay+1 to today-1 with a polite
  1.5s delay between successive day-fetches
- prisma.tender.upsert({ where: { dedupKey } }) — SCHEMA-02 change-detection
  seam: identical notice does not duplicate, changed contentHash updates in
  place
- pruneExpiredTenders(): marks active+past-deadline rows 'expired', deletes
  expired rows older than 90 days, never touches deadlineAt=null rows (D-05)
- Plain PrismaService throughout — no tenant RLS extension on the global
  Tender/TenderSourcePollConfig tables (D-03, T-10-09)
- Registered in TendersModule.providers
2026-07-21 11:08:59 +02:00
schalli f9e52ab95f test(10-04): add failing spec for TenderIngestionService (day-cursor gate, SCHEMA-02, D-05)
- Day-cursor no-op gate (Pitfall A)
- SCHEMA-02 change detection: fresh insert, identical no-dup, changed-content update
- Catch-up cursor advance across missed days
- D-05 retention: expire past-deadline, prune >90d expired, never touch null-deadline
- Multi-tenant safety: asserts no forTenant() call in the implementation
2026-07-21 11:07:49 +02:00
schalli 761077814f docs(10-03): complete DÖE source adapter & normalizer plan 2026-07-21 11:01:50 +02:00
schalli 31607df48c feat(10-03): implement TenderNormalizerService — fields, dedupKey, hash
- normalize(raw): eForms-DE XML primary for deadlineAt/estimatedValue/
  procedureType (RESEARCH Pattern 3); OCDS primary for ocid/buyerName/
  title/cpvCodes/region/plz
- deadlineAt/estimatedValue nullable by mandate (RESEARCH Pattern 4) —
  missing data normalizes to null, never thrown or zero
- dedupKey priority: ocid -> sourcePortal:sourceNoticeId fallback
- contentHash = sha256(title+deadlineAt+estimatedValue+status), stable
  across repeat calls, changes when the deadline changes (SCHEMA-02 hook)
- Register TenderNormalizerService in TendersModule.providers
- All tender-normalizer.service.spec.ts tests green (6/6); full API
  suite green (59/59); tsc --noEmit clean
2026-07-21 10:58:33 +02:00
schalli 3764feb50b feat(10-03): implement DoeOpenDataAdapter — fetch, extract, D-02 filter
- Native fetch + AbortController 15s timeout (icon-discovery idiom, no
  axios); URL host hardcoded, only the internally-computed dayCursor is
  interpolated (T-10-06)
- HTTP 400 treated as an expected no-op (pubDay today/future) -> []
- adm-zip extraction with a pre-extraction decompression-bomb ceiling
  check (sum entry.header.size vs ~50MB) before any entry buffer is read
  (T-10-07); entries are never written to disk
- D-02 open-tender filter: positive tag.includes('tender') match only —
  award/planning/untagged-with-awards excluded (Pitfall C)
- Register DoeOpenDataAdapter in TendersModule.providers
- All doe-opendata.adapter.spec.ts tests green (6/6)
2026-07-21 10:57:29 +02:00
schalli f88e2a8141 test(10-03): add failing specs + real DÖE fixtures for adapter/normalizer
- Capture real, trimmed DÖE day-export fixtures (pubDay=2026-07-19, 8
  notices spanning tender/award/planning/untagged-with-awards tag classes)
  under tenders/__fixtures__/, live-downloaded from oeffentlichevergabe.de
- Define RawTenderRecord/NormalizedTenderFields/SourceType (tender.types.ts)
  and TenderSourceAdapter (day-cursor fetchTenders signature per RESEARCH
  Pattern 1)
- RED: doe-opendata.adapter.spec.ts asserts D-02 exact-count filtering,
  HTTP-400 no-op, and a zip-bomb ceiling guard (T-10-07)
- RED: tender-normalizer.service.spec.ts asserts eForms-primary deadline
  recovery where OCDS is null (RESEARCH Pattern 3), nullable deadline/value,
  dedupKey priority (ocid -> sourcePortal:sourceNoticeId), and a stable
  sha256 contentHash
2026-07-21 10:56:13 +02:00
schalli f80d491adc docs(10-02): complete Ausschreibungs-Radar marketplace registration plan 2026-07-21 10:46:30 +02:00
schalli 9a7ed71d7a test(10-02): add seed registration coverage for tender-radar (CONFIG-01)
- asserts seedModule called once with slug 'tender-radar', isSystem: true,
  and a bilingual (de/en) description object
2026-07-21 10:43:43 +02:00
schalli 3292afb913 feat(10-02): add tender-radar module-loader whitelist entry + placeholder page
- MODULE_REGISTRY['tender-radar'] entry (mirrors cert-manager/dkv-fleet shape)
- minimal client-component placeholder page proving activation -> page-load path
- hardcoded German string is an intentional MVP stub; full i18n is CONFIG-03 (Phase 14)
2026-07-21 10:43:28 +02:00
schalli e7b40946c8 feat(10-02): register TendersModule with marketplace self-seed + singleton poll config
- tenders.seed.ts seeds slug 'tender-radar' (isSystem: true, category 'procurement')
- tenders.module.ts self-seeds registry on boot (mirrors DkvModule pattern)
- upserts singleton doe-opendata TenderSourcePollConfig row (global, no forTenant())
- TendersModule added to app.module.ts imports
2026-07-21 10:42:44 +02:00
schalli 6cfda90960 docs(10-01): complete tender-radar foundation plan 2026-07-21 10:40:15 +02:00
schalli 713b1eb748 feat(10-01): add Tender/TenderSourcePollConfig models + ingestion packages
- Install fast-xml-parser, adm-zip, csv-parse in @tessera/api
- Add global Tender model (no tenantId — D-03 platform-global data)
- Add singleton TenderSourcePollConfig (sourceType @unique)
- Handwritten additive migration for both tables
- Regenerate Prisma client
2026-07-21 10:33:24 +02:00
schalli b33213b04b docs(10): begin phase execution
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-21 10:24:28 +02:00
schalli 66efd08dc0 docs(10): mark phase planned (6 plans) + add pattern map
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-21 10:10:41 +02:00
schalli b52a0a796b docs(10): add admin config UI plan (INGEST-06 gap), resolve research open questions 2026-07-21 10:08:40 +02:00
schalli e7b59d16af docs(10): create phase plan (5 plans, DÖE ingestion foundation) 2026-07-21 10:03:19 +02:00
schalli 1a6061bc0e docs(phase-10): add validation strategy 2026-07-21 09:47:16 +02:00
schalli 3a51d9bfed docs(10): research DÖE OpenData API live verification and phase domain 2026-07-21 09:46:20 +02:00