Commit Graph

393 Commits

Author SHA1 Message Date
schalli fed5ecbc43 feat(11-01): implement tender-query.builder (GREEN)
buildTenderWhere: conditional Prisma where-builder covering keyword
(D-01), openOnly deadline default + explicit deadline range (D-04),
and NULL-graceful value filter (D-05, Kern-Test: value filter never
eliminates estimatedValue=null rows). buildOrderBy: sort whitelist
(deadline/value/published) defaulting to publishedAt desc (UI-01,
T-11-01 — no dynamic orderBy keys from user input).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-21 15:48:56 +02:00
schalli 426a1ea3b8 test(11-01): add failing tender-query.builder spec (RED)
Extends TenderQueryDto with validated filter/sort params (q, openOnly,
deadlineFrom/To, valueMin/Max, includeNullValue, sort) and adds the
RED-first spec for the not-yet-implemented tender-query.builder.ts:
NULL-graceful value filter (D-05), openOnly deadline default (D-04),
explicit deadline range, and sort whitelist (UI-01).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-21 15:47:40 +02:00
schalli 34b8df28d5 docs(11): validation strategy + state update — plans PASS, ready to execute
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-21 15:28:02 +02:00
schalli d7b8e9bf5f docs(11): revise phase plan per checker BLOCK — VALIDATION.md, FILTER-04 date range, serialized deps
- Add 11-VALIDATION.md (Nyquist Dimension 8: per-task test map, sampling, Wave-0 gaps)
- Implement FILTER-04 deadlineFrom/deadlineTo (DTO + builder branch + FilterPanel) in 11-01
- Serialize depends_on into a linear chain (11-04<-03, 11-05<-04, 11-06<-05) to prevent parallel shared-file/migration corruption
- Mark 11-RESEARCH Open Questions RESOLVED
- Fix 11-05 Task 2 <files> dto/ path typo

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-21 15:26:18 +02:00
schalli 4036991acd docs(11): create phase plan — 6 vertical-slice plans (filter engine, results UI, saved searches)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-21 15:17:48 +02:00
schalli 930a8d8f79 docs(11): phase research — live-DB findings, query patterns, new models
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-21 15:07:41 +02:00
schalli a4a661f829 docs(11): phase context from user requirements — filters, triage, saved searches
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-21 14:57:09 +02:00
schalli 48d1246043 fix(10): resolve GET /source-config 404 shadowed by :id route
The admin source-config settings form failed to load with "Failed to
fetch tender-radar source config". Network trace showed
GET /modules/tender-radar/source-config returning 404.

Root cause: NestJS RouterExplorer maps routes in method-declaration
order. `@Get(':id')` was declared before `@Get('source-config')`, so
the param route captured "source-config" as an id and shadowed the
static handler (401 unauthenticated, 404 past the guard — no Tender
with id "source-config").

Fix: declare `@Get('source-config')` before `@Get(':id')`. Add a
declaration-order regression test — unit tests call controller methods
directly, bypass routing, and could never catch route shadowing.

Verified live: settings form now loads real config, interval save
persists and live-re-registers the scheduler (INGEST-06). Phase 10
verification raised human_needed -> passed after full browser UAT.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-21 14:52:17 +02:00
schalli eb10bd3116 wip: pause phase 10 (code complete, UAT+rebuild pending) 2026-07-21 11:36:31 +02:00
schalli a3cef389df docs(10): phase verification — 5/5 must-haves verified, UAT pending docker rebuild
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-21 11:34:07 +02:00
schalli 4f3c6cf5e9 docs(10-06): complete tender-radar admin settings UI plan 2026-07-21 11:27:31 +02:00
schalli 8e3dfda64d test(10-06): SourceConfigForm component test — fetch, save, bounds
- fetch-on-mount populates pollIntervalMin input
- Speichern calls saveSourceConfig with edited interval + isActive
- interval below 5 or above 1440 rejected client-side, no save call
- automated proof for the INGEST-06 admin-UI slice
2026-07-21 11:25:43 +02:00
schalli 4360bc0c6b feat(10-06): tender-radar-api client + admin source-config settings form
- tender-radar-api.ts: fetchSourceConfig/saveSourceConfig hitting GET/PUT
  /modules/tender-radar/source-config (Plan 05 endpoint)
- SourceConfigForm: load-on-mount, numeric interval (5-1440 min, mirrors
  backend SourceConfigDto bounds) + isActive toggle, read-only sourceType/
  lastIngestedDay display
- settings/page.tsx: standard App Router route rendering the form,
  no module-loader whitelist change needed
2026-07-21 11:24:32 +02:00
schalli 6d6302294e docs(10-05): complete tenders controller plan 2026-07-21 11:21:52 +02:00
schalli eaff1d86bb test(10-05): controller spec — global read not tenant-scoped, admin config applies to scheduler
- GET / findMany where clause asserted to have no tenantId key
- GET /:id returns tender / throws NotFoundException for missing id
- PUT /source-config isActive=true+pollIntervalMin=30 -> setInterval(30) single-arg
- PUT /source-config isActive=false -> stopJob()
2026-07-21 11:18:35 +02:00
schalli 7b9b6b8c1c feat(10-05): wire TendersController — global read + admin source-config
- GET / and GET /🆔 paginated global Tender catalog, @UseModule('tender-radar')-gated, never row-scoped by tenant id
- GET/PUT /source-config: @Roles(ADMIN, SUPER_ADMIN)-guarded singleton doe-opendata config
- PUT /source-config live-applies pollIntervalMin/isActive to TenderSchedulerService (setInterval/stopJob, no tenant arg) — INGEST-06
- Registered TendersController in TendersModule.controllers
2026-07-21 11:17:13 +02:00
schalli f6e636c37d feat(10-05): add SourceConfigDto and TenderQueryDto
- SourceConfigDto: pollIntervalMin (@Min(5) @Max(1440)), isActive
- TenderQueryDto: page/limit pagination (copied from DkvHistoryQueryDto) + status filter
2026-07-21 11:15:36 +02:00
schalli 9227cc9e78 docs(10-04): complete ingestion orchestration & multi-tenant-safe scheduler plan 2026-07-21 11:13:39 +02:00
schalli 444c68b8ec test(10-04): add two-tenant safety integration test (Success Criteria 4 & 5)
Proves the phase's headline acceptance criterion: activating tender-radar
for a 2nd tenant triggers zero additional DÖE calls, zero additional cron
jobs (still exactly one 'tender-doe-poll'), and zero additional Tender rows.
Drives the real (unmocked) ModuleRegistryService against a fake prisma to
exercise the genuine activateForTenant() call path.

Passes immediately because Task 2's TenderSchedulerService already
implements the poll-once-fan-out-many invariant correctly — this test
locks in and regression-proofs that already-correct architecture rather
than driving new production code (documented in SUMMARY under TDD Gate
Compliance).
2026-07-21 11:11:09 +02:00
schalli 0ba74108db feat(10-04): implement TenderSchedulerService — single global cron (poll-once-fan-out-many)
- One named cron job 'tender-doe-poll' for the whole platform; setInterval()
  takes no tenant argument (INGEST-06) — reuses DkvSchedulerService's
  CronJob require()-resolution + SchedulerRegistry mechanics, drops the
  per-tenant activeTenantId framing entirely
- onModuleInit() loads the singleton doe-opendata config via findUnique on
  the fixed sourceType slug, never findFirst (Pitfall D)
- Day-cursor gate stays inside TenderIngestionService.pollDueSources() —
  this scheduler only controls cron-tick frequency (Pitfall A separation)
- Registered in TendersModule.providers; ScheduleModule already global via
  AppModule, no re-registration needed
2026-07-21 11:10:21 +02:00
schalli 6fb734bf3c feat(10-04): implement TenderIngestionService — day-cursor gate, upsert change-detect, D-05 retention
- pollDueSources(): singleton doe-opendata config via findUnique (fixed slug,
  not findFirst); day-cursor gate (nextDayToFetch) no-ops when nothing new
  (Pitfall A); catch-up loop from lastIngestedDay+1 to today-1 with a polite
  1.5s delay between successive day-fetches
- prisma.tender.upsert({ where: { dedupKey } }) — SCHEMA-02 change-detection
  seam: identical notice does not duplicate, changed contentHash updates in
  place
- pruneExpiredTenders(): marks active+past-deadline rows 'expired', deletes
  expired rows older than 90 days, never touches deadlineAt=null rows (D-05)
- Plain PrismaService throughout — no tenant RLS extension on the global
  Tender/TenderSourcePollConfig tables (D-03, T-10-09)
- Registered in TendersModule.providers
2026-07-21 11:08:59 +02:00
schalli f9e52ab95f test(10-04): add failing spec for TenderIngestionService (day-cursor gate, SCHEMA-02, D-05)
- Day-cursor no-op gate (Pitfall A)
- SCHEMA-02 change detection: fresh insert, identical no-dup, changed-content update
- Catch-up cursor advance across missed days
- D-05 retention: expire past-deadline, prune >90d expired, never touch null-deadline
- Multi-tenant safety: asserts no forTenant() call in the implementation
2026-07-21 11:07:49 +02:00
schalli 761077814f docs(10-03): complete DÖE source adapter & normalizer plan 2026-07-21 11:01:50 +02:00
schalli 31607df48c feat(10-03): implement TenderNormalizerService — fields, dedupKey, hash
- normalize(raw): eForms-DE XML primary for deadlineAt/estimatedValue/
  procedureType (RESEARCH Pattern 3); OCDS primary for ocid/buyerName/
  title/cpvCodes/region/plz
- deadlineAt/estimatedValue nullable by mandate (RESEARCH Pattern 4) —
  missing data normalizes to null, never thrown or zero
- dedupKey priority: ocid -> sourcePortal:sourceNoticeId fallback
- contentHash = sha256(title+deadlineAt+estimatedValue+status), stable
  across repeat calls, changes when the deadline changes (SCHEMA-02 hook)
- Register TenderNormalizerService in TendersModule.providers
- All tender-normalizer.service.spec.ts tests green (6/6); full API
  suite green (59/59); tsc --noEmit clean
2026-07-21 10:58:33 +02:00
schalli 3764feb50b feat(10-03): implement DoeOpenDataAdapter — fetch, extract, D-02 filter
- Native fetch + AbortController 15s timeout (icon-discovery idiom, no
  axios); URL host hardcoded, only the internally-computed dayCursor is
  interpolated (T-10-06)
- HTTP 400 treated as an expected no-op (pubDay today/future) -> []
- adm-zip extraction with a pre-extraction decompression-bomb ceiling
  check (sum entry.header.size vs ~50MB) before any entry buffer is read
  (T-10-07); entries are never written to disk
- D-02 open-tender filter: positive tag.includes('tender') match only —
  award/planning/untagged-with-awards excluded (Pitfall C)
- Register DoeOpenDataAdapter in TendersModule.providers
- All doe-opendata.adapter.spec.ts tests green (6/6)
2026-07-21 10:57:29 +02:00
schalli f88e2a8141 test(10-03): add failing specs + real DÖE fixtures for adapter/normalizer
- Capture real, trimmed DÖE day-export fixtures (pubDay=2026-07-19, 8
  notices spanning tender/award/planning/untagged-with-awards tag classes)
  under tenders/__fixtures__/, live-downloaded from oeffentlichevergabe.de
- Define RawTenderRecord/NormalizedTenderFields/SourceType (tender.types.ts)
  and TenderSourceAdapter (day-cursor fetchTenders signature per RESEARCH
  Pattern 1)
- RED: doe-opendata.adapter.spec.ts asserts D-02 exact-count filtering,
  HTTP-400 no-op, and a zip-bomb ceiling guard (T-10-07)
- RED: tender-normalizer.service.spec.ts asserts eForms-primary deadline
  recovery where OCDS is null (RESEARCH Pattern 3), nullable deadline/value,
  dedupKey priority (ocid -> sourcePortal:sourceNoticeId), and a stable
  sha256 contentHash
2026-07-21 10:56:13 +02:00
schalli f80d491adc docs(10-02): complete Ausschreibungs-Radar marketplace registration plan 2026-07-21 10:46:30 +02:00
schalli 9a7ed71d7a test(10-02): add seed registration coverage for tender-radar (CONFIG-01)
- asserts seedModule called once with slug 'tender-radar', isSystem: true,
  and a bilingual (de/en) description object
2026-07-21 10:43:43 +02:00
schalli 3292afb913 feat(10-02): add tender-radar module-loader whitelist entry + placeholder page
- MODULE_REGISTRY['tender-radar'] entry (mirrors cert-manager/dkv-fleet shape)
- minimal client-component placeholder page proving activation -> page-load path
- hardcoded German string is an intentional MVP stub; full i18n is CONFIG-03 (Phase 14)
2026-07-21 10:43:28 +02:00
schalli e7b40946c8 feat(10-02): register TendersModule with marketplace self-seed + singleton poll config
- tenders.seed.ts seeds slug 'tender-radar' (isSystem: true, category 'procurement')
- tenders.module.ts self-seeds registry on boot (mirrors DkvModule pattern)
- upserts singleton doe-opendata TenderSourcePollConfig row (global, no forTenant())
- TendersModule added to app.module.ts imports
2026-07-21 10:42:44 +02:00
schalli 6cfda90960 docs(10-01): complete tender-radar foundation plan 2026-07-21 10:40:15 +02:00
schalli 713b1eb748 feat(10-01): add Tender/TenderSourcePollConfig models + ingestion packages
- Install fast-xml-parser, adm-zip, csv-parse in @tessera/api
- Add global Tender model (no tenantId — D-03 platform-global data)
- Add singleton TenderSourcePollConfig (sourceType @unique)
- Handwritten additive migration for both tables
- Regenerate Prisma client
2026-07-21 10:33:24 +02:00
schalli b33213b04b docs(10): begin phase execution
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-21 10:24:28 +02:00
schalli 66efd08dc0 docs(10): mark phase planned (6 plans) + add pattern map
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-21 10:10:41 +02:00
schalli b52a0a796b docs(10): add admin config UI plan (INGEST-06 gap), resolve research open questions 2026-07-21 10:08:40 +02:00
schalli e7b59d16af docs(10): create phase plan (5 plans, DÖE ingestion foundation) 2026-07-21 10:03:19 +02:00
schalli 1a6061bc0e docs(phase-10): add validation strategy 2026-07-21 09:47:16 +02:00
schalli 3a51d9bfed docs(10): research DÖE OpenData API live verification and phase domain 2026-07-21 09:46:20 +02:00
schalli fe55e7c054 docs(state): record phase 10 context session 2026-07-17 13:49:36 +02:00
schalli b482c4aaa7 docs(10): capture phase context 2026-07-17 13:49:35 +02:00
schalli eb3ed949a9 docs: create milestone v1.1 roadmap (5 phases) 2026-07-17 12:54:48 +02:00
schalli 21cae97ca0 docs: define milestone v1.1 requirements 2026-07-17 10:44:58 +02:00
schalli d2ac1997d6 docs: complete project research
Ausschreibungs-Radar (v1.1) STACK/FEATURES/ARCHITECTURE/PITFALLS research plus SUMMARY.md synthesis.
2026-07-17 10:12:59 +02:00
schalli ae841205df docs: start milestone v1.1 Ausschreibungs-Radar
Tessera CI/CD / Lint & Type Check (push) Successful in 48s
Tessera CI/CD / Tests (push) Successful in 40s
Tessera CI/CD / Build & Publish Images (push) Successful in 2m50s
2026-07-17 09:58:09 +02:00
schalli 9b65ac63c3 fix(favorites): normalize scheme-less URLs so favicons resolve
Tessera CI/CD / Lint & Type Check (push) Successful in 38s
Tessera CI/CD / Tests (push) Successful in 40s
Tessera CI/CD / Build & Publish Images (push) Successful in 22s
A favorite entered as a bare host ("ctl.de") passed @IsUrl() but had no
scheme, so `new URL()` threw inside icon discovery and it silently fell
back to a relative "/favicon.ico" — which 502'd through the icon proxy
and left the widget showing the first-letter placeholder ("C").

- add normalizeUrl() (prepend https:// when no scheme present)
- apply it in discoverFavoriteIconUrl and when storing the favorite url,
  so both the link and discovery use the normalized value
- on update, re-run discovery when the icon field is cleared, so editing
  a previously-broken favorite repairs its icon
- tests: normalizeUrl cases + end-to-end discovery (apple-touch extraction,
  scheme-less fallback stays absolute)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-14 13:58:44 +02:00
schalli 9553e5304d chore(planning): mark LDAP exclude filter done, catch up STATE.md
Tessera CI/CD / Lint & Type Check (push) Successful in 43s
Tessera CI/CD / Tests (push) Successful in 46s
Tessera CI/CD / Build & Publish Images (push) Successful in 6s
Record the per-user exclude/denylist filter (9d1323f) and its live
verification as complete, close out the two other carried-over items
(full-sync verify, quick-tasks bookkeeping), and backfill the STATE.md
Quick Tasks table with the direct-fix commits that never got /gsd-quick
entries.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-14 13:48:26 +02:00
schalli 9d1323fe97 feat(ldap): per-user exclude/denylist filter for sync
Tessera CI/CD / Lint & Type Check (push) Successful in 43s
Tessera CI/CD / Tests (push) Successful in 43s
Tessera CI/CD / Build & Publish Images (push) Successful in 1m46s
Add a per-username denylist so individual accounts (service accounts like
administrator, krbtgt, guest, ldap$) can be excluded from LDAP sync,
independent of the group/OU include-filter which only scopes the search.

- schema: LdapConfig.userExcludeList String[] (+ migration)
- sync: skip excluded usernames (case-insensitive) before recording the DN,
  so an already-imported user added to the list gets deactivated next sync
- DTO / config service / controller / scheduler: thread userExcludeList through
- web: exclude-list admin UI section (add/remove/save) + de/en translations
- tests: 3 specs covering empty list, case-insensitive skip, deactivation

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-14 09:34:01 +02:00
schalli ef3e41769e wip: session paused — LDAP live-testing in progress
Tessera CI/CD / Lint & Type Check (push) Successful in 45s
Tessera CI/CD / Tests (push) Successful in 42s
Tessera CI/CD / Build & Publish Images (push) Successful in 7s
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-09 16:38:14 +02:00
schalli aaa29226c9 feat(ldap): search box for the discovered groups/OUs list
Tessera CI/CD / Lint & Type Check (push) Successful in 39s
Tessera CI/CD / Tests (push) Successful in 40s
Tessera CI/CD / Build & Publish Images (push) Successful in 1m25s
Real directories can return many groups/OUs, making the checkbox list
tedious to scroll through. Adds a client-side search input above the
list that filters by name or DN substring (case-insensitive) as you
type, so picking the right groups for the import filter is faster.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-09 16:06:10 +02:00
schalli 246dc89a98 fix(ldap): treat ldapts empty-array attributes as absent, not "undefined"
Tessera CI/CD / Lint & Type Check (push) Successful in 45s
Tessera CI/CD / Tests (push) Successful in 39s
Tessera CI/CD / Build & Publish Images (push) Successful in 25s
ldapts represents a missing/absent LDAP attribute as an empty array
([]), not undefined -- entry['mail'] is [] when an account has no mail
set. The field-mapping loop did Array.isArray(value) ? String(value[0])
: ..., and String(undefined) is the literal string "undefined". Every
synced entry without that attribute got mappedData['email'] = "undefined"
(a truthy string, so the `|| fallback` never kicked in), and the second
such entry onward crashed with a unique constraint violation on email
since they all shared the exact same literal string.

Found live: syncing against a real Zentyal/Samba AD directory failed
on every entry after the first (Kevin Schaller, krbtgt, Guest, the DC
computer object, etc.) with "Unique constraint failed on the fields:
(email)".

Fix: resolve array values to their first element (or use the raw
value for non-arrays) and only keep it when actually present and
non-empty, so a genuinely missing attribute falls through to the
`${username}@ldap.local` fallback instead of the string "undefined".

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-09 15:28:48 +02:00