Commit Graph

489 Commits

Author SHA1 Message Date
schalli 4036991acd docs(11): create phase plan — 6 vertical-slice plans (filter engine, results UI, saved searches)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-21 15:17:48 +02:00
schalli 930a8d8f79 docs(11): phase research — live-DB findings, query patterns, new models
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-21 15:07:41 +02:00
schalli a4a661f829 docs(11): phase context from user requirements — filters, triage, saved searches
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-21 14:57:09 +02:00
schalli 48d1246043 fix(10): resolve GET /source-config 404 shadowed by :id route
The admin source-config settings form failed to load with "Failed to
fetch tender-radar source config". Network trace showed
GET /modules/tender-radar/source-config returning 404.

Root cause: NestJS RouterExplorer maps routes in method-declaration
order. `@Get(':id')` was declared before `@Get('source-config')`, so
the param route captured "source-config" as an id and shadowed the
static handler (401 unauthenticated, 404 past the guard — no Tender
with id "source-config").

Fix: declare `@Get('source-config')` before `@Get(':id')`. Add a
declaration-order regression test — unit tests call controller methods
directly, bypass routing, and could never catch route shadowing.

Verified live: settings form now loads real config, interval save
persists and live-re-registers the scheduler (INGEST-06). Phase 10
verification raised human_needed -> passed after full browser UAT.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-21 14:52:17 +02:00
schalli eb10bd3116 wip: pause phase 10 (code complete, UAT+rebuild pending) 2026-07-21 11:36:31 +02:00
schalli a3cef389df docs(10): phase verification — 5/5 must-haves verified, UAT pending docker rebuild
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-21 11:34:07 +02:00
schalli 4f3c6cf5e9 docs(10-06): complete tender-radar admin settings UI plan 2026-07-21 11:27:31 +02:00
schalli 8e3dfda64d test(10-06): SourceConfigForm component test — fetch, save, bounds
- fetch-on-mount populates pollIntervalMin input
- Speichern calls saveSourceConfig with edited interval + isActive
- interval below 5 or above 1440 rejected client-side, no save call
- automated proof for the INGEST-06 admin-UI slice
2026-07-21 11:25:43 +02:00
schalli 4360bc0c6b feat(10-06): tender-radar-api client + admin source-config settings form
- tender-radar-api.ts: fetchSourceConfig/saveSourceConfig hitting GET/PUT
  /modules/tender-radar/source-config (Plan 05 endpoint)
- SourceConfigForm: load-on-mount, numeric interval (5-1440 min, mirrors
  backend SourceConfigDto bounds) + isActive toggle, read-only sourceType/
  lastIngestedDay display
- settings/page.tsx: standard App Router route rendering the form,
  no module-loader whitelist change needed
2026-07-21 11:24:32 +02:00
schalli 6d6302294e docs(10-05): complete tenders controller plan 2026-07-21 11:21:52 +02:00
schalli eaff1d86bb test(10-05): controller spec — global read not tenant-scoped, admin config applies to scheduler
- GET / findMany where clause asserted to have no tenantId key
- GET /:id returns tender / throws NotFoundException for missing id
- PUT /source-config isActive=true+pollIntervalMin=30 -> setInterval(30) single-arg
- PUT /source-config isActive=false -> stopJob()
2026-07-21 11:18:35 +02:00
schalli 7b9b6b8c1c feat(10-05): wire TendersController — global read + admin source-config
- GET / and GET /🆔 paginated global Tender catalog, @UseModule('tender-radar')-gated, never row-scoped by tenant id
- GET/PUT /source-config: @Roles(ADMIN, SUPER_ADMIN)-guarded singleton doe-opendata config
- PUT /source-config live-applies pollIntervalMin/isActive to TenderSchedulerService (setInterval/stopJob, no tenant arg) — INGEST-06
- Registered TendersController in TendersModule.controllers
2026-07-21 11:17:13 +02:00
schalli f6e636c37d feat(10-05): add SourceConfigDto and TenderQueryDto
- SourceConfigDto: pollIntervalMin (@Min(5) @Max(1440)), isActive
- TenderQueryDto: page/limit pagination (copied from DkvHistoryQueryDto) + status filter
2026-07-21 11:15:36 +02:00
schalli 9227cc9e78 docs(10-04): complete ingestion orchestration & multi-tenant-safe scheduler plan 2026-07-21 11:13:39 +02:00
schalli 444c68b8ec test(10-04): add two-tenant safety integration test (Success Criteria 4 & 5)
Proves the phase's headline acceptance criterion: activating tender-radar
for a 2nd tenant triggers zero additional DÖE calls, zero additional cron
jobs (still exactly one 'tender-doe-poll'), and zero additional Tender rows.
Drives the real (unmocked) ModuleRegistryService against a fake prisma to
exercise the genuine activateForTenant() call path.

Passes immediately because Task 2's TenderSchedulerService already
implements the poll-once-fan-out-many invariant correctly — this test
locks in and regression-proofs that already-correct architecture rather
than driving new production code (documented in SUMMARY under TDD Gate
Compliance).
2026-07-21 11:11:09 +02:00
schalli 0ba74108db feat(10-04): implement TenderSchedulerService — single global cron (poll-once-fan-out-many)
- One named cron job 'tender-doe-poll' for the whole platform; setInterval()
  takes no tenant argument (INGEST-06) — reuses DkvSchedulerService's
  CronJob require()-resolution + SchedulerRegistry mechanics, drops the
  per-tenant activeTenantId framing entirely
- onModuleInit() loads the singleton doe-opendata config via findUnique on
  the fixed sourceType slug, never findFirst (Pitfall D)
- Day-cursor gate stays inside TenderIngestionService.pollDueSources() —
  this scheduler only controls cron-tick frequency (Pitfall A separation)
- Registered in TendersModule.providers; ScheduleModule already global via
  AppModule, no re-registration needed
2026-07-21 11:10:21 +02:00
schalli 6fb734bf3c feat(10-04): implement TenderIngestionService — day-cursor gate, upsert change-detect, D-05 retention
- pollDueSources(): singleton doe-opendata config via findUnique (fixed slug,
  not findFirst); day-cursor gate (nextDayToFetch) no-ops when nothing new
  (Pitfall A); catch-up loop from lastIngestedDay+1 to today-1 with a polite
  1.5s delay between successive day-fetches
- prisma.tender.upsert({ where: { dedupKey } }) — SCHEMA-02 change-detection
  seam: identical notice does not duplicate, changed contentHash updates in
  place
- pruneExpiredTenders(): marks active+past-deadline rows 'expired', deletes
  expired rows older than 90 days, never touches deadlineAt=null rows (D-05)
- Plain PrismaService throughout — no tenant RLS extension on the global
  Tender/TenderSourcePollConfig tables (D-03, T-10-09)
- Registered in TendersModule.providers
2026-07-21 11:08:59 +02:00
schalli f9e52ab95f test(10-04): add failing spec for TenderIngestionService (day-cursor gate, SCHEMA-02, D-05)
- Day-cursor no-op gate (Pitfall A)
- SCHEMA-02 change detection: fresh insert, identical no-dup, changed-content update
- Catch-up cursor advance across missed days
- D-05 retention: expire past-deadline, prune >90d expired, never touch null-deadline
- Multi-tenant safety: asserts no forTenant() call in the implementation
2026-07-21 11:07:49 +02:00
schalli 761077814f docs(10-03): complete DÖE source adapter & normalizer plan 2026-07-21 11:01:50 +02:00
schalli 31607df48c feat(10-03): implement TenderNormalizerService — fields, dedupKey, hash
- normalize(raw): eForms-DE XML primary for deadlineAt/estimatedValue/
  procedureType (RESEARCH Pattern 3); OCDS primary for ocid/buyerName/
  title/cpvCodes/region/plz
- deadlineAt/estimatedValue nullable by mandate (RESEARCH Pattern 4) —
  missing data normalizes to null, never thrown or zero
- dedupKey priority: ocid -> sourcePortal:sourceNoticeId fallback
- contentHash = sha256(title+deadlineAt+estimatedValue+status), stable
  across repeat calls, changes when the deadline changes (SCHEMA-02 hook)
- Register TenderNormalizerService in TendersModule.providers
- All tender-normalizer.service.spec.ts tests green (6/6); full API
  suite green (59/59); tsc --noEmit clean
2026-07-21 10:58:33 +02:00
schalli 3764feb50b feat(10-03): implement DoeOpenDataAdapter — fetch, extract, D-02 filter
- Native fetch + AbortController 15s timeout (icon-discovery idiom, no
  axios); URL host hardcoded, only the internally-computed dayCursor is
  interpolated (T-10-06)
- HTTP 400 treated as an expected no-op (pubDay today/future) -> []
- adm-zip extraction with a pre-extraction decompression-bomb ceiling
  check (sum entry.header.size vs ~50MB) before any entry buffer is read
  (T-10-07); entries are never written to disk
- D-02 open-tender filter: positive tag.includes('tender') match only —
  award/planning/untagged-with-awards excluded (Pitfall C)
- Register DoeOpenDataAdapter in TendersModule.providers
- All doe-opendata.adapter.spec.ts tests green (6/6)
2026-07-21 10:57:29 +02:00
schalli f88e2a8141 test(10-03): add failing specs + real DÖE fixtures for adapter/normalizer
- Capture real, trimmed DÖE day-export fixtures (pubDay=2026-07-19, 8
  notices spanning tender/award/planning/untagged-with-awards tag classes)
  under tenders/__fixtures__/, live-downloaded from oeffentlichevergabe.de
- Define RawTenderRecord/NormalizedTenderFields/SourceType (tender.types.ts)
  and TenderSourceAdapter (day-cursor fetchTenders signature per RESEARCH
  Pattern 1)
- RED: doe-opendata.adapter.spec.ts asserts D-02 exact-count filtering,
  HTTP-400 no-op, and a zip-bomb ceiling guard (T-10-07)
- RED: tender-normalizer.service.spec.ts asserts eForms-primary deadline
  recovery where OCDS is null (RESEARCH Pattern 3), nullable deadline/value,
  dedupKey priority (ocid -> sourcePortal:sourceNoticeId), and a stable
  sha256 contentHash
2026-07-21 10:56:13 +02:00
schalli f80d491adc docs(10-02): complete Ausschreibungs-Radar marketplace registration plan 2026-07-21 10:46:30 +02:00
schalli 9a7ed71d7a test(10-02): add seed registration coverage for tender-radar (CONFIG-01)
- asserts seedModule called once with slug 'tender-radar', isSystem: true,
  and a bilingual (de/en) description object
2026-07-21 10:43:43 +02:00
schalli 3292afb913 feat(10-02): add tender-radar module-loader whitelist entry + placeholder page
- MODULE_REGISTRY['tender-radar'] entry (mirrors cert-manager/dkv-fleet shape)
- minimal client-component placeholder page proving activation -> page-load path
- hardcoded German string is an intentional MVP stub; full i18n is CONFIG-03 (Phase 14)
2026-07-21 10:43:28 +02:00
schalli e7b40946c8 feat(10-02): register TendersModule with marketplace self-seed + singleton poll config
- tenders.seed.ts seeds slug 'tender-radar' (isSystem: true, category 'procurement')
- tenders.module.ts self-seeds registry on boot (mirrors DkvModule pattern)
- upserts singleton doe-opendata TenderSourcePollConfig row (global, no forTenant())
- TendersModule added to app.module.ts imports
2026-07-21 10:42:44 +02:00
schalli 6cfda90960 docs(10-01): complete tender-radar foundation plan 2026-07-21 10:40:15 +02:00
schalli 713b1eb748 feat(10-01): add Tender/TenderSourcePollConfig models + ingestion packages
- Install fast-xml-parser, adm-zip, csv-parse in @tessera/api
- Add global Tender model (no tenantId — D-03 platform-global data)
- Add singleton TenderSourcePollConfig (sourceType @unique)
- Handwritten additive migration for both tables
- Regenerate Prisma client
2026-07-21 10:33:24 +02:00
schalli b33213b04b docs(10): begin phase execution
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-21 10:24:28 +02:00
schalli 66efd08dc0 docs(10): mark phase planned (6 plans) + add pattern map
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-21 10:10:41 +02:00
schalli b52a0a796b docs(10): add admin config UI plan (INGEST-06 gap), resolve research open questions 2026-07-21 10:08:40 +02:00
schalli e7b59d16af docs(10): create phase plan (5 plans, DÖE ingestion foundation) 2026-07-21 10:03:19 +02:00
schalli 1a6061bc0e docs(phase-10): add validation strategy 2026-07-21 09:47:16 +02:00
schalli 3a51d9bfed docs(10): research DÖE OpenData API live verification and phase domain 2026-07-21 09:46:20 +02:00
schalli fe55e7c054 docs(state): record phase 10 context session 2026-07-17 13:49:36 +02:00
schalli b482c4aaa7 docs(10): capture phase context 2026-07-17 13:49:35 +02:00
schalli eb3ed949a9 docs: create milestone v1.1 roadmap (5 phases) 2026-07-17 12:54:48 +02:00
schalli 21cae97ca0 docs: define milestone v1.1 requirements 2026-07-17 10:44:58 +02:00
schalli d2ac1997d6 docs: complete project research
Ausschreibungs-Radar (v1.1) STACK/FEATURES/ARCHITECTURE/PITFALLS research plus SUMMARY.md synthesis.
2026-07-17 10:12:59 +02:00
schalli ae841205df docs: start milestone v1.1 Ausschreibungs-Radar
Tessera CI/CD / Lint & Type Check (push) Successful in 48s
Tessera CI/CD / Tests (push) Successful in 40s
Tessera CI/CD / Build & Publish Images (push) Successful in 2m50s
2026-07-17 09:58:09 +02:00
schalli 9b65ac63c3 fix(favorites): normalize scheme-less URLs so favicons resolve
Tessera CI/CD / Lint & Type Check (push) Successful in 38s
Tessera CI/CD / Tests (push) Successful in 40s
Tessera CI/CD / Build & Publish Images (push) Successful in 22s
A favorite entered as a bare host ("ctl.de") passed @IsUrl() but had no
scheme, so `new URL()` threw inside icon discovery and it silently fell
back to a relative "/favicon.ico" — which 502'd through the icon proxy
and left the widget showing the first-letter placeholder ("C").

- add normalizeUrl() (prepend https:// when no scheme present)
- apply it in discoverFavoriteIconUrl and when storing the favorite url,
  so both the link and discovery use the normalized value
- on update, re-run discovery when the icon field is cleared, so editing
  a previously-broken favorite repairs its icon
- tests: normalizeUrl cases + end-to-end discovery (apple-touch extraction,
  scheme-less fallback stays absolute)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-14 13:58:44 +02:00
schalli 9553e5304d chore(planning): mark LDAP exclude filter done, catch up STATE.md
Tessera CI/CD / Lint & Type Check (push) Successful in 43s
Tessera CI/CD / Tests (push) Successful in 46s
Tessera CI/CD / Build & Publish Images (push) Successful in 6s
Record the per-user exclude/denylist filter (9d1323f) and its live
verification as complete, close out the two other carried-over items
(full-sync verify, quick-tasks bookkeeping), and backfill the STATE.md
Quick Tasks table with the direct-fix commits that never got /gsd-quick
entries.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-14 13:48:26 +02:00
schalli 9d1323fe97 feat(ldap): per-user exclude/denylist filter for sync
Tessera CI/CD / Lint & Type Check (push) Successful in 43s
Tessera CI/CD / Tests (push) Successful in 43s
Tessera CI/CD / Build & Publish Images (push) Successful in 1m46s
Add a per-username denylist so individual accounts (service accounts like
administrator, krbtgt, guest, ldap$) can be excluded from LDAP sync,
independent of the group/OU include-filter which only scopes the search.

- schema: LdapConfig.userExcludeList String[] (+ migration)
- sync: skip excluded usernames (case-insensitive) before recording the DN,
  so an already-imported user added to the list gets deactivated next sync
- DTO / config service / controller / scheduler: thread userExcludeList through
- web: exclude-list admin UI section (add/remove/save) + de/en translations
- tests: 3 specs covering empty list, case-insensitive skip, deactivation

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-14 09:34:01 +02:00
schalli ef3e41769e wip: session paused — LDAP live-testing in progress
Tessera CI/CD / Lint & Type Check (push) Successful in 45s
Tessera CI/CD / Tests (push) Successful in 42s
Tessera CI/CD / Build & Publish Images (push) Successful in 7s
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-09 16:38:14 +02:00
schalli aaa29226c9 feat(ldap): search box for the discovered groups/OUs list
Tessera CI/CD / Lint & Type Check (push) Successful in 39s
Tessera CI/CD / Tests (push) Successful in 40s
Tessera CI/CD / Build & Publish Images (push) Successful in 1m25s
Real directories can return many groups/OUs, making the checkbox list
tedious to scroll through. Adds a client-side search input above the
list that filters by name or DN substring (case-insensitive) as you
type, so picking the right groups for the import filter is faster.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-09 16:06:10 +02:00
schalli 246dc89a98 fix(ldap): treat ldapts empty-array attributes as absent, not "undefined"
Tessera CI/CD / Lint & Type Check (push) Successful in 45s
Tessera CI/CD / Tests (push) Successful in 39s
Tessera CI/CD / Build & Publish Images (push) Successful in 25s
ldapts represents a missing/absent LDAP attribute as an empty array
([]), not undefined -- entry['mail'] is [] when an account has no mail
set. The field-mapping loop did Array.isArray(value) ? String(value[0])
: ..., and String(undefined) is the literal string "undefined". Every
synced entry without that attribute got mappedData['email'] = "undefined"
(a truthy string, so the `|| fallback` never kicked in), and the second
such entry onward crashed with a unique constraint violation on email
since they all shared the exact same literal string.

Found live: syncing against a real Zentyal/Samba AD directory failed
on every entry after the first (Kevin Schaller, krbtgt, Guest, the DC
computer object, etc.) with "Unique constraint failed on the fields:
(email)".

Fix: resolve array values to their first element (or use the raw
value for non-arrays) and only keep it when actually present and
non-empty, so a genuinely missing attribute falls through to the
`${username}@ldap.local` fallback instead of the string "undefined".

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-09 15:28:48 +02:00
schalli baff7ce4db fix(auth): make usernames case-insensitive
Tessera CI/CD / Lint & Type Check (push) Successful in 47s
Tessera CI/CD / Tests (push) Successful in 42s
Tessera CI/CD / Build & Publish Images (push) Successful in 24s
Username lookups (login, admin seed, LDAP sync) compared case-sensitively
against a stored value with whatever casing it was created with, so
"Admin" and "admin" were treated as different accounts.

Normalizes at every write and read path: UserService.create/update
lowercase the username before persisting, findByUsername lowercases
the lookup input, AuthService.validateUser lowercases before the login
query, AdminSeedService lowercases the configured admin username, and
the LDAP sync loop lowercases the mapped sAMAccountName before using it
for lookup/create/update -- so AD casing differences don't create
duplicate accounts either.

Added a data migration to lowercase any existing mixed-case usernames.
It relies on the User.username unique constraint to fail loudly if two
existing accounts would collide after normalizing, rather than silently
merging them.

Verified locally: logged in with "ADMIN" (uppercase) against the
existing lowercase "admin" account after rebuilding the API image.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-09 15:20:49 +02:00
schalli 010aceb1ac feat(ldap): support anonymous bind (no bind DN/password required)
Tessera CI/CD / Lint & Type Check (push) Successful in 41s
Tessera CI/CD / Tests (push) Successful in 39s
Tessera CI/CD / Build & Publish Images (push) Successful in 1m40s
bindDn and bindPassword are now optional on LdapConfig (nullable
migration) and throughout the DTOs/service/client -- an admin can
leave both blank to connect to directories that permit anonymous
read access. LdapService.bind() falls back to an RFC 4513 anonymous
bind (empty DN + empty password) whenever either field is missing,
shared across testConnection, listGroups, and syncUsersForTenant.

Frontend: removed the required attribute from Bind-DN/Bind-Passwort,
added a placeholder hint ("leer = anonymous bind"), and the
"Verbindung testen" button now only needs a Server-URL to enable
(not bindDn+bindPassword). Config responses now return bindPassword
as null (not a misleading "********") when no password is set.

Verified locally: submitted only a Server-URL with both bind fields
empty and confirmed the request reached the anonymous-bind code path
(DNS failure for the unreachable test host, not a validation error).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-08 12:57:26 +02:00
schalli 39aa4bff2a feat(ldap): allow testing connection before saving a config
Tessera CI/CD / Lint & Type Check (push) Successful in 43s
Tessera CI/CD / Tests (push) Successful in 42s
Tessera CI/CD / Build & Publish Images (push) Successful in 1m35s
"Verbindung testen" only appeared after a config was already saved,
so a fresh/never-configured tenant had no way to validate connection
details at all. Now the button is always visible; POST /ldap/test-connection
accepts optional ad-hoc serverUrl/bindDn/bindPassword and falls back
field-by-field to the saved config for anything omitted (bindPassword
in particular, since the form never re-sends the masked saved password).

Verified locally: tested with typed-but-unsaved values (base DN left
blank, nothing saved) and got a real connection attempt/error back,
not a 404 "no config" response.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-08 10:55:46 +02:00
schalli 8e8305ce18 revert(ldap): remove CTL-specific AD connection prefill
Tessera CI/CD / Lint & Type Check (push) Successful in 42s
Tessera CI/CD / Tests (push) Successful in 46s
Tessera CI/CD / Build & Publish Images (push) Successful in 1m27s
The prior quick task (260707-csw) pre-filled the LDAP connection form
with one customer's specific Active Directory values (balios.ctl.local,
dc=ctl,dc=local) and a matching bind-DN hint. User clarified this was
never wanted -- Tessera is a generic multi-tenant product, and baking
one customer's infrastructure into the shared admin UI is wrong,
especially since a `dcdown -v` reinstall surfaced it unexpectedly as
seemingly-baked-in defaults on a fresh system.

Reverted to blank fields with generic example placeholders
(ldap.example.com / dc=example,dc=com / cn=admin,dc=example,dc=com),
matching the form's state before that change. Removed the now-unused
bindDnHint i18n key.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-08 09:33:34 +02:00