Files
tessera-ctl/.planning/STATE.md
T
schalli 9dd5038575
Tessera CI/CD / Lint & Type Check (push) Successful in 45s
Tessera CI/CD / Tests (push) Successful in 45s
Tessera CI/CD / Build & Publish Images (push) Successful in 26s
docs(state): record phase 14 planning session
2026-07-23 12:17:58 +02:00

252 lines
20 KiB
Markdown

---
gsd_state_version: 1.0
milestone: v1.1
milestone_name: Ausschreibungs-Radar
current_phase: 13
current_phase_name: scraping-adapters-cross-source-dedup
status: verifying
stopped_at: Phase 14 planned (5 plans, verification passed)
last_updated: "2026-07-23T10:17:57.937Z"
last_activity: 2026-07-23
last_activity_desc: Phase 13 execution started
progress:
total_phases: 14
completed_phases: 12
total_plans: 67
completed_plans: 61
---
# Project State
## Project Reference
See: .planning/PROJECT.md (updated 2026-07-17)
**Core value:** Eine zentrale Plattform, in der beliebige Workflow-Tools als Module lizenziert, aktiviert und genutzt werden koennen -- ohne zwischen verschiedenen Anwendungen wechseln zu muessen.
**Current focus:** Phase 13 — scraping-adapters-cross-source-dedup
## Current Position
Phase: 13 (scraping-adapters-cross-source-dedup) — EXECUTING
Plan: 6 of 6
Status: Phase complete — ready for verification
Last activity: 2026-07-23 — Phase 13 execution started
Progress: [██████████] 98%
## Performance Metrics
**Velocity:**
- Total plans completed: 2
- Average duration: 12 min
- Total execution time: 0.38 hours
**By Phase:**
| Phase | Plans | Total | Avg/Plan |
|-------|-------|-------|----------|
| 01-foundation-portal-shell | 2/3 | 23 min | 12 min |
**Recent Trend:**
- Last 5 plans: 01-01 (16 min), 01-02 (7 min)
- Trend: improving
*Updated after each plan completion*
| Phase 02-authentication-multi-tenancy P02 | 8min | 3 tasks | 26 files |
| Phase 02-authentication-multi-tenancy P03 | 5min | 2 tasks | 20 files |
| Phase 05-dashboard-calendar P01 | 14min | 4 tasks | 28 files |
| Phase 05-dashboard-calendar P02 | 4min | 4 tasks | 16 files |
| Phase 05-dashboard-calendar P03 | 11min | 4 tasks | 14 files |
| Phase 05-dashboard-calendar PP04 | 5min | 2 tasks | 11 files |
| Phase 06-desktop-client-ci-cd P01 | 5min | 2 tasks | 13 files |
| Phase 06 P03 | 3min | 3 tasks | 3 files |
| Phase 07-dkv-fleet-module P03 | 5min | 4 tasks | 8 files |
| Phase 07-dkv-fleet-module P04 | 7min | 3 tasks | 8 files |
| Phase 07-dkv-fleet-module P05 | 8min | 3 tasks | 11 files |
| Phase 07-dkv-fleet-module P06 | 5min | 2 tasks | 7 files |
| Phase 09 P03 | 17 | 3 tasks | 6 files |
| Phase 09-cert-manager-module P04 | 4 | 3 tasks | 5 files |
| Phase 09-cert-manager-module P05 | 8 | 3 tasks | 6 files |
| Phase 09 P06 | 7 | 3 tasks | 7 files |
| Phase 10 P01 | 15min | 3 tasks | 4 files |
| Phase 10 P02 | 20min | 3 tasks | 6 files |
| Phase 10 P03 | 35min | 3 tasks | 9 files |
| Phase 10 P04 | 25min | 3 tasks | 5 files |
| Phase 10 P05 | 20min | 3 tasks | 5 files |
| Phase 10 P06 | 3min | 2 tasks | 4 files |
**Per-Plan Metrics:**
| Plan | Duration | Tasks | Files |
|------|----------|-------|-------|
| Phase 11 P01 | 8min | 3 tasks | 11 files |
| Phase 11 P02 | 4min | 3 tasks | 12 files |
| Phase 11 P03 | 9min | 4 tasks | 12 files |
| Phase 11 P04 | 12min | 2 tasks | 5 files |
| Phase 11 P05 | 24min | 3 tasks | 15 files |
| Phase 11 P06 | 35min | 3 tasks | 12 files |
| Phase 12 P01 | 35min | 3 tasks | 7 files |
| Phase 12 P02 | 8min | 3 tasks | 5 files |
| Phase 12 P03 | 15min | 2 tasks | 3 files |
| Phase 12 P04 | 12min | 3 tasks | 10 files |
| Phase 13 P01 | 35min | 3 tasks | 6 files |
| Phase 13 P02 | 20min | 2 tasks | 4 files |
| Phase 13 P03 | 45min | 3 tasks | 5 files |
| Phase 13 P06 | 15min | 2 tasks | 5 files |
| Phase 13 P04 | 55min | 3 tasks | 6 files |
| Phase 13 P05 | 40min | 2 tasks | 4 files |
## Accumulated Context
### Decisions
Decisions are logged in PROJECT.md Key Decisions table.
Recent decisions affecting current work:
- [Roadmap v1.1]: 5 phases (10-14) derived from 29 v1.1 requirements, standard granularity — DÖE-only MVP (10: ingestion, 11: filter/UI, 12: notifications) ships first as legally-clean demoable slice, then scraping adapters + cross-source dedup (13), then RSS + email-alert long tail (14)
- [Roadmap v1.1]: Tender data modeled as platform-global (no tenantId on Tender) — only TenderSavedSearch/TenderMatch are tenant+user-scoped; deviates deliberately from the DKV per-tenant template
- [Roadmap v1.1]: Scheduler must be poll-once-fan-out-many from Phase 10 onward — explicitly NOT the DKV `findFirst()` single-tenant pattern (documented pitfall)
- [Roadmap v1.1]: Notification phase (12) requires an explicit matched-vs-notified state with backfill suppression to avoid first-activation email floods and duplicate digest+instant sends
- [Roadmap v1.1]: INGEST-07 (vergabe24/aumass hard denylist) enforced in the adapter registry in Phase 13, not just documented
- [Roadmap v1.1]: inbox/ module extraction (ImapProvider/ExchangeInboxProvider out of dkv/) scoped as a one-time prerequisite refactor inside Phase 14, not done earlier
- [Roadmap]: 6 phases derived from 44 v1.0 requirements, standard granularity
- [Roadmap]: Research recommends NestJS + Next.js + PostgreSQL RLS + Keycloak + Tauri stack
- [01-01]: Used Traefik v2.11 instead of v3.4 due to Docker API version incompatibility on host
- [01-01]: Traefik placed on frontend-net + backend-net for routing to both web and api services
- [01-01]: API Dockerfile copies full monorepo node_modules structure for pnpm workspace compatibility
- [01-02]: OKLCH color space for all design tokens (Tailwind v4 native, perceptually uniform)
- [01-02]: Dark mode uses oklch(0.17 0.01 260) dark gray-blue for comfortable contrast with yellow primary
- [01-02]: Cookie-based locale (NEXT_LOCALE) instead of URL routing for portal app
- [01-02]: CSS custom property --current-sidebar-width for responsive main content margin
- [Phase ?]: Route groups (auth)/(portal) for layout separation: auth pages standalone, portal pages wrapped in AppShell
- [Phase ?]: Controller-level tenant isolation for ADMIN role as defense-in-depth alongside RLS
- [Phase ?]: Remember-me controls cookie maxAge (30d session vs browser-session) not separate token type
- [Phase ?]: react-grid-layout v2 uses dragConfig/resizeConfig instead of isDraggable/isResizable
- [05-02]: SearchProvider defaults as constants merged with user DB rows (no seed migration)
- [05-02]: useRef with explicit undefined initial value for React 19 strict TypeScript
- [05-03]: DAVClient class constructor instead of createDAVClient factory (tsdav v2 type compatibility)
- [05-03]: Dynamic imports for ews-javascript-api and @microsoft/microsoft-graph-client (lazy-load)
- [05-03]: In-memory Map cache with 5-min TTL for calendar events (Redis not needed at current scale)
- [05-03]: ews-javascript-api imported as any (no TypeScript definitions available)
- [Phase ?]: Optimistic UI for calendar visibility toggle — reverts on API error
- [Phase ?]: Auto-run testSource after adding new calendar source for immediate feedback
- [Phase ?]: URL constructor for client-side https-only validation (T-05-14)
- [Phase ?]: StoreExt trait import required for app.store() in Tauri 2.x
- [Phase ?]: frontendDist=../src local page, navigate() for runtime URL override
- [Phase ?]: CSP connect-src wildcard for configurable server URL (D-02)
- [Phase ?]: Plain docker compose build statt build-push-action (Gitea JWT Pitfall 4)
- [Phase ?]: Ephemeral runner mode (GITEA_RUNNER_EPHEMERAL=1) fuer Credential-Revokation pro Job
- [Phase ?]: Separate docker-compose.ci.yml fuer opt-in CI-Infrastruktur
- [07-01]: DKV PDF uses two extraction formats: single-tx (tab-separated) vs multi-tx (columnar) — both handled in DkvParserService
- [07-01]: Research Pattern 4 regex replaced with empirical dual-format tab/columnar parser after testing against real invoice.pdf
- [07-01]: CalendarCryptoService exported from CalendarModule for DKV credential encryption reuse
- [07-02]: Max attachment size 25MB enforced in both ImapProvider and ExchangeInboxProvider before buffering (T-07-05)
- [07-02]: ExchangeInboxProvider uses WellKnownFolderName.Inbox + FindItems (not FindAppointments — email vs calendar EWS API)
- [07-02]: export type {} required for type-only re-exports under isolatedModules TypeScript setting
- [07-03]: SettingsService.getStartupSmtpConfig uses findFirst (tenant-agnostic) for MailModule startup transport
- [07-03]: MailModule forRootAsync factory priority: DB SmtpConfig → MAIL_* env → TESSERA_SMTP_* env → localhost:1025 fallback
- [07-03]: DkvMailService injects SettingsService (not PrismaService directly) to reuse decryption logic
- [07-03]: user-files/ path resolved via path.resolve(__dirname, 4 levels up) from apps/api/dist/dkv/ to monorepo root
- [07-03]: Export prune sorted by mtime ascending (oldest first), delete all beyond last 10
- [07-04]: DkvScheduler v1 uses findFirst() — single-tenant; multi-tenant scheduling deferred
- [07-04]: Circular dep DkvService<->DkvScheduler avoided via controller coordination after PUT config
- [07-04]: CronJob resolved via require() workaround (pnpm strict isolation: transitive dep)
- [07-04]: rechnungsnummer from email subject regex /d{2}-d{9}-d{3}/; fallback=email-{uid}
- [07-05]: refreshKey lift: parent increments on checkNow success; InvoiceHistoryTable reruns useEffect
- [07-05]: onItemsLoaded callback: InvoiceHistoryTable notifies parent; parent passes items to ExportFileList (avoids second fetch)
- [07-05]: Password blank on load: configToForm() always sets password=''; hasPassword boolean drives UX hint only
- [07-05]: CsvImportButton replace: two-step inline confirm (not full modal); accept=".csv" client-side guard
- [Phase ?]: T-09-01/T-09-02
- [Phase ?]: 09-03
- [Phase ?]: 09-03
- [Phase ?]: 09-03
- [Phase ?]: splitCerts PEM path reuses parsePemChain; P7B sniffs first bytes
- [Phase ?]: splitCerts format crt comparison removed — detectFormat returns pem|der|pfx|p7b only
- [Phase 10]: Tender ist plattform-global (D-03): kein tenantId, keine RLS/forTenant() — Verhindert versehentliches Ausblenden globaler Daten fuer einen zweiten Mandanten
- [Phase 10-02]: category: 'procurement' fuer Ausschreibungs-Radar im Marketplace gewaehlt (freies kebab-case, keine Enum-Beschraenkung)
- [Phase 10-02]: Singleton doe-opendata Poll-Config wird direkt in TendersModule.onModuleInit() upserted, isActive:true per Default (D-04)
- [Phase 10-02]: Platzhalter-Seite tender-radar/page.tsx nutzt hartkodierten deutschen Text statt next-intl (volle i18n ist CONFIG-03, Phase 14)
- [Phase 10-03]: Real DÖE fixtures live-captured (pubDay=2026-07-19), not synthetic — 8 notices spanning all D-02 tag classes
- [Phase 10-03]: sourceNoticeId = OCDS release.id (stable, no version suffix), not the zip entry filename
- [Phase 10-03]: eForms-DE XML primary for deadlineAt/estimatedValue/procedureType; OCDS primary for ocid/buyerName/title/cpvCodes/region/plz
- [Phase 10-03]: bundesland left null this plan — NUTS-to-Bundesland mapping deferred to Phase 11 filter UI
- [Phase ?]: Poll-once-fan-out-many scheduler: single named cron job, no tenant parameter — deliberately drops DKV's activeTenantId/findFirst per-tenant framing (Pitfall D)
- [Phase ?]: SCHEMA-02 change detection implemented via prisma.tender.upsert({ where: { dedupKey } }) — identical notice never duplicates, changed contentHash updates in place
- [Phase ?]: D-05 retention as two-phase updateMany/deleteMany with deadlineAt:{lt} filters — null-deadline rows structurally excluded, never auto-expired
- [Phase ?]: TendersController talks to PrismaService directly (no intermediate service layer) — source-config upsert and global read are simple enough for this plan's scope
- [Phase ?]: Comment wording avoids the literal tenantId token in tenders.controller.ts to prevent false-positive grep-gate failures (same pattern as Plan 10-04)
- [Phase ?]: TenderQueryDto.status defaults to active at the controller call site, not baked into the DTO, mirroring DkvController's page/limit default-at-usage pattern
- [Phase ?]: Admin-Settings-Formular fuer den DOE-Poll (Intervall 5-1440, Aktiv-Toggle) spiegelt InboxConfigForm ohne Credential-Felder, da die DOE-Quelle keine Auth-Oberflaeche hat
- [Phase 10]: Keine module-loader-Whitelist noetig fuer settings/page.tsx (verschachtelte Route unter bereits whitelisteter tender-radar-Seite)
- [Phase ?]: estimatedValue kommt als String (Prisma Decimal) im JSON-Response — formatValue() im Frontend prüft explizit auf null/NaN statt zu koerzieren
- [Phase ?]: openOnly/includeNullValue Default-Semantik lebt im Builder, nicht im DTO
- [Phase ?]: deadlineFrom/deadlineTo URL-Param-Namen sind 1:1 identisch zu TenderQueryDto-Feldnamen fuer den Saved-Search-Vertrag aus Plan 11-06
- [Phase ?]: bundesland-Filter matcht exakt gegen die befüllte, indexierte Spalte (statt region-startsWith); region bleibt als eigenständiger Präfixfilter erhalten.
- [Phase ?]: BUNDESLAND_OPTIONS im Web als kleine Konstante gespiegelt (kein Shared-Package) — web nutzt @tessera/shared nicht, 16-Werte-Katalog rechtfertigt keine neue Cross-Package-Abhängigkeit.
- [Phase ?]: CPV-Divisions-Kurzkatalog (2-stellig, ~45 Einträge) statt EU-Vollkatalog; hasSome-Match gegen precomputed cpvDivisions-Spalte statt Raw-SQL-Präfix-Match
- [Phase ?]: cpv-URL-Param als wiederholter Key (?cpv=45&cpv=71) statt Komma-Join; DTO normalisiert Einzelwert per @Transform zu Array
- [Phase ?]: TenderDetail fetcht selbstständig via getTender(tenderId) im useEffect (Muster SourceConfigForm); page.tsx bleibt reiner ?tender-Param-Reader
- [Phase ?]: ResultsList.tsx modifiziert (nicht im Plan gelistet) — Rule 3: Zeilen-Klick-Handler war notwendig, um den Plan-eigenen key_link/Done-Kriterium zu erfüllen
- [Phase ?]: TenderTriage (neu, per-user, kein forTenant/RLS) + favOnly-Sentinel-ID '__none__' für garantierten Zero-Match statt versehentlich ungefilterter Liste
- [Phase ?]: TenderSavedSearch hat keinen Tender-FK — speichert nur Filterkriterien, unkritisch bei Retention.
- [Phase ?]: page/tender-URL-Params bewusst aus dem Suchprofil-Payload ausgeschlossen (Navigations-/View-State, kein Filter-State).
- [Phase ?]: Single notifiedAt field (not two per-channel timestamps) as the matched-vs-notified eligibility gate (12-01, D-06)
- [Phase ?]: Delta-only matching (no backfill/suppression table) structurally prevents backfill-flood for new saved-search profiles (12-01, D-07)
- [Phase ?]: TenderMailService swallows missing-SmtpConfig and send-failure into a single boolean (never throws) so the digest cron gets one clean success signal for stamping notifiedAt
- [Phase ?]: TenderDigestScheduler.runDigest(now) takes an injectable clock parameter for testable Monday-only weekly-digest gating
- [Phase ?]: Instant-Dispatch filtert die bereits geladenen savedSearches (kein zweiter Query); notifiedAt/channel='instant' nur nach Erfolg gestempelt — identisches Gate wie Digest (D-06)
- [Phase ?]: Digest-interval selector inline in settings/page.tsx (already 'use client'); instantAlert toggle uses plain checkbox for chip-based SavedSearchBar UI
- [Phase ?]: SCHEMA-03 fingerprint: title+buyer dominant, CPV division, value-bucket, deadline-day, sha256; dedupKey untouched, fingerprint additive
- [Phase ?]: One-time TS backfill scripts run via compiled dist/ output (not raw .ts execution) to keep tsc --noEmit clean
- [Phase ?]: SourceRegistry.register() throws DeniedPortalError for any portal in DENYLISTED_PORTALS (vergabe24, aumass), enforced at DI-registration time not just documented (INGEST-07)
- [Phase ?]: TenderDedupService tier-2 (source:noticeId) match is unconditional (not gated by dedupActive) — idempotent same-source re-poll must work even with a single active source
- [Phase ?]: Fingerprint always computed on tender.create regardless of dedupActive, so DÖE-only tenders become fingerprint-matchable the instant a 2nd source activates without further backfill
- [Phase ?]: 13-06: portalLabel()-Map bleibt lokal in TenderDetail.tsx (nicht geteilt) — einziger Consumer bisher; Fallback auf sourceUrl-Block bei fehlendem/leerem sources[].
- [Phase ?]: 13-04: cheerio (nicht node-html-parser) fuer NetServer-HTML-Parsing gewaehlt — Legitimacy-Gate flaggte node-html-parser als 'too-new' (Fehlmessung: Latest-Version-Datum statt Package-Alter); cheerio bestand das Gate sauber, Human-Approval eingeholt
- [Phase ?]: cosinex/DTVP-Selektoren voll befuellt statt needs-JS-deferred (D-01) — Trefferliste ist server-gerendert, live geprueft 2026-07-23
- [Phase ?]: cosinex/DTVP liefert echte Notice-Deep-Links (pid=) — besser als NetServer-Fallback (Such-URL); Rule-1-Fix: arrayBuffer()+TextDecoder('iso-8859-1') statt res.text(), da Portal ISO-8859-1 mit rohen Latin-1-Bytes sendet
- [Phase ?]: [quick-260723-e7i]: TenderNormalizerService dispatcht per sourceType (normalizeDoe/normalizeBag) mit geteiltem assemble()-Tail — schliesst den Normalizer-Gap aus 13-VERIFICATION.md fuer NetServer/Cosinex-Bag-Records
### Pending Todos
None yet.
### Blockers/Concerns
- [Roadmap v1.1]: DÖE OpenData API pagination/rate-limit parameters unverified (Swagger UI is JS-rendered) — resolve via a live API call during Phase 10 planning, not assumed from docs.
- [Roadmap v1.1]: Whether AI-AG NetServer / cosinex VMP search pages require JS rendering is unverified — needs a Phase 13 start-of-phase spike before committing to playwright.
### Quick Tasks Completed
| # | Description | Date | Commit | Directory |
|---|-------------|------|--------|-----------|
| 260630-gbh | User Settings: Passwort ändern (nur non-LDAP) + Profilbild setzen | 2026-06-30 | merge | [260630-gbh-user-settings-passwort-ndern-nur-non-lda](.planning/quick/260630-gbh-user-settings-passwort-ndern-nur-non-lda/) |
| 260701-abc | Fix i18n: marketplace.accessDenied + calendar form hardcoded EN strings | 2026-07-01 | e5b76b7 | [260701-abc-i18n-missing-keys](.planning/quick/260701-abc-i18n-missing-keys/) |
| 260707-csw | LDAP AD Anbindung: Zugangsdaten aus XWiki vorbefuellen und Import-Filter fuer Benutzer/Gruppen | 2026-07-07 | a5c500d | [260707-csw-ldap-ad-anbindung-zugangsdaten-aus-xwiki](.planning/quick/260707-csw-ldap-ad-anbindung-zugangsdaten-aus-xwiki/) |
| 260707-lgh | Favoriten-Widget: Icon-Proxy fuer Cross-Origin-Resource-Policy-Seiten (claude.ai) | 2026-07-07 | f06a2ff | [260707-lgh-favoriten-widget-icon-proxy-fuer-cross-o](.planning/quick/260707-lgh-favoriten-widget-icon-proxy-fuer-cross-o/) |
| 260708-cuc | Fix: FavoriteLink-Tabelle fehlt in Prod-DB, nie als Migration committed (500 auf GET /favorites) | 2026-07-08 | afef9b2 | [260708-cuc-fix-favoritelink-tabelle-fehlt-in-prod-d](.planning/quick/260708-cuc-fix-favoritelink-tabelle-fehlt-in-prod-d/) |
| 260708-rev | LDAP: CTL-spezifisches AD-Prefill entfernt (Multi-Tenant, "das war nie das Ziel") | 2026-07-08 | 8e8305c | (direct) |
| 260708-tst | LDAP: Verbindung testen vor dem Speichern einer Config moeglich | 2026-07-08 | 39aa4bf | (direct) |
| 260709-abd | LDAP: anonymous bind (bindDn/bindPassword optional, Schema nullable + Migration) | 2026-07-09 | 010aceb | (direct) |
| 260709-ciu | Auth: Benutzernamen ueberall case-insensitive (Login, Seed, LDAP-Sync + Daten-Migration) | 2026-07-09 | baff7ce | (direct) |
| 260709-lda | LDAP: ldapts empty-array-Attribut-Bug (E-Mail-Kollision auf Unique-Constraint) | 2026-07-09 | 246dc89 | (direct) |
| 260709-sbx | LDAP: Suchbox fuer die entdeckten Gruppen/OUs-Liste | 2026-07-09 | aaa2922 | (direct) |
| 260714-lex | LDAP: Per-User Exclude/Denylist-Filter (Service-Accounts vom Sync ausschliessen) — live verifiziert: deaktiviert 4 Accounts, 2 echte User aktiv | 2026-07-14 | 9d1323f | (direct) |
| 13 | Normalizer-Gap Phase 13 schliessen: NetServer/Cosinex-Bag-Dispatch (TenderNormalizerService) | 2026-07-23 | 9881005 | — |
## Deferred Items
Items acknowledged and carried forward from previous milestone close:
| Category | Item | Status | Deferred At |
|----------|------|--------|-------------|
| *(none)* | | | |
## Session Continuity
Last session: 2026-07-23T10:17:57.923Z
Stopped at: Phase 14 planned (5 plans, verification passed)
Resume file: .planning/phases/14-rss-email-alert-ingestion-module-rollout/14-01-PLAN.md
Last activity: 2026-07-14 - Built LDAP per-user exclude/denylist filter (9d1323f), migration applied on live DB, verified via Playwright: sync deactivated 4 excluded service accounts (administrator/krbtgt/guest/dns-ldap), 2 real LDAP users stay active, 0 wrongly created