Files

T

schalli 761077814f docs(10-03): complete DÖE source adapter & normalizer plan

2026-07-21 11:01:50 +02:00

13 KiB

Raw Blame History

gsd_state_version, milestone, milestone_name, current_phase, current_phase_name, status, stopped_at, last_updated, last_activity, last_activity_desc, progress

gsd_state_version

milestone

milestone_name

current_phase

current_phase_name

status

stopped_at

last_updated

last_activity

last_activity_desc

progress

1.0

v1.1

Ausschreibungs-Radar

ausschreibungs-radar-foundation-d-e-ingestion

executing

Completed 10-02-PLAN.md

2026-07-21T09:01:03.926Z

2026-07-21

Phase 10 execution started

total_phases	completed_phases	total_plans	completed_plans	percent
14	8	46	42	57

Project State

Project Reference

See: .planning/PROJECT.md (updated 2026-07-17)

Core value: Eine zentrale Plattform, in der beliebige Workflow-Tools als Module lizenziert, aktiviert und genutzt werden koennen -- ohne zwischen verschiedenen Anwendungen wechseln zu muessen. Current focus: Phase 10 — ausschreibungs-radar-foundation-d-e-ingestion

Current Position

Phase: 10 (ausschreibungs-radar-foundation-d-e-ingestion) — EXECUTING Plan: 4 of 6 Status: Ready to execute Last activity: 2026-07-21 — Phase 10 execution started

Progress: [░░░░░░░░░░] 0%

Performance Metrics

Velocity:

Total plans completed: 2
Average duration: 12 min
Total execution time: 0.38 hours

By Phase:

Phase	Plans	Total	Avg/Plan
01-foundation-portal-shell	2/3	23 min	12 min

Recent Trend:

Last 5 plans: 01-01 (16 min), 01-02 (7 min)
Trend: improving

Accumulated Context

Decisions

Decisions are logged in PROJECT.md Key Decisions table. Recent decisions affecting current work:

[Roadmap v1.1]: 5 phases (10-14) derived from 29 v1.1 requirements, standard granularity — DÖE-only MVP (10: ingestion, 11: filter/UI, 12: notifications) ships first as legally-clean demoable slice, then scraping adapters + cross-source dedup (13), then RSS + email-alert long tail (14)
[Roadmap v1.1]: Tender data modeled as platform-global (no tenantId on Tender) — only TenderSavedSearch/TenderMatch are tenant+user-scoped; deviates deliberately from the DKV per-tenant template
[Roadmap v1.1]: Scheduler must be poll-once-fan-out-many from Phase 10 onward — explicitly NOT the DKV findFirst() single-tenant pattern (documented pitfall)
[Roadmap v1.1]: Notification phase (12) requires an explicit matched-vs-notified state with backfill suppression to avoid first-activation email floods and duplicate digest+instant sends
[Roadmap v1.1]: INGEST-07 (vergabe24/aumass hard denylist) enforced in the adapter registry in Phase 13, not just documented
[Roadmap v1.1]: inbox/ module extraction (ImapProvider/ExchangeInboxProvider out of dkv/) scoped as a one-time prerequisite refactor inside Phase 14, not done earlier
[Roadmap]: 6 phases derived from 44 v1.0 requirements, standard granularity
[Roadmap]: Research recommends NestJS + Next.js + PostgreSQL RLS + Keycloak + Tauri stack
[01-01]: Used Traefik v2.11 instead of v3.4 due to Docker API version incompatibility on host
[01-01]: Traefik placed on frontend-net + backend-net for routing to both web and api services
[01-01]: API Dockerfile copies full monorepo node_modules structure for pnpm workspace compatibility
[01-02]: OKLCH color space for all design tokens (Tailwind v4 native, perceptually uniform)
[01-02]: Dark mode uses oklch(0.17 0.01 260) dark gray-blue for comfortable contrast with yellow primary
[01-02]: Cookie-based locale (NEXT_LOCALE) instead of URL routing for portal app
[01-02]: CSS custom property --current-sidebar-width for responsive main content margin
Phase ?: Route groups (auth)/(portal) for layout separation: auth pages standalone, portal pages wrapped in AppShell
Phase ?: Controller-level tenant isolation for ADMIN role as defense-in-depth alongside RLS
Phase ?: Remember-me controls cookie maxAge (30d session vs browser-session) not separate token type
Phase ?: react-grid-layout v2 uses dragConfig/resizeConfig instead of isDraggable/isResizable
[05-02]: SearchProvider defaults as constants merged with user DB rows (no seed migration)
[05-02]: useRef with explicit undefined initial value for React 19 strict TypeScript
[05-03]: DAVClient class constructor instead of createDAVClient factory (tsdav v2 type compatibility)
[05-03]: Dynamic imports for ews-javascript-api and @microsoft/microsoft-graph-client (lazy-load)
[05-03]: In-memory Map cache with 5-min TTL for calendar events (Redis not needed at current scale)
[05-03]: ews-javascript-api imported as any (no TypeScript definitions available)
Phase ?: Optimistic UI for calendar visibility toggle — reverts on API error
Phase ?: Auto-run testSource after adding new calendar source for immediate feedback
Phase ?: URL constructor for client-side https-only validation (T-05-14)
Phase ?: StoreExt trait import required for app.store() in Tauri 2.x
Phase ?: frontendDist=../src local page, navigate() for runtime URL override
Phase ?: CSP connect-src wildcard for configurable server URL (D-02)
Phase ?: Plain docker compose build statt build-push-action (Gitea JWT Pitfall 4)
Phase ?: Ephemeral runner mode (GITEA_RUNNER_EPHEMERAL=1) fuer Credential-Revokation pro Job
Phase ?: Separate docker-compose.ci.yml fuer opt-in CI-Infrastruktur
[07-01]: DKV PDF uses two extraction formats: single-tx (tab-separated) vs multi-tx (columnar) — both handled in DkvParserService
[07-01]: Research Pattern 4 regex replaced with empirical dual-format tab/columnar parser after testing against real invoice.pdf
[07-01]: CalendarCryptoService exported from CalendarModule for DKV credential encryption reuse
[07-02]: Max attachment size 25MB enforced in both ImapProvider and ExchangeInboxProvider before buffering (T-07-05)
[07-02]: ExchangeInboxProvider uses WellKnownFolderName.Inbox + FindItems (not FindAppointments — email vs calendar EWS API)
[07-02]: export type {} required for type-only re-exports under isolatedModules TypeScript setting
[07-03]: SettingsService.getStartupSmtpConfig uses findFirst (tenant-agnostic) for MailModule startup transport
[07-03]: MailModule forRootAsync factory priority: DB SmtpConfig → MAIL_* env → TESSERA_SMTP_* env → localhost:1025 fallback
[07-03]: DkvMailService injects SettingsService (not PrismaService directly) to reuse decryption logic
[07-03]: user-files/ path resolved via path.resolve(__dirname, 4 levels up) from apps/api/dist/dkv/ to monorepo root
[07-03]: Export prune sorted by mtime ascending (oldest first), delete all beyond last 10
[07-04]: DkvScheduler v1 uses findFirst() — single-tenant; multi-tenant scheduling deferred
[07-04]: Circular dep DkvService<->DkvScheduler avoided via controller coordination after PUT config
[07-04]: CronJob resolved via require() workaround (pnpm strict isolation: transitive dep)
[07-04]: rechnungsnummer from email subject regex /d{2}-d{9}-d{3}/; fallback=email-{uid}
[07-05]: refreshKey lift: parent increments on checkNow success; InvoiceHistoryTable reruns useEffect
[07-05]: onItemsLoaded callback: InvoiceHistoryTable notifies parent; parent passes items to ExportFileList (avoids second fetch)
[07-05]: Password blank on load: configToForm() always sets password=''; hasPassword boolean drives UX hint only
[07-05]: CsvImportButton replace: two-step inline confirm (not full modal); accept=".csv" client-side guard
Phase ?: splitCerts PEM path reuses parsePemChain; P7B sniffs first bytes
Phase ?: splitCerts format crt comparison removed — detectFormat returns pem|der|pfx|p7b only
[Phase 10]: Tender ist plattform-global (D-03): kein tenantId, keine RLS/forTenant() — Verhindert versehentliches Ausblenden globaler Daten fuer einen zweiten Mandanten
[Phase 10-02]: category: 'procurement' fuer Ausschreibungs-Radar im Marketplace gewaehlt (freies kebab-case, keine Enum-Beschraenkung)
[Phase 10-02]: Singleton doe-opendata Poll-Config wird direkt in TendersModule.onModuleInit() upserted, isActive:true per Default (D-04)
[Phase 10-02]: Platzhalter-Seite tender-radar/page.tsx nutzt hartkodierten deutschen Text statt next-intl (volle i18n ist CONFIG-03, Phase 14)
[Phase 10-03]: Real DÖE fixtures live-captured (pubDay=2026-07-19), not synthetic — 8 notices spanning all D-02 tag classes
[Phase 10-03]: sourceNoticeId = OCDS release.id (stable, no version suffix), not the zip entry filename
[Phase 10-03]: eForms-DE XML primary for deadlineAt/estimatedValue/procedureType; OCDS primary for ocid/buyerName/title/cpvCodes/region/plz
[Phase 10-03]: bundesland left null this plan — NUTS-to-Bundesland mapping deferred to Phase 11 filter UI

Pending Todos

None yet.

Blockers/Concerns

[Roadmap v1.1]: DÖE OpenData API pagination/rate-limit parameters unverified (Swagger UI is JS-rendered) — resolve via a live API call during Phase 10 planning, not assumed from docs.
[Roadmap v1.1]: Whether AI-AG NetServer / cosinex VMP search pages require JS rendering is unverified — needs a Phase 13 start-of-phase spike before committing to playwright.

Quick Tasks Completed

#	Description	Date	Commit	Directory
260630-gbh	User Settings: Passwort ändern (nur non-LDAP) + Profilbild setzen	2026-06-30	merge	260630-gbh-user-settings-passwort-ndern-nur-non-lda
260701-abc	Fix i18n: marketplace.accessDenied + calendar form hardcoded EN strings	2026-07-01	`e5b76b7`	260701-abc-i18n-missing-keys
260707-csw	LDAP AD Anbindung: Zugangsdaten aus XWiki vorbefuellen und Import-Filter fuer Benutzer/Gruppen	2026-07-07	`a5c500d`	260707-csw-ldap-ad-anbindung-zugangsdaten-aus-xwiki
260707-lgh	Favoriten-Widget: Icon-Proxy fuer Cross-Origin-Resource-Policy-Seiten (claude.ai)	2026-07-07	`f06a2ff`	260707-lgh-favoriten-widget-icon-proxy-fuer-cross-o
260708-cuc	Fix: FavoriteLink-Tabelle fehlt in Prod-DB, nie als Migration committed (500 auf GET /favorites)	2026-07-08	`afef9b2`	260708-cuc-fix-favoritelink-tabelle-fehlt-in-prod-d
260708-rev	LDAP: CTL-spezifisches AD-Prefill entfernt (Multi-Tenant, "das war nie das Ziel")	2026-07-08	`8e8305c`	(direct)
260708-tst	LDAP: Verbindung testen vor dem Speichern einer Config moeglich	2026-07-08	`39aa4bf`	(direct)
260709-abd	LDAP: anonymous bind (bindDn/bindPassword optional, Schema nullable + Migration)	2026-07-09	`010aceb`	(direct)
260709-ciu	Auth: Benutzernamen ueberall case-insensitive (Login, Seed, LDAP-Sync + Daten-Migration)	2026-07-09	`baff7ce`	(direct)
260709-lda	LDAP: ldapts empty-array-Attribut-Bug (E-Mail-Kollision auf Unique-Constraint)	2026-07-09	`246dc89`	(direct)
260709-sbx	LDAP: Suchbox fuer die entdeckten Gruppen/OUs-Liste	2026-07-09	`aaa2922`	(direct)
260714-lex	LDAP: Per-User Exclude/Denylist-Filter (Service-Accounts vom Sync ausschliessen) — live verifiziert: deaktiviert 4 Accounts, 2 echte User aktiv	2026-07-14	`9d1323f`	(direct)

Deferred Items

Items acknowledged and carried forward from previous milestone close:

Category	Item	Status	Deferred At
(none)

Session Continuity

Last session: 2026-07-21T09:00:23.857Z Stopped at: Completed 10-02-PLAN.md Resume file: None Last activity: 2026-07-14 - Built LDAP per-user exclude/denylist filter (9d1323f), migration applied on live DB, verified via Playwright: sync deactivated 4 excluded service accounts (administrator/krbtgt/guest/dns-ldap), 2 real LDAP users stay active, 0 wrongly created

13 KiB Raw Blame History