diff --git a/.planning/REQUIREMENTS.md b/.planning/REQUIREMENTS.md index 5142fcb..d0b4714 100644 --- a/.planning/REQUIREMENTS.md +++ b/.planning/REQUIREMENTS.md @@ -14,7 +14,7 @@ - [x] **INGEST-02**: Das System importiert Ausschreibungen von Administration-Intelligence-NetServer-Portalen (lhs-vpbw, tender24, vergabe.landbw) über einen konfigurierbaren Adapter. - [x] **INGEST-03**: Das System importiert Ausschreibungen vom cosinex-Vergabemarktplatz (DTVP) über einen Adapter. - [ ] **INGEST-04**: Das System importiert Ausschreibungen aus RSS-Feeds (subreport-elvis, service.bund.de). -- [ ] **INGEST-05**: Das System liest Portal-Benachrichtigungs-E-Mails aus einem konfigurierten Postfach ein (nutzt bestehende DKV-Inbox-Infrastruktur) und extrahiert daraus Ausschreibungen. +- [x] **INGEST-05**: Das System liest Portal-Benachrichtigungs-E-Mails aus einem konfigurierten Postfach ein (nutzt bestehende DKV-Inbox-Infrastruktur) und extrahiert daraus Ausschreibungen. - [x] **INGEST-06**: Jede Quelle wird einmal zentral pro Zeitplan abgefragt (poll-once-fan-out-many), Intervall pro Quelle im Admin-Bereich konfigurierbar; das Ergebnis wird an alle passenden Mandanten-Suchprofile verteilt. - [x] **INGEST-07**: vergabe24 und aumass sind als harte Denylist hinterlegt und können nicht als automatische Scraping-Quelle registriert werden (AGB-Verbot). @@ -101,7 +101,7 @@ | INGEST-07 | Phase 13 | Complete | | SCHEMA-03 | Phase 13 | Complete | | INGEST-04 | Phase 14 | Pending | -| INGEST-05 | Phase 14 | Pending | +| INGEST-05 | Phase 14 | Complete | | CONFIG-02 | Phase 14 | Pending | | CONFIG-03 | Phase 14 | Pending | | UI-06 | Phase 14 | Pending | diff --git a/.planning/ROADMAP.md b/.planning/ROADMAP.md index f7b17a0..72b9495 100644 --- a/.planning/ROADMAP.md +++ b/.planning/ROADMAP.md @@ -464,11 +464,11 @@ Plans: 4. vergabe24 and aumass are shown in the UI as "manually monitor" with a direct link, instead of appearing as a silent coverage gap 5. The entire module UI (results list, filters, saved searches, settings) is fully usable in both German and English -**Plans**: 5 plans +**Plans**: 1/5 plans executed **Wave 1** *(parallel — disjoint files)* -- [ ] 14-01-PLAN.md — Inbox-Modul-Extraktion (ImapProvider/ExchangeInboxProvider → apps/api/src/inbox/) + additive fetchMessages() (INGEST-05 prerequisite) +- [x] 14-01-PLAN.md — Inbox-Modul-Extraktion (ImapProvider/ExchangeInboxProvider → apps/api/src/inbox/) + additive fetchMessages() (INGEST-05 prerequisite) - [ ] 14-02-PLAN.md — RSS-Ingestion-Slice: RssAdapter + globale Feed-Liste-CRUD (Denylist/SSRF-Guard) + pollGranularity-Tick-Gate + Admin-UI (INGEST-04, CONFIG-02) **Wave 2** *(blocked on 14-01 + 14-02)* @@ -505,4 +505,4 @@ Phases execute in numeric order: 1 -> 2 -> 3 -> 4 -> 5 -> 6 -> 7 -> 8 -> 9 -> 10 | 11. Filter Engine, Results UI & Saved Searches | 6/6 | In Progress| | | 12. Tender Notifications | 4/4 | In Progress| | | 13. Scraping Adapters & Cross-Source Deduplication | 6/6 | In Progress| | -| 14. RSS, Email-Alert Ingestion & Module Rollout | 0/5 | Not started | - | +| 14. RSS, Email-Alert Ingestion & Module Rollout | 1/5 | In Progress| | diff --git a/.planning/STATE.md b/.planning/STATE.md index d52941e..d1c3e56 100644 --- a/.planning/STATE.md +++ b/.planning/STATE.md @@ -5,15 +5,15 @@ milestone_name: Ausschreibungs-Radar current_phase: 13 current_phase_name: scraping-adapters-cross-source-dedup status: verifying -stopped_at: Phase 14 planned (5 plans, verification passed) -last_updated: "2026-07-23T10:17:57.937Z" +stopped_at: Completed 14-01-PLAN.md +last_updated: "2026-07-23T11:11:11.104Z" last_activity: 2026-07-23 last_activity_desc: Phase 13 execution started progress: total_phases: 14 completed_phases: 12 total_plans: 67 - completed_plans: 61 + completed_plans: 62 --- # Project State @@ -32,7 +32,7 @@ Plan: 6 of 6 Status: Phase complete — ready for verification Last activity: 2026-07-23 — Phase 13 execution started -Progress: [██████████] 98% +Progress: [█████████░] 93% ## Performance Metrics @@ -96,6 +96,7 @@ Progress: [██████████] 98% | Phase 13 P06 | 15min | 2 tasks | 5 files | | Phase 13 P04 | 55min | 3 tasks | 6 files | | Phase 13 P05 | 40min | 2 tasks | 4 files | +| Phase 14 P01 | 13min | 2 tasks | 10 files | ## Accumulated Context @@ -207,6 +208,9 @@ Recent decisions affecting current work: - [Phase ?]: cosinex/DTVP-Selektoren voll befuellt statt needs-JS-deferred (D-01) — Trefferliste ist server-gerendert, live geprueft 2026-07-23 - [Phase ?]: cosinex/DTVP liefert echte Notice-Deep-Links (pid=) — besser als NetServer-Fallback (Such-URL); Rule-1-Fix: arrayBuffer()+TextDecoder('iso-8859-1') statt res.text(), da Portal ISO-8859-1 mit rohen Latin-1-Bytes sendet - [Phase ?]: [quick-260723-e7i]: TenderNormalizerService dispatcht per sourceType (normalizeDoe/normalizeBag) mit geteiltem assemble()-Tail — schliesst den Normalizer-Gap aus 13-VERIFICATION.md fuer NetServer/Cosinex-Bag-Records +- [Phase ?]: 14-01: DKV inbox providers moved verbatim into shared apps/api/src/inbox/ module; dkv.types.ts re-exports InboxConfig/InboxAttachment/InboxEmail so no DKV consumer import changed (D-01) +- [Phase ?]: 14-01: fetchMessages() added as a sibling method (findBodyParts/getItemBodySoap) on both providers without touching fetchPdfAttachments (D-02); DKV not switched to it +- [Phase ?]: 14-01: httpntlm loaded via raw require() bypasses vi.mock — tests seed require.cache with a stub before dynamically importing the provider ### Pending Todos @@ -245,7 +249,7 @@ Items acknowledged and carried forward from previous milestone close: ## Session Continuity -Last session: 2026-07-23T10:17:57.923Z -Stopped at: Phase 14 planned (5 plans, verification passed) -Resume file: .planning/phases/14-rss-email-alert-ingestion-module-rollout/14-01-PLAN.md +Last session: 2026-07-23T11:11:11.091Z +Stopped at: Completed 14-01-PLAN.md +Resume file: None Last activity: 2026-07-14 - Built LDAP per-user exclude/denylist filter (9d1323f), migration applied on live DB, verified via Playwright: sync deactivated 4 excluded service accounts (administrator/krbtgt/guest/dns-ldap), 2 real LDAP users stay active, 0 wrongly created diff --git a/.planning/phases/14-rss-email-alert-ingestion-module-rollout/14-01-SUMMARY.md b/.planning/phases/14-rss-email-alert-ingestion-module-rollout/14-01-SUMMARY.md new file mode 100644 index 0000000..7603305 --- /dev/null +++ b/.planning/phases/14-rss-email-alert-ingestion-module-rollout/14-01-SUMMARY.md @@ -0,0 +1,136 @@ +--- +phase: 14-rss-email-alert-ingestion-module-rollout +plan: 01 +subsystem: api +tags: [nestjs, imapflow, httpntlm, ews, vitest, refactor] + +# Dependency graph +requires: [] +provides: + - "apps/api/src/inbox/ module: InboxModule, InboxProvider, ImapProvider, ExchangeInboxProvider, InboxConfig/InboxAttachment/InboxEmail/InboxMessage" + - "InboxProvider.fetchMessages(config) seam — subject + HTML/text body per unread message, shared by any future module needing mailbox polling" + - "DKV switched to the shared inbox/ import path with zero behavior change (regression-gated by full API suite)" +affects: [14-03-email-alert-adapter] + +# Tech tracking +tech-stack: + added: [] + patterns: + - "Shared connection-only module extraction: inbox/ shares IMAP/EWS mechanics, each consuming module keeps its own independent config (D-03)" + - "require.cache stub for testing raw CommonJS require() dependencies that vi.mock cannot intercept (httpntlm)" + +key-files: + created: + - apps/api/src/inbox/inbox.module.ts + - apps/api/src/inbox/inbox.types.ts + - apps/api/src/inbox/inbox-provider.interface.ts + - apps/api/src/inbox/imap.provider.ts + - apps/api/src/inbox/exchange-inbox.provider.ts + - apps/api/src/inbox/imap.provider.spec.ts + - apps/api/src/inbox/exchange-inbox.provider.spec.ts + modified: + - apps/api/src/dkv/dkv.service.ts + - apps/api/src/dkv/dkv.module.ts + - apps/api/src/dkv/dkv.types.ts + +key-decisions: + - "Pure move + import-path swap for Task 1 — no logic changes to fetchPdfAttachments, InboxConfig fields, or DKV pipeline (D-01/D-02 regression guard)" + - "dkv.types.ts re-exports InboxConfig/InboxAttachment/InboxEmail from ../inbox/inbox.types instead of declaring them, so no DKV consumer import needed to change" + - "fetchMessages implemented as a sibling method (findBodyParts for IMAP MIME walk, getItemBodySoap for EWS) — does not touch or call into the existing PDF-attachment code paths" + - "httpntlm is loaded via a raw CommonJS require() in exchange-inbox.provider.ts, which vi.mock cannot intercept (verified via repro) — tests instead pre-seed Node's require.cache for the resolved httpntlm path with a stub before dynamically importing the provider in beforeAll" + +patterns-established: + - "Shared inbox/ module is the seam for any future mailbox-polling module (Plan 14-03's EmailAlertAdapter will inject ImapProvider/ExchangeInboxProvider from InboxModule)" + +requirements-completed: [INGEST-05] + +coverage: + - id: D1 + description: "DKV's IMAP/Exchange providers moved into shared apps/api/src/inbox/ module; DKV imports switched to the new path with zero behavior change" + requirement: "INGEST-05" + verification: + - kind: unit + ref: "cd apps/api && npx tsc --noEmit -p tsconfig.json && npx vitest run — full 301-test suite green (no DKV spec files exist yet, so this is a compile + no-regression-elsewhere gate)" + status: pass + human_judgment: false + - id: D2 + description: "New fetchMessages(config) method on InboxProvider/ImapProvider/ExchangeInboxProvider returns each unread message's subject + HTML/text body, without touching fetchPdfAttachments" + requirement: "INGEST-05" + verification: + - kind: unit + ref: "apps/api/src/inbox/imap.provider.spec.ts (7 tests) + apps/api/src/inbox/exchange-inbox.provider.spec.ts (6 tests)" + status: pass + human_judgment: false + +duration: 13min +completed: 2026-07-23 +status: complete +--- + +# Phase 14 Plan 01: Inbox Module Extraction Summary + +**Extracted DKV's IMAP/Exchange inbox providers into a shared `apps/api/src/inbox/` module and added an additive `fetchMessages()` method returning subject + HTML/text body per unread message — the seam Plan 14-03's EmailAlertAdapter will consume.** + +## Performance + +- **Duration:** ~13 min +- **Started:** 2026-07-23T12:59:00+02:00 (approx.) +- **Completed:** 2026-07-23T13:09:45+02:00 +- **Tasks:** 2/2 completed +- **Files modified:** 10 (5 new, 3 modified in Task 1; 2 new specs + 3 further modified in Task 2) + +## Accomplishments +- Moved `InboxProvider`, `ImapProvider`, `ExchangeInboxProvider`, and `InboxConfig`/`InboxAttachment`/`InboxEmail` verbatim from `apps/api/src/dkv/providers/` into `apps/api/src/inbox/`, with `dkv.types.ts` re-exporting the moved types so no other DKV file's import needed to change +- New `InboxModule` (NestJS `@Module`) provides + exports both providers; `DkvModule` now imports `InboxModule` instead of declaring the providers directly +- Added `fetchMessages(config): Promise` to the interface and both providers, entirely additive — `fetchPdfAttachments` bodies are byte-identical to before the move (verified via `git diff`) +- Net-new unit coverage: 13 tests across `imap.provider.spec.ts` (7) and `exchange-inbox.provider.spec.ts` (6), covering body extraction, idempotent \Seen/IsRead marking, sender filtering, and graceful `[]` returns on connect/search/HTTP errors + +## Task Commits + +Each task was committed atomically: + +1. **Task 1: Move inbox providers into apps/api/src/inbox/ (behavior-preserving)** - `eb668fd` (refactor) +2. **Task 2: Add fetchMessages() to InboxProvider + both implementations (net-new coverage)** - `c404954` (feat, TDD RED confirmed via 13 failing "not a function" assertions before implementation) + +**Plan metadata:** (this commit, to follow) + +## Files Created/Modified +- `apps/api/src/inbox/inbox.module.ts` - New NestJS module providing/exporting ImapProvider + ExchangeInboxProvider +- `apps/api/src/inbox/inbox.types.ts` - InboxConfig, InboxAttachment, InboxEmail (moved) + new InboxMessage type +- `apps/api/src/inbox/inbox-provider.interface.ts` - InboxProvider interface, now with fetchMessages +- `apps/api/src/inbox/imap.provider.ts` - ImapProvider, moved + fetchMessages/findBodyParts added +- `apps/api/src/inbox/exchange-inbox.provider.ts` - ExchangeInboxProvider, moved + fetchMessages/getItemBodySoap/fetchMessagesViaEws added +- `apps/api/src/inbox/imap.provider.spec.ts` - New: mocks ImapFlow, covers fetchMessages +- `apps/api/src/inbox/exchange-inbox.provider.spec.ts` - New: mocks httpntlm.post via require.cache stub, covers fetchMessages +- `apps/api/src/dkv/dkv.service.ts` - Import paths for ImapProvider/ExchangeInboxProvider changed to `../inbox/...` +- `apps/api/src/dkv/dkv.module.ts` - Imports InboxModule instead of declaring the two providers directly +- `apps/api/src/dkv/dkv.types.ts` - Re-exports InboxConfig/InboxAttachment/InboxEmail from `../inbox/inbox.types` + +## Decisions Made +- **Pure move, no logic changes (Task 1):** `fetchPdfAttachments`, `testConnection`, and all EWS SOAP builders were copied verbatim; only import paths changed. Verified via `git diff HEAD~1 HEAD` showing zero edits inside the pre-existing method bodies. +- **Re-export shim in dkv.types.ts:** rather than updating every DKV file that imports `InboxConfig`/`InboxAttachment`/`InboxEmail`, `dkv.types.ts` now re-exports them from `../inbox/inbox.types`, so only the two files that imported the *providers* (dkv.service.ts, dkv.module.ts) needed edits. +- **fetchMessages as a true sibling, not a refactor of fetchPdfAttachments:** IMAP's `findBodyParts` mirrors `collectPdfParts`'s tree-walk shape but is a separate function; EWS's `getItemBodySoap` is a separate SOAP builder from `getItemSoap`. Neither touches the PDF-attachment call path — this was required by D-01/D-02 (DKV must stay behavior-identical) and verified by grepping the diff for edits inside `fetchPdfAttachments`/`fetchViaEws`. +- **Testing httpntlm's raw `require()`:** `vi.mock('httpntlm', ...)` was verified (via a throwaway repro test) to have zero effect on `exchange-inbox.provider.ts`'s `const httpntlm = require('httpntlm')` — Vitest's mock interception only covers the ESM module graph, and a literal `require()` call resolves through Node's real module cache regardless. The spec instead pre-seeds `require.cache[require.resolve('httpntlm')]` with a stub `{ post }` in a `beforeAll` (dynamic `import()` of the provider, deferred out of module scope so the file stays compatible with the project's `commonjs` tsconfig which rejects top-level `await`/`import.meta`). + +## Deviations from Plan + +None — plan executed exactly as written. The `require.cache` testing approach for `exchange-inbox.provider.spec.ts` is a test-infrastructure detail within Task 2's stated scope ("mirroring cosinex.adapter.spec.ts's mock style ... stub httpntlm.post") — the plan did not prescribe *how* to intercept a raw CJS `require()`, and the chosen approach achieves the same outcome (httpntlm.post fully mocked, zero real network calls) without touching the provider's production code (which the plan explicitly forbids "improving" or refactoring). + +## Issues Encountered +- Discovered mid-Task-2 that `vi.mock('httpntlm', ...)` does not intercept `exchange-inbox.provider.ts`'s raw `require('httpntlm')` call (confirmed via a minimal repro: `vi.isMockFunction(require('httpntlm').post)` returned `false`, and all 6 initial test attempts hit a real DNS lookup — `getaddrinfo ENOTFOUND mail.example.com` — with 3 "passing" only by coincidence since their assertions expected `[]`, which a failed network call also produces). Resolved via the `require.cache` stub approach described above; re-verified all 6 exchange-inbox tests genuinely exercise the mocked `httpntlm.post` (assertions on `messages[0]` content, not just `[]`). + +## User Setup Required + +None - no external service configuration required. + +## Next Phase Readiness +- `apps/api/src/inbox/` is ready for Plan 14-03's `EmailAlertAdapter` to inject `ImapProvider`/`ExchangeInboxProvider` (via `InboxModule`) and call `fetchMessages(config)` for the tender-alert mailbox — with its own independent mailbox config (D-03), not DKV's. +- DKV's regression surface (fetchPdfAttachments, the full pipeline) is unchanged and the full API suite (301 tests) is green; no DKV-specific spec file exists yet, so the regression guard for this plan was the compile (`tsc --noEmit`) plus the rest of the suite not regressing — a gap the plan itself flagged as pre-existing (no `dkv/*.spec.ts` files in the repo). + +--- +*Phase: 14-rss-email-alert-ingestion-module-rollout* +*Completed: 2026-07-23* + +## Self-Check: PASSED + +All 7 created/modified inbox files verified on disk; both task commits (`eb668fd`, `c404954`) verified in `git log --oneline --all`.