666 lines
66 KiB
Markdown
666 lines
66 KiB
Markdown
# Phase 14: RSS, Email-Alert Ingestion & Module Rollout - Research
|
|
|
|
**Researched:** 2026-07-23
|
|
**Domain:** NestJS backend ingestion adapters (RSS/XML, IMAP/EWS email), per-tenant encrypted config, Next.js i18n (next-intl)
|
|
**Confidence:** MEDIUM (HIGH for code-pattern reuse, MEDIUM for live-feed shapes captured this session, LOW/flagged where a concrete decision is still open)
|
|
|
|
<user_constraints>
|
|
## User Constraints (from CONTEXT.md)
|
|
|
|
### Locked Decisions
|
|
|
|
**Inbox-Modul-Extraktion (INGEST-05, Refactor)**
|
|
- **D-01:** Additiver Refactor, DKV bleibt verhaltensgleich. Die bestehenden IMAP-/Exchange-Provider (`ImapProvider`, `ExchangeInboxProvider`) und das `InboxProvider`-Interface werden aus `apps/api/src/dkv/providers/` in ein geteiltes `apps/api/src/inbox/`-Modul gehoben. Geteilt wird ausschließlich die **Verbindungsmechanik** (IMAP/Exchange-Anmeldung, TLS, NTLM via httpntlm, Timeouts, Mail-Abruf).
|
|
- **D-02:** Die bestehende PDF-Anhang-Methode (`fetchPdfAttachments`) bleibt unverändert erhalten. Für die Ausschreibungs-Ingestion wird eine **zusätzliche** Methode ergänzt, die den Mail-Body (HTML/Text) und ganze Nachrichten liefert. DKV wird NICHT auf die neue Methode umgestellt — nur der Import-Pfad wechselt auf `inbox/`.
|
|
- **D-03:** Jedes Modul behält seine **eigene, unabhängige** Postfach-Konfiguration (eigene Adresse, eigene Credentials). Geteilt ist nur der Code, nicht die Config.
|
|
|
|
**E-Mail-Alert-Parsing (INGEST-05)**
|
|
- **D-04:** Generische Auswertung statt portalspezifischer Parser. Aus jeder Alert-Mail werden die enthaltenen Links (auf die Ausschreibungs-Detailseite) plus Betreff/erste Textzeile als Titel extrahiert. Feldarmut (kein CPV/Buyer/Frist) wird bewusst und offen dokumentiert.
|
|
- **D-05:** Konsequenz für Cross-Source-Dedup: E-Mail-/RSS-Records erben die Fingerprint-Asymmetrie aus Phase 13 (residual_gap_decision = Option C, dormant). Kein neuer Dedup-Mechanismus in Phase 14.
|
|
|
|
**Config-Modell + Admin-UI (CONFIG-02, CONFIG-03)**
|
|
- **D-06:** Postfach-Konfiguration ist **pro Mandant** (tenantId-scoped), Muster wie `DkvModuleConfig`. Neues Prisma-Modell, kein Ausbau des globalen `TenderSourcePollConfig`-Singletons.
|
|
- **D-07:** Zugangsdaten werden mit `CalendarCryptoService` (AES-256-GCM) verschlüsselt. Verschlüsselte Felder werden aus GET-Responses ausgeschlossen (Safe-Select-Muster, T-07-12).
|
|
- **D-08:** RSS-Feeds sind öffentlich und für alle gleich → globale Plattform-Konfiguration (bestehendes Singleton-Muster), nicht pro Mandant.
|
|
- **D-09:** Die bestehende Admin-Seite (`apps/web/.../tender-radar/settings/`) wird **erweitert**, nicht neu gebaut: ein Abschnitt „E-Mail-Alerts" (pro Mandant) und einer „RSS-Feeds" (global). Rollen-Trennung über bestehendes `@Roles(ADMIN, SUPER_ADMIN)`.
|
|
|
|
**i18n (UI-06)**
|
|
- **D-10:** Neuer `tenderRadar`-Namensraum in `de.json`/`en.json`; alle hartcodierten deutschen Strings der tender-radar-Komponenten in `useTranslations`-Keys überführt.
|
|
- **D-11:** KEIN sichtbarer Sprachumschalter. Die Sprachwahl folgt dem bestehenden next-intl-Mechanismus.
|
|
|
|
**Denylist-Transparenz (UI-06)**
|
|
- **D-12:** vergabe24 und aumass werden im UI als „manuell beobachten" mit Direktlink dargestellt. `CoverageBanner.tsx` ist der natürliche Ort. Denylist-Quelle bleibt der code-level Gate `DENYLISTED_PORTALS` (`source-registry.ts:12`).
|
|
|
|
### Claude's Discretion
|
|
- Wahl der RSS-Parsing-Bibliothek und das genaue Feld-Mapping der RSS-Items auf `NormalizedTenderFields`.
|
|
- Genaue Struktur des neuen `inbox/`-Modul-Interfaces und der zusätzlichen Message-Methode.
|
|
- Scheduler-Anbindung der neuen Quellen (RSS-Adapter, E-Mail-Adapter) über das bestehende `SourceRegistry` + `pollDueSources`-Muster.
|
|
- EN-Übersetzungen erstellt Claude selbst (Fachbegriffe der Vergabe-Domäne konsistent halten).
|
|
|
|
### Deferred Ideas (OUT OF SCOPE)
|
|
- **App-weiter Sprachumschalter** — eigene Plattform-Phase, nicht Teil von Phase 14.
|
|
- **Portalspezifische E-Mail-Parser** — erst sinnvoll, wenn konkrete Alert-Portale/echte Mail-Samples bekannt sind.
|
|
- **Cross-Source-Fingerprint für dünne Quellen** — erst bei Live-Aktivierung einer zweiten Quelle.
|
|
</user_constraints>
|
|
|
|
<phase_requirements>
|
|
## Phase Requirements
|
|
|
|
| ID | Description | Research Support |
|
|
|----|-------------|------------------|
|
|
| INGEST-04 | System importiert Ausschreibungen aus RSS-Feeds (subreport-elvis, service.bund.de) | Live feed shapes captured this session (service.bund.de + a subreport-elvis instance); `fast-xml-parser` (already installed) recommended over adding `rss-parser`; adapter design mirrors `NetServerAdapter`'s multi-portal-in-one-adapter pattern; RSS feed URLs recommended as an admin-CRUD list (new global model), not hardcoded — see "RSS Ingestion" below |
|
|
| INGEST-05 | System liest Portal-Benachrichtigungs-E-Mails aus einem konfigurierten Postfach und extrahiert Ausschreibungen (nutzt bestehende DKV-Inbox-Infrastruktur) | Concrete file-move list, new `fetchMessages()` method design (IMAP via imapflow body-part download, EWS via `t:Body` FieldURI addition), generic link+subject extraction algorithm, per-tenant fan-out design that reuses the poll-once-fan-out-many pattern **inside** a single adapter (avoids touching `TenderIngestionService`) — see "Inbox Module Extraction" and "Email-Alert Parsing" below |
|
|
| CONFIG-02 | Admin verwaltet Quellen-Poll-Konfiguration sowie das E-Mail-Ingestion-Postfach pro Mandant (Zugangsdaten verschlüsselt) | Exact Prisma model shape (mirrors `DkvModuleConfig`), Safe-Select pattern, `CalendarCryptoService` injection path, new `GET/PUT /modules/tender-radar/email-config` route pattern (mirrors `DkvController`) — see "Per-Tenant Config + Encryption" below |
|
|
| CONFIG-03 | Modul-UI ist mehrsprachig (DE/EN) | Confirmed next-intl locale mechanism (cookie `NEXT_LOCALE`, default `de`, no code change needed for D-11); concrete list of 8 files/~90 string-sites requiring conversion; namespace shape and key-parity test recommendation — see "i18n" below |
|
|
| UI-06 | Ausgeschlossene Portale werden als „manuell zu überwachen" mit Direktlink angezeigt | Confirmed real URLs (vergabe24.de, aumass.de); confirmed `DENYLISTED_PORTALS` has no API exposure today; recommended new small backend endpoint sourced from the same constant — see "Denylist UI Transparency" below |
|
|
</phase_requirements>
|
|
|
|
## Summary
|
|
|
|
Phase 14 is almost entirely **pattern replication**, not new architecture: every one of the five requirements has a directly analogous precedent already built and tested in this codebase (NetServer/cosinex adapters for scraping-into-bag ingestion; DKV for encrypted per-tenant mailbox config; the existing `dkvFleet` i18n namespace for the message-file shape). The main net-new engineering is (1) a `fetchMessages()` method on the inbox providers that doesn't exist yet, and (2) a per-tenant fan-out happening *inside* a single new adapter rather than at the `TenderSourcePollConfig`/`TenderIngestionService` level — because `Tender` and `TenderSourcePollConfig` are both structurally singleton/global (D-03 from Phase 10), and re-opening that architecture is explicitly out of scope.
|
|
|
|
Two real, previously-undocumented technical findings surfaced during this research and materially affect planning:
|
|
|
|
1. **Day-cursor throttling would silently cap RSS/email polling to once per calendar day**, regardless of the admin-configured `pollIntervalMin`, if the new sources are plugged into `TenderIngestionService.pollDueSources()` unmodified — the day-cursor gate (`nextDayToFetch`) was built for DÖE's daily batch export and is reused as-is by NetServer/cosinex today. This needs an explicit decision (see Common Pitfalls #1).
|
|
2. **subreport-elvis has no single global feed URL** — each municipality/Vergabestelle runs its own feed instance (e.g. `subreport-elvis.de/elvis/secure/rss.pl?id=4615` for Stadt Neuss). D-08 calls RSS "global, same for all," which is true for service.bund.de but not for subreport-elvis. The admin-CRUD "RSS-Feeds" list already implied by D-09 resolves this cleanly (admin adds whichever feed URLs are relevant) — this is a design confirmation, not a re-litigation of D-08.
|
|
|
|
**Primary recommendation:** Reuse `fast-xml-parser` (already a dependency, already configured with `removeNSPrefix`/CDATA-merge semantics in `doe-opendata.adapter.ts`) for RSS/Atom parsing instead of adding `rss-parser` as a new dependency. Build both new adapters (`RssAdapter`, `EmailAlertAdapter`) to route through the existing generic `normalizeBag()` normalizer path unchanged. Extract `ImapProvider`/`ExchangeInboxProvider`/`InboxProvider` into `apps/api/src/inbox/` with only import-path changes in `dkv.service.ts`/`dkv.module.ts`, and add one new interface method (`fetchMessages`) implemented in both providers without touching `fetchPdfAttachments`.
|
|
|
|
## Architectural Responsibility Map
|
|
|
|
| Capability | Primary Tier | Secondary Tier | Rationale |
|
|
|------------|-------------|----------------|-----------|
|
|
| RSS feed fetch + XML parse | API / Backend | — | Server-side scheduled job (`TenderSourceAdapter`), no browser involvement, mirrors `DoeOpenDataAdapter`/`NetServerAdapter` |
|
|
| RSS feed URL admin CRUD | API / Backend | Frontend Server (SSR form) | New global Prisma model + `@Roles(ADMIN,SUPER_ADMIN)` routes; frontend is a thin form, same tier split as existing `SourceConfigForm` |
|
|
| Email mailbox connect + fetch | API / Backend | — | IMAP/EWS network I/O must happen server-side (credentials, TLS/NTLM); mirrors `ImapProvider`/`ExchangeInboxProvider` |
|
|
| Email mailbox config CRUD (per tenant) | API / Backend | Frontend Server (SSR form) | Mirrors `DkvModuleConfig` + `DkvController`/`InboxConfigForm` split exactly |
|
|
| Credential encryption | API / Backend | Database (encrypted-at-rest column) | `CalendarCryptoService`, never touches browser or CDN tier |
|
|
| Denylist portal transparency | Frontend Server (SSR/client component) | API / Backend (data source) | Static-ish informational content; backend only needs to expose the denylist list, no new business logic |
|
|
| i18n message resolution | Frontend Server (SSR, next-intl `getRequestConfig`) | Browser (client components read via `useTranslations`) | Locale already resolved server-side from a cookie; client components consume via the existing `NextIntlClientProvider` |
|
|
| Cross-source dedup for new sources | API / Backend | Database (fingerprint column) | Explicitly **not** touched this phase (D-05) — inherits the dormant Phase-13 mechanism unchanged |
|
|
|
|
## Standard Stack
|
|
|
|
### Core
|
|
|
|
| Library | Version (installed) | Purpose | Why Standard (for this codebase) |
|
|
|---------|---------|---------|--------------|
|
|
| `fast-xml-parser` | `^5.10.1` [VERIFIED: already in apps/api/package.json, used live in `doe-opendata.adapter.ts`] | Parse RSS 2.0 XML into JS objects | Zero new dependency; already configured for `removeNSPrefix`/attribute handling; RSS 2.0 is plain XML, no need for a dedicated RSS-specific parser |
|
|
| `imapflow` | `^1.4.3` [VERIFIED: already in apps/api/package.json, used in `imap.provider.ts`] | IMAP protocol client | Already the codebase's IMAP client; the new `fetchMessages()` method is an additive use of the same client |
|
|
| `httpntlm` | `^1.8.13` [VERIFIED: already in apps/api/package.json, used in `exchange-inbox.provider.ts`] | NTLM-authenticated HTTP for EWS | Already the codebase's EWS/NTLM transport; reused as-is for the new EWS body-fetch SOAP call |
|
|
| `cheerio` | `^1.2.0` [VERIFIED: already in apps/api/package.json, used in `netserver.adapter.ts`] | HTML parsing (link extraction from HTML email bodies; optional RSS `<description>` sub-parsing) | Already the codebase's HTML-parsing tool (chosen over `node-html-parser` in Phase 13 after a package-legitimacy false flag — see 13-04 decision log); reused for extracting `<a href>` links from HTML email bodies and, optionally, structured sub-fields from RSS `<description>` HTML fragments |
|
|
|
|
### Supporting
|
|
|
|
| Library | Version | Purpose | When to Use |
|
|
|---------|---------|---------|-------------|
|
|
| `class-validator` | (already a dep, used throughout DTOs) | Validate new `TenderEmailConfigDto`/`TenderRssFeedSourceDto` | Same `@IsIn`/`@IsInt`/`@Min`/`@Max`/`@IsUrl` pattern as `DkvConfigDto`/`SourceConfigDto` |
|
|
|
|
### Alternatives Considered
|
|
|
|
| Instead of | Could Use | Tradeoff |
|
|
|------------|-----------|----------|
|
|
| `fast-xml-parser` (reuse) | `rss-parser` | `rss-parser` [OK verdict, see Package Legitimacy Audit] is purpose-built (handles more RSS/Atom edge cases, e.g. `<atom:link>`, out of the box) but adds a new dependency for a format (RSS 2.0) that is plain XML the codebase already parses successfully elsewhere. Only worth it if Atom-format feeds (not confirmed for either target source) turn out to need handling. |
|
|
| Generic bag + regex link extraction (email) | A dedicated HTML-email-parsing library (e.g. `mailparser`'s `simpleParser` output, or `linkifyjs`) | Not needed: `cheerio` (already installed) extracts `<a href>` from HTML bodies directly; a plain regex covers the plaintext-body fallback. Adding a new library for this narrow, already-covered need is unjustified. |
|
|
| Admin-CRUD list of RSS feed URLs (recommended) | Hardcoded feed URLs mirroring `NETSERVER_PORTALS` | Rejected because subreport-elvis has **no single URL** to hardcode (see Summary finding #2) — an admin-managed list is the only way to support "subreport-elvis" as a named source without picking one arbitrary municipality's feed at code-time. |
|
|
|
|
**Installation:**
|
|
```bash
|
|
# No new packages required — fast-xml-parser, imapflow, httpntlm, cheerio,
|
|
# and class-validator are all already dependencies of apps/api.
|
|
```
|
|
|
|
**Version verification:** All four libraries above are already installed and in live production use elsewhere in this codebase (confirmed via `grep` on `apps/api/package.json` and direct file reads of `doe-opendata.adapter.ts`, `imap.provider.ts`, `exchange-inbox.provider.ts`, `netserver.adapter.ts`). No `npm view` was needed since no new package is proposed; `rss-parser` (the one alternative considered) was checked via the package-legitimacy tool for completeness — see Package Legitimacy Audit.
|
|
|
|
## Package Legitimacy Audit
|
|
|
|
**No new external packages are proposed for this phase** — all required libraries (`fast-xml-parser`, `imapflow`, `httpntlm`, `cheerio`, `class-validator`) are already installed dependencies of `apps/api`, confirmed via direct inspection of `apps/api/package.json` and their live usage in existing adapters/providers.
|
|
|
|
One alternative package was checked for completeness (considered and rejected — see Alternatives Considered):
|
|
|
|
| Package | Registry | Age | Downloads | Source Repo | Verdict | Disposition |
|
|
|---------|----------|-----|-----------|-------------|---------|-------------|
|
|
| `rss-parser` | npm | published 2023-04-11 (~3 yrs) | ~825k/week | github.com/bobby-brennan/rss-parser | OK | Not adopted — `fast-xml-parser` (already installed) covers this phase's need with zero new dependency |
|
|
|
|
**Packages removed due to [SLOP] verdict:** none
|
|
**Packages flagged as suspicious [SUS]:** none
|
|
|
|
## Architecture Patterns
|
|
|
|
### System Architecture Diagram
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────────────────┐
|
|
│ TenderSchedulerService (cron) │
|
|
│ single global tick — no tenant dimension (unchanged) │
|
|
└───────────────────────────────┬───────────────────────────────────────┘
|
|
│ tick
|
|
▼
|
|
┌─────────────────────────────────────────────────────────────────────┐
|
|
│ TenderIngestionService.pollDueSources() │
|
|
│ findMany(TenderSourcePollConfig where isActive) → fan-out per row │
|
|
└───┬─────────────┬─────────────┬─────────────┬─────────────┬─────────┘
|
|
│ doe-opendata│ ai-netserver│ cosinex-dtvp│ rss (NEW) │ email-alert (NEW)
|
|
▼ ▼ ▼ ▼ ▼
|
|
┌────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌───────────────────┐
|
|
│DoeAdap-│ │NetServer │ │Cosinex │ │RssAdapter│ │EmailAlertAdapter │
|
|
│ter │ │Adapter │ │Adapter │ │(NEW) │ │(NEW) │
|
|
└────────┘ └──────────┘ └──────────┘ └────┬─────┘ └────────┬───────────┘
|
|
│ │ internal fan-out:
|
|
│ fetch each │ findMany(TenderEmailConfig
|
|
│ admin-added │ where isActive) — ONE tenant
|
|
│ feed URL │ mailbox per row, catch-per-tenant
|
|
▼ ▼
|
|
┌──────────────┐ ┌──────────────────────┐
|
|
│TenderRssFeed │ │inbox/ module │
|
|
│Source (NEW, │ │ ImapProvider / │
|
|
│global, admin │ │ ExchangeInboxProvider│
|
|
│CRUD list) │ │ .fetchMessages(cfg) │
|
|
└──────────────┘ │ → decrypt creds via │
|
|
│ CalendarCryptoService│
|
|
│ from TenderEmailConfig│
|
|
│ (per-tenant, NEW) │
|
|
└───────────┬───────────┘
|
|
│ subject + body
|
|
▼
|
|
┌──────────────────────┐
|
|
│ generic link+subject │
|
|
│ extraction (cheerio/ │
|
|
│ regex) → RawTender- │
|
|
│ Record per link │
|
|
└───────────┬───────────┘
|
|
└──────────────────────────┬──────────────────────────────────────┘
|
|
▼
|
|
TenderNormalizerService.normalize()
|
|
(dispatches 'rss'/'email-alert' → normalizeBag(),
|
|
UNCHANGED from Phase 13's NetServer/cosinex path)
|
|
│
|
|
▼
|
|
TenderDedupService.resolve() → Tender (global, D-03)
|
|
│
|
|
▼
|
|
TendersController (GET/PUT /modules/tender-radar/*)
|
|
│
|
|
▼
|
|
Next.js: ResultsList / TenderDetail / CoverageBanner (+ denylist block)
|
|
all converted to useTranslations('tenderRadar') (D-10)
|
|
```
|
|
|
|
### Recommended Project Structure
|
|
|
|
```
|
|
apps/api/src/
|
|
├── inbox/ # NEW — extracted shared module (D-01/D-02/D-03)
|
|
│ ├── inbox.module.ts # provides+exports ImapProvider, ExchangeInboxProvider
|
|
│ ├── inbox.types.ts # InboxConfig/InboxAttachment/InboxEmail/InboxMessage (moved+extended)
|
|
│ ├── inbox-provider.interface.ts # moved from dkv/providers/, +fetchMessages()
|
|
│ ├── imap.provider.ts # moved from dkv/providers/, +fetchMessages()
|
|
│ └── exchange-inbox.provider.ts # moved from dkv/providers/, +fetchMessages()
|
|
├── dkv/
|
|
│ ├── dkv.service.ts # import path only: '../inbox/...' instead of './providers/...'
|
|
│ ├── dkv.module.ts # imports InboxModule instead of declaring the 2 providers directly
|
|
│ ├── dkv.types.ts # re-exports InboxConfig/InboxAttachment/InboxEmail from '../inbox/inbox.types'
|
|
│ └── providers/ # now empty of Inbox* files — directory removed if nothing else lives there
|
|
└── tenders/
|
|
├── adapters/
|
|
│ ├── rss.adapter.ts # NEW
|
|
│ ├── rss.adapter.spec.ts # NEW (fixture-based, mirrors cosinex.adapter.spec.ts)
|
|
│ ├── email-alert.adapter.ts # NEW
|
|
│ └── email-alert.adapter.spec.ts # NEW (mocked InboxProvider)
|
|
├── __fixtures__/
|
|
│ ├── service-bund-feed.xml # NEW — live-captured fixture
|
|
│ └── subreport-elvis-feed.xml # NEW — live-captured fixture (see Open Questions)
|
|
├── dto/
|
|
│ ├── tender-email-config.dto.ts # NEW
|
|
│ └── tender-rss-feed.dto.ts # NEW
|
|
├── tender-email-config.service.ts # NEW — mirrors DkvService's config half
|
|
└── tender.types.ts # SourceType extended: + 'rss' | 'email-alert'
|
|
|
|
apps/web/src/app/(portal)/modules/tender-radar/
|
|
├── settings/components/
|
|
│ ├── EmailAlertConfigForm.tsx # NEW — mirrors InboxConfigForm.tsx (per-tenant, password field)
|
|
│ └── RssFeedListForm.tsx # NEW — admin CRUD list (add/remove feed URL + label)
|
|
└── components/CoverageBanner.tsx # extended with a denylist block (UI-06)
|
|
|
|
apps/web/src/messages/
|
|
├── de.json # + "tenderRadar": { ... } namespace
|
|
└── en.json # + "tenderRadar": { ... } namespace (EN translations)
|
|
```
|
|
|
|
### Pattern 1: Multi-source adapter with internal fan-out (RSS + per-tenant email)
|
|
|
|
**What:** A single `TenderSourceAdapter` implementation internally iterates over multiple concrete targets (multiple RSS feed URLs, or multiple tenants' mailboxes) rather than the fan-out happening at the `TenderSourcePollConfig`/`SourceRegistry` level.
|
|
|
|
**When to use:** When the natural "one row per pollable thing" doesn't fit the existing singleton/global `TenderSourcePollConfig` (one row per `sourceType`) — exactly the situation for both RSS (multiple feed URLs under one `sourceType: 'rss'`) and email-alert (multiple tenant mailboxes under one `sourceType: 'email-alert'`).
|
|
|
|
**Example (already proven in this codebase, `netserver.adapter.ts:75-94`):**
|
|
```typescript
|
|
// Source: apps/api/src/tenders/adapters/netserver.adapter.ts (existing, Phase 13)
|
|
async fetchTenders(_dayCursor: string): Promise<RawTenderRecord[]> {
|
|
const records: RawTenderRecord[] = [];
|
|
for (const portal of this.portals) {
|
|
try {
|
|
const html = await this.fetchPortalHtml(NETSERVER_PORTALS[portal].baseUrl);
|
|
records.push(...this.parseSearchResults(html, portal, new Date()));
|
|
} catch (error) {
|
|
this.logger.warn(`NetServer portal '${portal}' fetch failed, skipping: ${(error as Error).message}`);
|
|
// catch-per-target — one broken target never aborts the others (D-01 from Phase 13)
|
|
}
|
|
}
|
|
return records;
|
|
}
|
|
```
|
|
Recommended `EmailAlertAdapter.fetchTenders()` shape (new, mirrors the above exactly, substituting "portal" with "tenant mailbox config row"):
|
|
```typescript
|
|
async fetchTenders(_dayCursor: string): Promise<RawTenderRecord[]> {
|
|
const configs = await this.prisma.tenderEmailConfig.findMany({ where: { isActive: true } });
|
|
const records: RawTenderRecord[] = [];
|
|
for (const cfg of configs) {
|
|
try {
|
|
const inboxConfig = await this.resolveDecryptedConfig(cfg); // CalendarCryptoService.decrypt
|
|
const provider = cfg.protocol === 'exchange' ? this.exchangeProvider : this.imapProvider;
|
|
const messages = await provider.fetchMessages(inboxConfig);
|
|
records.push(...this.extractCandidates(messages, cfg.tenantId));
|
|
} catch (error) {
|
|
this.logger.warn(`Email-alert mailbox for tenant ${cfg.tenantId} failed, skipping: ${(error as Error).message}`);
|
|
// catch-per-tenant — one tenant's broken mailbox never blocks the others
|
|
}
|
|
}
|
|
return records;
|
|
}
|
|
```
|
|
This keeps `TenderIngestionService.pollDueSources()`, `TenderSourcePollConfig`, and `SourceRegistry` **completely untouched** except for two additive `registry.register()` calls and two additive `tenderSourcePollConfig.upsert()` seed calls in `tenders.module.ts` — exactly the same additive pattern Plans 13-04/13-05 already used for NetServer/cosinex.
|
|
|
|
### Pattern 2: Additive inbox-provider method (D-02)
|
|
|
|
**What:** Add `fetchMessages(config): Promise<InboxMessage[]>` to `InboxProvider` alongside the existing `fetchPdfAttachments`, without modifying it.
|
|
|
|
**IMAP implementation sketch** (extends the existing `imap.provider.ts` MIME-tree-walking pattern used by `collectPdfParts`):
|
|
```typescript
|
|
// New sibling to collectPdfParts() in imap.provider.ts / inbox/imap.provider.ts
|
|
function findBodyParts(node: MessageStructureObject | undefined): { htmlPart?: string; textPart?: string } {
|
|
// Walk node.childNodes exactly like collectPdfParts, but match
|
|
// type === 'text/html' → htmlPart, type === 'text/plain' → textPart (first match wins for each).
|
|
}
|
|
|
|
async fetchMessages(config: InboxConfig): Promise<InboxMessage[]> {
|
|
// Same connect/lock/search/fetchAll(envelope+bodyStructure) skeleton as fetchPdfAttachments,
|
|
// but instead of collectPdfParts + client.download() for PDF parts, use findBodyParts +
|
|
// client.download() for the html/text part IDs, then streamToBuffer(...).toString('utf8').
|
|
// Mark \Seen after processing (same as fetchPdfAttachments) — this IS the idempotency
|
|
// mechanism for re-polls (D-04's "re-polling the mailbox doesn't create duplicates").
|
|
}
|
|
```
|
|
|
|
**EWS implementation sketch** (extends `exchange-inbox.provider.ts`'s SOAP builders):
|
|
```typescript
|
|
// getItemSoap() needs one more AdditionalProperties FieldURI:
|
|
// <t:FieldURI FieldURI="item:Body"/>
|
|
// The GetItem response then contains <t:Body BodyType="HTML">...escaped HTML...</t:Body>
|
|
// or BodyType="Text". Extract via:
|
|
const bodyType = extractAttr(block, 't:Body', 'BodyType'); // 'HTML' | 'Text'
|
|
const bodyRaw = extractAll(block, 't:Body')[0] ?? '';
|
|
// EWS SOAP-escapes the body content (already handled by the existing escapeXml/extractAll
|
|
// pair used elsewhere in this file) — verify against a captured fixture, not live, per
|
|
// Validation Architecture below (EWS/NTLM is a documented fragile path, see Pitfall 2).
|
|
```
|
|
|
|
### Pattern 3: RSS item → generic bag mapping (reuses Phase 13's `normalizeBag()` unchanged)
|
|
|
|
Given the live-captured shapes (see "RSS Ingestion" findings below), map each `<item>` to the SAME flat bag shape `NetServerAdapter`/`CosinexAdapter` already produce — no normalizer changes needed:
|
|
|
|
```typescript
|
|
// Source: pattern mirrors apps/api/src/tenders/adapters/netserver.adapter.ts's ocdsPayload bag
|
|
records.push({
|
|
sourceType: 'rss',
|
|
sourcePortal: feedLabel, // admin-provided label or hostname-derived slug, e.g. 'service-bund'
|
|
sourceNoticeId: guid || sha256(link).slice(0, 40),
|
|
sourceUrl: link,
|
|
fetchedAt: new Date(),
|
|
publishedAt: pubDate ? new Date(pubDate) : null,
|
|
eformsPayload: null,
|
|
ocdsPayload: {
|
|
title, // item.title, CDATA-merged automatically by fast-xml-parser
|
|
buyerName: null, // OPTIONAL enhancement: regex/cheerio-extract from <description>
|
|
procedureType: null,
|
|
legalFramework: null,
|
|
deadlineAt: null, // OPTIONAL enhancement: regex-extract "Angebotsfrist: ..." from <description>
|
|
},
|
|
});
|
|
```
|
|
|
|
**Optional enhancement (higher data quality, Claude's discretion):** service.bund.de's `<description>` is a CDATA-wrapped HTML fragment with labeled fields (`Erfüllungsort: <strong>PLZ Ort</strong>`, `Vergabestelle: <strong>Name</strong>`, `Angebotsfrist: <strong>DD.MM.YYYY HH:MM</strong>` — confirmed live, see below). Loading this fragment with `cheerio.load()` and matching label text via `:contains()` can populate `buyerName`/`deadlineAt`/`region` beyond the bare-minimum bag — the same technique `NetServerAdapter` uses for its HTML table. This is worth doing for service.bund.de specifically (structure confirmed); do **not** assume the same label set for subreport-elvis without a live fixture capture first (its `<description>` was observed to be an HTML `<table>`, structure not fully characterized this session — see Open Questions).
|
|
|
|
### Anti-Patterns to Avoid
|
|
- **Extending `TenderSourcePollConfig` with a `tenantId` column:** Would break the "singleton, RLS-exempt, global" invariant that `TenderIngestionService`/`TenderSchedulerService` depend on throughout (D-03 from Phase 10). Use the internal-fan-out pattern above instead.
|
|
- **Wrapping the new per-tenant queries in `forTenant()`/RLS:** `TenderEmailConfig` rows ARE per-tenant data, but the `EmailAlertAdapter`'s poll-time read needs to see **all** tenants' active configs in one query (`findMany` across all tenants) — this is a deliberate, audited cross-tenant read at the platform-scheduler level, analogous to how `TenderIngestionService` itself never uses `forTenant()`. Document this exception clearly in the adapter's docstring (mirroring the existing docstring convention) so it isn't "fixed" into an RLS-wrapped query later.
|
|
- **Portal-specific email parsers:** explicitly deferred (D-04, Deferred Ideas) — do not add sender-domain-based branching logic even if it seems easy for one obvious case.
|
|
|
|
## Don't Hand-Roll
|
|
|
|
| Problem | Don't Build | Use Instead | Why |
|
|
|---------|-------------|-------------|-----|
|
|
| RSS/XML parsing | A custom regex-based RSS tag extractor | `fast-xml-parser` (already installed, already configured) | Already handles CDATA-merge, attributes, namespace stripping — a hand-rolled regex parser would re-solve a problem this dependency already solves correctly elsewhere in the codebase |
|
|
| Credential encryption | A new AES implementation or a different crypto library | `CalendarCryptoService` (D-07, already exported by `CalendarModule`) | Exact same AES-256-GCM/`iv:authTag:ciphertext` format already used by DKV and SMTP; introducing a second crypto scheme would fragment key management and audit surface |
|
|
| HTML link extraction from email bodies | A regex-only HTML tag stripper | `cheerio` (already installed) | Regex-based HTML parsing is a well-known correctness trap (nested tags, malformed markup, HTML entities); cheerio already handles this correctly for NetServer's table parsing |
|
|
| NTLM/EWS SOAP construction | A new EWS client library | The existing hand-rolled SOAP builders in `exchange-inbox.provider.ts` | Extending the existing, already-battle-tested (if fragile) SOAP-string builders is lower-risk than introducing a second EWS abstraction layer for one new field |
|
|
|
|
**Key insight:** every "don't hand-roll" item in this phase already has a proven, in-repo solution from a previous phase — the engineering discipline this phase requires is **restraint** (reuse the existing tool) more than net-new tool selection.
|
|
|
|
## Runtime State Inventory
|
|
|
|
> Scoped to the INGEST-05 inbox-module-extraction refactor only (the rest of this phase is greenfield).
|
|
|
|
| Category | Items Found | Action Required |
|
|
|----------|-------------|------------------|
|
|
| Stored data | None — no database rows reference the file path `dkv/providers/` or the class names by string; Prisma models (`DkvModuleConfig`) reference no file paths | None |
|
|
| Live service config | None — IMAP/EWS server configuration lives entirely in `DkvModuleConfig` rows (host/port/credentials), not tied to which TS file implements the connection | None |
|
|
| OS-registered state | None — no OS-level task/service registration references these file paths | None |
|
|
| Secrets/env vars | None — `CALENDAR_ENCRYPTION_KEY` and DKV mailbox credentials are addressed by `tenantId`, not by import path | None |
|
|
| Build artifacts | `apps/api/dist/dkv/providers/*.js` will simply stop being emitted once the source files move; nothing external references the old `dist/` path (DKV's `__dirname`-relative `user-files/` path resolution in `dkv.service.ts` is unrelated to the providers/ subfolder) | Standard rebuild after the move — no migration script needed |
|
|
|
|
**Confirmed via:** direct reads of `dkv.service.ts`, `dkv.module.ts`, `dkv.types.ts`, and `inbox-provider.interface.ts` — the only two files that need an import-path edit are `dkv.service.ts` (2 import lines) and `dkv.module.ts` (2 import lines + swap direct-provider-declaration for `InboxModule` import).
|
|
|
|
## Common Pitfalls
|
|
|
|
### Pitfall 1: Day-cursor gate silently throttles RSS/email to once per calendar day
|
|
**What goes wrong:** `TenderIngestionService.pollDueSources()` computes `nextDayToFetch(config.lastIngestedDay)` per source row and only calls `adapter.fetchTenders(cursorDay)` when the result is a day strictly before "today" — then advances `lastIngestedDay` and stops. This means each source gets **at most one fetch per calendar day**, no matter how frequently the cron tick fires or what `pollIntervalMin` is configured to. This is correct for DÖE (a genuine daily batch export) and is currently *also* silently applied to `ai-netserver`/`cosinex-dtvp` (dormant, unconfirmed live) — but for RSS and especially email-alerts, this defeats the purpose of a configurable poll interval: an admin setting "check every 30 minutes" would still only see new items once every 24 hours.
|
|
**Why it happens:** The day-cursor gate and the cron-tick interval are two independent mechanisms (documented as "Pitfall A" in `tender-scheduler.service.ts`) — the existing code was written exclusively for a day-batch-export source and adapters since have inherited the same gate without re-examining the fit.
|
|
**How to avoid:** Add an explicit flag distinguishing day-batch sources from tick-driven sources, e.g. a `pollGranularity: String @default("day")` column on `TenderSourcePollConfig` (`'day' | 'tick'`), seeded `'tick'` for `rss` and `'email-alert'`. In `pollDueSources()`, branch: `'day'` sources keep the existing `nextDayToFetch` gate unchanged (zero regression risk for `doe-opendata`/`ai-netserver`/`cosinex-dtvp`); `'tick'` sources call `adapter.fetchTenders(berlinToday)` unconditionally on every tick where `isActive`, relying purely on the cron interval (already bounded to a minimum by `SourceConfigDto`'s validation, and "sub-hourly polling" is already out of scope per REQUIREMENTS.md) for pacing, and on IMAP `\Seen`/EWS `IsRead` marking (email) or feed-item GUID dedup (RSS, via `dedupKey` upsert — already idempotent) for not reprocessing.
|
|
**Warning signs:** A newly-activated RSS/email source shows zero new tenders for up to 24h after activation even though the feed/mailbox clearly has new items; `lastIngestedDay` advancing by exactly one calendar day per admin-configured interval regardless of the interval's actual value.
|
|
|
|
### Pitfall 2: subreport-elvis has no single canonical feed URL
|
|
**What goes wrong:** Treating "subreport-elvis" as one hardcoded portal (like `NETSERVER_PORTALS`) is structurally impossible — the platform issues one feed per contracting authority/municipality (confirmed live: Stadt Neuss's feed is `https://www.subreport-elvis.de/elvis/secure/rss.pl?id=4615`; other municipalities have different numeric `id` values on the same base path). Hardcoding one municipality's feed would silently scope "subreport-elvis ingestion" to that one city only, contradicting the intent of a platform-wide long-tail source.
|
|
**Why it happens:** The phase description names "subreport-elvis" as if it were a single source, matching the `service.bund.de` pattern (which genuinely IS one national feed) — but subreport-elvis's product model is per-customer feed subscriptions.
|
|
**How to avoid:** Build the admin-managed list (`TenderRssFeedSource`, already implied by D-09's "RSS-Feeds" admin section) so an admin can add whichever concrete subreport-elvis feed URL(s) are relevant, rather than baking one into code. Seed the DB with zero subreport-elvis rows by default (there is no universally-correct one); optionally seed the confirmed service.bund.de national feed as a sane global default since that one genuinely is Germany-wide.
|
|
**Warning signs:** A hardcoded `SUBREPORT_ELVIS_URL` constant with only one municipality's feed id.
|
|
|
|
### Pitfall 3: Runtime-added RSS feed URLs bypass the INGEST-07 code-level denylist gate
|
|
**What goes wrong:** `SourceRegistry.register()` checks `DENYLISTED_PORTALS` against an adapter's **statically declared** `portals: readonly string[]` array at DI-boot time. An admin-CRUD RSS feed URL is added at **runtime** (via the new `TenderRssFeedSource` model), long after boot — the `RssAdapter`'s `portals` array can only contain a symbolic placeholder (e.g. `['rss']`), not the actual runtime feed hostnames. This means the code-level denylist gate provides **zero protection** against an admin accidentally adding an RSS feed URL that points at vergabe24.de/aumass.de.
|
|
**Why it happens:** INGEST-07's denylist gate was designed for adapter-registration-time enforcement (a developer registering a whole new scraping adapter), not for runtime, per-row admin input — RSS is the first source type in this codebase where the "portal" is admin-supplied data rather than code.
|
|
**How to avoid:** Add a parallel, save-time hostname validation in the `TenderRssFeedSource` create/update service (reject any URL whose hostname contains `vergabe24` or `aumass`, mirroring the spirit of `DENYLISTED_PORTALS` even though it's enforced in a different code path). Document in the DTO/service that this is a **second, independent** enforcement point, not a bypass of the existing one.
|
|
**Warning signs:** A `TenderRssFeedSource` row pointing at a vergabe24/aumass hostname existing in the DB without any validation error at save time.
|
|
|
|
### Pitfall 4: EWS body-fetch is a new, untested addition to a documented-fragile file
|
|
**What goes wrong:** `exchange-inbox.provider.ts` implements NTLM/EWS SOAP by hand (no EWS client library) and is called out in project memory as fragile (`project_calendar_ews_fix`: "Exchange EWS nutzt NTLM via httpntlm... nie ews-javascript-api"). Adding a new `fetchMessages()` method here — even though additive and not touching `fetchPdfAttachments` — introduces new hand-written SOAP/XML-extraction code (`t:Body`/`BodyType` handling) with **zero existing test coverage** to fall back on (there are currently no `.spec.ts` files for either `ImapProvider` or `ExchangeInboxProvider` in this codebase at all).
|
|
**Why it happens:** DKV's EWS path has historically been validated live against a real Exchange server rather than via unit tests (confirmed: `find apps/api/src/dkv -iname "*.spec.ts"` returns nothing).
|
|
**How to avoid:** Do not skip unit tests for the new method just because the existing methods also lack them — this phase is the right moment to add fixture/mock-based coverage for `fetchMessages()` on both providers (mock `ImapFlow`/`httpntlm.post`), even if `fetchPdfAttachments` remains untested (out of scope to backfill). Flag the EWS body-fetch specifically for a `checkpoint:human-verify` against a real Exchange mailbox before considering INGEST-05's Exchange path production-ready — this mirrors Phase 13's human-verification-required pattern for anything requiring live external access.
|
|
**Warning signs:** `fetchMessages()` on the Exchange path passing all mocked unit tests but returning empty/garbled bodies against a real mailbox (EWS body escaping/whitespace handling is a common source of subtle bugs).
|
|
|
|
### Pitfall 5: Email-derived Tender rows become visible to every tenant, not just the one whose mailbox produced them
|
|
**What goes wrong:** `Tender` has deliberately no `tenantId` column (D-03, Phase 10: "Tender ist plattform-global"). Every prior source (DÖE, NetServer, cosinex) is a public portal anyway, so global visibility was never in tension with the data's origin. An email-alert record, by contrast, is discovered via **one specific tenant's private mailbox subscription** — under the current architecture it will still be inserted as a global `Tender` row, visible to every other tenant that has the tender-radar module activated, even though they never configured that mailbox.
|
|
**Why it happens:** This is an intentional consequence of D-03 (locked in Phase 10) combined with D-06 (email config is per-tenant) — CONTEXT.md for this phase does not revisit or discuss this interaction explicitly.
|
|
**How to avoid:** This is **not** a code bug to fix silently — it is a product-policy question that should be explicitly confirmed with the user before/while planning INGEST-05, since it means one tenant's paid/subscribed alert channel effectively becomes a shared benefit for all tenants on the platform. Surface this plainly to the planner/user (see Open Questions #1) rather than assuming either "this is fine" or "this needs per-tenant scoping" — both are defensible depending on product intent, but only one matches what CONTEXT.md's silence implies.
|
|
**Warning signs:** None at the code level (it will work exactly as designed) — the risk is entirely in unstated product expectations surfacing after Tenant B starts seeing tenders Tenant A's mailbox produced.
|
|
|
|
## Code Examples
|
|
|
|
### RSS parsing with the existing `fast-xml-parser` configuration
|
|
```typescript
|
|
// Source: mirrors apps/api/src/tenders/adapters/doe-opendata.adapter.ts's XMLParser config
|
|
import { XMLParser } from 'fast-xml-parser';
|
|
|
|
const xmlParser = new XMLParser({
|
|
removeNSPrefix: true,
|
|
ignoreAttributes: false,
|
|
attributeNamePrefix: '@_',
|
|
// cdataPropName defaults to `false`, which means CDATA content is merged
|
|
// directly into the tag's text value — confirmed in fast-xml-parser v5.10.1
|
|
// source (OptionsBuilder.js `cdataPropName: false`), no extra config needed
|
|
// for RSS <title><![CDATA[...]]></title> to parse straight to a string.
|
|
});
|
|
|
|
const parsed = xmlParser.parse(rssXmlText) as {
|
|
rss?: { channel?: { item?: unknown | unknown[] } };
|
|
};
|
|
const channel = parsed.rss?.channel;
|
|
const items = Array.isArray(channel?.item) ? channel.item : channel?.item ? [channel.item] : [];
|
|
```
|
|
|
|
### Live-captured RSS shapes (this session, both fetched directly against the real endpoints)
|
|
|
|
**service.bund.de** — `https://www.service.bund.de/Content/Globals/Functions/RSSFeed/RSSGenerator_Ausschreibungen.xml` [CITED: live fetch, 2026-07-23]:
|
|
```xml
|
|
<?xml version="1.0" encoding="UTF-8"?>
|
|
<rss version="2.0">
|
|
<channel>
|
|
<title>service.bund.de - öffentliche Ausschreibungen</title>
|
|
<link>http://www.service.bund.de/ausschreibungen</link>
|
|
<description>RSS-Feed mit aktuellen Ausschreibungen der Vergabestellen...</description>
|
|
<language>de-de</language>
|
|
<pubDate>Thu, 23 Jul 2026 11:27:44 +0200</pubDate>
|
|
<ttl>60</ttl>
|
|
<item>
|
|
<title>Vhv 120_26 UFZ - Bench-scale Anaerobic Bioreactor System</title>
|
|
<link>https://www.service.bund.de/IMPORTE/Ausschreibungen/editor/Helmholtz-Zentrum-fuer-Umweltforschung-GmbH/2026/07/6585334.html#track=feed-callforbids</link>
|
|
<description><![CDATA[Erfüllungsort: <strong>04318 Leipzig</strong><br />Vergabestelle: <strong>Helmholtz-Zentrum für Umweltforschung GmbH - UFZ</strong><br />Angebotsfrist: <strong>07.08.2026 10:00</strong>...]]></description>
|
|
<pubDate>Thu, 23 Jul 2026 11:00:00 +0200</pubDate>
|
|
<guid>https://www.service.bund.de/IMPORTE/Ausschreibungen/editor/Helmholtz-Zentrum-fuer-Umweltforschung-GmbH/2026/07/6585334.html</guid>
|
|
</item>
|
|
</channel>
|
|
</rss>
|
|
```
|
|
Field mapping: `title`→title, `link`/`guid`→sourceUrl/sourceNoticeId, `pubDate`→publishedAt, `description` CDATA contains labeled HTML fields (`Erfüllungsort`, `Vergabestelle`, `Angebotsfrist`) extractable via cheerio for the optional enhancement (see Pattern 3).
|
|
|
|
**subreport-elvis** (one live municipal instance, Stadt Neuss) — `https://www.subreport-elvis.de/elvis/secure/rss.pl?id=4615` [CITED: live fetch, 2026-07-23]:
|
|
```xml
|
|
<?xml version='1.0' encoding='utf-8'?>
|
|
<rss version='2.0'>
|
|
<channel>
|
|
<title><![CDATA[subreport ELViS]]></title>
|
|
<link>https://www.subreport-elvis.de/</link>
|
|
<description><![CDATA[Aktuelle Ausschreibungen von Stadt Neuss, 41460 Neuss]]></description>
|
|
<item>
|
|
<title><![CDATA[E73433797:Objektbezogene Schadensanalysen...]]></title>
|
|
<description><![CDATA[<table>...</table>]]></description>
|
|
<link>https://www.subreport-elvis.de/browseVerdingungsunterlagen.html#ELVISID:E73433797</link>
|
|
<author><![CDATA[Stadt Neuss, vergabe@stadt.neuss.de]]></author>
|
|
<guid isPermaLink="false">E73433797-558464</guid>
|
|
</item>
|
|
</channel>
|
|
</rss>
|
|
```
|
|
**Notable difference from service.bund.de:** no `pubDate` on items (⇒ `publishedAt: null`, already nullable-tolerant everywhere downstream); `<description>` is an HTML `<table>` whose internal field structure was **not** fully characterized this session (see Open Questions #2) — do not assume the same label-based extraction as service.bund.de without a fixture capture first.
|
|
|
|
### Generic link extraction from an alert email body (D-04)
|
|
```typescript
|
|
import * as cheerio from 'cheerio';
|
|
import { createHash } from 'crypto';
|
|
|
|
const FOOTER_NOISE = ['unsubscribe', 'abmelden', 'einstellungen', 'preferences'];
|
|
const MAX_LINKS_PER_EMAIL = 5; // safety valve against link-spam producing excess records
|
|
|
|
function extractCandidateLinks(bodyHtml: string | null, bodyText: string): string[] {
|
|
let links: string[];
|
|
if (bodyHtml) {
|
|
const $ = cheerio.load(bodyHtml);
|
|
links = $('a[href]')
|
|
.map((_, el) => $(el).attr('href'))
|
|
.get()
|
|
.filter((href): href is string => typeof href === 'string' && /^https?:\/\//i.test(href));
|
|
} else {
|
|
links = Array.from(bodyText.matchAll(/https?:\/\/[^\s"'<>]+/g)).map((m) => m[0]);
|
|
}
|
|
return Array.from(new Set(links))
|
|
.filter((url) => !FOOTER_NOISE.some((noise) => url.toLowerCase().includes(noise)))
|
|
.slice(0, MAX_LINKS_PER_EMAIL);
|
|
}
|
|
|
|
function titleFromEmail(subject: string, bodyText: string): string {
|
|
if (subject.trim()) return subject.trim();
|
|
const firstLine = bodyText.split('\n').map((l) => l.trim()).find(Boolean);
|
|
return firstLine || 'Unbenannte Ausschreibung'; // matches normalizeBag()'s existing fallback
|
|
}
|
|
|
|
function sourceNoticeIdFor(link: string): string {
|
|
return createHash('sha256').update(link).digest('hex').slice(0, 40);
|
|
}
|
|
```
|
|
|
|
### Per-tenant config service (mirrors `DkvService`'s config half + `SettingsService`'s Safe-Select)
|
|
```typescript
|
|
// Source: pattern mirrors apps/api/src/dkv/dkv.service.ts §21-41, §95-125
|
|
const EMAIL_CONFIG_SAFE_SELECT = {
|
|
id: true, tenantId: true, protocol: true, host: true, port: true,
|
|
encryption: true, folder: true, senderFilter: true, domain: true,
|
|
isActive: true, createdAt: true, updatedAt: true,
|
|
// encryptedInboxCreds: NEVER included — T-07-12 pattern
|
|
} as const;
|
|
|
|
async function getConfigForApi(prisma: PrismaService, crypto: CalendarCryptoService, tenantId: string) {
|
|
const safe = await prisma.tenderEmailConfig.findUnique({ where: { tenantId }, select: EMAIL_CONFIG_SAFE_SELECT });
|
|
if (!safe) return null;
|
|
let username: string | null = null;
|
|
let hasPassword = false;
|
|
try {
|
|
const raw = await prisma.tenderEmailConfig.findUnique({ where: { tenantId } });
|
|
if (raw?.encryptedInboxCreds) {
|
|
const creds = JSON.parse(crypto.decrypt(raw.encryptedInboxCreds)) as { username?: string; password?: string };
|
|
username = creds.username ?? null;
|
|
hasPassword = Boolean(creds.password);
|
|
}
|
|
} catch { /* ignore decrypt errors — return empty username, matches DkvService.getConfigForApi */ }
|
|
return { ...safe, username, hasPassword };
|
|
}
|
|
```
|
|
|
|
## State of the Art
|
|
|
|
| Old Approach | Current Approach | When Changed | Impact |
|
|
|--------------|------------------|---------------|--------|
|
|
| ews-javascript-api (Basic Auth only) | Hand-rolled NTLM/EWS SOAP via `httpntlm` | Phase 7 (DKV build) | The new `fetchMessages()` EWS path must extend the hand-rolled SOAP builders, not reach for a library — consistent with `project_calendar_ews_fix` memory |
|
|
| `node-html-parser` (initially considered for NetServer HTML) | `cheerio` | Phase 13, Plan 13-04 (package-legitimacy false-positive on `node-html-parser`'s age check, human-approved switch to `cheerio`) | Confirms `cheerio` is the codebase's settled HTML-parsing choice — use it for email-body link extraction and optional RSS description parsing, don't reconsider `node-html-parser` |
|
|
| Hardcoded tender-radar UI strings ("MVP-stub convention," explicitly noted in `SourceConfigForm.tsx`/`settings/page.tsx` comments) | `useTranslations`-driven i18n | Phase 14 (this phase, CONFIG-03) | This phase is the point where the previously-accepted stub convention is retired for the tender-radar module specifically |
|
|
|
|
**Deprecated/outdated:** none beyond the above — this is a young, actively-developed module with no legacy cruft to remove.
|
|
|
|
## Assumptions Log
|
|
|
|
| # | Claim | Section | Risk if Wrong |
|
|
|---|-------|---------|---------------|
|
|
| A1 | subreport-elvis's `<description>` HTML `<table>` structure was not fully captured this session (only a summarized excerpt was obtained via WebFetch, not the complete raw table markup) | RSS Ingestion / Code Examples | If the planner assumes a specific field-label structure without a fresh live fixture capture, the optional enhancement (buyerName/deadlineAt extraction) could silently extract wrong/garbled data for subreport-elvis specifically. The bare-minimum bag mapping (title/link/guid only) is NOT at risk — it only needs `<item><title>`/`<link>`/`<guid>`, all confirmed. |
|
|
| A2 | The exact EWS `<t:Body>`/`BodyType` XML shape returned by a real Exchange server for the new `fetchMessages()` GetItem call was not verified live this session (based on Microsoft EWS schema documentation / training knowledge, not a live capture against this codebase's actual target mailbox) | Architecture Patterns / Pattern 2 | If the real server's response shape differs from the assumed `<t:Body BodyType="HTML">...</t:Body>` structure (e.g. additional wrapping elements, different escaping), the EWS body-fetch will silently return empty/garbled content rather than throwing — exactly the failure mode flagged in Pitfall 4. Requires a `checkpoint:human-verify` against a real mailbox. |
|
|
| A3 | Whether "email-derived Tender rows visible platform-wide" (Pitfall 5) is the intended product behavior or an unstated gap in this phase's scoping — not discussed in CONTEXT.md | Common Pitfalls / Pitfall 5 | If wrongly assumed acceptable, one tenant's private mailbox subscription becomes free platform-wide data for every other tenant — a potential customer-trust/contract issue for a soon-to-be-sold multi-tenant product. If wrongly assumed to need per-tenant scoping, unnecessary architecture work (breaking D-03) may be attempted. |
|
|
|
|
**If this table is empty:** N/A — see rows above; all three should be confirmed with the user or resolved via a `checkpoint:human-verify`/explicit discuss-phase note before the planner locks task-level detail for the affected areas.
|
|
|
|
## Open Questions (RESOLVED)
|
|
|
|
1. **Email-alert Tender visibility across tenants (see Pitfall 5 / A3)** — **RESOLVED via CONTEXT.md D-13 (User-Entscheidung 2026-07-23):** E-Mail-Tender sind nur für den konfigurierenden Mandanten sichtbar (nullable `Tender.ownerTenantId`), öffentliche Quellen inkl. RSS bleiben global. Umgesetzt in Plan 14-03.
|
|
- What we know: `Tender` is architecturally platform-global (D-03); `TenderEmailConfig` is architecturally per-tenant (D-06). CONTEXT.md doesn't reconcile the two for this specific case.
|
|
- What's unclear: Whether this is accepted product behavior (matches the existing "all discovered tenders are shared" philosophy) or an oversight.
|
|
- Recommendation: Surface explicitly to the user in `/gsd-discuss-phase` follow-up or as a plan-time `checkpoint:human-verify` before implementing INGEST-05's dedup/storage step — this is a five-minute confirmation that prevents a much larger later rework if the answer is "no, that's not okay."
|
|
|
|
2. **subreport-elvis `<description>` field structure (see A1)** — **RESOLVED via Plan 14-02 Task 1:** Baseline schickt nur die bare-minimum Bag (title/link/guid), reicht für INGEST-04; optionale Feld-Anreicherung aus der HTML-Tabelle ist nicht Teil des Baselines. Live-Fixture-Capture bei Bedarf zur Ausführungszeit.
|
|
- What we know: It's an HTML `<table>`; the outer feed/item shape (title/link/guid/author) is fully confirmed live.
|
|
- What's unclear: The internal table structure (column labels, whether "Angebotsfrist"-equivalent data exists at all for subreport-elvis, unlike service.bund.de where it's confirmed present in the description).
|
|
- Recommendation: Capture a fresh, complete raw XML fixture from a live subreport-elvis feed at Wave-0/plan-time (mirroring Phase 13's live-fixture-capture convention for NetServer/cosinex HTML) before deciding whether to attempt the optional field-enrichment for this source, or to ship it with the bare-minimum bag only (title/link/guid — already fully sufficient for INGEST-04's baseline requirement).
|
|
|
|
3. **Default seeded RSS feeds**
|
|
- **RESOLVED via Plan 14-02 Task 2:** service.bund.de wird active-by-default geseedet, die Feed-Liste sonst leer für admin-getriebene Ergänzungen (matcht die „framework ready, activation deferred"-Haltung).
|
|
- What we know: service.bund.de's feed is genuinely national/global and safe to seed by default; subreport-elvis has no such single URL.
|
|
- What's unclear: Whether to seed zero rows and require an admin action before any RSS ingestion happens, or seed the service.bund.de URL as an always-on default.
|
|
- Recommendation: Seed service.bund.de active-by-default (mirrors DÖE's `isActive: true` seed from Phase 10), leave the RSS feed list otherwise empty for admin-driven additions — matches the "framework ready, activation deferred" stance already established for ai-netserver/cosinex-dtvp in Phase 13.
|
|
|
|
## Environment Availability
|
|
|
|
| Dependency | Required By | Available | Version | Fallback |
|
|
|------------|------------|-----------|---------|----------|
|
|
| `fast-xml-parser` | RSS parsing (INGEST-04) | ✓ | ^5.10.1 (installed) | — |
|
|
| `imapflow` | Email fetchMessages (INGEST-05) | ✓ | ^1.4.3 (installed) | — |
|
|
| `httpntlm` | EWS fetchMessages (INGEST-05) | ✓ | ^1.8.13 (installed) | — |
|
|
| `cheerio` | Link extraction / optional RSS enrichment | ✓ | ^1.2.0 (installed) | — |
|
|
| Live subreport-elvis test feed | RSS adapter fixture capture (Wave 0) | ✓ (public, one instance confirmed reachable) | — | If a specific tenant's real feed is unavailable at plan/build time, use the Stadt Neuss instance captured this session as a placeholder fixture, clearly marked as a third-party example, not a Tessera-owned endpoint |
|
|
| Live Exchange/EWS mailbox for `fetchMessages()` verification | Pitfall 4 human-verify | ✗ (not available in this research session — no live mailbox credentials) | — | Build against mocked SOAP responses (fixture-based unit tests); require a `checkpoint:human-verify` against a real mailbox before considering the Exchange email-alert path production-ready |
|
|
|
|
**Missing dependencies with no fallback:** none blocking — the one genuinely missing environment piece (a live Exchange mailbox to verify EWS body-fetch against) has an accepted fallback (mock-based tests + human-verify checkpoint).
|
|
|
|
**Missing dependencies with fallback:** live Exchange/EWS mailbox (see above).
|
|
|
|
## Validation Architecture
|
|
|
|
### Test Framework
|
|
|
|
| Property | Value |
|
|
|----------|-------|
|
|
| Framework | Vitest (apps/api: node environment; apps/web: jsdom environment) |
|
|
| Config file | `apps/api/vitest.config.ts` (`include: ['src/**/*.spec.ts']`), `apps/web/vitest.config.ts` |
|
|
| Quick run command | `cd apps/api && npx vitest run src/tenders/adapters/rss.adapter.spec.ts` (swap path per file) |
|
|
| Full suite command | `cd apps/api && npx vitest run` / `cd apps/web && npx vitest run` |
|
|
|
|
### Phase Requirements → Test Map
|
|
|
|
| Req ID | Behavior | Test Type | Automated Command | File Exists? |
|
|
|--------|----------|-----------|-------------------|-------------|
|
|
| INGEST-04 | RSS feed fetch+parse maps real feed XML to `RawTenderRecord[]` correctly (both confirmed shapes: service.bund.de with pubDate, subreport-elvis without) | unit (fixture-based, mirrors `cosinex.adapter.spec.ts`) | `npx vitest run src/tenders/adapters/rss.adapter.spec.ts` | ❌ Wave 0 — new file, fixtures need live capture (see Environment Availability) |
|
|
| INGEST-04 | Extended normalizer dispatch for `'rss'` sourceType (should reuse existing `normalizeBag()` — extend the `switch` in `tender-normalizer.service.ts` and its spec) | unit | `npx vitest run src/tenders/tender-normalizer.service.spec.ts` | ✅ file exists, needs 1-2 new test cases added |
|
|
| INGEST-05 | `fetchMessages()` on `ImapProvider` extracts html/text body from a mocked `ImapFlow` MIME tree | unit (mocked, no existing coverage to extend — net new) | `npx vitest run src/inbox/imap.provider.spec.ts` | ❌ Wave 0 — **no `.spec.ts` exists for either inbox provider today** (confirmed via directory search) |
|
|
| INGEST-05 | `fetchMessages()` on `ExchangeInboxProvider` extracts body from a mocked `httpntlm.post` SOAP response | unit (mocked) | `npx vitest run src/inbox/exchange-inbox.provider.spec.ts` | ❌ Wave 0 — new file |
|
|
| INGEST-05 | Generic link+subject extraction (`extractCandidateLinks`/`titleFromEmail`) — pure functions, HTML and plaintext fallback both covered | unit (pure function, no I/O) | `npx vitest run src/tenders/adapters/email-alert.adapter.spec.ts` | ❌ Wave 0 — new file |
|
|
| INGEST-05 | Per-tenant fan-out: one broken tenant mailbox doesn't block others (catch-per-tenant) | unit (mocked `prisma.tenderEmailConfig.findMany` + mocked provider) | `npx vitest run src/tenders/adapters/email-alert.adapter.spec.ts` | ❌ same file as above |
|
|
| INGEST-05 | DKV regression: moving `Imap`/`ExchangeInboxProvider` into `inbox/` does not change DKV's existing `fetchPdfAttachments` behavior | manual/full-suite | `cd apps/api && npx vitest run` (full suite green) + `npx tsc --noEmit` | ✅ existing DKV spec files (0 today) mean this is really "no regression in the full suite," not a dedicated regression test — see Wave 0 Gaps |
|
|
| CONFIG-02 | Safe-Select excludes `encryptedInboxCreds`; encrypt/decrypt round-trips correctly | unit | `npx vitest run src/tenders/tender-email-config.service.spec.ts` | ❌ Wave 0 — new file, mirrors absent-but-should-exist `dkv.service.spec.ts` pattern |
|
|
| CONFIG-02 | `GET/PUT /modules/tender-radar/email-config` and `/rss-feeds` are `@Roles(ADMIN,SUPER_ADMIN)`-guarded | integration (controller test) | `npx vitest run src/tenders/tenders.controller.spec.ts` | ✅ file exists, needs new route test cases added |
|
|
| CONFIG-02 | Runtime host-based denylist check rejects a vergabe24/aumass RSS feed URL at save time (Pitfall 3) | unit | `npx vitest run src/tenders/tender-rss-feed.service.spec.ts` | ❌ Wave 0 — new file |
|
|
| CONFIG-03 | `de.json`/`en.json` have identical key sets under the new `tenderRadar` namespace (no missing EN/DE translation) | unit (web side, pure JSON structural comparison) | `cd apps/web && npx vitest run src/messages/tenderRadar-parity.spec.ts` | ❌ Wave 0 — no existing i18n key-parity test in the codebase at all; recommend adding one (cheap, catches the exact class of bug fixed by quick task 260701-abc) |
|
|
| UI-06 | `CoverageBanner` renders the denylist "manually monitor" block with correct vergabe24/aumass links | component test | `cd apps/web && npx vitest run src/app/\(portal\)/modules/tender-radar/components/CoverageBanner.test.tsx` | ❌ Wave 0 — no test file exists for `CoverageBanner` today |
|
|
| UI-06 | New denylist read endpoint returns the two portals with correct URLs | integration | `npx vitest run src/tenders/tenders.controller.spec.ts` | ✅ extend existing file |
|
|
|
|
### Sampling Rate
|
|
- **Per task commit:** run the single most relevant spec file from the table above (`npx vitest run <file>`)
|
|
- **Per wave merge:** `cd apps/api && npx vitest run && npx tsc --noEmit -p tsconfig.json`, then `cd apps/web && npx vitest run`
|
|
- **Phase gate:** both full suites green + `tsc --noEmit` clean before `/gsd-verify-work`, matching Phase 13's verification convention
|
|
|
|
### Wave 0 Gaps
|
|
- [ ] `apps/api/src/tenders/__fixtures__/service-bund-feed.xml` — live-captured, full (not summarized) raw XML; the excerpt in this RESEARCH.md is representative but should be re-captured in full at plan/build time for the actual fixture file
|
|
- [ ] `apps/api/src/tenders/__fixtures__/subreport-elvis-feed.xml` — live-captured, full raw XML including the complete `<description>` `<table>` markup (currently only a summarized excerpt exists — see Open Questions #2)
|
|
- [ ] `apps/api/src/inbox/imap.provider.spec.ts` + `exchange-inbox.provider.spec.ts` — genuinely net-new test infrastructure; no prior DKV provider tests exist to extend
|
|
- [ ] `apps/web/src/messages/` key-parity check — no existing test enforces `de.json`/`en.json` structural parity; recommend adding one as part of this phase given the module now doubles the message-file surface area
|
|
- [ ] Prisma migration for `TenderEmailConfig` (per-tenant) and `TenderRssFeedSource` (global) — neither model exists yet
|
|
|
|
## Security Domain
|
|
|
|
### Applicable ASVS Categories
|
|
|
|
| ASVS Category | Applies | Standard Control |
|
|
|---------------|---------|-----------------|
|
|
| V2 Authentication | no (new routes reuse existing session auth, no new auth mechanism) | — |
|
|
| V3 Session Management | no | — |
|
|
| V4 Access Control | yes | `@Roles(Role.ADMIN, Role.SUPER_ADMIN)` on all new config routes (email-config, rss-feeds), mirroring `DkvController`/existing `source-config` route |
|
|
| V5 Input Validation | yes | `class-validator` DTOs for the new `TenderEmailConfigDto`/`TenderRssFeedSourceDto` (`@IsIn` for protocol/encryption, `@IsUrl` for feed URLs, `@IsEmail` for senderFilter — mirrors `DkvConfigDto`); host-based denylist validation on RSS feed URL save (Pitfall 3); email body HTML must never be rendered raw in any UI — only extracted plain-text title/links are ever stored/displayed |
|
|
| V6 Cryptography | yes | `CalendarCryptoService` (AES-256-GCM) — never a new/different crypto implementation for the new mailbox credentials |
|
|
|
|
### Known Threat Patterns for this stack
|
|
|
|
| Pattern | STRIDE | Standard Mitigation |
|
|
|---------|--------|---------------------|
|
|
| SSRF via admin-supplied RSS feed URL (admin could enter an internal/private-network URL) | Tampering / Information Disclosure | Validate feed URL scheme is `https?://` and reject private/loopback IP ranges before fetching, mirroring the SSRF-guard intent already documented for `NETSERVER_PORTALS` (hardcoded there; here it must be a runtime check since the URL is admin input) — new for this phase since it's the first admin-input-driven fetch target in `tenders/` |
|
|
| Email-injection / stored-XSS via alert email body (HTML) | Tampering | Only extract plain-text `title` and raw `href` URL strings — never store or render the HTML body itself; mirrors the existing `.text()`-not-`.html()` discipline already documented in `NetServerAdapter` (V5) |
|
|
| Credential leakage via logs (IMAP/EWS) | Information Disclosure | Reuse the existing `logger: false` (imapflow) / generic-error-message (EWS) conventions for any new log statement in `fetchMessages()` — do not introduce a new logging call that includes `config.password`/`config.username` |
|
|
| Decompression/parsing DoS via a malicious/huge RSS feed response | Denial of Service | Apply the same `AbortController` 15s timeout pattern already used by `DoeOpenDataAdapter`/`NetServerAdapter`; consider a response-size ceiling for RSS fetches (RSS feeds are normally small, but an admin-supplied URL is less trusted than a hardcoded one) |
|
|
|
|
## Sources
|
|
|
|
### Primary (HIGH confidence)
|
|
- Direct codebase reads (this session): `inbox-provider.interface.ts`, `imap.provider.ts`, `exchange-inbox.provider.ts`, `dkv.types.ts`, `dkv.service.ts`, `dkv.module.ts`, `crypto.service.ts`, `dkv-config.dto.ts`, `dkv.controller.ts`, `tender-source-adapter.interface.ts`, `source-registry.ts`, `tenders.module.ts`, `tender.types.ts`, `tender-normalizer.service.ts`, `tender-ingestion.service.ts`, `tender-scheduler.service.ts`, `netserver.adapter.ts`, `doe-opendata.adapter.ts`, `tender-mail.service.ts`, `settings.service.ts`, `prisma-tenant.extension.ts`, `tenders.controller.ts`, `SourceConfigForm.tsx`, `InboxConfigForm.tsx`, `tender-radar-api.ts`, `CoverageBanner.tsx`, `layout.tsx`, `i18n/request.ts`, `middleware.ts`, `schema.prisma` (DkvModuleConfig/TenderSourcePollConfig/Tender models), `apps/api/package.json`, `apps/web/src/messages/de.json`/`en.json`
|
|
- `gsd-tools query package-legitimacy check` — `rss-parser` verdict OK (825k/wk, github.com/bobby-brennan/rss-parser, not deprecated, no postinstall)
|
|
- fast-xml-parser v5.10.1 source (`OptionsBuilder.js`, `fxp.d.ts`) — confirmed `cdataPropName` default behavior
|
|
|
|
### Secondary (MEDIUM confidence)
|
|
- Live WebFetch of `https://www.service.bund.de/Content/Globals/Functions/RSSFeed/RSSGenerator_Ausschreibungen.xml` (2026-07-23) — full channel/item XML shape
|
|
- Live WebFetch of `https://www.subreport-elvis.de/elvis/secure/rss.pl?id=4615` (2026-07-23) — channel/item XML shape (description table content not fully captured, see A1)
|
|
- WebSearch confirming subreport-elvis's per-municipality feed model (Neuss city page) and vergabe24.de/aumass.de canonical URLs
|
|
|
|
### Tertiary (LOW confidence)
|
|
- EWS `<t:Body BodyType="...">` response shape for the new `fetchMessages()` GetItem call — based on general EWS schema knowledge, not verified against a live Exchange server this session (A2)
|
|
|
|
## Metadata
|
|
|
|
**Confidence breakdown:**
|
|
- Standard stack: HIGH — zero new dependencies, all four libraries already proven in this exact codebase
|
|
- Architecture: HIGH for the adapter/config/scheduler patterns (direct precedent in Phase 13); MEDIUM for the exact EWS body-fetch shape (A2)
|
|
- Pitfalls: HIGH for #1/#3/#5 (derived from direct code reading, not speculation); MEDIUM for #2/#4 (well-grounded but partially forward-looking)
|
|
|
|
**Research date:** 2026-07-23
|
|
**Valid until:** 30 days for code-pattern findings (stable, in-repo); the two live-fetched RSS feed snapshots should be re-verified at plan/build time regardless of age, since external feed structures can change without notice
|