66 KiB
Phase 14: RSS, Email-Alert Ingestion & Module Rollout - Research
Researched: 2026-07-23 Domain: NestJS backend ingestion adapters (RSS/XML, IMAP/EWS email), per-tenant encrypted config, Next.js i18n (next-intl) Confidence: MEDIUM (HIGH for code-pattern reuse, MEDIUM for live-feed shapes captured this session, LOW/flagged where a concrete decision is still open)
<user_constraints>
User Constraints (from CONTEXT.md)
Locked Decisions
Inbox-Modul-Extraktion (INGEST-05, Refactor)
- D-01: Additiver Refactor, DKV bleibt verhaltensgleich. Die bestehenden IMAP-/Exchange-Provider (
ImapProvider,ExchangeInboxProvider) und dasInboxProvider-Interface werden ausapps/api/src/dkv/providers/in ein geteiltesapps/api/src/inbox/-Modul gehoben. Geteilt wird ausschließlich die Verbindungsmechanik (IMAP/Exchange-Anmeldung, TLS, NTLM via httpntlm, Timeouts, Mail-Abruf). - D-02: Die bestehende PDF-Anhang-Methode (
fetchPdfAttachments) bleibt unverändert erhalten. Für die Ausschreibungs-Ingestion wird eine zusätzliche Methode ergänzt, die den Mail-Body (HTML/Text) und ganze Nachrichten liefert. DKV wird NICHT auf die neue Methode umgestellt — nur der Import-Pfad wechselt aufinbox/. - D-03: Jedes Modul behält seine eigene, unabhängige Postfach-Konfiguration (eigene Adresse, eigene Credentials). Geteilt ist nur der Code, nicht die Config.
E-Mail-Alert-Parsing (INGEST-05)
- D-04: Generische Auswertung statt portalspezifischer Parser. Aus jeder Alert-Mail werden die enthaltenen Links (auf die Ausschreibungs-Detailseite) plus Betreff/erste Textzeile als Titel extrahiert. Feldarmut (kein CPV/Buyer/Frist) wird bewusst und offen dokumentiert.
- D-05: Konsequenz für Cross-Source-Dedup: E-Mail-/RSS-Records erben die Fingerprint-Asymmetrie aus Phase 13 (residual_gap_decision = Option C, dormant). Kein neuer Dedup-Mechanismus in Phase 14.
Config-Modell + Admin-UI (CONFIG-02, CONFIG-03)
- D-06: Postfach-Konfiguration ist pro Mandant (tenantId-scoped), Muster wie
DkvModuleConfig. Neues Prisma-Modell, kein Ausbau des globalenTenderSourcePollConfig-Singletons. - D-07: Zugangsdaten werden mit
CalendarCryptoService(AES-256-GCM) verschlüsselt. Verschlüsselte Felder werden aus GET-Responses ausgeschlossen (Safe-Select-Muster, T-07-12). - D-08: RSS-Feeds sind öffentlich und für alle gleich → globale Plattform-Konfiguration (bestehendes Singleton-Muster), nicht pro Mandant.
- D-09: Die bestehende Admin-Seite (
apps/web/.../tender-radar/settings/) wird erweitert, nicht neu gebaut: ein Abschnitt „E-Mail-Alerts" (pro Mandant) und einer „RSS-Feeds" (global). Rollen-Trennung über bestehendes@Roles(ADMIN, SUPER_ADMIN).
i18n (UI-06)
- D-10: Neuer
tenderRadar-Namensraum inde.json/en.json; alle hartcodierten deutschen Strings der tender-radar-Komponenten inuseTranslations-Keys überführt. - D-11: KEIN sichtbarer Sprachumschalter. Die Sprachwahl folgt dem bestehenden next-intl-Mechanismus.
Denylist-Transparenz (UI-06)
- D-12: vergabe24 und aumass werden im UI als „manuell beobachten" mit Direktlink dargestellt.
CoverageBanner.tsxist der natürliche Ort. Denylist-Quelle bleibt der code-level GateDENYLISTED_PORTALS(source-registry.ts:12).
Claude's Discretion
- Wahl der RSS-Parsing-Bibliothek und das genaue Feld-Mapping der RSS-Items auf
NormalizedTenderFields. - Genaue Struktur des neuen
inbox/-Modul-Interfaces und der zusätzlichen Message-Methode. - Scheduler-Anbindung der neuen Quellen (RSS-Adapter, E-Mail-Adapter) über das bestehende
SourceRegistry+pollDueSources-Muster. - EN-Übersetzungen erstellt Claude selbst (Fachbegriffe der Vergabe-Domäne konsistent halten).
Deferred Ideas (OUT OF SCOPE)
- App-weiter Sprachumschalter — eigene Plattform-Phase, nicht Teil von Phase 14.
- Portalspezifische E-Mail-Parser — erst sinnvoll, wenn konkrete Alert-Portale/echte Mail-Samples bekannt sind.
- Cross-Source-Fingerprint für dünne Quellen — erst bei Live-Aktivierung einer zweiten Quelle. </user_constraints>
<phase_requirements>
Phase Requirements
| ID | Description | Research Support |
|---|---|---|
| INGEST-04 | System importiert Ausschreibungen aus RSS-Feeds (subreport-elvis, service.bund.de) | Live feed shapes captured this session (service.bund.de + a subreport-elvis instance); fast-xml-parser (already installed) recommended over adding rss-parser; adapter design mirrors NetServerAdapter's multi-portal-in-one-adapter pattern; RSS feed URLs recommended as an admin-CRUD list (new global model), not hardcoded — see "RSS Ingestion" below |
| INGEST-05 | System liest Portal-Benachrichtigungs-E-Mails aus einem konfigurierten Postfach und extrahiert Ausschreibungen (nutzt bestehende DKV-Inbox-Infrastruktur) | Concrete file-move list, new fetchMessages() method design (IMAP via imapflow body-part download, EWS via t:Body FieldURI addition), generic link+subject extraction algorithm, per-tenant fan-out design that reuses the poll-once-fan-out-many pattern inside a single adapter (avoids touching TenderIngestionService) — see "Inbox Module Extraction" and "Email-Alert Parsing" below |
| CONFIG-02 | Admin verwaltet Quellen-Poll-Konfiguration sowie das E-Mail-Ingestion-Postfach pro Mandant (Zugangsdaten verschlüsselt) | Exact Prisma model shape (mirrors DkvModuleConfig), Safe-Select pattern, CalendarCryptoService injection path, new GET/PUT /modules/tender-radar/email-config route pattern (mirrors DkvController) — see "Per-Tenant Config + Encryption" below |
| CONFIG-03 | Modul-UI ist mehrsprachig (DE/EN) | Confirmed next-intl locale mechanism (cookie NEXT_LOCALE, default de, no code change needed for D-11); concrete list of 8 files/~90 string-sites requiring conversion; namespace shape and key-parity test recommendation — see "i18n" below |
| UI-06 | Ausgeschlossene Portale werden als „manuell zu überwachen" mit Direktlink angezeigt | Confirmed real URLs (vergabe24.de, aumass.de); confirmed DENYLISTED_PORTALS has no API exposure today; recommended new small backend endpoint sourced from the same constant — see "Denylist UI Transparency" below |
| </phase_requirements> |
Summary
Phase 14 is almost entirely pattern replication, not new architecture: every one of the five requirements has a directly analogous precedent already built and tested in this codebase (NetServer/cosinex adapters for scraping-into-bag ingestion; DKV for encrypted per-tenant mailbox config; the existing dkvFleet i18n namespace for the message-file shape). The main net-new engineering is (1) a fetchMessages() method on the inbox providers that doesn't exist yet, and (2) a per-tenant fan-out happening inside a single new adapter rather than at the TenderSourcePollConfig/TenderIngestionService level — because Tender and TenderSourcePollConfig are both structurally singleton/global (D-03 from Phase 10), and re-opening that architecture is explicitly out of scope.
Two real, previously-undocumented technical findings surfaced during this research and materially affect planning:
- Day-cursor throttling would silently cap RSS/email polling to once per calendar day, regardless of the admin-configured
pollIntervalMin, if the new sources are plugged intoTenderIngestionService.pollDueSources()unmodified — the day-cursor gate (nextDayToFetch) was built for DÖE's daily batch export and is reused as-is by NetServer/cosinex today. This needs an explicit decision (see Common Pitfalls #1). - subreport-elvis has no single global feed URL — each municipality/Vergabestelle runs its own feed instance (e.g.
subreport-elvis.de/elvis/secure/rss.pl?id=4615for Stadt Neuss). D-08 calls RSS "global, same for all," which is true for service.bund.de but not for subreport-elvis. The admin-CRUD "RSS-Feeds" list already implied by D-09 resolves this cleanly (admin adds whichever feed URLs are relevant) — this is a design confirmation, not a re-litigation of D-08.
Primary recommendation: Reuse fast-xml-parser (already a dependency, already configured with removeNSPrefix/CDATA-merge semantics in doe-opendata.adapter.ts) for RSS/Atom parsing instead of adding rss-parser as a new dependency. Build both new adapters (RssAdapter, EmailAlertAdapter) to route through the existing generic normalizeBag() normalizer path unchanged. Extract ImapProvider/ExchangeInboxProvider/InboxProvider into apps/api/src/inbox/ with only import-path changes in dkv.service.ts/dkv.module.ts, and add one new interface method (fetchMessages) implemented in both providers without touching fetchPdfAttachments.
Architectural Responsibility Map
| Capability | Primary Tier | Secondary Tier | Rationale |
|---|---|---|---|
| RSS feed fetch + XML parse | API / Backend | — | Server-side scheduled job (TenderSourceAdapter), no browser involvement, mirrors DoeOpenDataAdapter/NetServerAdapter |
| RSS feed URL admin CRUD | API / Backend | Frontend Server (SSR form) | New global Prisma model + @Roles(ADMIN,SUPER_ADMIN) routes; frontend is a thin form, same tier split as existing SourceConfigForm |
| Email mailbox connect + fetch | API / Backend | — | IMAP/EWS network I/O must happen server-side (credentials, TLS/NTLM); mirrors ImapProvider/ExchangeInboxProvider |
| Email mailbox config CRUD (per tenant) | API / Backend | Frontend Server (SSR form) | Mirrors DkvModuleConfig + DkvController/InboxConfigForm split exactly |
| Credential encryption | API / Backend | Database (encrypted-at-rest column) | CalendarCryptoService, never touches browser or CDN tier |
| Denylist portal transparency | Frontend Server (SSR/client component) | API / Backend (data source) | Static-ish informational content; backend only needs to expose the denylist list, no new business logic |
| i18n message resolution | Frontend Server (SSR, next-intl getRequestConfig) |
Browser (client components read via useTranslations) |
Locale already resolved server-side from a cookie; client components consume via the existing NextIntlClientProvider |
| Cross-source dedup for new sources | API / Backend | Database (fingerprint column) | Explicitly not touched this phase (D-05) — inherits the dormant Phase-13 mechanism unchanged |
Standard Stack
Core
| Library | Version (installed) | Purpose | Why Standard (for this codebase) |
|---|---|---|---|
fast-xml-parser |
^5.10.1 [VERIFIED: already in apps/api/package.json, used live in doe-opendata.adapter.ts] |
Parse RSS 2.0 XML into JS objects | Zero new dependency; already configured for removeNSPrefix/attribute handling; RSS 2.0 is plain XML, no need for a dedicated RSS-specific parser |
imapflow |
^1.4.3 [VERIFIED: already in apps/api/package.json, used in imap.provider.ts] |
IMAP protocol client | Already the codebase's IMAP client; the new fetchMessages() method is an additive use of the same client |
httpntlm |
^1.8.13 [VERIFIED: already in apps/api/package.json, used in exchange-inbox.provider.ts] |
NTLM-authenticated HTTP for EWS | Already the codebase's EWS/NTLM transport; reused as-is for the new EWS body-fetch SOAP call |
cheerio |
^1.2.0 [VERIFIED: already in apps/api/package.json, used in netserver.adapter.ts] |
HTML parsing (link extraction from HTML email bodies; optional RSS <description> sub-parsing) |
Already the codebase's HTML-parsing tool (chosen over node-html-parser in Phase 13 after a package-legitimacy false flag — see 13-04 decision log); reused for extracting <a href> links from HTML email bodies and, optionally, structured sub-fields from RSS <description> HTML fragments |
Supporting
| Library | Version | Purpose | When to Use |
|---|---|---|---|
class-validator |
(already a dep, used throughout DTOs) | Validate new TenderEmailConfigDto/TenderRssFeedSourceDto |
Same @IsIn/@IsInt/@Min/@Max/@IsUrl pattern as DkvConfigDto/SourceConfigDto |
Alternatives Considered
| Instead of | Could Use | Tradeoff |
|---|---|---|
fast-xml-parser (reuse) |
rss-parser |
rss-parser [OK verdict, see Package Legitimacy Audit] is purpose-built (handles more RSS/Atom edge cases, e.g. <atom:link>, out of the box) but adds a new dependency for a format (RSS 2.0) that is plain XML the codebase already parses successfully elsewhere. Only worth it if Atom-format feeds (not confirmed for either target source) turn out to need handling. |
| Generic bag + regex link extraction (email) | A dedicated HTML-email-parsing library (e.g. mailparser's simpleParser output, or linkifyjs) |
Not needed: cheerio (already installed) extracts <a href> from HTML bodies directly; a plain regex covers the plaintext-body fallback. Adding a new library for this narrow, already-covered need is unjustified. |
| Admin-CRUD list of RSS feed URLs (recommended) | Hardcoded feed URLs mirroring NETSERVER_PORTALS |
Rejected because subreport-elvis has no single URL to hardcode (see Summary finding #2) — an admin-managed list is the only way to support "subreport-elvis" as a named source without picking one arbitrary municipality's feed at code-time. |
Installation:
# No new packages required — fast-xml-parser, imapflow, httpntlm, cheerio,
# and class-validator are all already dependencies of apps/api.
Version verification: All four libraries above are already installed and in live production use elsewhere in this codebase (confirmed via grep on apps/api/package.json and direct file reads of doe-opendata.adapter.ts, imap.provider.ts, exchange-inbox.provider.ts, netserver.adapter.ts). No npm view was needed since no new package is proposed; rss-parser (the one alternative considered) was checked via the package-legitimacy tool for completeness — see Package Legitimacy Audit.
Package Legitimacy Audit
No new external packages are proposed for this phase — all required libraries (fast-xml-parser, imapflow, httpntlm, cheerio, class-validator) are already installed dependencies of apps/api, confirmed via direct inspection of apps/api/package.json and their live usage in existing adapters/providers.
One alternative package was checked for completeness (considered and rejected — see Alternatives Considered):
| Package | Registry | Age | Downloads | Source Repo | Verdict | Disposition |
|---|---|---|---|---|---|---|
rss-parser |
npm | published 2023-04-11 (~3 yrs) | ~825k/week | github.com/bobby-brennan/rss-parser | OK | Not adopted — fast-xml-parser (already installed) covers this phase's need with zero new dependency |
Packages removed due to [SLOP] verdict: none Packages flagged as suspicious [SUS]: none
Architecture Patterns
System Architecture Diagram
┌─────────────────────────────────────────────────────────────────────┐
│ TenderSchedulerService (cron) │
│ single global tick — no tenant dimension (unchanged) │
└───────────────────────────────┬───────────────────────────────────────┘
│ tick
▼
┌─────────────────────────────────────────────────────────────────────┐
│ TenderIngestionService.pollDueSources() │
│ findMany(TenderSourcePollConfig where isActive) → fan-out per row │
└───┬─────────────┬─────────────┬─────────────┬─────────────┬─────────┘
│ doe-opendata│ ai-netserver│ cosinex-dtvp│ rss (NEW) │ email-alert (NEW)
▼ ▼ ▼ ▼ ▼
┌────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌───────────────────┐
│DoeAdap-│ │NetServer │ │Cosinex │ │RssAdapter│ │EmailAlertAdapter │
│ter │ │Adapter │ │Adapter │ │(NEW) │ │(NEW) │
└────────┘ └──────────┘ └──────────┘ └────┬─────┘ └────────┬───────────┘
│ │ internal fan-out:
│ fetch each │ findMany(TenderEmailConfig
│ admin-added │ where isActive) — ONE tenant
│ feed URL │ mailbox per row, catch-per-tenant
▼ ▼
┌──────────────┐ ┌──────────────────────┐
│TenderRssFeed │ │inbox/ module │
│Source (NEW, │ │ ImapProvider / │
│global, admin │ │ ExchangeInboxProvider│
│CRUD list) │ │ .fetchMessages(cfg) │
└──────────────┘ │ → decrypt creds via │
│ CalendarCryptoService│
│ from TenderEmailConfig│
│ (per-tenant, NEW) │
└───────────┬───────────┘
│ subject + body
▼
┌──────────────────────┐
│ generic link+subject │
│ extraction (cheerio/ │
│ regex) → RawTender- │
│ Record per link │
└───────────┬───────────┘
└──────────────────────────┬──────────────────────────────────────┘
▼
TenderNormalizerService.normalize()
(dispatches 'rss'/'email-alert' → normalizeBag(),
UNCHANGED from Phase 13's NetServer/cosinex path)
│
▼
TenderDedupService.resolve() → Tender (global, D-03)
│
▼
TendersController (GET/PUT /modules/tender-radar/*)
│
▼
Next.js: ResultsList / TenderDetail / CoverageBanner (+ denylist block)
all converted to useTranslations('tenderRadar') (D-10)
Recommended Project Structure
apps/api/src/
├── inbox/ # NEW — extracted shared module (D-01/D-02/D-03)
│ ├── inbox.module.ts # provides+exports ImapProvider, ExchangeInboxProvider
│ ├── inbox.types.ts # InboxConfig/InboxAttachment/InboxEmail/InboxMessage (moved+extended)
│ ├── inbox-provider.interface.ts # moved from dkv/providers/, +fetchMessages()
│ ├── imap.provider.ts # moved from dkv/providers/, +fetchMessages()
│ └── exchange-inbox.provider.ts # moved from dkv/providers/, +fetchMessages()
├── dkv/
│ ├── dkv.service.ts # import path only: '../inbox/...' instead of './providers/...'
│ ├── dkv.module.ts # imports InboxModule instead of declaring the 2 providers directly
│ ├── dkv.types.ts # re-exports InboxConfig/InboxAttachment/InboxEmail from '../inbox/inbox.types'
│ └── providers/ # now empty of Inbox* files — directory removed if nothing else lives there
└── tenders/
├── adapters/
│ ├── rss.adapter.ts # NEW
│ ├── rss.adapter.spec.ts # NEW (fixture-based, mirrors cosinex.adapter.spec.ts)
│ ├── email-alert.adapter.ts # NEW
│ └── email-alert.adapter.spec.ts # NEW (mocked InboxProvider)
├── __fixtures__/
│ ├── service-bund-feed.xml # NEW — live-captured fixture
│ └── subreport-elvis-feed.xml # NEW — live-captured fixture (see Open Questions)
├── dto/
│ ├── tender-email-config.dto.ts # NEW
│ └── tender-rss-feed.dto.ts # NEW
├── tender-email-config.service.ts # NEW — mirrors DkvService's config half
└── tender.types.ts # SourceType extended: + 'rss' | 'email-alert'
apps/web/src/app/(portal)/modules/tender-radar/
├── settings/components/
│ ├── EmailAlertConfigForm.tsx # NEW — mirrors InboxConfigForm.tsx (per-tenant, password field)
│ └── RssFeedListForm.tsx # NEW — admin CRUD list (add/remove feed URL + label)
└── components/CoverageBanner.tsx # extended with a denylist block (UI-06)
apps/web/src/messages/
├── de.json # + "tenderRadar": { ... } namespace
└── en.json # + "tenderRadar": { ... } namespace (EN translations)
Pattern 1: Multi-source adapter with internal fan-out (RSS + per-tenant email)
What: A single TenderSourceAdapter implementation internally iterates over multiple concrete targets (multiple RSS feed URLs, or multiple tenants' mailboxes) rather than the fan-out happening at the TenderSourcePollConfig/SourceRegistry level.
When to use: When the natural "one row per pollable thing" doesn't fit the existing singleton/global TenderSourcePollConfig (one row per sourceType) — exactly the situation for both RSS (multiple feed URLs under one sourceType: 'rss') and email-alert (multiple tenant mailboxes under one sourceType: 'email-alert').
Example (already proven in this codebase, netserver.adapter.ts:75-94):
// Source: apps/api/src/tenders/adapters/netserver.adapter.ts (existing, Phase 13)
async fetchTenders(_dayCursor: string): Promise<RawTenderRecord[]> {
const records: RawTenderRecord[] = [];
for (const portal of this.portals) {
try {
const html = await this.fetchPortalHtml(NETSERVER_PORTALS[portal].baseUrl);
records.push(...this.parseSearchResults(html, portal, new Date()));
} catch (error) {
this.logger.warn(`NetServer portal '${portal}' fetch failed, skipping: ${(error as Error).message}`);
// catch-per-target — one broken target never aborts the others (D-01 from Phase 13)
}
}
return records;
}
Recommended EmailAlertAdapter.fetchTenders() shape (new, mirrors the above exactly, substituting "portal" with "tenant mailbox config row"):
async fetchTenders(_dayCursor: string): Promise<RawTenderRecord[]> {
const configs = await this.prisma.tenderEmailConfig.findMany({ where: { isActive: true } });
const records: RawTenderRecord[] = [];
for (const cfg of configs) {
try {
const inboxConfig = await this.resolveDecryptedConfig(cfg); // CalendarCryptoService.decrypt
const provider = cfg.protocol === 'exchange' ? this.exchangeProvider : this.imapProvider;
const messages = await provider.fetchMessages(inboxConfig);
records.push(...this.extractCandidates(messages, cfg.tenantId));
} catch (error) {
this.logger.warn(`Email-alert mailbox for tenant ${cfg.tenantId} failed, skipping: ${(error as Error).message}`);
// catch-per-tenant — one tenant's broken mailbox never blocks the others
}
}
return records;
}
This keeps TenderIngestionService.pollDueSources(), TenderSourcePollConfig, and SourceRegistry completely untouched except for two additive registry.register() calls and two additive tenderSourcePollConfig.upsert() seed calls in tenders.module.ts — exactly the same additive pattern Plans 13-04/13-05 already used for NetServer/cosinex.
Pattern 2: Additive inbox-provider method (D-02)
What: Add fetchMessages(config): Promise<InboxMessage[]> to InboxProvider alongside the existing fetchPdfAttachments, without modifying it.
IMAP implementation sketch (extends the existing imap.provider.ts MIME-tree-walking pattern used by collectPdfParts):
// New sibling to collectPdfParts() in imap.provider.ts / inbox/imap.provider.ts
function findBodyParts(node: MessageStructureObject | undefined): { htmlPart?: string; textPart?: string } {
// Walk node.childNodes exactly like collectPdfParts, but match
// type === 'text/html' → htmlPart, type === 'text/plain' → textPart (first match wins for each).
}
async fetchMessages(config: InboxConfig): Promise<InboxMessage[]> {
// Same connect/lock/search/fetchAll(envelope+bodyStructure) skeleton as fetchPdfAttachments,
// but instead of collectPdfParts + client.download() for PDF parts, use findBodyParts +
// client.download() for the html/text part IDs, then streamToBuffer(...).toString('utf8').
// Mark \Seen after processing (same as fetchPdfAttachments) — this IS the idempotency
// mechanism for re-polls (D-04's "re-polling the mailbox doesn't create duplicates").
}
EWS implementation sketch (extends exchange-inbox.provider.ts's SOAP builders):
// getItemSoap() needs one more AdditionalProperties FieldURI:
// <t:FieldURI FieldURI="item:Body"/>
// The GetItem response then contains <t:Body BodyType="HTML">...escaped HTML...</t:Body>
// or BodyType="Text". Extract via:
const bodyType = extractAttr(block, 't:Body', 'BodyType'); // 'HTML' | 'Text'
const bodyRaw = extractAll(block, 't:Body')[0] ?? '';
// EWS SOAP-escapes the body content (already handled by the existing escapeXml/extractAll
// pair used elsewhere in this file) — verify against a captured fixture, not live, per
// Validation Architecture below (EWS/NTLM is a documented fragile path, see Pitfall 2).
Pattern 3: RSS item → generic bag mapping (reuses Phase 13's normalizeBag() unchanged)
Given the live-captured shapes (see "RSS Ingestion" findings below), map each <item> to the SAME flat bag shape NetServerAdapter/CosinexAdapter already produce — no normalizer changes needed:
// Source: pattern mirrors apps/api/src/tenders/adapters/netserver.adapter.ts's ocdsPayload bag
records.push({
sourceType: 'rss',
sourcePortal: feedLabel, // admin-provided label or hostname-derived slug, e.g. 'service-bund'
sourceNoticeId: guid || sha256(link).slice(0, 40),
sourceUrl: link,
fetchedAt: new Date(),
publishedAt: pubDate ? new Date(pubDate) : null,
eformsPayload: null,
ocdsPayload: {
title, // item.title, CDATA-merged automatically by fast-xml-parser
buyerName: null, // OPTIONAL enhancement: regex/cheerio-extract from <description>
procedureType: null,
legalFramework: null,
deadlineAt: null, // OPTIONAL enhancement: regex-extract "Angebotsfrist: ..." from <description>
},
});
Optional enhancement (higher data quality, Claude's discretion): service.bund.de's <description> is a CDATA-wrapped HTML fragment with labeled fields (Erfüllungsort: <strong>PLZ Ort</strong>, Vergabestelle: <strong>Name</strong>, Angebotsfrist: <strong>DD.MM.YYYY HH:MM</strong> — confirmed live, see below). Loading this fragment with cheerio.load() and matching label text via :contains() can populate buyerName/deadlineAt/region beyond the bare-minimum bag — the same technique NetServerAdapter uses for its HTML table. This is worth doing for service.bund.de specifically (structure confirmed); do not assume the same label set for subreport-elvis without a live fixture capture first (its <description> was observed to be an HTML <table>, structure not fully characterized this session — see Open Questions).
Anti-Patterns to Avoid
- Extending
TenderSourcePollConfigwith atenantIdcolumn: Would break the "singleton, RLS-exempt, global" invariant thatTenderIngestionService/TenderSchedulerServicedepend on throughout (D-03 from Phase 10). Use the internal-fan-out pattern above instead. - Wrapping the new per-tenant queries in
forTenant()/RLS:TenderEmailConfigrows ARE per-tenant data, but theEmailAlertAdapter's poll-time read needs to see all tenants' active configs in one query (findManyacross all tenants) — this is a deliberate, audited cross-tenant read at the platform-scheduler level, analogous to howTenderIngestionServiceitself never usesforTenant(). Document this exception clearly in the adapter's docstring (mirroring the existing docstring convention) so it isn't "fixed" into an RLS-wrapped query later. - Portal-specific email parsers: explicitly deferred (D-04, Deferred Ideas) — do not add sender-domain-based branching logic even if it seems easy for one obvious case.
Don't Hand-Roll
| Problem | Don't Build | Use Instead | Why |
|---|---|---|---|
| RSS/XML parsing | A custom regex-based RSS tag extractor | fast-xml-parser (already installed, already configured) |
Already handles CDATA-merge, attributes, namespace stripping — a hand-rolled regex parser would re-solve a problem this dependency already solves correctly elsewhere in the codebase |
| Credential encryption | A new AES implementation or a different crypto library | CalendarCryptoService (D-07, already exported by CalendarModule) |
Exact same AES-256-GCM/iv:authTag:ciphertext format already used by DKV and SMTP; introducing a second crypto scheme would fragment key management and audit surface |
| HTML link extraction from email bodies | A regex-only HTML tag stripper | cheerio (already installed) |
Regex-based HTML parsing is a well-known correctness trap (nested tags, malformed markup, HTML entities); cheerio already handles this correctly for NetServer's table parsing |
| NTLM/EWS SOAP construction | A new EWS client library | The existing hand-rolled SOAP builders in exchange-inbox.provider.ts |
Extending the existing, already-battle-tested (if fragile) SOAP-string builders is lower-risk than introducing a second EWS abstraction layer for one new field |
Key insight: every "don't hand-roll" item in this phase already has a proven, in-repo solution from a previous phase — the engineering discipline this phase requires is restraint (reuse the existing tool) more than net-new tool selection.
Runtime State Inventory
Scoped to the INGEST-05 inbox-module-extraction refactor only (the rest of this phase is greenfield).
| Category | Items Found | Action Required |
|---|---|---|
| Stored data | None — no database rows reference the file path dkv/providers/ or the class names by string; Prisma models (DkvModuleConfig) reference no file paths |
None |
| Live service config | None — IMAP/EWS server configuration lives entirely in DkvModuleConfig rows (host/port/credentials), not tied to which TS file implements the connection |
None |
| OS-registered state | None — no OS-level task/service registration references these file paths | None |
| Secrets/env vars | None — CALENDAR_ENCRYPTION_KEY and DKV mailbox credentials are addressed by tenantId, not by import path |
None |
| Build artifacts | apps/api/dist/dkv/providers/*.js will simply stop being emitted once the source files move; nothing external references the old dist/ path (DKV's __dirname-relative user-files/ path resolution in dkv.service.ts is unrelated to the providers/ subfolder) |
Standard rebuild after the move — no migration script needed |
Confirmed via: direct reads of dkv.service.ts, dkv.module.ts, dkv.types.ts, and inbox-provider.interface.ts — the only two files that need an import-path edit are dkv.service.ts (2 import lines) and dkv.module.ts (2 import lines + swap direct-provider-declaration for InboxModule import).
Common Pitfalls
Pitfall 1: Day-cursor gate silently throttles RSS/email to once per calendar day
What goes wrong: TenderIngestionService.pollDueSources() computes nextDayToFetch(config.lastIngestedDay) per source row and only calls adapter.fetchTenders(cursorDay) when the result is a day strictly before "today" — then advances lastIngestedDay and stops. This means each source gets at most one fetch per calendar day, no matter how frequently the cron tick fires or what pollIntervalMin is configured to. This is correct for DÖE (a genuine daily batch export) and is currently also silently applied to ai-netserver/cosinex-dtvp (dormant, unconfirmed live) — but for RSS and especially email-alerts, this defeats the purpose of a configurable poll interval: an admin setting "check every 30 minutes" would still only see new items once every 24 hours.
Why it happens: The day-cursor gate and the cron-tick interval are two independent mechanisms (documented as "Pitfall A" in tender-scheduler.service.ts) — the existing code was written exclusively for a day-batch-export source and adapters since have inherited the same gate without re-examining the fit.
How to avoid: Add an explicit flag distinguishing day-batch sources from tick-driven sources, e.g. a pollGranularity: String @default("day") column on TenderSourcePollConfig ('day' | 'tick'), seeded 'tick' for rss and 'email-alert'. In pollDueSources(), branch: 'day' sources keep the existing nextDayToFetch gate unchanged (zero regression risk for doe-opendata/ai-netserver/cosinex-dtvp); 'tick' sources call adapter.fetchTenders(berlinToday) unconditionally on every tick where isActive, relying purely on the cron interval (already bounded to a minimum by SourceConfigDto's validation, and "sub-hourly polling" is already out of scope per REQUIREMENTS.md) for pacing, and on IMAP \Seen/EWS IsRead marking (email) or feed-item GUID dedup (RSS, via dedupKey upsert — already idempotent) for not reprocessing.
Warning signs: A newly-activated RSS/email source shows zero new tenders for up to 24h after activation even though the feed/mailbox clearly has new items; lastIngestedDay advancing by exactly one calendar day per admin-configured interval regardless of the interval's actual value.
Pitfall 2: subreport-elvis has no single canonical feed URL
What goes wrong: Treating "subreport-elvis" as one hardcoded portal (like NETSERVER_PORTALS) is structurally impossible — the platform issues one feed per contracting authority/municipality (confirmed live: Stadt Neuss's feed is https://www.subreport-elvis.de/elvis/secure/rss.pl?id=4615; other municipalities have different numeric id values on the same base path). Hardcoding one municipality's feed would silently scope "subreport-elvis ingestion" to that one city only, contradicting the intent of a platform-wide long-tail source.
Why it happens: The phase description names "subreport-elvis" as if it were a single source, matching the service.bund.de pattern (which genuinely IS one national feed) — but subreport-elvis's product model is per-customer feed subscriptions.
How to avoid: Build the admin-managed list (TenderRssFeedSource, already implied by D-09's "RSS-Feeds" admin section) so an admin can add whichever concrete subreport-elvis feed URL(s) are relevant, rather than baking one into code. Seed the DB with zero subreport-elvis rows by default (there is no universally-correct one); optionally seed the confirmed service.bund.de national feed as a sane global default since that one genuinely is Germany-wide.
Warning signs: A hardcoded SUBREPORT_ELVIS_URL constant with only one municipality's feed id.
Pitfall 3: Runtime-added RSS feed URLs bypass the INGEST-07 code-level denylist gate
What goes wrong: SourceRegistry.register() checks DENYLISTED_PORTALS against an adapter's statically declared portals: readonly string[] array at DI-boot time. An admin-CRUD RSS feed URL is added at runtime (via the new TenderRssFeedSource model), long after boot — the RssAdapter's portals array can only contain a symbolic placeholder (e.g. ['rss']), not the actual runtime feed hostnames. This means the code-level denylist gate provides zero protection against an admin accidentally adding an RSS feed URL that points at vergabe24.de/aumass.de.
Why it happens: INGEST-07's denylist gate was designed for adapter-registration-time enforcement (a developer registering a whole new scraping adapter), not for runtime, per-row admin input — RSS is the first source type in this codebase where the "portal" is admin-supplied data rather than code.
How to avoid: Add a parallel, save-time hostname validation in the TenderRssFeedSource create/update service (reject any URL whose hostname contains vergabe24 or aumass, mirroring the spirit of DENYLISTED_PORTALS even though it's enforced in a different code path). Document in the DTO/service that this is a second, independent enforcement point, not a bypass of the existing one.
Warning signs: A TenderRssFeedSource row pointing at a vergabe24/aumass hostname existing in the DB without any validation error at save time.
Pitfall 4: EWS body-fetch is a new, untested addition to a documented-fragile file
What goes wrong: exchange-inbox.provider.ts implements NTLM/EWS SOAP by hand (no EWS client library) and is called out in project memory as fragile (project_calendar_ews_fix: "Exchange EWS nutzt NTLM via httpntlm... nie ews-javascript-api"). Adding a new fetchMessages() method here — even though additive and not touching fetchPdfAttachments — introduces new hand-written SOAP/XML-extraction code (t:Body/BodyType handling) with zero existing test coverage to fall back on (there are currently no .spec.ts files for either ImapProvider or ExchangeInboxProvider in this codebase at all).
Why it happens: DKV's EWS path has historically been validated live against a real Exchange server rather than via unit tests (confirmed: find apps/api/src/dkv -iname "*.spec.ts" returns nothing).
How to avoid: Do not skip unit tests for the new method just because the existing methods also lack them — this phase is the right moment to add fixture/mock-based coverage for fetchMessages() on both providers (mock ImapFlow/httpntlm.post), even if fetchPdfAttachments remains untested (out of scope to backfill). Flag the EWS body-fetch specifically for a checkpoint:human-verify against a real Exchange mailbox before considering INGEST-05's Exchange path production-ready — this mirrors Phase 13's human-verification-required pattern for anything requiring live external access.
Warning signs: fetchMessages() on the Exchange path passing all mocked unit tests but returning empty/garbled bodies against a real mailbox (EWS body escaping/whitespace handling is a common source of subtle bugs).
Pitfall 5: Email-derived Tender rows become visible to every tenant, not just the one whose mailbox produced them
What goes wrong: Tender has deliberately no tenantId column (D-03, Phase 10: "Tender ist plattform-global"). Every prior source (DÖE, NetServer, cosinex) is a public portal anyway, so global visibility was never in tension with the data's origin. An email-alert record, by contrast, is discovered via one specific tenant's private mailbox subscription — under the current architecture it will still be inserted as a global Tender row, visible to every other tenant that has the tender-radar module activated, even though they never configured that mailbox.
Why it happens: This is an intentional consequence of D-03 (locked in Phase 10) combined with D-06 (email config is per-tenant) — CONTEXT.md for this phase does not revisit or discuss this interaction explicitly.
How to avoid: This is not a code bug to fix silently — it is a product-policy question that should be explicitly confirmed with the user before/while planning INGEST-05, since it means one tenant's paid/subscribed alert channel effectively becomes a shared benefit for all tenants on the platform. Surface this plainly to the planner/user (see Open Questions #1) rather than assuming either "this is fine" or "this needs per-tenant scoping" — both are defensible depending on product intent, but only one matches what CONTEXT.md's silence implies.
Warning signs: None at the code level (it will work exactly as designed) — the risk is entirely in unstated product expectations surfacing after Tenant B starts seeing tenders Tenant A's mailbox produced.
Code Examples
RSS parsing with the existing fast-xml-parser configuration
// Source: mirrors apps/api/src/tenders/adapters/doe-opendata.adapter.ts's XMLParser config
import { XMLParser } from 'fast-xml-parser';
const xmlParser = new XMLParser({
removeNSPrefix: true,
ignoreAttributes: false,
attributeNamePrefix: '@_',
// cdataPropName defaults to `false`, which means CDATA content is merged
// directly into the tag's text value — confirmed in fast-xml-parser v5.10.1
// source (OptionsBuilder.js `cdataPropName: false`), no extra config needed
// for RSS <title><![CDATA[...]]></title> to parse straight to a string.
});
const parsed = xmlParser.parse(rssXmlText) as {
rss?: { channel?: { item?: unknown | unknown[] } };
};
const channel = parsed.rss?.channel;
const items = Array.isArray(channel?.item) ? channel.item : channel?.item ? [channel.item] : [];
Live-captured RSS shapes (this session, both fetched directly against the real endpoints)
service.bund.de — https://www.service.bund.de/Content/Globals/Functions/RSSFeed/RSSGenerator_Ausschreibungen.xml [CITED: live fetch, 2026-07-23]:
<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0">
<channel>
<title>service.bund.de - öffentliche Ausschreibungen</title>
<link>http://www.service.bund.de/ausschreibungen</link>
<description>RSS-Feed mit aktuellen Ausschreibungen der Vergabestellen...</description>
<language>de-de</language>
<pubDate>Thu, 23 Jul 2026 11:27:44 +0200</pubDate>
<ttl>60</ttl>
<item>
<title>Vhv 120_26 UFZ - Bench-scale Anaerobic Bioreactor System</title>
<link>https://www.service.bund.de/IMPORTE/Ausschreibungen/editor/Helmholtz-Zentrum-fuer-Umweltforschung-GmbH/2026/07/6585334.html#track=feed-callforbids</link>
<description><![CDATA[Erfüllungsort: <strong>04318 Leipzig</strong><br />Vergabestelle: <strong>Helmholtz-Zentrum für Umweltforschung GmbH - UFZ</strong><br />Angebotsfrist: <strong>07.08.2026 10:00</strong>...]]></description>
<pubDate>Thu, 23 Jul 2026 11:00:00 +0200</pubDate>
<guid>https://www.service.bund.de/IMPORTE/Ausschreibungen/editor/Helmholtz-Zentrum-fuer-Umweltforschung-GmbH/2026/07/6585334.html</guid>
</item>
</channel>
</rss>
Field mapping: title→title, link/guid→sourceUrl/sourceNoticeId, pubDate→publishedAt, description CDATA contains labeled HTML fields (Erfüllungsort, Vergabestelle, Angebotsfrist) extractable via cheerio for the optional enhancement (see Pattern 3).
subreport-elvis (one live municipal instance, Stadt Neuss) — https://www.subreport-elvis.de/elvis/secure/rss.pl?id=4615 [CITED: live fetch, 2026-07-23]:
<?xml version='1.0' encoding='utf-8'?>
<rss version='2.0'>
<channel>
<title><![CDATA[subreport ELViS]]></title>
<link>https://www.subreport-elvis.de/</link>
<description><![CDATA[Aktuelle Ausschreibungen von Stadt Neuss, 41460 Neuss]]></description>
<item>
<title><![CDATA[E73433797:Objektbezogene Schadensanalysen...]]></title>
<description><![CDATA[<table>...</table>]]></description>
<link>https://www.subreport-elvis.de/browseVerdingungsunterlagen.html#ELVISID:E73433797</link>
<author><![CDATA[Stadt Neuss, vergabe@stadt.neuss.de]]></author>
<guid isPermaLink="false">E73433797-558464</guid>
</item>
</channel>
</rss>
Notable difference from service.bund.de: no pubDate on items (⇒ publishedAt: null, already nullable-tolerant everywhere downstream); <description> is an HTML <table> whose internal field structure was not fully characterized this session (see Open Questions #2) — do not assume the same label-based extraction as service.bund.de without a fixture capture first.
Generic link extraction from an alert email body (D-04)
import * as cheerio from 'cheerio';
import { createHash } from 'crypto';
const FOOTER_NOISE = ['unsubscribe', 'abmelden', 'einstellungen', 'preferences'];
const MAX_LINKS_PER_EMAIL = 5; // safety valve against link-spam producing excess records
function extractCandidateLinks(bodyHtml: string | null, bodyText: string): string[] {
let links: string[];
if (bodyHtml) {
const $ = cheerio.load(bodyHtml);
links = $('a[href]')
.map((_, el) => $(el).attr('href'))
.get()
.filter((href): href is string => typeof href === 'string' && /^https?:\/\//i.test(href));
} else {
links = Array.from(bodyText.matchAll(/https?:\/\/[^\s"'<>]+/g)).map((m) => m[0]);
}
return Array.from(new Set(links))
.filter((url) => !FOOTER_NOISE.some((noise) => url.toLowerCase().includes(noise)))
.slice(0, MAX_LINKS_PER_EMAIL);
}
function titleFromEmail(subject: string, bodyText: string): string {
if (subject.trim()) return subject.trim();
const firstLine = bodyText.split('\n').map((l) => l.trim()).find(Boolean);
return firstLine || 'Unbenannte Ausschreibung'; // matches normalizeBag()'s existing fallback
}
function sourceNoticeIdFor(link: string): string {
return createHash('sha256').update(link).digest('hex').slice(0, 40);
}
Per-tenant config service (mirrors DkvService's config half + SettingsService's Safe-Select)
// Source: pattern mirrors apps/api/src/dkv/dkv.service.ts §21-41, §95-125
const EMAIL_CONFIG_SAFE_SELECT = {
id: true, tenantId: true, protocol: true, host: true, port: true,
encryption: true, folder: true, senderFilter: true, domain: true,
isActive: true, createdAt: true, updatedAt: true,
// encryptedInboxCreds: NEVER included — T-07-12 pattern
} as const;
async function getConfigForApi(prisma: PrismaService, crypto: CalendarCryptoService, tenantId: string) {
const safe = await prisma.tenderEmailConfig.findUnique({ where: { tenantId }, select: EMAIL_CONFIG_SAFE_SELECT });
if (!safe) return null;
let username: string | null = null;
let hasPassword = false;
try {
const raw = await prisma.tenderEmailConfig.findUnique({ where: { tenantId } });
if (raw?.encryptedInboxCreds) {
const creds = JSON.parse(crypto.decrypt(raw.encryptedInboxCreds)) as { username?: string; password?: string };
username = creds.username ?? null;
hasPassword = Boolean(creds.password);
}
} catch { /* ignore decrypt errors — return empty username, matches DkvService.getConfigForApi */ }
return { ...safe, username, hasPassword };
}
State of the Art
| Old Approach | Current Approach | When Changed | Impact |
|---|---|---|---|
| ews-javascript-api (Basic Auth only) | Hand-rolled NTLM/EWS SOAP via httpntlm |
Phase 7 (DKV build) | The new fetchMessages() EWS path must extend the hand-rolled SOAP builders, not reach for a library — consistent with project_calendar_ews_fix memory |
node-html-parser (initially considered for NetServer HTML) |
cheerio |
Phase 13, Plan 13-04 (package-legitimacy false-positive on node-html-parser's age check, human-approved switch to cheerio) |
Confirms cheerio is the codebase's settled HTML-parsing choice — use it for email-body link extraction and optional RSS description parsing, don't reconsider node-html-parser |
Hardcoded tender-radar UI strings ("MVP-stub convention," explicitly noted in SourceConfigForm.tsx/settings/page.tsx comments) |
useTranslations-driven i18n |
Phase 14 (this phase, CONFIG-03) | This phase is the point where the previously-accepted stub convention is retired for the tender-radar module specifically |
Deprecated/outdated: none beyond the above — this is a young, actively-developed module with no legacy cruft to remove.
Assumptions Log
| # | Claim | Section | Risk if Wrong |
|---|---|---|---|
| A1 | subreport-elvis's <description> HTML <table> structure was not fully captured this session (only a summarized excerpt was obtained via WebFetch, not the complete raw table markup) |
RSS Ingestion / Code Examples | If the planner assumes a specific field-label structure without a fresh live fixture capture, the optional enhancement (buyerName/deadlineAt extraction) could silently extract wrong/garbled data for subreport-elvis specifically. The bare-minimum bag mapping (title/link/guid only) is NOT at risk — it only needs <item><title>/<link>/<guid>, all confirmed. |
| A2 | The exact EWS <t:Body>/BodyType XML shape returned by a real Exchange server for the new fetchMessages() GetItem call was not verified live this session (based on Microsoft EWS schema documentation / training knowledge, not a live capture against this codebase's actual target mailbox) |
Architecture Patterns / Pattern 2 | If the real server's response shape differs from the assumed <t:Body BodyType="HTML">...</t:Body> structure (e.g. additional wrapping elements, different escaping), the EWS body-fetch will silently return empty/garbled content rather than throwing — exactly the failure mode flagged in Pitfall 4. Requires a checkpoint:human-verify against a real mailbox. |
| A3 | Whether "email-derived Tender rows visible platform-wide" (Pitfall 5) is the intended product behavior or an unstated gap in this phase's scoping — not discussed in CONTEXT.md | Common Pitfalls / Pitfall 5 | If wrongly assumed acceptable, one tenant's private mailbox subscription becomes free platform-wide data for every other tenant — a potential customer-trust/contract issue for a soon-to-be-sold multi-tenant product. If wrongly assumed to need per-tenant scoping, unnecessary architecture work (breaking D-03) may be attempted. |
If this table is empty: N/A — see rows above; all three should be confirmed with the user or resolved via a checkpoint:human-verify/explicit discuss-phase note before the planner locks task-level detail for the affected areas.
Open Questions (RESOLVED)
-
Email-alert Tender visibility across tenants (see Pitfall 5 / A3) — RESOLVED via CONTEXT.md D-13 (User-Entscheidung 2026-07-23): E-Mail-Tender sind nur für den konfigurierenden Mandanten sichtbar (nullable
Tender.ownerTenantId), öffentliche Quellen inkl. RSS bleiben global. Umgesetzt in Plan 14-03.- What we know:
Tenderis architecturally platform-global (D-03);TenderEmailConfigis architecturally per-tenant (D-06). CONTEXT.md doesn't reconcile the two for this specific case. - What's unclear: Whether this is accepted product behavior (matches the existing "all discovered tenders are shared" philosophy) or an oversight.
- Recommendation: Surface explicitly to the user in
/gsd-discuss-phasefollow-up or as a plan-timecheckpoint:human-verifybefore implementing INGEST-05's dedup/storage step — this is a five-minute confirmation that prevents a much larger later rework if the answer is "no, that's not okay."
- What we know:
-
subreport-elvis
<description>field structure (see A1) — RESOLVED via Plan 14-02 Task 1: Baseline schickt nur die bare-minimum Bag (title/link/guid), reicht für INGEST-04; optionale Feld-Anreicherung aus der HTML-Tabelle ist nicht Teil des Baselines. Live-Fixture-Capture bei Bedarf zur Ausführungszeit.- What we know: It's an HTML
<table>; the outer feed/item shape (title/link/guid/author) is fully confirmed live. - What's unclear: The internal table structure (column labels, whether "Angebotsfrist"-equivalent data exists at all for subreport-elvis, unlike service.bund.de where it's confirmed present in the description).
- Recommendation: Capture a fresh, complete raw XML fixture from a live subreport-elvis feed at Wave-0/plan-time (mirroring Phase 13's live-fixture-capture convention for NetServer/cosinex HTML) before deciding whether to attempt the optional field-enrichment for this source, or to ship it with the bare-minimum bag only (title/link/guid — already fully sufficient for INGEST-04's baseline requirement).
- What we know: It's an HTML
-
Default seeded RSS feeds
- RESOLVED via Plan 14-02 Task 2: service.bund.de wird active-by-default geseedet, die Feed-Liste sonst leer für admin-getriebene Ergänzungen (matcht die „framework ready, activation deferred"-Haltung).
- What we know: service.bund.de's feed is genuinely national/global and safe to seed by default; subreport-elvis has no such single URL.
- What's unclear: Whether to seed zero rows and require an admin action before any RSS ingestion happens, or seed the service.bund.de URL as an always-on default.
- Recommendation: Seed service.bund.de active-by-default (mirrors DÖE's
isActive: trueseed from Phase 10), leave the RSS feed list otherwise empty for admin-driven additions — matches the "framework ready, activation deferred" stance already established for ai-netserver/cosinex-dtvp in Phase 13.
Environment Availability
| Dependency | Required By | Available | Version | Fallback |
|---|---|---|---|---|
fast-xml-parser |
RSS parsing (INGEST-04) | ✓ | ^5.10.1 (installed) | — |
imapflow |
Email fetchMessages (INGEST-05) | ✓ | ^1.4.3 (installed) | — |
httpntlm |
EWS fetchMessages (INGEST-05) | ✓ | ^1.8.13 (installed) | — |
cheerio |
Link extraction / optional RSS enrichment | ✓ | ^1.2.0 (installed) | — |
| Live subreport-elvis test feed | RSS adapter fixture capture (Wave 0) | ✓ (public, one instance confirmed reachable) | — | If a specific tenant's real feed is unavailable at plan/build time, use the Stadt Neuss instance captured this session as a placeholder fixture, clearly marked as a third-party example, not a Tessera-owned endpoint |
Live Exchange/EWS mailbox for fetchMessages() verification |
Pitfall 4 human-verify | ✗ (not available in this research session — no live mailbox credentials) | — | Build against mocked SOAP responses (fixture-based unit tests); require a checkpoint:human-verify against a real mailbox before considering the Exchange email-alert path production-ready |
Missing dependencies with no fallback: none blocking — the one genuinely missing environment piece (a live Exchange mailbox to verify EWS body-fetch against) has an accepted fallback (mock-based tests + human-verify checkpoint).
Missing dependencies with fallback: live Exchange/EWS mailbox (see above).
Validation Architecture
Test Framework
| Property | Value |
|---|---|
| Framework | Vitest (apps/api: node environment; apps/web: jsdom environment) |
| Config file | apps/api/vitest.config.ts (include: ['src/**/*.spec.ts']), apps/web/vitest.config.ts |
| Quick run command | cd apps/api && npx vitest run src/tenders/adapters/rss.adapter.spec.ts (swap path per file) |
| Full suite command | cd apps/api && npx vitest run / cd apps/web && npx vitest run |
Phase Requirements → Test Map
| Req ID | Behavior | Test Type | Automated Command | File Exists? |
|---|---|---|---|---|
| INGEST-04 | RSS feed fetch+parse maps real feed XML to RawTenderRecord[] correctly (both confirmed shapes: service.bund.de with pubDate, subreport-elvis without) |
unit (fixture-based, mirrors cosinex.adapter.spec.ts) |
npx vitest run src/tenders/adapters/rss.adapter.spec.ts |
❌ Wave 0 — new file, fixtures need live capture (see Environment Availability) |
| INGEST-04 | Extended normalizer dispatch for 'rss' sourceType (should reuse existing normalizeBag() — extend the switch in tender-normalizer.service.ts and its spec) |
unit | npx vitest run src/tenders/tender-normalizer.service.spec.ts |
✅ file exists, needs 1-2 new test cases added |
| INGEST-05 | fetchMessages() on ImapProvider extracts html/text body from a mocked ImapFlow MIME tree |
unit (mocked, no existing coverage to extend — net new) | npx vitest run src/inbox/imap.provider.spec.ts |
❌ Wave 0 — no .spec.ts exists for either inbox provider today (confirmed via directory search) |
| INGEST-05 | fetchMessages() on ExchangeInboxProvider extracts body from a mocked httpntlm.post SOAP response |
unit (mocked) | npx vitest run src/inbox/exchange-inbox.provider.spec.ts |
❌ Wave 0 — new file |
| INGEST-05 | Generic link+subject extraction (extractCandidateLinks/titleFromEmail) — pure functions, HTML and plaintext fallback both covered |
unit (pure function, no I/O) | npx vitest run src/tenders/adapters/email-alert.adapter.spec.ts |
❌ Wave 0 — new file |
| INGEST-05 | Per-tenant fan-out: one broken tenant mailbox doesn't block others (catch-per-tenant) | unit (mocked prisma.tenderEmailConfig.findMany + mocked provider) |
npx vitest run src/tenders/adapters/email-alert.adapter.spec.ts |
❌ same file as above |
| INGEST-05 | DKV regression: moving Imap/ExchangeInboxProvider into inbox/ does not change DKV's existing fetchPdfAttachments behavior |
manual/full-suite | cd apps/api && npx vitest run (full suite green) + npx tsc --noEmit |
✅ existing DKV spec files (0 today) mean this is really "no regression in the full suite," not a dedicated regression test — see Wave 0 Gaps |
| CONFIG-02 | Safe-Select excludes encryptedInboxCreds; encrypt/decrypt round-trips correctly |
unit | npx vitest run src/tenders/tender-email-config.service.spec.ts |
❌ Wave 0 — new file, mirrors absent-but-should-exist dkv.service.spec.ts pattern |
| CONFIG-02 | GET/PUT /modules/tender-radar/email-config and /rss-feeds are @Roles(ADMIN,SUPER_ADMIN)-guarded |
integration (controller test) | npx vitest run src/tenders/tenders.controller.spec.ts |
✅ file exists, needs new route test cases added |
| CONFIG-02 | Runtime host-based denylist check rejects a vergabe24/aumass RSS feed URL at save time (Pitfall 3) | unit | npx vitest run src/tenders/tender-rss-feed.service.spec.ts |
❌ Wave 0 — new file |
| CONFIG-03 | de.json/en.json have identical key sets under the new tenderRadar namespace (no missing EN/DE translation) |
unit (web side, pure JSON structural comparison) | cd apps/web && npx vitest run src/messages/tenderRadar-parity.spec.ts |
❌ Wave 0 — no existing i18n key-parity test in the codebase at all; recommend adding one (cheap, catches the exact class of bug fixed by quick task 260701-abc) |
| UI-06 | CoverageBanner renders the denylist "manually monitor" block with correct vergabe24/aumass links |
component test | cd apps/web && npx vitest run src/app/\(portal\)/modules/tender-radar/components/CoverageBanner.test.tsx |
❌ Wave 0 — no test file exists for CoverageBanner today |
| UI-06 | New denylist read endpoint returns the two portals with correct URLs | integration | npx vitest run src/tenders/tenders.controller.spec.ts |
✅ extend existing file |
Sampling Rate
- Per task commit: run the single most relevant spec file from the table above (
npx vitest run <file>) - Per wave merge:
cd apps/api && npx vitest run && npx tsc --noEmit -p tsconfig.json, thencd apps/web && npx vitest run - Phase gate: both full suites green +
tsc --noEmitclean before/gsd-verify-work, matching Phase 13's verification convention
Wave 0 Gaps
apps/api/src/tenders/__fixtures__/service-bund-feed.xml— live-captured, full (not summarized) raw XML; the excerpt in this RESEARCH.md is representative but should be re-captured in full at plan/build time for the actual fixture fileapps/api/src/tenders/__fixtures__/subreport-elvis-feed.xml— live-captured, full raw XML including the complete<description><table>markup (currently only a summarized excerpt exists — see Open Questions #2)apps/api/src/inbox/imap.provider.spec.ts+exchange-inbox.provider.spec.ts— genuinely net-new test infrastructure; no prior DKV provider tests exist to extendapps/web/src/messages/key-parity check — no existing test enforcesde.json/en.jsonstructural parity; recommend adding one as part of this phase given the module now doubles the message-file surface area- Prisma migration for
TenderEmailConfig(per-tenant) andTenderRssFeedSource(global) — neither model exists yet
Security Domain
Applicable ASVS Categories
| ASVS Category | Applies | Standard Control |
|---|---|---|
| V2 Authentication | no (new routes reuse existing session auth, no new auth mechanism) | — |
| V3 Session Management | no | — |
| V4 Access Control | yes | @Roles(Role.ADMIN, Role.SUPER_ADMIN) on all new config routes (email-config, rss-feeds), mirroring DkvController/existing source-config route |
| V5 Input Validation | yes | class-validator DTOs for the new TenderEmailConfigDto/TenderRssFeedSourceDto (@IsIn for protocol/encryption, @IsUrl for feed URLs, @IsEmail for senderFilter — mirrors DkvConfigDto); host-based denylist validation on RSS feed URL save (Pitfall 3); email body HTML must never be rendered raw in any UI — only extracted plain-text title/links are ever stored/displayed |
| V6 Cryptography | yes | CalendarCryptoService (AES-256-GCM) — never a new/different crypto implementation for the new mailbox credentials |
Known Threat Patterns for this stack
| Pattern | STRIDE | Standard Mitigation |
|---|---|---|
| SSRF via admin-supplied RSS feed URL (admin could enter an internal/private-network URL) | Tampering / Information Disclosure | Validate feed URL scheme is https?:// and reject private/loopback IP ranges before fetching, mirroring the SSRF-guard intent already documented for NETSERVER_PORTALS (hardcoded there; here it must be a runtime check since the URL is admin input) — new for this phase since it's the first admin-input-driven fetch target in tenders/ |
| Email-injection / stored-XSS via alert email body (HTML) | Tampering | Only extract plain-text title and raw href URL strings — never store or render the HTML body itself; mirrors the existing .text()-not-.html() discipline already documented in NetServerAdapter (V5) |
| Credential leakage via logs (IMAP/EWS) | Information Disclosure | Reuse the existing logger: false (imapflow) / generic-error-message (EWS) conventions for any new log statement in fetchMessages() — do not introduce a new logging call that includes config.password/config.username |
| Decompression/parsing DoS via a malicious/huge RSS feed response | Denial of Service | Apply the same AbortController 15s timeout pattern already used by DoeOpenDataAdapter/NetServerAdapter; consider a response-size ceiling for RSS fetches (RSS feeds are normally small, but an admin-supplied URL is less trusted than a hardcoded one) |
Sources
Primary (HIGH confidence)
- Direct codebase reads (this session):
inbox-provider.interface.ts,imap.provider.ts,exchange-inbox.provider.ts,dkv.types.ts,dkv.service.ts,dkv.module.ts,crypto.service.ts,dkv-config.dto.ts,dkv.controller.ts,tender-source-adapter.interface.ts,source-registry.ts,tenders.module.ts,tender.types.ts,tender-normalizer.service.ts,tender-ingestion.service.ts,tender-scheduler.service.ts,netserver.adapter.ts,doe-opendata.adapter.ts,tender-mail.service.ts,settings.service.ts,prisma-tenant.extension.ts,tenders.controller.ts,SourceConfigForm.tsx,InboxConfigForm.tsx,tender-radar-api.ts,CoverageBanner.tsx,layout.tsx,i18n/request.ts,middleware.ts,schema.prisma(DkvModuleConfig/TenderSourcePollConfig/Tender models),apps/api/package.json,apps/web/src/messages/de.json/en.json gsd-tools query package-legitimacy check—rss-parserverdict OK (825k/wk, github.com/bobby-brennan/rss-parser, not deprecated, no postinstall)- fast-xml-parser v5.10.1 source (
OptionsBuilder.js,fxp.d.ts) — confirmedcdataPropNamedefault behavior
Secondary (MEDIUM confidence)
- Live WebFetch of
https://www.service.bund.de/Content/Globals/Functions/RSSFeed/RSSGenerator_Ausschreibungen.xml(2026-07-23) — full channel/item XML shape - Live WebFetch of
https://www.subreport-elvis.de/elvis/secure/rss.pl?id=4615(2026-07-23) — channel/item XML shape (description table content not fully captured, see A1) - WebSearch confirming subreport-elvis's per-municipality feed model (Neuss city page) and vergabe24.de/aumass.de canonical URLs
Tertiary (LOW confidence)
- EWS
<t:Body BodyType="...">response shape for the newfetchMessages()GetItem call — based on general EWS schema knowledge, not verified against a live Exchange server this session (A2)
Metadata
Confidence breakdown:
- Standard stack: HIGH — zero new dependencies, all four libraries already proven in this exact codebase
- Architecture: HIGH for the adapter/config/scheduler patterns (direct precedent in Phase 13); MEDIUM for the exact EWS body-fetch shape (A2)
- Pitfalls: HIGH for #1/#3/#5 (derived from direct code reading, not speculation); MEDIUM for #2/#4 (well-grounded but partially forward-looking)
Research date: 2026-07-23 Valid until: 30 days for code-pattern findings (stable, in-repo); the two live-fetched RSS feed snapshots should be re-verified at plan/build time regardless of age, since external feed structures can change without notice