--- phase: 14-rss-email-alert-ingestion-module-rollout plan: 02 subsystem: api tags: [nestjs, prisma, fast-xml-parser, rss, ssrf, cron, nextjs] # Dependency graph requires: - phase: 13-additional-tender-sources-cross-source-dedup provides: TenderSourceAdapter interface, SourceRegistry denylist gate, normalizeBag() generic-bag normalizer path, pollDueSources fan-out provides: - RssAdapter (fast-xml-parser, admin-feed internal fan-out) parsing RSS 2.0 into RawTenderRecord[] - 'rss' SourceType + normalizer dispatch through normalizeBag() - TenderRssFeedSource Prisma model (global admin-managed feed list) + save-time hostname/SSRF guard service - TenderSourcePollConfig.pollGranularity ('day' | 'tick') gate in pollDueSources — RSS bypasses the day-cursor entirely - GET/POST/DELETE /modules/tender-radar/rss-feeds admin routes + RssFeedListForm settings UI affects: [14-03-email-alert-ingestion, 14-04-admin-config-encryption, 14-05-i18n] # Tech tracking tech-stack: added: [] patterns: - "RSS parsing reuses the existing fast-xml-parser XMLParser config (removeNSPrefix/ignoreAttributes/attributeNamePrefix) from doe-opendata.adapter.ts — no new dependency" - "Internal-fan-out adapter: RssAdapter.fetchTenders() reads all active TenderRssFeedSource rows and fetches each independently (catch-per-feed), same shape as NetServerAdapter's multi-portal loop" - "pollGranularity branch in pollDueSources(): 'day' sources keep the byte-unchanged lastIngestedDay gate; 'tick' sources fetch unconditionally every active tick" - "Save-time hostname/SSRF guard as a SEPARATE enforcement point from the code-level SourceRegistry denylist — required whenever a source's target is runtime admin input rather than a hardcoded portal" key-files: created: - apps/api/src/tenders/adapters/rss.adapter.ts - apps/api/src/tenders/adapters/rss.adapter.spec.ts - apps/api/src/tenders/__fixtures__/service-bund-feed.xml - apps/api/src/tenders/__fixtures__/subreport-elvis-feed.xml - apps/api/src/tenders/tender-rss-feed.service.ts - apps/api/src/tenders/tender-rss-feed.service.spec.ts - apps/api/src/tenders/dto/tender-rss-feed.dto.ts - apps/api/prisma/migrations/20260723111946_add_rss_feed_source_and_poll_granularity/migration.sql - apps/web/src/app/(portal)/modules/tender-radar/settings/components/RssFeedListForm.tsx - apps/web/src/app/(portal)/modules/tender-radar/settings/components/RssFeedListForm.test.tsx modified: - apps/api/src/tenders/tender.types.ts - apps/api/src/tenders/tender-normalizer.service.ts - apps/api/src/tenders/tender-normalizer.service.spec.ts - apps/api/src/tenders/tender-ingestion.service.ts - apps/api/src/tenders/tender-ingestion.service.spec.ts - apps/api/src/tenders/tenders.module.ts - apps/api/src/tenders/tenders.controller.ts - apps/api/src/tenders/tenders.controller.spec.ts - apps/api/prisma/schema.prisma - apps/web/src/lib/tender-radar-api.ts - apps/web/src/app/(portal)/modules/tender-radar/settings/page.tsx key-decisions: - "fast-xml-parser only decodes the 5 predefined XML entities, not numeric character refs (Ü) — added an explicit decodeNumericEntities() step so service.bund.de titles render correctly instead of leaking raw entity syntax (Rule 1 fix)" - "Seeded service.bund.de as an active-by-default TenderRssFeedSource row; zero subreport-elvis rows seeded (no single canonical URL — per-municipality instances, admin adds relevant feeds)" - "'rss' TenderSourcePollConfig seeded isActive:true/pollGranularity:'tick' from day one — unlike ai-netserver/cosinex-dtvp's 'framework ready, activation deferred' stance from Phase 13" - "Save-time SSRF guard is string/IP-literal-based only, no DNS resolution — matches the existing T-10-06 scope; resolving a hostname to check its IP would itself be an SSRF-adjacent action" patterns-established: - "pollGranularity as the mechanism the upcoming email-alert adapter (Plan 14-03) reuses for its own tick-driven polling need" requirements-completed: [INGEST-04, CONFIG-02] coverage: - id: D1 description: "RssAdapter.parseFeed maps live-captured service.bund.de (pubDate present, numeric-entity titles) and subreport-elvis (pubDate absent, CDATA titles) shapes into RawTenderRecord[]; malformed/empty XML and single-item link-loss never throw" requirement: INGEST-04 verification: - kind: unit ref: "apps/api/src/tenders/adapters/rss.adapter.spec.ts (18 tests)" status: pass human_judgment: false - id: D2 description: "'rss' sourceType routes through TenderNormalizerService.normalizeBag() with title-through mapping and empty cpvDivisions" requirement: INGEST-04 verification: - kind: unit ref: "apps/api/src/tenders/tender-normalizer.service.spec.ts (rss describe block, 2 tests)" status: pass human_judgment: false - id: D3 description: "Admin can add/list/remove global RSS feed URLs; vergabe24/aumass and private/loopback hosts are rejected at save time with a domain-specific 400 message" requirement: CONFIG-02 verification: - kind: unit ref: "apps/api/src/tenders/tender-rss-feed.service.spec.ts (14 tests)" status: pass - kind: unit ref: "apps/api/src/tenders/tenders.controller.spec.ts (RSS-feeds describe block, 4 tests + 1 route-order test)" status: pass human_judgment: false - id: D4 description: "RSS is polled every active scheduler tick honoring pollIntervalMin (via the single global cron), NOT gated to once per calendar day like doe-opendata/ai-netserver/cosinex-dtvp" requirement: INGEST-04 verification: - kind: unit ref: "apps/api/src/tenders/tender-ingestion.service.spec.ts (pollGranularity 'tick' gate describe block, 3 tests)" status: pass human_judgment: false - id: D5 description: "RssFeedListForm renders an admin-manageable RSS feed list on the tender-radar settings page: load, add (denylist error surfaced inline), remove" requirement: CONFIG-02 verification: - kind: unit ref: "apps/web/src/app/(portal)/modules/tender-radar/settings/components/RssFeedListForm.test.tsx (6 tests)" status: pass human_judgment: false duration: 30min completed: 2026-07-23 status: complete --- # Phase 14 Plan 02: RSS Ingestion Slice Summary **RssAdapter (fast-xml-parser, admin-CRUD feed fan-out) + global TenderRssFeedSource list with a save-time denylist/SSRF guard + pollGranularity='tick' gate so RSS honors its poll interval instead of DÖE's day-cursor** ## Performance - **Duration:** ~30 min (including live-fixture capture and a pre-existing migration checksum-drift fix) - **Started:** 2026-07-23T13:09:00Z (context reads) / first commit 13:17:02 - **Completed:** 2026-07-23T13:29:07Z - **Tasks:** 3/3 completed - **Files modified:** 21 (10 created, 11 modified) ## Accomplishments - `RssAdapter.parseFeed()` maps two live-captured RSS shapes (service.bund.de with `pubDate` + numeric-HTML-entity titles; subreport-elvis without `pubDate`, CDATA titles) into `RawTenderRecord[]`, routed through the existing generic `normalizeBag()` normalizer path (D-04/D-05, zero normalizer-specific RSS code) - `TenderRssFeedSource` Prisma model (global, no `tenantId`, D-08) with full admin CRUD (`TenderRssFeedSourceService`) gated by a save-time hostname/SSRF guard (D-14) — rejects `DENYLISTED_PORTALS` hostnames (vergabe24/aumass, same constant as the code-level gate), non-http(s) schemes, and private/loopback/link-local hosts - `TenderSourcePollConfig.pollGranularity` (`'day' | 'tick'`) added; `pollDueSources()` branches per source — 'day' sources (doe-opendata/ai-netserver/cosinex-dtvp) keep the exact pre-existing `lastIngestedDay` gate, 'tick' sources (rss) fetch unconditionally every active tick (D-15) - `RssAdapter` registered in `tenders.module.ts`; 'rss' poll config seeded `isActive: true, pollGranularity: 'tick'`; a default-active `service.bund.de` feed row seeded (zero subreport-elvis rows — no single canonical URL to default to) - `GET/POST/DELETE /modules/tender-radar/rss-feeds` admin routes (`@Roles(ADMIN, SUPER_ADMIN)`), declared before `@Get(':id')` per the project's NestJS route-order convention - Web: `tender-radar-api.ts` RSS feed client functions (relaying the backend's specific denylist/SSRF rejection message) + `RssFeedListForm.tsx` rendering an "RSS-Feeds" section on the existing tender-radar settings page (D-09) ## Task Commits 1. **Task 1: RssAdapter.parseFeed + 'rss' normalizer dispatch (fixture-first)** - `3a96cbb` (feat) 2. **Task 2: TenderRssFeedSource model + CRUD service (denylist guard) + tick-gate + wiring** - `e812738` (feat) 3. **Task 3: RSS feed admin routes + client + RssFeedListForm settings section** - `48a2dc1` (feat) _No separate plan-metadata commit yet — this SUMMARY/STATE/ROADMAP update is the final commit for this plan._ ## Files Created/Modified - `apps/api/src/tenders/adapters/rss.adapter.ts` - `RssAdapter implements TenderSourceAdapter`; pure `parseFeed()` + internal fan-out `fetchTenders()` over active `TenderRssFeedSource` rows (native fetch + AbortController 15s + 10MB size ceiling, catch-per-feed) - `apps/api/src/tenders/adapters/rss.adapter.spec.ts` - Fixture-driven parseFeed tests (both feed shapes) + fetchTenders fan-out/catch-per-feed/size-ceiling tests (18 tests) - `apps/api/src/tenders/__fixtures__/service-bund-feed.xml` / `subreport-elvis-feed.xml` - Live-captured (2026-07-23) real RSS XML, trimmed to a representative item subset (6 and 5 items respectively) - `apps/api/src/tenders/tender.types.ts` - `SourceType` extended with `'rss'` - `apps/api/src/tenders/tender-normalizer.service.ts` - `normalize()` dispatches `'rss'` through `normalizeBag()` - `apps/api/src/tenders/tender-rss-feed.service.ts` - Admin CRUD (`list`/`create`/`remove`) + `assertUrlAllowed()` save-time hostname/SSRF guard - `apps/api/src/tenders/dto/tender-rss-feed.dto.ts` - `TenderRssFeedDto` (coarse `@IsUrl` check; substantive guard lives in the service) - `apps/api/src/tenders/tender-ingestion.service.ts` - `pollDueSources()` `pollGranularity` branch (D-15) - `apps/api/src/tenders/tenders.module.ts` - `RssAdapter`/`TenderRssFeedSourceService` providers, registry registration, poll-config + default-feed seeds - `apps/api/src/tenders/tenders.controller.ts` - `rss-feeds` routes (declared before `:id`) - `apps/api/prisma/schema.prisma` + migration - `TenderRssFeedSource` model, `TenderSourcePollConfig.pollGranularity` column - `apps/web/src/lib/tender-radar-api.ts` - `RssFeedSource`/`CreateRssFeedPayload` types + `listRssFeeds`/`createRssFeed`/`deleteRssFeed` + `extractErrorMessage` helper - `apps/web/src/app/(portal)/modules/tender-radar/settings/components/RssFeedListForm.tsx` + `.test.tsx` - Admin feed-list UI (load/add/remove, inline denylist error) - `apps/web/src/app/(portal)/modules/tender-radar/settings/page.tsx` - "RSS-Feeds" section added below `SourceConfigForm` ## Decisions Made - Trimmed the live-captured service.bund.de fixture (500 items, 360KB) down to the first 6 items (5.3KB) — a representative real-data sample, not a fabricated one, matching the sizing convention of the existing cosinex fixture (20 rows) - Chose a string/IP-literal-based private-host check (no DNS resolution) for the SSRF guard — resolving a hostname at save time would itself be an SSRF-adjacent action and is out of scope; matches the existing T-10-06 pattern's scope - `pollIntervalMin: 60` seeded for the 'rss' poll config for consistency with other seeds, even though the single global cron's actual tick rate is still governed by the doe-opendata config's `setInterval()` call (pre-existing architecture, unchanged this plan) ## Deviations from Plan ### Auto-fixed Issues **1. [Rule 1 - Bug] fast-xml-parser does not decode numeric HTML character references** - **Found during:** Task 1 (writing rss.adapter.ts against the live-captured service.bund.de fixture) - **Issue:** `fast-xml-parser` v5.10.1 only decodes the 5 predefined XML entities (`&` `<` `>` `"` `'`); service.bund.de item titles use raw numeric character references (`Übermittlung...`) rather than CDATA — without a fix, RSS-derived tender titles would literally display `Übermittlung...` to users instead of `Übermittlung...`. - **Fix:** Added `decodeNumericEntities()` (decimal `&#NNN;` and hex `&#xHH;`) applied to the extracted title only (never to ``, which is never read at all — no HTML risk). - **Files modified:** `apps/api/src/tenders/adapters/rss.adapter.ts` - **Verification:** `rss.adapter.spec.ts` asserts the decoded title contains "Ü" and no residual `&#` sequence. - **Committed in:** `3a96cbb` (part of Task 1 commit) **2. [Rule 3 - Blocking] Pre-existing Prisma migration checksum drift blocked `migrate dev`** - **Found during:** Task 2 (running the RSS schema migration against the local dev DB) - **Issue:** `prisma migrate dev` refused to proceed ("The migration `20260722100000_add_tender_notifications` was modified after it was applied... reset the schema"), because the `_prisma_migrations.checksum` column for that pre-existing migration was empty in the DB — a leftover from an earlier out-of-band resolution (see project memory: "Failed Prisma migration resolved and marked as applied", 2026-07-21). `migrate status` reported "up to date" (no pending drift), confirming this was purely a stale-checksum bookkeeping issue, not an actual schema mismatch. - **Fix:** Verified via `migrate status` that no real drift existed, then updated the recorded checksum in `_prisma_migrations` to match the on-disk migration file's sha256 (no `migrate reset`, no data loss) before re-running `migrate dev` for this plan's new migration. - **Files modified:** none (DB-only fix, no source file changes) - **Verification:** `prisma migrate dev --name add_rss_feed_source_and_poll_granularity` then succeeded cleanly; `prisma validate` passes; `\d "TenderRssFeedSource"` / `\d "TenderSourcePollConfig"` confirmed the new schema live. - **Committed in:** N/A (database-only, no commit) --- **Total deviations:** 2 auto-fixed (1 Rule 1 bug fix, 1 Rule 3 blocking-issue fix) **Impact on plan:** Both fixes were necessary for correctness (Rule 1: user-visible garbled titles) or to unblock the required local migration (Rule 3: pre-existing DB bookkeeping issue, unrelated to this plan's schema changes). No scope creep. ## Issues Encountered None beyond the two deviations documented above. ## User Setup Required None - no external service configuration required. The migration was applied locally per the project's documented convention (host → container IP `172.19.0.2`, `tessera:tessera_dev`); no Docker rebuild/restart was performed (per CLAUDE.md instruction). ## Next Phase Readiness - The `pollGranularity` mechanism (D-15) is now proven end-to-end and ready for Plan 14-03 (email-alert ingestion) to reuse for its own tick-driven polling need, as anticipated by this plan's purpose statement. - Full API suite: 343/343 tests passing (29 files). Full web suite: 139/139 tests passing (24 files). Both `tsc --noEmit` clean. - No blockers for Plan 14-03/14-04/14-05. --- *Phase: 14-rss-email-alert-ingestion-module-rollout* *Completed: 2026-07-23* ## Self-Check: PASSED All created files verified present on disk; all 3 task commit hashes (3a96cbb, e812738, 48a2dc1) verified present in git history.