docs(14-02): complete RSS ingestion slice plan

2026-07-23 13:33:01 +02:00
parent 48a2dc1026
commit 6e3789a0b0
4 changed files with 238 additions and 22 deletions
@@ -0,0 +1,211 @@
+---
+phase: 14-rss-email-alert-ingestion-module-rollout
+plan: 02
+subsystem: api
+tags: [nestjs, prisma, fast-xml-parser, rss, ssrf, cron, nextjs]
+
+# Dependency graph
+requires:
+  - phase: 13-additional-tender-sources-cross-source-dedup
+    provides: TenderSourceAdapter interface, SourceRegistry denylist gate, normalizeBag() generic-bag normalizer path, pollDueSources fan-out
+provides:
+  - RssAdapter (fast-xml-parser, admin-feed internal fan-out) parsing RSS 2.0 into RawTenderRecord[]
+  - 'rss' SourceType + normalizer dispatch through normalizeBag()
+  - TenderRssFeedSource Prisma model (global admin-managed feed list) + save-time hostname/SSRF guard service
+  - TenderSourcePollConfig.pollGranularity ('day' | 'tick') gate in pollDueSources — RSS bypasses the day-cursor entirely
+  - GET/POST/DELETE /modules/tender-radar/rss-feeds admin routes + RssFeedListForm settings UI
+affects: [14-03-email-alert-ingestion, 14-04-admin-config-encryption, 14-05-i18n]
+
+# Tech tracking
+tech-stack:
+  added: []
+  patterns:
+    - "RSS parsing reuses the existing fast-xml-parser XMLParser config (removeNSPrefix/ignoreAttributes/attributeNamePrefix) from doe-opendata.adapter.ts — no new dependency"
+    - "Internal-fan-out adapter: RssAdapter.fetchTenders() reads all active TenderRssFeedSource rows and fetches each independently (catch-per-feed), same shape as NetServerAdapter's multi-portal loop"
+    - "pollGranularity branch in pollDueSources(): 'day' sources keep the byte-unchanged lastIngestedDay gate; 'tick' sources fetch unconditionally every active tick"
+    - "Save-time hostname/SSRF guard as a SEPARATE enforcement point from the code-level SourceRegistry denylist — required whenever a source's target is runtime admin input rather than a hardcoded portal"
+
+key-files:
+  created:
+    - apps/api/src/tenders/adapters/rss.adapter.ts
+    - apps/api/src/tenders/adapters/rss.adapter.spec.ts
+    - apps/api/src/tenders/__fixtures__/service-bund-feed.xml
+    - apps/api/src/tenders/__fixtures__/subreport-elvis-feed.xml
+    - apps/api/src/tenders/tender-rss-feed.service.ts
+    - apps/api/src/tenders/tender-rss-feed.service.spec.ts
+    - apps/api/src/tenders/dto/tender-rss-feed.dto.ts
+    - apps/api/prisma/migrations/20260723111946_add_rss_feed_source_and_poll_granularity/migration.sql
+    - apps/web/src/app/(portal)/modules/tender-radar/settings/components/RssFeedListForm.tsx
+    - apps/web/src/app/(portal)/modules/tender-radar/settings/components/RssFeedListForm.test.tsx
+  modified:
+    - apps/api/src/tenders/tender.types.ts
+    - apps/api/src/tenders/tender-normalizer.service.ts
+    - apps/api/src/tenders/tender-normalizer.service.spec.ts
+    - apps/api/src/tenders/tender-ingestion.service.ts
+    - apps/api/src/tenders/tender-ingestion.service.spec.ts
+    - apps/api/src/tenders/tenders.module.ts
+    - apps/api/src/tenders/tenders.controller.ts
+    - apps/api/src/tenders/tenders.controller.spec.ts
+    - apps/api/prisma/schema.prisma
+    - apps/web/src/lib/tender-radar-api.ts
+    - apps/web/src/app/(portal)/modules/tender-radar/settings/page.tsx
+
+key-decisions:
+  - "fast-xml-parser only decodes the 5 predefined XML entities, not numeric character refs (&#220;) — added an explicit decodeNumericEntities() step so service.bund.de titles render correctly instead of leaking raw entity syntax (Rule 1 fix)"
+  - "Seeded service.bund.de as an active-by-default TenderRssFeedSource row; zero subreport-elvis rows seeded (no single canonical URL — per-municipality instances, admin adds relevant feeds)"
+  - "'rss' TenderSourcePollConfig seeded isActive:true/pollGranularity:'tick' from day one — unlike ai-netserver/cosinex-dtvp's 'framework ready, activation deferred' stance from Phase 13"
+  - "Save-time SSRF guard is string/IP-literal-based only, no DNS resolution — matches the existing T-10-06 scope; resolving a hostname to check its IP would itself be an SSRF-adjacent action"
+
+patterns-established:
+  - "pollGranularity as the mechanism the upcoming email-alert adapter (Plan 14-03) reuses for its own tick-driven polling need"
+
+requirements-completed: [INGEST-04, CONFIG-02]
+
+coverage:
+  - id: D1
+    description: "RssAdapter.parseFeed maps live-captured service.bund.de (pubDate present, numeric-entity titles) and subreport-elvis (pubDate absent, CDATA titles) shapes into RawTenderRecord[]; malformed/empty XML and single-item link-loss never throw"
+    requirement: INGEST-04
+    verification:
+      - kind: unit
+        ref: "apps/api/src/tenders/adapters/rss.adapter.spec.ts (18 tests)"
+        status: pass
+    human_judgment: false
+  - id: D2
+    description: "'rss' sourceType routes through TenderNormalizerService.normalizeBag() with title-through mapping and empty cpvDivisions"
+    requirement: INGEST-04
+    verification:
+      - kind: unit
+        ref: "apps/api/src/tenders/tender-normalizer.service.spec.ts (rss describe block, 2 tests)"
+        status: pass
+    human_judgment: false
+  - id: D3
+    description: "Admin can add/list/remove global RSS feed URLs; vergabe24/aumass and private/loopback hosts are rejected at save time with a domain-specific 400 message"
+    requirement: CONFIG-02
+    verification:
+      - kind: unit
+        ref: "apps/api/src/tenders/tender-rss-feed.service.spec.ts (14 tests)"
+        status: pass
+      - kind: unit
+        ref: "apps/api/src/tenders/tenders.controller.spec.ts (RSS-feeds describe block, 4 tests + 1 route-order test)"
+        status: pass
+    human_judgment: false
+  - id: D4
+    description: "RSS is polled every active scheduler tick honoring pollIntervalMin (via the single global cron), NOT gated to once per calendar day like doe-opendata/ai-netserver/cosinex-dtvp"
+    requirement: INGEST-04
+    verification:
+      - kind: unit
+        ref: "apps/api/src/tenders/tender-ingestion.service.spec.ts (pollGranularity 'tick' gate describe block, 3 tests)"
+        status: pass
+    human_judgment: false
+  - id: D5
+    description: "RssFeedListForm renders an admin-manageable RSS feed list on the tender-radar settings page: load, add (denylist error surfaced inline), remove"
+    requirement: CONFIG-02
+    verification:
+      - kind: unit
+        ref: "apps/web/src/app/(portal)/modules/tender-radar/settings/components/RssFeedListForm.test.tsx (6 tests)"
+        status: pass
+    human_judgment: false
+
+duration: 30min
+completed: 2026-07-23
+status: complete
+---
+
+# Phase 14 Plan 02: RSS Ingestion Slice Summary
+
+**RssAdapter (fast-xml-parser, admin-CRUD feed fan-out) + global TenderRssFeedSource list with a save-time denylist/SSRF guard + pollGranularity='tick' gate so RSS honors its poll interval instead of DÖE's day-cursor**
+
+## Performance
+
+- **Duration:** ~30 min (including live-fixture capture and a pre-existing migration checksum-drift fix)
+- **Started:** 2026-07-23T13:09:00Z (context reads) / first commit 13:17:02
+- **Completed:** 2026-07-23T13:29:07Z
+- **Tasks:** 3/3 completed
+- **Files modified:** 21 (10 created, 11 modified)
+
+## Accomplishments
+
+- `RssAdapter.parseFeed()` maps two live-captured RSS shapes (service.bund.de with `pubDate` + numeric-HTML-entity titles; subreport-elvis without `pubDate`, CDATA titles) into `RawTenderRecord[]`, routed through the existing generic `normalizeBag()` normalizer path (D-04/D-05, zero normalizer-specific RSS code)
+- `TenderRssFeedSource` Prisma model (global, no `tenantId`, D-08) with full admin CRUD (`TenderRssFeedSourceService`) gated by a save-time hostname/SSRF guard (D-14) — rejects `DENYLISTED_PORTALS` hostnames (vergabe24/aumass, same constant as the code-level gate), non-http(s) schemes, and private/loopback/link-local hosts
+- `TenderSourcePollConfig.pollGranularity` (`'day' | 'tick'`) added; `pollDueSources()` branches per source — 'day' sources (doe-opendata/ai-netserver/cosinex-dtvp) keep the exact pre-existing `lastIngestedDay` gate, 'tick' sources (rss) fetch unconditionally every active tick (D-15)
+- `RssAdapter` registered in `tenders.module.ts`; 'rss' poll config seeded `isActive: true, pollGranularity: 'tick'`; a default-active `service.bund.de` feed row seeded (zero subreport-elvis rows — no single canonical URL to default to)
+- `GET/POST/DELETE /modules/tender-radar/rss-feeds` admin routes (`@Roles(ADMIN, SUPER_ADMIN)`), declared before `@Get(':id')` per the project's NestJS route-order convention
+- Web: `tender-radar-api.ts` RSS feed client functions (relaying the backend's specific denylist/SSRF rejection message) + `RssFeedListForm.tsx` rendering an "RSS-Feeds" section on the existing tender-radar settings page (D-09)
+
+## Task Commits
+
+1. **Task 1: RssAdapter.parseFeed + 'rss' normalizer dispatch (fixture-first)** - `3a96cbb` (feat)
+2. **Task 2: TenderRssFeedSource model + CRUD service (denylist guard) + tick-gate + wiring** - `e812738` (feat)
+3. **Task 3: RSS feed admin routes + client + RssFeedListForm settings section** - `48a2dc1` (feat)
+
+_No separate plan-metadata commit yet — this SUMMARY/STATE/ROADMAP update is the final commit for this plan._
+
+## Files Created/Modified
+
+- `apps/api/src/tenders/adapters/rss.adapter.ts` - `RssAdapter implements TenderSourceAdapter`; pure `parseFeed()` + internal fan-out `fetchTenders()` over active `TenderRssFeedSource` rows (native fetch + AbortController 15s + 10MB size ceiling, catch-per-feed)
+- `apps/api/src/tenders/adapters/rss.adapter.spec.ts` - Fixture-driven parseFeed tests (both feed shapes) + fetchTenders fan-out/catch-per-feed/size-ceiling tests (18 tests)
+- `apps/api/src/tenders/__fixtures__/service-bund-feed.xml` / `subreport-elvis-feed.xml` - Live-captured (2026-07-23) real RSS XML, trimmed to a representative item subset (6 and 5 items respectively)
+- `apps/api/src/tenders/tender.types.ts` - `SourceType` extended with `'rss'`
+- `apps/api/src/tenders/tender-normalizer.service.ts` - `normalize()` dispatches `'rss'` through `normalizeBag()`
+- `apps/api/src/tenders/tender-rss-feed.service.ts` - Admin CRUD (`list`/`create`/`remove`) + `assertUrlAllowed()` save-time hostname/SSRF guard
+- `apps/api/src/tenders/dto/tender-rss-feed.dto.ts` - `TenderRssFeedDto` (coarse `@IsUrl` check; substantive guard lives in the service)
+- `apps/api/src/tenders/tender-ingestion.service.ts` - `pollDueSources()` `pollGranularity` branch (D-15)
+- `apps/api/src/tenders/tenders.module.ts` - `RssAdapter`/`TenderRssFeedSourceService` providers, registry registration, poll-config + default-feed seeds
+- `apps/api/src/tenders/tenders.controller.ts` - `rss-feeds` routes (declared before `:id`)
+- `apps/api/prisma/schema.prisma` + migration - `TenderRssFeedSource` model, `TenderSourcePollConfig.pollGranularity` column
+- `apps/web/src/lib/tender-radar-api.ts` - `RssFeedSource`/`CreateRssFeedPayload` types + `listRssFeeds`/`createRssFeed`/`deleteRssFeed` + `extractErrorMessage` helper
+- `apps/web/src/app/(portal)/modules/tender-radar/settings/components/RssFeedListForm.tsx` + `.test.tsx` - Admin feed-list UI (load/add/remove, inline denylist error)
+- `apps/web/src/app/(portal)/modules/tender-radar/settings/page.tsx` - "RSS-Feeds" section added below `SourceConfigForm`
+
+## Decisions Made
+
+- Trimmed the live-captured service.bund.de fixture (500 items, 360KB) down to the first 6 items (5.3KB) — a representative real-data sample, not a fabricated one, matching the sizing convention of the existing cosinex fixture (20 rows)
+- Chose a string/IP-literal-based private-host check (no DNS resolution) for the SSRF guard — resolving a hostname at save time would itself be an SSRF-adjacent action and is out of scope; matches the existing T-10-06 pattern's scope
+- `pollIntervalMin: 60` seeded for the 'rss' poll config for consistency with other seeds, even though the single global cron's actual tick rate is still governed by the doe-opendata config's `setInterval()` call (pre-existing architecture, unchanged this plan)
+
+## Deviations from Plan
+
+### Auto-fixed Issues
+
+**1. [Rule 1 - Bug] fast-xml-parser does not decode numeric HTML character references**
+- **Found during:** Task 1 (writing rss.adapter.ts against the live-captured service.bund.de fixture)
+- **Issue:** `fast-xml-parser` v5.10.1 only decodes the 5 predefined XML entities (`&amp;` `&lt;` `&gt;` `&quot;` `&apos;`); service.bund.de item titles use raw numeric character references (`&#220;bermittlung...`) rather than CDATA — without a fix, RSS-derived tender titles would literally display `&#220;bermittlung...` to users instead of `Übermittlung...`.
+- **Fix:** Added `decodeNumericEntities()` (decimal `&#NNN;` and hex `&#xHH;`) applied to the extracted title only (never to `<description>`, which is never read at all — no HTML risk).
+- **Files modified:** `apps/api/src/tenders/adapters/rss.adapter.ts`
+- **Verification:** `rss.adapter.spec.ts` asserts the decoded title contains "Ü" and no residual `&#` sequence.
+- **Committed in:** `3a96cbb` (part of Task 1 commit)
+
+**2. [Rule 3 - Blocking] Pre-existing Prisma migration checksum drift blocked `migrate dev`**
+- **Found during:** Task 2 (running the RSS schema migration against the local dev DB)
+- **Issue:** `prisma migrate dev` refused to proceed ("The migration `20260722100000_add_tender_notifications` was modified after it was applied... reset the schema"), because the `_prisma_migrations.checksum` column for that pre-existing migration was empty in the DB — a leftover from an earlier out-of-band resolution (see project memory: "Failed Prisma migration resolved and marked as applied", 2026-07-21). `migrate status` reported "up to date" (no pending drift), confirming this was purely a stale-checksum bookkeeping issue, not an actual schema mismatch.
+- **Fix:** Verified via `migrate status` that no real drift existed, then updated the recorded checksum in `_prisma_migrations` to match the on-disk migration file's sha256 (no `migrate reset`, no data loss) before re-running `migrate dev` for this plan's new migration.
+- **Files modified:** none (DB-only fix, no source file changes)
+- **Verification:** `prisma migrate dev --name add_rss_feed_source_and_poll_granularity` then succeeded cleanly; `prisma validate` passes; `\d "TenderRssFeedSource"` / `\d "TenderSourcePollConfig"` confirmed the new schema live.
+- **Committed in:** N/A (database-only, no commit)
+
+---
+
+**Total deviations:** 2 auto-fixed (1 Rule 1 bug fix, 1 Rule 3 blocking-issue fix)
+**Impact on plan:** Both fixes were necessary for correctness (Rule 1: user-visible garbled titles) or to unblock the required local migration (Rule 3: pre-existing DB bookkeeping issue, unrelated to this plan's schema changes). No scope creep.
+
+## Issues Encountered
+
+None beyond the two deviations documented above.
+
+## User Setup Required
+
+None - no external service configuration required. The migration was applied locally per the project's documented convention (host → container IP `172.19.0.2`, `tessera:tessera_dev`); no Docker rebuild/restart was performed (per CLAUDE.md instruction).
+
+## Next Phase Readiness
+
+- The `pollGranularity` mechanism (D-15) is now proven end-to-end and ready for Plan 14-03 (email-alert ingestion) to reuse for its own tick-driven polling need, as anticipated by this plan's purpose statement.
+- Full API suite: 343/343 tests passing (29 files). Full web suite: 139/139 tests passing (24 files). Both `tsc --noEmit` clean.
+- No blockers for Plan 14-03/14-04/14-05.
+
+---
+*Phase: 14-rss-email-alert-ingestion-module-rollout*
+*Completed: 2026-07-23*
+
+## Self-Check: PASSED
+
+All created files verified present on disk; all 3 task commit hashes (3a96cbb, e812738, 48a2dc1) verified present in git history.