212 lines
15 KiB
Markdown
212 lines
15 KiB
Markdown
---
|
|
phase: 14-rss-email-alert-ingestion-module-rollout
|
|
plan: 02
|
|
subsystem: api
|
|
tags: [nestjs, prisma, fast-xml-parser, rss, ssrf, cron, nextjs]
|
|
|
|
# Dependency graph
|
|
requires:
|
|
- phase: 13-additional-tender-sources-cross-source-dedup
|
|
provides: TenderSourceAdapter interface, SourceRegistry denylist gate, normalizeBag() generic-bag normalizer path, pollDueSources fan-out
|
|
provides:
|
|
- RssAdapter (fast-xml-parser, admin-feed internal fan-out) parsing RSS 2.0 into RawTenderRecord[]
|
|
- 'rss' SourceType + normalizer dispatch through normalizeBag()
|
|
- TenderRssFeedSource Prisma model (global admin-managed feed list) + save-time hostname/SSRF guard service
|
|
- TenderSourcePollConfig.pollGranularity ('day' | 'tick') gate in pollDueSources — RSS bypasses the day-cursor entirely
|
|
- GET/POST/DELETE /modules/tender-radar/rss-feeds admin routes + RssFeedListForm settings UI
|
|
affects: [14-03-email-alert-ingestion, 14-04-admin-config-encryption, 14-05-i18n]
|
|
|
|
# Tech tracking
|
|
tech-stack:
|
|
added: []
|
|
patterns:
|
|
- "RSS parsing reuses the existing fast-xml-parser XMLParser config (removeNSPrefix/ignoreAttributes/attributeNamePrefix) from doe-opendata.adapter.ts — no new dependency"
|
|
- "Internal-fan-out adapter: RssAdapter.fetchTenders() reads all active TenderRssFeedSource rows and fetches each independently (catch-per-feed), same shape as NetServerAdapter's multi-portal loop"
|
|
- "pollGranularity branch in pollDueSources(): 'day' sources keep the byte-unchanged lastIngestedDay gate; 'tick' sources fetch unconditionally every active tick"
|
|
- "Save-time hostname/SSRF guard as a SEPARATE enforcement point from the code-level SourceRegistry denylist — required whenever a source's target is runtime admin input rather than a hardcoded portal"
|
|
|
|
key-files:
|
|
created:
|
|
- apps/api/src/tenders/adapters/rss.adapter.ts
|
|
- apps/api/src/tenders/adapters/rss.adapter.spec.ts
|
|
- apps/api/src/tenders/__fixtures__/service-bund-feed.xml
|
|
- apps/api/src/tenders/__fixtures__/subreport-elvis-feed.xml
|
|
- apps/api/src/tenders/tender-rss-feed.service.ts
|
|
- apps/api/src/tenders/tender-rss-feed.service.spec.ts
|
|
- apps/api/src/tenders/dto/tender-rss-feed.dto.ts
|
|
- apps/api/prisma/migrations/20260723111946_add_rss_feed_source_and_poll_granularity/migration.sql
|
|
- apps/web/src/app/(portal)/modules/tender-radar/settings/components/RssFeedListForm.tsx
|
|
- apps/web/src/app/(portal)/modules/tender-radar/settings/components/RssFeedListForm.test.tsx
|
|
modified:
|
|
- apps/api/src/tenders/tender.types.ts
|
|
- apps/api/src/tenders/tender-normalizer.service.ts
|
|
- apps/api/src/tenders/tender-normalizer.service.spec.ts
|
|
- apps/api/src/tenders/tender-ingestion.service.ts
|
|
- apps/api/src/tenders/tender-ingestion.service.spec.ts
|
|
- apps/api/src/tenders/tenders.module.ts
|
|
- apps/api/src/tenders/tenders.controller.ts
|
|
- apps/api/src/tenders/tenders.controller.spec.ts
|
|
- apps/api/prisma/schema.prisma
|
|
- apps/web/src/lib/tender-radar-api.ts
|
|
- apps/web/src/app/(portal)/modules/tender-radar/settings/page.tsx
|
|
|
|
key-decisions:
|
|
- "fast-xml-parser only decodes the 5 predefined XML entities, not numeric character refs (Ü) — added an explicit decodeNumericEntities() step so service.bund.de titles render correctly instead of leaking raw entity syntax (Rule 1 fix)"
|
|
- "Seeded service.bund.de as an active-by-default TenderRssFeedSource row; zero subreport-elvis rows seeded (no single canonical URL — per-municipality instances, admin adds relevant feeds)"
|
|
- "'rss' TenderSourcePollConfig seeded isActive:true/pollGranularity:'tick' from day one — unlike ai-netserver/cosinex-dtvp's 'framework ready, activation deferred' stance from Phase 13"
|
|
- "Save-time SSRF guard is string/IP-literal-based only, no DNS resolution — matches the existing T-10-06 scope; resolving a hostname to check its IP would itself be an SSRF-adjacent action"
|
|
|
|
patterns-established:
|
|
- "pollGranularity as the mechanism the upcoming email-alert adapter (Plan 14-03) reuses for its own tick-driven polling need"
|
|
|
|
requirements-completed: [INGEST-04, CONFIG-02]
|
|
|
|
coverage:
|
|
- id: D1
|
|
description: "RssAdapter.parseFeed maps live-captured service.bund.de (pubDate present, numeric-entity titles) and subreport-elvis (pubDate absent, CDATA titles) shapes into RawTenderRecord[]; malformed/empty XML and single-item link-loss never throw"
|
|
requirement: INGEST-04
|
|
verification:
|
|
- kind: unit
|
|
ref: "apps/api/src/tenders/adapters/rss.adapter.spec.ts (18 tests)"
|
|
status: pass
|
|
human_judgment: false
|
|
- id: D2
|
|
description: "'rss' sourceType routes through TenderNormalizerService.normalizeBag() with title-through mapping and empty cpvDivisions"
|
|
requirement: INGEST-04
|
|
verification:
|
|
- kind: unit
|
|
ref: "apps/api/src/tenders/tender-normalizer.service.spec.ts (rss describe block, 2 tests)"
|
|
status: pass
|
|
human_judgment: false
|
|
- id: D3
|
|
description: "Admin can add/list/remove global RSS feed URLs; vergabe24/aumass and private/loopback hosts are rejected at save time with a domain-specific 400 message"
|
|
requirement: CONFIG-02
|
|
verification:
|
|
- kind: unit
|
|
ref: "apps/api/src/tenders/tender-rss-feed.service.spec.ts (14 tests)"
|
|
status: pass
|
|
- kind: unit
|
|
ref: "apps/api/src/tenders/tenders.controller.spec.ts (RSS-feeds describe block, 4 tests + 1 route-order test)"
|
|
status: pass
|
|
human_judgment: false
|
|
- id: D4
|
|
description: "RSS is polled every active scheduler tick honoring pollIntervalMin (via the single global cron), NOT gated to once per calendar day like doe-opendata/ai-netserver/cosinex-dtvp"
|
|
requirement: INGEST-04
|
|
verification:
|
|
- kind: unit
|
|
ref: "apps/api/src/tenders/tender-ingestion.service.spec.ts (pollGranularity 'tick' gate describe block, 3 tests)"
|
|
status: pass
|
|
human_judgment: false
|
|
- id: D5
|
|
description: "RssFeedListForm renders an admin-manageable RSS feed list on the tender-radar settings page: load, add (denylist error surfaced inline), remove"
|
|
requirement: CONFIG-02
|
|
verification:
|
|
- kind: unit
|
|
ref: "apps/web/src/app/(portal)/modules/tender-radar/settings/components/RssFeedListForm.test.tsx (6 tests)"
|
|
status: pass
|
|
human_judgment: false
|
|
|
|
duration: 30min
|
|
completed: 2026-07-23
|
|
status: complete
|
|
---
|
|
|
|
# Phase 14 Plan 02: RSS Ingestion Slice Summary
|
|
|
|
**RssAdapter (fast-xml-parser, admin-CRUD feed fan-out) + global TenderRssFeedSource list with a save-time denylist/SSRF guard + pollGranularity='tick' gate so RSS honors its poll interval instead of DÖE's day-cursor**
|
|
|
|
## Performance
|
|
|
|
- **Duration:** ~30 min (including live-fixture capture and a pre-existing migration checksum-drift fix)
|
|
- **Started:** 2026-07-23T13:09:00Z (context reads) / first commit 13:17:02
|
|
- **Completed:** 2026-07-23T13:29:07Z
|
|
- **Tasks:** 3/3 completed
|
|
- **Files modified:** 21 (10 created, 11 modified)
|
|
|
|
## Accomplishments
|
|
|
|
- `RssAdapter.parseFeed()` maps two live-captured RSS shapes (service.bund.de with `pubDate` + numeric-HTML-entity titles; subreport-elvis without `pubDate`, CDATA titles) into `RawTenderRecord[]`, routed through the existing generic `normalizeBag()` normalizer path (D-04/D-05, zero normalizer-specific RSS code)
|
|
- `TenderRssFeedSource` Prisma model (global, no `tenantId`, D-08) with full admin CRUD (`TenderRssFeedSourceService`) gated by a save-time hostname/SSRF guard (D-14) — rejects `DENYLISTED_PORTALS` hostnames (vergabe24/aumass, same constant as the code-level gate), non-http(s) schemes, and private/loopback/link-local hosts
|
|
- `TenderSourcePollConfig.pollGranularity` (`'day' | 'tick'`) added; `pollDueSources()` branches per source — 'day' sources (doe-opendata/ai-netserver/cosinex-dtvp) keep the exact pre-existing `lastIngestedDay` gate, 'tick' sources (rss) fetch unconditionally every active tick (D-15)
|
|
- `RssAdapter` registered in `tenders.module.ts`; 'rss' poll config seeded `isActive: true, pollGranularity: 'tick'`; a default-active `service.bund.de` feed row seeded (zero subreport-elvis rows — no single canonical URL to default to)
|
|
- `GET/POST/DELETE /modules/tender-radar/rss-feeds` admin routes (`@Roles(ADMIN, SUPER_ADMIN)`), declared before `@Get(':id')` per the project's NestJS route-order convention
|
|
- Web: `tender-radar-api.ts` RSS feed client functions (relaying the backend's specific denylist/SSRF rejection message) + `RssFeedListForm.tsx` rendering an "RSS-Feeds" section on the existing tender-radar settings page (D-09)
|
|
|
|
## Task Commits
|
|
|
|
1. **Task 1: RssAdapter.parseFeed + 'rss' normalizer dispatch (fixture-first)** - `3a96cbb` (feat)
|
|
2. **Task 2: TenderRssFeedSource model + CRUD service (denylist guard) + tick-gate + wiring** - `e812738` (feat)
|
|
3. **Task 3: RSS feed admin routes + client + RssFeedListForm settings section** - `48a2dc1` (feat)
|
|
|
|
_No separate plan-metadata commit yet — this SUMMARY/STATE/ROADMAP update is the final commit for this plan._
|
|
|
|
## Files Created/Modified
|
|
|
|
- `apps/api/src/tenders/adapters/rss.adapter.ts` - `RssAdapter implements TenderSourceAdapter`; pure `parseFeed()` + internal fan-out `fetchTenders()` over active `TenderRssFeedSource` rows (native fetch + AbortController 15s + 10MB size ceiling, catch-per-feed)
|
|
- `apps/api/src/tenders/adapters/rss.adapter.spec.ts` - Fixture-driven parseFeed tests (both feed shapes) + fetchTenders fan-out/catch-per-feed/size-ceiling tests (18 tests)
|
|
- `apps/api/src/tenders/__fixtures__/service-bund-feed.xml` / `subreport-elvis-feed.xml` - Live-captured (2026-07-23) real RSS XML, trimmed to a representative item subset (6 and 5 items respectively)
|
|
- `apps/api/src/tenders/tender.types.ts` - `SourceType` extended with `'rss'`
|
|
- `apps/api/src/tenders/tender-normalizer.service.ts` - `normalize()` dispatches `'rss'` through `normalizeBag()`
|
|
- `apps/api/src/tenders/tender-rss-feed.service.ts` - Admin CRUD (`list`/`create`/`remove`) + `assertUrlAllowed()` save-time hostname/SSRF guard
|
|
- `apps/api/src/tenders/dto/tender-rss-feed.dto.ts` - `TenderRssFeedDto` (coarse `@IsUrl` check; substantive guard lives in the service)
|
|
- `apps/api/src/tenders/tender-ingestion.service.ts` - `pollDueSources()` `pollGranularity` branch (D-15)
|
|
- `apps/api/src/tenders/tenders.module.ts` - `RssAdapter`/`TenderRssFeedSourceService` providers, registry registration, poll-config + default-feed seeds
|
|
- `apps/api/src/tenders/tenders.controller.ts` - `rss-feeds` routes (declared before `:id`)
|
|
- `apps/api/prisma/schema.prisma` + migration - `TenderRssFeedSource` model, `TenderSourcePollConfig.pollGranularity` column
|
|
- `apps/web/src/lib/tender-radar-api.ts` - `RssFeedSource`/`CreateRssFeedPayload` types + `listRssFeeds`/`createRssFeed`/`deleteRssFeed` + `extractErrorMessage` helper
|
|
- `apps/web/src/app/(portal)/modules/tender-radar/settings/components/RssFeedListForm.tsx` + `.test.tsx` - Admin feed-list UI (load/add/remove, inline denylist error)
|
|
- `apps/web/src/app/(portal)/modules/tender-radar/settings/page.tsx` - "RSS-Feeds" section added below `SourceConfigForm`
|
|
|
|
## Decisions Made
|
|
|
|
- Trimmed the live-captured service.bund.de fixture (500 items, 360KB) down to the first 6 items (5.3KB) — a representative real-data sample, not a fabricated one, matching the sizing convention of the existing cosinex fixture (20 rows)
|
|
- Chose a string/IP-literal-based private-host check (no DNS resolution) for the SSRF guard — resolving a hostname at save time would itself be an SSRF-adjacent action and is out of scope; matches the existing T-10-06 pattern's scope
|
|
- `pollIntervalMin: 60` seeded for the 'rss' poll config for consistency with other seeds, even though the single global cron's actual tick rate is still governed by the doe-opendata config's `setInterval()` call (pre-existing architecture, unchanged this plan)
|
|
|
|
## Deviations from Plan
|
|
|
|
### Auto-fixed Issues
|
|
|
|
**1. [Rule 1 - Bug] fast-xml-parser does not decode numeric HTML character references**
|
|
- **Found during:** Task 1 (writing rss.adapter.ts against the live-captured service.bund.de fixture)
|
|
- **Issue:** `fast-xml-parser` v5.10.1 only decodes the 5 predefined XML entities (`&` `<` `>` `"` `'`); service.bund.de item titles use raw numeric character references (`Übermittlung...`) rather than CDATA — without a fix, RSS-derived tender titles would literally display `Übermittlung...` to users instead of `Übermittlung...`.
|
|
- **Fix:** Added `decodeNumericEntities()` (decimal `&#NNN;` and hex `&#xHH;`) applied to the extracted title only (never to `<description>`, which is never read at all — no HTML risk).
|
|
- **Files modified:** `apps/api/src/tenders/adapters/rss.adapter.ts`
|
|
- **Verification:** `rss.adapter.spec.ts` asserts the decoded title contains "Ü" and no residual `&#` sequence.
|
|
- **Committed in:** `3a96cbb` (part of Task 1 commit)
|
|
|
|
**2. [Rule 3 - Blocking] Pre-existing Prisma migration checksum drift blocked `migrate dev`**
|
|
- **Found during:** Task 2 (running the RSS schema migration against the local dev DB)
|
|
- **Issue:** `prisma migrate dev` refused to proceed ("The migration `20260722100000_add_tender_notifications` was modified after it was applied... reset the schema"), because the `_prisma_migrations.checksum` column for that pre-existing migration was empty in the DB — a leftover from an earlier out-of-band resolution (see project memory: "Failed Prisma migration resolved and marked as applied", 2026-07-21). `migrate status` reported "up to date" (no pending drift), confirming this was purely a stale-checksum bookkeeping issue, not an actual schema mismatch.
|
|
- **Fix:** Verified via `migrate status` that no real drift existed, then updated the recorded checksum in `_prisma_migrations` to match the on-disk migration file's sha256 (no `migrate reset`, no data loss) before re-running `migrate dev` for this plan's new migration.
|
|
- **Files modified:** none (DB-only fix, no source file changes)
|
|
- **Verification:** `prisma migrate dev --name add_rss_feed_source_and_poll_granularity` then succeeded cleanly; `prisma validate` passes; `\d "TenderRssFeedSource"` / `\d "TenderSourcePollConfig"` confirmed the new schema live.
|
|
- **Committed in:** N/A (database-only, no commit)
|
|
|
|
---
|
|
|
|
**Total deviations:** 2 auto-fixed (1 Rule 1 bug fix, 1 Rule 3 blocking-issue fix)
|
|
**Impact on plan:** Both fixes were necessary for correctness (Rule 1: user-visible garbled titles) or to unblock the required local migration (Rule 3: pre-existing DB bookkeeping issue, unrelated to this plan's schema changes). No scope creep.
|
|
|
|
## Issues Encountered
|
|
|
|
None beyond the two deviations documented above.
|
|
|
|
## User Setup Required
|
|
|
|
None - no external service configuration required. The migration was applied locally per the project's documented convention (host → container IP `172.19.0.2`, `tessera:tessera_dev`); no Docker rebuild/restart was performed (per CLAUDE.md instruction).
|
|
|
|
## Next Phase Readiness
|
|
|
|
- The `pollGranularity` mechanism (D-15) is now proven end-to-end and ready for Plan 14-03 (email-alert ingestion) to reuse for its own tick-driven polling need, as anticipated by this plan's purpose statement.
|
|
- Full API suite: 343/343 tests passing (29 files). Full web suite: 139/139 tests passing (24 files). Both `tsc --noEmit` clean.
|
|
- No blockers for Plan 14-03/14-04/14-05.
|
|
|
|
---
|
|
*Phase: 14-rss-email-alert-ingestion-module-rollout*
|
|
*Completed: 2026-07-23*
|
|
|
|
## Self-Check: PASSED
|
|
|
|
All created files verified present on disk; all 3 task commit hashes (3a96cbb, e812738, 48a2dc1) verified present in git history.
|