173 lines
16 KiB
Markdown
173 lines
16 KiB
Markdown
---
|
|
phase: 14-rss-email-alert-ingestion-module-rollout
|
|
plan: 03
|
|
subsystem: api,web
|
|
tags: [nestjs, prisma, cheerio, imap, exchange-ews, aes-256-gcm, multi-tenant, nextjs]
|
|
|
|
requires:
|
|
- phase: 14-01
|
|
provides: "shared inbox/ module (InboxProvider.fetchMessages, ImapProvider/ExchangeInboxProvider)"
|
|
- phase: 14-02
|
|
provides: "SourceType union widening pattern, normalizeBag() generic bag dispatch, pollGranularity='tick' mechanism"
|
|
provides:
|
|
- "EmailAlertAdapter: generic link/subject extraction + real per-tenant fan-out over active TenderEmailConfig rows"
|
|
- "TenderEmailConfig Prisma model (per-tenant, encrypted creds) + TenderEmailConfigService admin CRUD"
|
|
- "Tender.ownerTenantId nullable column (D-13) threaded write-side through dedup CREATE"
|
|
- "D-13 read-side visibility filter in buildTenderWhere/listTenders/getTender"
|
|
- "GET/PUT /modules/tender-radar/email-config routes + EmailAlertConfigForm UI"
|
|
affects: [14-04, 14-05, tender-radar-verification]
|
|
|
|
tech-stack:
|
|
added: []
|
|
patterns:
|
|
- "Per-tenant internal fan-out inside a single adapter (findMany({isActive:true}) across ALL tenants, deliberate cross-tenant platform-scheduler read, catch-per-tenant)"
|
|
- "D-13 private-source visibility: nullable ownerTenantId on a platform-global model, OR[null,mine] read filter, CREATE-only write, fail-closed on unresolved requester"
|
|
|
|
key-files:
|
|
created:
|
|
- apps/api/src/tenders/adapters/email-alert.adapter.ts
|
|
- apps/api/src/tenders/adapters/email-alert.adapter.spec.ts
|
|
- apps/api/src/tenders/tender-email-config.service.ts
|
|
- apps/api/src/tenders/tender-email-config.service.spec.ts
|
|
- apps/api/src/tenders/dto/tender-email-config.dto.ts
|
|
- apps/api/prisma/migrations/20260723113917_tender_email_config_owner_tenant_id/migration.sql
|
|
- apps/web/src/app/(portal)/modules/tender-radar/settings/components/EmailAlertConfigForm.tsx
|
|
- apps/web/src/app/(portal)/modules/tender-radar/settings/components/EmailAlertConfigForm.test.tsx
|
|
modified:
|
|
- apps/api/prisma/schema.prisma
|
|
- apps/api/src/tenders/tender.types.ts
|
|
- apps/api/src/tenders/tender-normalizer.service.ts
|
|
- apps/api/src/tenders/tender-normalizer.service.spec.ts
|
|
- apps/api/src/tenders/tender-dedup.service.ts
|
|
- apps/api/src/tenders/tender-dedup.service.spec.ts
|
|
- apps/api/src/tenders/tender-query.builder.ts
|
|
- apps/api/src/tenders/tender-query.builder.spec.ts
|
|
- apps/api/src/tenders/tenders.module.ts
|
|
- apps/api/src/tenders/tenders.controller.ts
|
|
- apps/api/src/tenders/tenders.controller.spec.ts
|
|
- apps/web/src/lib/tender-radar-api.ts
|
|
- apps/web/src/app/(portal)/modules/tender-radar/settings/page.tsx
|
|
|
|
key-decisions:
|
|
- "D-13 read filter fails CLOSED: an unresolved requesting tenant (no auth context) sees ONLY global (ownerTenantId=null) tenders, never an accidental private-tender leak — this is a Rule 2 (missing critical functionality) addition beyond the plan's literal OR-clause spec, since the plan didn't specify the no-context case"
|
|
- "getTender's D-13 guard throws the SAME NotFoundException as a missing id (not a distinct Forbidden), per plan's explicit no-detail-leak requirement"
|
|
- "email-alert TenderSourcePollConfig seeded isActive=false (unlike rss's isActive=true) — no safe default mailbox exists to activate; framework-ready-activation-deferred stance, same as ai-netserver/cosinex-dtvp"
|
|
- "EmailAlertConfigForm omits a Test-Connection button and an Abrufintervall field — no test-connection endpoint was built for this config, and poll cadence is governed by the shared platform-wide TenderSourcePollConfig row, not a per-tenant setting"
|
|
|
|
patterns-established:
|
|
- "TDD RED/GREEN split for Task 1: spec committed first against a moved-away adapter.ts (confirmed failing), then adapter.ts restored (confirmed passing) — two separate commits"
|
|
|
|
requirements-completed: []
|
|
|
|
coverage: []
|
|
|
|
duration: ~70min
|
|
completed: 2026-07-23
|
|
status: incomplete
|
|
---
|
|
|
|
# Phase 14 Plan 03: E-Mail-Alert Ingestion (Tasks 1-3) Summary
|
|
|
|
**Per-tenant encrypted portal-alert mailbox config + EmailAlertAdapter with real IMAP/EWS fan-out, tagging every extracted tender with a private `ownerTenantId` (D-13) that's enforced on both write (dedup CREATE-only) and read (OR[global,mine] filter, fail-closed) — Task 4's live-mailbox human-verify remains open.**
|
|
|
|
## Status: INCOMPLETE — Task 4 (human-verify) is the only remaining item
|
|
|
|
This plan's Tasks 1-3 are fully implemented, tested, and committed. **Task 4 — a `checkpoint:human-verify` requiring a real portal-alert mailbox (including the Exchange/EWS live path) — was deliberately NOT executed**, per explicit scope instructions. No code changes are needed for Task 4; it is a pure operator verification step (see PLAN.md Task 4 for the exact steps) that must be performed once a real mailbox is available.
|
|
|
|
## Performance
|
|
|
|
- **Tasks completed:** 3 of 4 (Task 4 open)
|
|
- **Files modified:** 21 (8 created, 13 modified)
|
|
- **Migration:** `20260723113917_tender_email_config_owner_tenant_id` (applied locally)
|
|
|
|
## Accomplishments
|
|
|
|
- **Task 1 (TDD RED/GREEN):** `EmailAlertAdapter`'s three pure helpers — `extractCandidateLinks` (cheerio `a[href]` extraction with footer-noise filtering and a `MAX_LINKS_PER_EMAIL=5` cap, plaintext-regex fallback when no HTML body), `titleFromEmail` (subject → first body line → fallback), `sourceNoticeIdFor` (sha256 link hash) — with zero portal/sender-domain branching (D-04 restraint, verified by a source-scan test). `SourceType` gained `'email-alert'`; `TenderNormalizerService.normalize()` routes it through the existing `normalizeBag()` path unchanged.
|
|
- **Task 2:** Real per-tenant fan-out in `EmailAlertAdapter.fetchTenders()` — one `prisma.tenderEmailConfig.findMany({where:{isActive:true}})` across ALL tenants (documented, deliberate cross-tenant platform-scheduler read, never `forTenant()`/RLS), per-config credential decryption via `CalendarCryptoService`, protocol-based provider routing (imap/exchange), catch-per-tenant so one broken mailbox never blocks others. New `TenderEmailConfig` Prisma model (per-tenant, `tenantId @unique`, encrypted creds) mirrors `DkvModuleConfig`. New `Tender.ownerTenantId` nullable column (D-13): `RawTenderRecord`/`NormalizedTenderFields` gained an optional `ownerTenantId`, threaded unchanged through `TenderNormalizerService.assemble()`, and written ONLY by `TenderDedupService`'s CREATE branch (never UPDATE — a tender later also seen on a public source is never retroactively hidden).
|
|
- **Task 3:** D-13 read-side visibility — `buildTenderWhere` gained an optional `ownerTenantId` param producing `OR[{ownerTenantId:null},{ownerTenantId:<tenant>}]`; `listTenders` resolves the requesting tenant leniently from the auth context (never throws — degrades to global-only when absent); `getTender` 404s (same exception as a missing id) when a tender's non-null `ownerTenantId` doesn't match the requester. New `GET`/`PUT /modules/tender-radar/email-config` routes (Roles ADMIN/SUPER_ADMIN, tenantId from auth context, declared before `@Get(':id')`). Web: `EmailAlertConfig` type + `fetchEmailConfig`/`saveEmailConfig` client functions, `EmailAlertConfigForm.tsx` (password blank-on-load, T-07-12) added as a new "E-Mail-Alerts" section on the tender-radar settings page.
|
|
|
|
## Task Commits
|
|
|
|
1. **Task 1 RED — failing spec** - `4d6fbb1` (test) — email-alert.adapter.spec.ts committed against a temporarily-removed adapter.ts, confirmed failing (module-not-found).
|
|
2. **Task 1 GREEN — implementation** - `8983231` (feat) — email-alert.adapter.ts + SourceType/normalizer dispatch; spec confirmed passing (14 tests).
|
|
3. **Task 2** - `1be6b15` (feat) — TenderEmailConfig model/migration + TenderEmailConfigService + ownerTenantId write-side (tender.types.ts, tender-normalizer.service.ts, tender-dedup.service.ts) + real EmailAlertAdapter fan-out + tenders.module.ts wiring.
|
|
4. **Task 3** - `48e1252` (feat) — tender-query.builder.ts D-13 filter, tenders.controller.ts email-config routes + visibility guards, web EmailAlertConfigForm + settings page section.
|
|
|
|
**Plan metadata:** *(this SUMMARY's own commit — see final commit below)*
|
|
|
|
_Task 1 followed the TDD RED→GREEN gate sequence exactly (`test(14-03): ...` then `feat(14-03): ...`); Tasks 2-3 are single `feat` commits per the plan's `type="auto"` (non-TDD) declaration._
|
|
|
|
## Files Created/Modified
|
|
|
|
- `apps/api/src/tenders/adapters/email-alert.adapter.ts` - Pure extraction helpers + real per-tenant fan-out `fetchTenders()`
|
|
- `apps/api/src/tenders/adapters/email-alert.adapter.spec.ts` - 21 tests: pure functions + fan-out (routing, decrypt, catch-per-tenant, ownerTenantId tagging)
|
|
- `apps/api/src/tenders/tender-email-config.service.ts` - Safe-select admin CRUD (encrypt-preserve-empty semantics, mirrors DkvService)
|
|
- `apps/api/src/tenders/tender-email-config.service.spec.ts` - 6 tests: round-trip, safe-select, credential-preserve-on-partial-update
|
|
- `apps/api/src/tenders/dto/tender-email-config.dto.ts` - class-validator DTO mirroring DkvConfigDto
|
|
- `apps/api/prisma/schema.prisma` - `TenderEmailConfig` model + `Tender.ownerTenantId` nullable column + index
|
|
- `apps/api/prisma/migrations/20260723113917_tender_email_config_owner_tenant_id/` - Applied locally against `172.19.0.2`
|
|
- `apps/api/src/tenders/tender.types.ts` - `SourceType` +`'email-alert'`; `RawTenderRecord`/`NormalizedTenderFields` + optional `ownerTenantId`
|
|
- `apps/api/src/tenders/tender-normalizer.service.ts` - `'email-alert'` case added to `normalizeBag()` dispatch; `assemble()` passes `ownerTenantId` through
|
|
- `apps/api/src/tenders/tender-dedup.service.ts` - CREATE writes `ownerTenantId ?? null`; UPDATE branch untouched
|
|
- `apps/api/src/tenders/tender-query.builder.ts` - `buildTenderWhere(dto, favIds, ownerTenantId)` — D-13 OR-clause, fail-closed default
|
|
- `apps/api/src/tenders/tenders.module.ts` - Imports `CalendarModule`/`InboxModule`; registers `EmailAlertAdapter`/`TenderEmailConfigService`; seeds `email-alert` poll config (inactive, tick)
|
|
- `apps/api/src/tenders/tenders.controller.ts` - `GET`/`PUT /email-config` routes; `listTenders`/`getTender` D-13 visibility
|
|
- `apps/api/src/tenders/tenders.controller.spec.ts` - +10 tests: email-config wiring + D-13 list/detail visibility proofs
|
|
- `apps/web/src/lib/tender-radar-api.ts` - `EmailAlertConfig` type + `fetchEmailConfig`/`saveEmailConfig`
|
|
- `apps/web/src/app/(portal)/modules/tender-radar/settings/components/EmailAlertConfigForm.tsx` - Per-tenant mailbox config form
|
|
- `apps/web/src/app/(portal)/modules/tender-radar/settings/components/EmailAlertConfigForm.test.tsx` - 5 tests: load/save/password-blank/protocol-switch
|
|
- `apps/web/src/app/(portal)/modules/tender-radar/settings/page.tsx` - New "E-Mail-Alerts" section
|
|
|
|
## Decisions Made
|
|
|
|
- D-13 read filter fails CLOSED when the requesting tenant can't be resolved (no auth context available) — only global tenders are ever returned to an unidentified requester. The plan specified the OR-clause shape for a resolved tenant but was silent on the no-context case; fail-closed is the only safe default for a security-critical visibility boundary (Rule 2).
|
|
- `getTender`'s D-13 guard reuses the exact same `NotFoundException` as a genuinely-missing id — confirmed via test that a mismatched-tenant request gets the identical error, so no existence-leak.
|
|
- `email-alert` poll config seeded `isActive: false` (unlike `rss`'s `isActive: true`) since there's no safe universal default mailbox — matches the "framework ready, activation deferred" stance already used for `ai-netserver`/`cosinex-dtvp`.
|
|
- No Test-Connection button or Abrufintervall field in `EmailAlertConfigForm` — neither exists on the backend for this per-tenant config (poll cadence is the shared platform-wide `TenderSourcePollConfig` row); adding either would have been scope creep beyond what Task 3 specified.
|
|
|
|
## Deviations from Plan
|
|
|
|
### Auto-fixed Issues
|
|
|
|
**1. [Rule 2 - Missing Critical] Added D-13 fail-closed default for unresolved tenant context**
|
|
- **Found during:** Task 3 (`buildTenderWhere`/`listTenders` D-13 filter)
|
|
- **Issue:** The plan specifies the `OR[{ownerTenantId:null},{ownerTenantId:<tenant>}]` shape for a resolved requester, but doesn't address what `buildTenderWhere` should do when no tenant context is resolvable (e.g. a request the auth layer somehow let through without full context).
|
|
- **Fix:** Added an explicit `else` branch: `{ ownerTenantId: null }` only — no private tenders are ever visible to an unidentified requester.
|
|
- **Files modified:** `apps/api/src/tenders/tender-query.builder.ts`
|
|
- **Verification:** `tender-query.builder.spec.ts` — "an unresolved ownerTenantId (no auth context) fails closed to global-only tenders"; `tenders.controller.spec.ts` — "GET / with no resolvable tenant context (no req) fails closed to only the global tender", "GET /:id 404s for an unauthenticated request (no req) to a private tender"
|
|
- **Committed in:** `48e1252` (Task 3 commit)
|
|
|
|
**2. [Rule 2 - Missing Critical] Added D-13 test coverage in `tender-dedup.service.spec.ts` and web `EmailAlertConfigForm.test.tsx`**
|
|
- **Found during:** Task 2 (dedup ownerTenantId write-side) and Task 3 (form component)
|
|
- **Issue:** The plan lists `tender-dedup.service.ts` and `EmailAlertConfigForm.tsx` as files to modify/create, but doesn't explicitly list their spec files among Task 2/3's `<files>`. Given the acceptance criteria explicitly require proving "dedup create writes ownerTenantId; update branch does NOT reference ownerTenantId" and "EmailAlertConfigForm renders in settings", dedicated tests were added to make these claims verifiable rather than asserted only in prose.
|
|
- **Files modified:** `apps/api/src/tenders/tender-dedup.service.spec.ts`, `apps/web/.../EmailAlertConfigForm.test.tsx` (new)
|
|
- **Verification:** Both spec files pass (3 new dedup tests, 5 new form tests).
|
|
- **Committed in:** `1be6b15` (dedup spec), `48e1252` (form spec)
|
|
|
|
---
|
|
|
|
**Total deviations:** 2 auto-fixed (both Rule 2 — missing critical functionality: a security-relevant default and missing verification coverage for explicit acceptance criteria).
|
|
**Impact on plan:** Both auto-fixes are narrowly scoped, additive, and directly required by either D-13's security intent or the plan's own stated acceptance criteria. No scope creep — no test-connection endpoint, no Abrufintervall field, and no i18n conversion were added (all explicitly out of scope per the plan/D-09/CONFIG-03).
|
|
|
|
## Issues Encountered
|
|
|
|
None beyond the deviations above. Local Prisma migration applied cleanly against the running `tessera-ctl-db-1` container (`172.19.0.2`) on the first attempt; no Docker service restart was performed (per project convention).
|
|
|
|
## User Setup Required
|
|
|
|
**Task 4 (live EWS/mailbox verification) requires manual operator action — see PLAN.md's Task 4 for exact steps.** In summary: an operator must configure a real portal-alert mailbox (try the Exchange/EWS path specifically) via Tessera → Ausschreibungs-Radar → Einstellungen → E-Mail-Alerts, activate the `email-alert` poll config, and confirm (a) a real alert email produces a tenant-private tender, (b) a different tenant does NOT see it, (c) the EWS body returns non-empty/non-garbled content. No environment variables or dashboard configuration beyond that form are required — the `CALENDAR_ENCRYPTION_KEY` env var (shared with DKV/SMTP) already exists in this environment.
|
|
|
|
## Next Phase Readiness
|
|
|
|
- **NOT ready to close this plan.** Task 4 (`checkpoint:human-verify`, gate=`blocking`) is the sole remaining item — it requires a real mailbox the operator will provide separately, per the calling instructions. Do not mark INGEST-05's Exchange path production-ready, and do not advance `.planning/STATE.md`'s plan counter, until Task 4 is completed by a follow-up execution against this same PLAN.md.
|
|
- All other acceptance criteria for Tasks 1-3 are met and automated-test-proven: full API suite 387/387 green (up from 343/343 at plan start), full web suite 144/144 green (up from 139/139), both `tsc --noEmit` clean.
|
|
- Migration `20260723113917_tender_email_config_owner_tenant_id` is applied on the local dev DB; existing `Tender` rows retain `ownerTenantId = NULL` (verified via `\d "Tender"` — no backfill, no visibility change for any existing global tender).
|
|
|
|
---
|
|
*Phase: 14-rss-email-alert-ingestion-module-rollout*
|
|
*Completed: 2026-07-23 (Tasks 1-3 only — Task 4 OPEN)*
|
|
|
|
## Self-Check: PASSED
|
|
|
|
All 8 created/key files confirmed present on disk; all 4 task commits (4d6fbb1, 8983231, 1be6b15, 48e1252) confirmed present in git log.
|