# Architecture Research — Ausschreibungs-Radar Module Integration **Domain:** Multi-source tender/procurement-notice ingestion module for an existing NestJS 11 modular-monolith platform (Tessera) **Researched:** 2026-07-17 **Confidence:** HIGH (module registration, tenant scoping, inbox/mail reuse — verified against existing `dkv/` and `module-registry/` source) / MEDIUM (OCDS field mapping, DÖE API pagination — verified via OCDS spec + DÖE Swagger listing, not a live API call) > **Note:** This supersedes the v1.0 platform-level `ARCHITECTURE.md` (2026-06-18, described a generic Fastify/Traefik shell that predates the stack decisions now recorded in `CLAUDE.md`). This file is scoped to the v1.1 Ausschreibungs-Radar milestone and describes how the new module integrates into the actual, current NestJS 11 + Next.js codebase — not a from-scratch platform design. ## Summary The Ausschreibungs-Radar module ("tenders" module) is a **new, self-contained NestJS module** built strictly on top of existing Tessera infrastructure — module-registry self-seeding, `@nestjs/schedule` dynamic cron jobs, the `ImapProvider`/`ExchangeInboxProvider` inbox abstraction, tenant-scoped SMTP via `SmtpConfig` + fresh-transport-per-send, and the module-loader-driven Next.js portal route. The one architectural decision that **breaks from the DKV template** is deliberate and important: **tender data is platform-global, not tenant-owned.** DKV invoices belong to one tenant; a DÖE/AI-NetServer/cosinex tender notice is a public fact relevant to *every* tenant. Ingestion, normalization, and dedup therefore run once for the whole platform; only **saved searches, matches, and notification preferences** are tenant-scoped. Getting this split right is the single highest-leverage decision in this design — getting it wrong means N-times redundant scraping/storage and N-times the anti-bot exposure per portal. ## Standard Architecture ### System Overview ``` +---------------------------------------------------------------------------+ | SOURCE ADAPTERS (new) | | +-----------+ +--------------+ +-----------+ +--------+ +--------------+| | |DoeOpenData| |AiNetServer | |Cosinex | |Rss | |EmailAlert || | |Adapter | |Adapter | |Adapter | |Adapter | |Adapter || | |(API,auth- | |(HTML scrape, | |(HTML | |(feed | |(reuses Imap/ || | | free) | | public | | scrape, | | parse) | | Exchange || | | | | search) | | public) | | | | InboxProvider|| | +-----+-----+ +------+-------+ +-----+-----+ +---+----+ +------+-------+| | | all implement TenderSourceAdapter -> RawTenderRecord[] | | +--------+--------------+---------------+-----------+-----------------+---+ `--------------`-------+-------`-----------`-------------' v +---------------------------------------------------------------------------+ | TenderIngestionService (new, global -- no tenantId) | | normalize(raw, sourceType) -> Tender (OCDS-oriented) | | computeDedupKey() -> upsert by dedupKey -> diff contentHash -> mark changed| +-------------------------------+-------------------------------------------+ v (only NEW / CHANGED tenders this poll) +---------------------------------------------------------------------------+ | TenderMatchingService (new) -- DB-query filter evaluation | | for each active TenderSavedSearch: Prisma `where` over the delta batch | | (structured filters) + Postgres full-text search (keywords) -> TenderMatch| +-------------------------------+-------------------------------------------+ v +---------------------------------------------------------------------------+ | TenderMailService (new) -- reuses SmtpConfig + fresh-transport | | instant: send on TenderMatch create digest: cron batch per SavedSearch | +---------------------------------------------------------------------------+ | +------------------------+---------------------------------------------------+ | TendersController (new) -- GET /tenders (search+filter+paginate), | | /tenders/:id, /tenders/saved-searches (CRUD, tenant-scoped), | | /tenders/source-config (admin, global), /tenders/check-now | +------------------------+---------------------------------------------------+ v apps/web/.../modules/tender-radar/ (new Next.js module UI) ``` ### Component Responsibilities | Component | Responsibility | Scope | New/Modified | |-----------|----------------|-------|---------------| | `TenderSourceAdapter` implementations | Fetch raw records from one source, no normalization | Global | New | | `TenderNormalizerService` | Map each source's raw shape → unified `Tender` fields, compute `dedupKey`/`contentHash` | Global | New | | `TenderIngestionService` | Orchestrate poll → normalize → upsert → change-detect per source | Global | New | | `TenderSchedulerService` | Dynamic cron per global source + per-tenant cron for email-alert ingestion | Mixed | New | | `TenderMatchingService` | Evaluate active saved searches against the new/changed delta | Per-tenant read, global data | New | | `TenderMailService` | Send instant/digest notification emails via tenant SMTP | Per-tenant | New | | `TendersController` | REST endpoints for list/detail/saved-search CRUD/admin source-config | Mixed | New | | Shared `InboxModule` (relocated) | `InboxProvider` interface + `ImapProvider`/`ExchangeInboxProvider` | Platform-shared | New (extracted from `dkv/`) | | `ModuleRegistryService` | Self-seed `tender-radar` module row | Platform | Reused unmodified | | `SettingsService` / `SmtpConfig` | Decrypted per-tenant SMTP for notification sends | Per-tenant | Reused unmodified | | `CalendarCryptoService` | AES-256-GCM encryption for any stored credentials | Platform | Reused unmodified | | `module-loader.ts` `MODULE_REGISTRY` | Whitelist entry mapping slug → lazy component | Platform | Modified (one entry) | | `AppModule` | Import `TendersModule` | Platform | Modified (one import) | ## Recommended Project Structure ``` apps/api/src/ ├── inbox/ # NEW -- extracted shared module (was dkv/providers/) │ ├── inbox.module.ts # exports ImapProvider, ExchangeInboxProvider │ ├── inbox-provider.interface.ts # InboxProvider contract (moved from dkv.types.ts) │ ├── imap.provider.ts # moved verbatim from dkv/providers/ │ ├── exchange-inbox.provider.ts # moved verbatim from dkv/providers/ │ └── inbox.types.ts # InboxConfig / InboxEmail / InboxAttachment │ ├── dkv/ # MODIFIED -- imports InboxModule instead of local providers/ │ └── ... # (providers/ folder removed, dkv.types.ts trimmed) │ ├── tenders/ # NEW -- Ausschreibungs-Radar module │ ├── tenders.module.ts │ ├── tenders.controller.ts # public list/detail + tenant saved-search CRUD + admin source-config │ ├── tenders.seed.ts # seeds 'tender-radar' into ModuleRegistry │ ├── tender-ingestion.service.ts # poll → normalize → upsert → change-detect (per source) │ ├── tender-matching.service.ts # SavedSearch → Prisma where-clause → TenderMatch │ ├── tender-mail.service.ts # instant + digest notification sends │ ├── tender-scheduler.service.ts # SchedulerRegistry cron: 1 per global source + 1 per tenant (email-alert) │ ├── tender.types.ts # RawTenderRecord, NormalizedTenderFields, SourceType │ ├── dto/ │ │ ├── saved-search.dto.ts │ │ ├── source-config.dto.ts │ │ └── tender-query.dto.ts # pagination + filter query params for GET /tenders │ └── adapters/ │ ├── tender-source-adapter.interface.ts # fetchTenders(config, since) → RawTenderRecord[] │ ├── doe-opendata.adapter.ts # Build order Phase A │ ├── ai-netserver.adapter.ts # Build order Phase B │ ├── cosinex.adapter.ts # Build order Phase B │ ├── rss.adapter.ts # Build order Phase C │ └── email-alert.adapter.ts # Build order Phase C (uses InboxModule) │ apps/web/src/app/(portal)/modules/ ├── tender-radar/ # follow the existing static per-module convention (dkv-fleet, cert-manager) │ ├── page.tsx # searchable trefferliste + filter sidebar │ ├── [id]/page.tsx # tender detail view │ ├── saved-searches/page.tsx # saved search CRUD UI │ └── settings/page.tsx # admin: source poll config, email-alert inbox config ``` ### Structure Rationale - **`inbox/` extraction is a prerequisite, not optional.** `ImapProvider`/`ExchangeInboxProvider` are already generic over `InboxConfig`/`InboxEmail` — nothing in them is DKV-specific. Today they live in `dkv/providers/` and their types live in `dkv.types.ts`, so `tenders/` would otherwise have to import from inside another feature module's internals (`../dkv/providers/imap.provider`), which couples two unrelated features and breaks if DKV is ever restructured. Moving them to a shared `inbox/` module once, and updating `dkv.module.ts` to import `InboxModule` instead, costs one small refactor now and pays for every future module that needs inbox polling (already two: DKV, Tenders). - **`tenders/adapters/` mirrors `dkv/providers/`** — same rationale as DKV: consumers (`TenderIngestionService`) depend only on the `TenderSourceAdapter` interface, never on a concrete adapter. This is what makes "ship DÖE first, add AI-NetServer/cosinex/RSS/email later" possible without touching the ingestion/matching/notification pipeline. - **Normalizer is separate from adapters**, unlike DKV where `DkvParserService` is a single PDF parser. Here there are 5 structurally incompatible raw shapes (eForms/OCDS JSON, two flavors of scraped HTML, RSS/Atom XML, free-text alert emails). Each adapter can either normalize inline or delegate to a per-source mapping function inside `TenderNormalizerService` — either way, the *interface boundary* is `RawTenderRecord[] → Tender[]`, so the ingestion orchestrator never branches on source type. - **Web module UI follows the existing static `modules//` folder pattern** seen in `dkv-fleet/` and `cert-manager/` (not the dynamic `[category]/[moduleSlug]/` route also present in the codebase for vehicle/settings sub-pages) — match whichever of the two conventions the team is actively converging on at execution time; both are already present, so this is a phase-planning decision, not an open architectural question. ## Architectural Patterns ### Pattern 1: Source-Adapter Abstraction (`TenderSourceAdapter`) **What:** One interface, N implementations — directly analogous to `InboxProvider` (`fetchPdfAttachments` → `fetchTenders`). ```typescript // tenders/adapters/tender-source-adapter.interface.ts export interface RawTenderRecord { sourceType: SourceType; // 'doe-opendata' | 'ai-netserver' | 'cosinex' | 'rss' | 'email-alert' sourcePortal: string; // 'doe' | 'lhs-vpbw' | 'tender24' | 'vergabe.landbw' | 'dtvp' | 'subreport-elvis' | 'service.bund.de' sourceRawId: string; // portal-native id/notice number, pre-normalization sourceUrl: string; fetchedAt: Date; payload: unknown; // raw JSON/HTML-extract/RSS-item/email-body — kept for rawPayload + reprocessing } export interface TenderSourceAdapter { readonly sourceType: SourceType; /** since: only fetch records new/changed after this timestamp (cursor from TenderSourcePollConfig.lastPolledAt) */ fetchTenders(config: TenderSourceConfig, since?: Date): Promise; testConnection?(config: TenderSourceConfig): Promise<{ success: boolean; message?: string }>; } ``` **When to use:** Any time a pipeline must ingest structurally different sources into one output shape without the orchestrator knowing about each source. Same pattern Tessera already uses for `ImapProvider`/`ExchangeInboxProvider`. **Trade-offs:** Each adapter owns its own retry/rate-limit/anti-bot logic (AI-NetServer and cosinex adapters need polite scraping delays; DÖE/RSS don't). The interface intentionally does *not* prescribe HTTP client or scraping library — `TenderSourceConfig` is adapter-specific (a discriminated union or `Json` blob per `sourceType`), same as `InboxConfig` covers both IMAP and Exchange with one shape only because both providers happen to share fields; here they mostly won't, so `TenderSourceConfig` should be `Json` on the Prisma side with adapter-specific Zod/DTO validation, not a single flat interface. ### Pattern 2: Normalized OCDS-Oriented Schema + Cross-Source Dedup Key **What:** One `Tender` Prisma model absorbing eForms/OCDS (DÖE), scraped HTML (AI-NetServer, cosinex), RSS, and email-alert text — with a **stable dedup key** so the same real-world procurement notice appearing on multiple sources (e.g. an AI-NetServer notice that later also appears on DÖE once it crosses the EU threshold, or the same DÖE OCID reappearing on a poll re-run) collapses to one row. ```prisma model Tender { id String @id @default(uuid()) // OCDS-oriented core (see standard.open-contracting.org/latest/en/schema/reference/) ocid String? // Open Contracting ID, e.g. "ocds-mnwr74-XXXXXXXX" — present when sourced via DÖE/TED noticeId String? // portal-native notice/procedure number (AI-NetServer, cosinex, RSS items) title String description String? @db.Text buyerName String? buyerId String? // e.g. Vergabestelle-ID if the source exposes it procedureType String? // OCDS tender.procurementMethod / procurementMethodDetails status String @default("active") // 'active' | 'awarded' | 'cancelled' | 'expired' cpvCodes String[] @default([]) region String? plz String? bundesland String? estimatedValue Decimal? @db.Decimal(14, 2) currency String? @default("EUR") publishedAt DateTime? deadlineAt DateTime? // Source + dedup sourceType String // 'doe-opendata' | 'ai-netserver' | 'cosinex' | 'rss' | 'email-alert' sourcePortal String // 'doe' | 'lhs-vpbw' | 'dtvp' | 'subreport-elvis' | ... sourceUrl String? dedupKey String @unique // see dedup strategy below contentHash String // hash of normalized fields — detects "changed" vs "identical re-poll" rawPayload Json // original adapter payload — debugging + future re-normalization firstSeenAt DateTime @default(now()) lastSeenAt DateTime @updatedAt createdAt DateTime @default(now()) matches TenderMatch[] @@index([sourceType]) @@index([deadlineAt]) @@index([publishedAt]) @@index([bundesland]) } ``` **Dedup key strategy (priority order, computed by `TenderNormalizerService`):** 1. `ocid` — when the source provides an OCDS Open Contracting ID (always true for DÖE/TED; the platform's registered OCDS prefix is `ocds-mnwr74`). OCID is designed exactly for this — joining the same contracting process across publishers. 2. `${sourcePortal}:${noticeId}` — when the portal exposes a stable native notice/procedure number (AI-NetServer Bietercockpit ID, cosinex Vergabenummer, RSS item guid, email-alert reference number). This is the *primary* key for scraped/RSS/email sources, since they never carry an OCID. 3. `sha256(sourcePortal + normalizedTitle + buyerName + deadlineAt)` — last-resort fallback only when a source gives neither an OCID nor a stable ID (should be rare; flag these rows for manual review via a `dedupConfidence: 'low'` marker if this path is hit). `dedupKey` is the `@unique` upsert target: `prisma.tender.upsert({ where: { dedupKey }, ... })`. `contentHash` (hash of title+deadline+value+status) is separate from `dedupKey` — it answers "did anything about this same notice change since we last saw it" (deadline extension, cancellation), which is what should trigger re-matching and potentially a "notice updated" notification, whereas an unchanged re-poll should just bump `lastSeenAt` and stop. **Trade-offs:** A single wide table is simpler to query/filter/index than per-source tables + a union view, and matches how the UI wants to browse ("one trefferliste across all sources"). The cost is that source-specific fields that don't map cleanly (e.g. cosinex-specific metadata) live only in `rawPayload` (Json, unindexed) — acceptable, since the feasibility research shows the cross-source overlap (title, buyer, deadline, value, CPV, region) covers what filtering/notification actually need. ### Pattern 3: Global Data, Per-Tenant Filtering (the key deviation from the DKV template) **What:** `Tender` rows carry **no `tenantId`** — they are platform-wide. Per-tenant scoping happens one layer up, in `TenderSavedSearch` and `TenderMatch`. ```prisma model TenderSavedSearch { id String @id @default(uuid()) tenantId String userId String? // null = tenant-wide search, set = personal search name String keywords String[] @default([]) // full-text match against title+description bundeslaender String[] @default([]) plzPrefixes String[] @default([]) cpvCodes String[] @default([]) minValue Decimal? @db.Decimal(14, 2) maxValue Decimal? @db.Decimal(14, 2) deadlineWithinDays Int? notifyMode String @default("digest") // 'none' | 'digest' | 'instant' digestHour Int? @default(7) // for digest mode: hour-of-day to send isActive Boolean @default(true) createdAt DateTime @default(now()) updatedAt DateTime @updatedAt matches TenderMatch[] @@index([tenantId]) } model TenderMatch { id String @id @default(uuid()) tenderId String tender Tender @relation(fields: [tenderId], references: [id], onDelete: Cascade) savedSearchId String savedSearch TenderSavedSearch @relation(fields: [savedSearchId], references: [id], onDelete: Cascade) tenantId String // denormalized for fast tenant-scoped queries/RLS matchedAt DateTime @default(now()) notifiedAt DateTime? // null = not yet sent (instant) or not yet in a digest @@unique([tenderId, savedSearchId]) @@index([tenantId]) @@index([notifiedAt]) } ``` **Why this beats a `tenantId` on `Tender`:** DÖE alone publishes thousands of Oberschwelle notices; duplicating that table N times (once per tenant) multiplies storage for zero benefit — every tenant sees the same underlying notice, just filtered differently. It also means the DÖE/AI-NetServer/cosinex/RSS pollers run **once for the whole platform**, not once per active tenant — critical for the scraping sources, where running the same scrape N times per tenant multiplies anti-bot/ToS exposure on portals that already sit at "Niedrig-Mittel" risk per the feasibility research. `TenantModuleActivation` still gates whether a tenant sees the module at all (standard Tessera marketplace pattern) — but activation controls *visibility*, not a second data copy. **When this pattern does NOT apply:** email-alert ingestion. A tenant's alert emails arrive in *that tenant's own mailbox* (their own registered "gespeicherte Suche" on a portal) — so `TenderInboxConfig` (credentials, mirroring `DkvModuleConfig.encryptedInboxCreds`) is legitimately per-tenant, even though the `Tender` rows it produces still land in the same global table (deduped against whatever DÖE/AI-NetServer/cosinex already ingested for the same notice). ### Pattern 4: Filter Evaluation — DB Query Against the Delta, Not In-Memory Full-Table Scan **What:** Filtering happens as a **Postgres query scoped to the just-ingested batch**, not (a) a full in-memory scan of all tenders per saved search, nor (b) a full re-scan of the entire `Tender` table on every poll. ```typescript // tender-matching.service.ts (sketch) async matchDelta(newOrChangedTenderIds: string[]): Promise { const savedSearches = await this.prisma.tenderSavedSearch.findMany({ where: { isActive: true } }); for (const search of savedSearches) { const where = this._buildWhereClause(search, newOrChangedTenderIds); // structured filters const matches = await this.prisma.tender.findMany({ where }); // DB does the heavy lifting // full-text keyword refinement, if keywords present, folded into the same query via // a raw `to_tsvector('german', title || ' ' || description) @@ plainto_tsquery(...)` for (const tender of matches) { await this.prisma.tenderMatch.upsert({ where: { tenderId_savedSearchId: { tenderId: tender.id, savedSearchId: search.id } }, create: { tenderId: tender.id, savedSearchId: search.id, tenantId: search.tenantId }, update: {}, // matchedAt stays as first-seen; re-match is idempotent }); } } } ``` **Why DB, not in-memory:** structured filters (CPV array overlap, numeric value range, date range, region) are exactly what Postgres indexes/arrays/range queries are built for, and the corpus (all German public tenders) will reach tens of thousands of rows within the first year — loading that into Node to filter per saved search per tenant does not scale and duplicates work Postgres already does better. Keyword matching specifically should use a **Postgres full-text `tsvector` GIN index** on `title || description` (German text-search config) rather than `ILIKE '%term%'` scans — `ILIKE` on a growing table degrades linearly, GIN full-text does not. **Why "scoped to the delta" and not the whole table every poll:** re-evaluating every saved search against the *entire* `Tender` table on every poll cycle is O(searches × table-size) repeated hourly — wasteful and re-creates matches that already exist (idempotent upsert hides the waste but not the cost). Instead, `TenderIngestionService` passes the list of tender IDs that were newly created or had a `contentHash` change in *this* poll run to `TenderMatchingService.matchDelta()`, so the query is `WHERE id IN (delta) AND ` — bounded by poll batch size (dozens to low hundreds), not table size. **Trade-off:** if a saved search is created or edited *after* a tender was ingested, that tender won't retroactively appear in the search's matches until the next time it's re-touched (re-poll sees no change → no delta → not re-evaluated). Mitigate with an explicit "backfill" action: when a `TenderSavedSearch` is created/edited, run `matchDelta()` once against *all* tenders from the last N days (bounded, on-demand, not a recurring cost) rather than the full historical table. ### Pattern 5: Scheduled Polling — Global Cron Per Source + Per-Tenant Cron Only for Email-Alerts **What:** Extends `DkvSchedulerService`'s `SchedulerRegistry.addCronJob()` pattern, but with two distinct job populations: - **Global source jobs** (DÖE, RSS feeds, AI-NetServer, cosinex): one cron job per row in `TenderSourcePollConfig` (admin-managed, not tenant-scoped), named `tender-poll-${sourceConfigId}`. Interval is source-appropriate — DÖE can poll frequently (auth-free API, cheap), scraping sources should poll less often and with jitter to stay polite. - **Per-tenant email-alert jobs**: one cron job per tenant with an active `TenderInboxConfig`, named `tender-inbox-poll-${tenantId}` — this is the one place Tessera's existing per-tenant scheduling gap (DKV's scheduler is documented as "v1 single-tenant, `findFirst()`") must actually be solved properly, since two tenants could each have their own mailbox subscribed to different portal alerts. Loop `TenderInboxConfig.findMany({ where: { isActive: true } })` on `onModuleInit` and register one job per row, mirroring `setInterval(intervalMin, tenantId)` but keyed by tenant instead of a single global slot. - **Digest cron**: one platform-wide cron (e.g. hourly) that queries `TenderMatch` rows where `notifiedAt IS NULL` and the owning `TenderSavedSearch.notifyMode = 'digest'` and `digestHour` matches the current hour, groups by tenant+savedSearch, and hands off to `TenderMailService`. **Change detection:** `contentHash` (see Pattern 2) is the mechanism — `TenderIngestionService.upsert()` compares the freshly computed hash against the stored one; identical → touch `lastSeenAt` only, skip matching; different → update fields, recompute `contentHash`, add to this poll's "changed" delta so it flows through matching/notification again (a deadline extension should re-surface in a saved search, a brand-new notice obviously should). ### Pattern 6: Notification Dispatch Reusing `SmtpConfig` + Fresh-Transport, Not the Global `MailerService` **What:** `TenderMailService` is built exactly like `DkvMailService` — `nodemailer.createTransport()` freshly per send, using `SettingsService.getDecryptedSmtpConfig(tenantId)` — **not** the platform's global `MailerService`/`@nestjs-modules/mailer` (which is reserved for system emails: password reset, welcome mail, configured once at bootstrap with a static transport). This distinction already exists in the codebase (`DkvMailService` vs `MailService`) and should hold here too: tender notifications are tenant-directed business content, and the tenant may have configured their own outbound SMTP relay that differs from the platform's. - **Instant:** triggered synchronously (or via a lightweight in-process queue, matching DKV's direct-call style — no message broker in this stack) right after `TenderMatchingService` creates a `TenderMatch` with `savedSearch.notifyMode === 'instant'`. - **Digest:** triggered by the digest cron (Pattern 5), batching all unnotified matches for a tenant+savedSearch into one summary email, then setting `notifiedAt` on each included `TenderMatch` — same "mark as sent" idempotency DKV uses for `DkvInvoiceHistory`. **Trade-off:** reusing per-send transport creation means every notification email opens/closes its own SMTP connection (as DKV already accepts) — fine at Tessera's realistic tender-volume/tenant-count, and it guarantees an admin's SMTP config change takes effect on the very next send without a service restart (same Pitfall-3 mitigation DKV already documents). ## Data Flow ### Ingestion → Notification Flow ``` [Cron tick: TenderSchedulerService] v [Adapter.fetchTenders(config, since)] -> RawTenderRecord[] v [TenderNormalizerService.normalize()] -> { fields, dedupKey, contentHash } v [TenderIngestionService.upsert()] -> prisma.tender.upsert({ where: { dedupKey } }) v (only rows that were newly created OR whose contentHash changed) [TenderMatchingService.matchDelta(deltaIds)] v (per active TenderSavedSearch, DB-scoped query) [prisma.tenderMatch.upsert()] v +--------------------------+---------------------------+ | notifyMode='instant' | notifyMode='digest' | v v [TenderMailService.sendInstant()] [Digest cron batches unnotified matches -> sendDigest()] v v [nodemailer via tenant SmtpConfig, fresh transport per send] ``` ### Read Flow (Portal UI) ``` [User opens Ausschreibungs-Radar] v GET /tenders?keywords=&bundesland=&cpv=&minValue=&deadlineBefore= (TendersController) v prisma.tender.findMany({ where: }) v [Trefferliste UI] -> click -> GET /tenders/:id -> [Detail view, incl. rawPayload debug panel for admins] [User manages saved searches] v POST/PUT/DELETE /tenders/saved-searches (tenant-scoped, req.tenantId) v prisma.tenderSavedSearch.upsert({ ..., tenantId }) v (on create/edit) -> one-off matchDelta() backfill against recent tenders (Pattern 4 mitigation) ``` ## Scaling Considerations | Scale | Architecture Adjustments | |-------|---------------------------| | Single tenant (current, internal test phase) | Exactly as designed above — global `Tender` table, one poller per source, no extra work needed even though only one tenant exists yet, because the schema is already tenant-agnostic at the data layer. | | Multiple tenants, few saved searches each | `matchDelta()` cost scales with (poll batch size × active saved searches), both small — no changes needed. | | Many tenants, many saved searches, high tender volume | Add a GIN full-text index on `title`/`description` (Pattern 4) before this becomes necessary, not after. If `matchDelta()` ever becomes a bottleneck, batch saved-search evaluation into a single SQL query per poll (`Tender × SavedSearch` cross-join filtered in one statement) instead of one query per search — straightforward migration since the where-clause builder is already centralized. | ### Scaling Priorities 1. **First bottleneck:** keyword filtering via `ILIKE` if the full-text index is skipped in the first slice — fix before it matters, it's a single migration (`CREATE INDEX ... USING GIN (to_tsvector('german', title || ' ' || coalesce(description,'')))`). 2. **Second bottleneck:** AI-NetServer/cosinex scraping adapters getting rate-limited or blocked as tender volume/poll frequency grows — mitigate with per-adapter jittered intervals and respecting any `Retry-After`, not a platform-wide fix. ## Anti-Patterns ### Anti-Pattern 1: Tenant-Scoping the `Tender` Table Like DKV Data **What people do:** Copy the DKV template literally — add `tenantId` to `Tender`, run every adapter poll once per active tenant (matching `DkvSchedulerService`'s per-tenant cron intent). **Why it's wrong:** Multiplies scraping requests against AI-NetServer/cosinex by tenant count (worse ToS exposure on portals already flagged "Niedrig-Mittel" risk), multiplies storage for identical public data, and makes cross-tenant dedup impossible (the same DÖE notice would need deduping *and* tenant-duplicating, which is incoherent). **Do this instead:** Global `Tender` table (Pattern 3); tenant scoping lives one layer up in `TenderSavedSearch`/`TenderMatch`. Only the email-alert path (genuinely per-tenant mailbox) needs per-tenant scheduling. ### Anti-Pattern 2: One Adapter Per Portal Instead of Per Platform **What people do:** Build a `LhsVpbwAdapter`, `Tender24Adapter`, `VergabeLandbwAdapter` as three separate classes because they're three separate portal URLs. **Why it's wrong:** The feasibility research already established these three (plus many unlisted others) share the same AI AG NetServer fingerprint (`/NetServer/…ControllerServlet`) — one HTML/DOM shape. Three adapter classes triple the maintenance burden for zero behavioral difference; only the base URL and possibly a search-form parameter differ, which belongs in `TenderSourceConfig`, not in three code paths. **Do this instead:** One `AiNetServerAdapter`, config-driven per portal instance (base URL + optional search params), same for a future `CosinexAdapter` covering DTVP and other cosinex Vergabemarktplatz instances. ### Anti-Pattern 3: In-Memory Filtering Across the Whole Table **What people do:** `prisma.tender.findMany()` with no `where`, then `.filter()` in TypeScript per saved search. **Why it's wrong:** Works fine in a demo with 50 rows, degrades badly once DÖE's Oberschwelle backbone plus scraped Unterschwelle notices accumulate over months, and re-does full-table work on every poll instead of scoping to the delta. **Do this instead:** Pattern 4 — structured Prisma `where` + Postgres full-text index, scoped to the newly-changed batch. ## Integration Points ### External Services | Service | Integration Pattern | Notes | |---------|----------------------|-------| | DÖE OpenData API (`oeffentlichevergabe.de`) | Auth-free HTTP GET against the OpenData/Swagger-documented endpoints; paginate; filter by `publishedAt`/last-poll cursor | eForms-DE / **OCDS `ocds-mnwr74`** / CSV formats available — prefer the OCDS export for direct field alignment with the `Tender` schema; ~75% of market value by € per feasibility doc, zero scraping/ToS risk | | AI AG NetServer portals (lhs-vpbw, tender24, vergabe.landbw, + others) | HTML scrape of the public search result pages (no login required for search) | One adapter, config-driven per instance; feasibility doc rates ToS risk "Niedrig-Mittel" — implement politely (rate limit, honest UA string, cache ETags if offered) | | cosinex Vergabemarktplatz (DTVP + other Länder instances) | HTML scrape of public search, structurally incompatible with AI-NetServer — separate adapter | Reusable across NRW/BB/NI/RLP cosinex instances per feasibility doc | | RSS feeds (subreport-elvis, service.bund.de) | Standard RSS/Atom parse (e.g. `rss-parser` or `fast-xml-parser`), config-driven feed URL list | Lowest ToS risk of the scraped sources; service.bund.de rated "Niedrig" | | Portal email alerts (Unterschwelle long tail, 8 of 10 portals) | Reuses `InboxProvider` (extracted `ImapProvider`/`ExchangeInboxProvider`) — per-tenant mailbox subscribed to each portal's native "gespeicherte Suche" alert | Requires manual one-time setup per portal (register a saved search on the portal itself); the module only ingests+parses the resulting alert emails, does not create the portal-side saved search | | TED API v3 | Optional, deprioritized — keyless, EU-wide, largely redundant to DÖE for DE-only coverage | Build order: only if EU-wide coverage becomes a requirement later | ### Internal Boundaries | Boundary | Communication | Notes | |----------|----------------|-------| | `TendersModule` ↔ `ModuleRegistryModule` | Direct DI import, `OnModuleInit` self-seed (`tenders.seed.ts`) — identical to `dkv.seed.ts` | New module import, no registry changes needed beyond the self-seed call | | `TendersModule` ↔ `InboxModule` (new, extracted) | Direct DI import; `EmailAlertAdapter` depends on `ImapProvider`/`ExchangeInboxProvider` | Requires the one-time extraction of `inbox/` out of `dkv/` (see Structure Rationale) | | `TendersModule` ↔ `SettingsModule` | Direct DI import, reused unmodified — `SettingsService.getDecryptedSmtpConfig(tenantId)` | Same pattern as `DkvModule` | | `TendersModule` ↔ `TenantMiddleware`/`TenantGuard` | `req.tenantId` extraction for saved-search/notification endpoints only — **not** for `GET /tenders` list/detail, which is platform-global read access gated only by `TenantModuleActivation` (module licensing), not by tenant-owned data | This is the one controller where "tenant-scoped" and "tenant-gated" genuinely differ — worth flagging explicitly in the phase plan so it isn't implemented as a blanket `where: { tenantId }` by habit | | `TendersController` ↔ `module-loader.ts` (web) | New `MODULE_REGISTRY['tender-radar']` entry, same as `cert-manager`/`dkv-fleet` | One-line addition, whitelist pattern (`T-03-09`) — must not be skipped or the module page 404s even if activated | | `AppModule` ↔ `TendersModule` | New import in `app.module.ts` | One line | ## Build Order — Ships DÖE-First as a Usable Slice Ordered by dependency; each step after step 8 is additive and doesn't touch the pipeline built before it (the point of the adapter abstraction). **Phase A — DÖE-only usable slice (schema + one source + filter + UI + notification, end-to-end):** 1. Prisma migration: `Tender`, `TenderSourcePollConfig`, `TenderSavedSearch`, `TenderMatch` (+ GIN full-text index on `title`/`description`). 2. `TendersModule` skeleton + `tenders.seed.ts` (module-registry self-seed, category e.g. `procurement`). 3. `TenderSourceAdapter` interface + `DoeOpenDataAdapter` (auth-free, OCDS-formatted fetch). 4. `TenderNormalizerService` (OCDS release → `Tender` fields, `ocid`-based dedup key, `contentHash`). 5. `TenderIngestionService` (poll → normalize → upsert → change-detect) + `TenderSchedulerService` (single global cron for DÖE). 6. `TenderMatchingService` (DB-query filter evaluation, Pattern 4) + `TendersController` (`GET /tenders`, `GET /tenders/:id`, saved-search CRUD). 7. `apps/web/.../modules/tender-radar/` — trefferliste + filter UI + saved-search management + `MODULE_REGISTRY` entry. **This alone is already a usable, demoable slice** — DÖE covers ~75% of market value by € per feasibility doc. 8. `TenderMailService` (instant + digest, reusing `SmtpConfig`) + digest cron. Closes the loop on the milestone's "optional E-Mail-Versand" requirement using only the DÖE source. **Phase B — Scraping adapters for the Unterschwelle long tail (pipeline unchanged, adapters only):** 9. `AiNetServerAdapter` (covers lhs-vpbw, tender24, vergabe.landbw + any future AI AG portal via config). 10. `CosinexAdapter` (covers DTVP; reusable for other cosinex Länder marketplaces later). **Phase C — RSS + email-alert (lowest ROI per feasibility doc, do last):** 11. `RssAdapter` (subreport-elvis, service.bund.de). 12. Extract `inbox/` shared module out of `dkv/` (prerequisite refactor). 13. `TenderInboxConfig` (per-tenant, mirrors `DkvModuleConfig` credential pattern) + `EmailAlertAdapter` + per-tenant scheduler jobs. **Explicitly out of this build order:** TED API v3 (redundant to DÖE for DE-only), vergabe24/aumass (AGB-prohibited scraping — feasibility doc flags these as avoid). ## New vs Modified — Explicit Inventory **New:** - `apps/api/src/tenders/` — entire module (controller, services, scheduler, seed, dto/, adapters/, types) - `apps/api/src/inbox/` — extracted shared inbox module - Prisma models: `Tender`, `TenderSourcePollConfig`, `TenderSavedSearch`, `TenderMatch`, `TenderInboxConfig` - `apps/web/src/app/(portal)/modules/tender-radar/` — list, detail, saved-search, settings pages **Modified:** - `apps/api/prisma/schema.prisma` — add the 5 new models + migration - `apps/api/src/dkv/` — `providers/` folder removed, imports `InboxModule` instead; `dkv.types.ts` trimmed of `InboxConfig`/`InboxEmail`/`InboxAttachment` (moved to `inbox/inbox.types.ts`) - `apps/api/src/app.module.ts` — import `TendersModule` (and `InboxModule` if not auto-imported via `TendersModule`'s own imports) - `apps/web/src/lib/module-loader.ts` — add `'tender-radar'` entry to `MODULE_REGISTRY` **Explicitly NOT modified:** `module-registry.service.ts`, `prisma-tenant.extension.ts` (`forTenant`), `mail.module.ts`/`mail.service.ts` (global system mailer stays untouched — tender notifications use the DKV-style per-tenant transport pattern instead), `settings.service.ts`. ## Sources - Existing codebase (verified by direct read): `apps/api/src/dkv/*`, `apps/api/src/module-registry/*`, `apps/api/src/mail/*`, `apps/api/src/prisma/prisma-tenant.extension.ts`, `apps/api/src/tenant/tenant.middleware.ts`, `apps/api/prisma/schema.prisma`, `apps/web/src/lib/module-loader.ts`, `apps/web/src/app/(portal)/modules/dkv-fleet/*` — HIGH confidence, ground truth. - `.planning/research/ausschreibungs-portale-feasibility.md` (2026-07-16) — portal platform fingerprints, ToS risk ratings, DÖE/TED coverage estimates — HIGH confidence (project's own prior research). - [OCDS Release Reference — Open Contracting Data Standard 1.1.5](https://standard.open-contracting.org/latest/en/schema/reference/) — core schema fields (ocid, release, tender, parties, buyer) — MEDIUM confidence (public spec, not project-specific). - [OCDS Building Blocks](https://standard.open-contracting.org/latest/en/getting_started/building_blocks/) — OCID composition (registered prefix + publisher-chosen process id) — MEDIUM confidence. - [oeffentlichevergabe.de OpenData Swagger UI](https://oeffentlichevergabe.de/documentation/swagger-ui/opendata/index.html) — confirms `ocds-mnwr74` as the registered German federal OCDS prefix (registered 2023-02-06) and CC-Zero licensing — MEDIUM confidence (page requires JS to render full endpoint/pagination detail; exact pagination parameters remain an open verification point, already flagged in the feasibility doc). --- *Architecture research for: Tessera Ausschreibungs-Radar module (v1.1 milestone)* *Researched: 2026-07-17*