Ausschreibungs-Radar (v1.1) STACK/FEATURES/ARCHITECTURE/PITFALLS research plus SUMMARY.md synthesis.
40 KiB
Architecture Research — Ausschreibungs-Radar Module Integration
Domain: Multi-source tender/procurement-notice ingestion module for an existing NestJS 11 modular-monolith platform (Tessera)
Researched: 2026-07-17
Confidence: HIGH (module registration, tenant scoping, inbox/mail reuse — verified against existing dkv/ and module-registry/ source) / MEDIUM (OCDS field mapping, DÖE API pagination — verified via OCDS spec + DÖE Swagger listing, not a live API call)
Note: This supersedes the v1.0 platform-level
ARCHITECTURE.md(2026-06-18, described a generic Fastify/Traefik shell that predates the stack decisions now recorded inCLAUDE.md). This file is scoped to the v1.1 Ausschreibungs-Radar milestone and describes how the new module integrates into the actual, current NestJS 11 + Next.js codebase — not a from-scratch platform design.
Summary
The Ausschreibungs-Radar module ("tenders" module) is a new, self-contained NestJS module built strictly on top of existing Tessera infrastructure — module-registry self-seeding, @nestjs/schedule dynamic cron jobs, the ImapProvider/ExchangeInboxProvider inbox abstraction, tenant-scoped SMTP via SmtpConfig + fresh-transport-per-send, and the module-loader-driven Next.js portal route. The one architectural decision that breaks from the DKV template is deliberate and important: tender data is platform-global, not tenant-owned. DKV invoices belong to one tenant; a DÖE/AI-NetServer/cosinex tender notice is a public fact relevant to every tenant. Ingestion, normalization, and dedup therefore run once for the whole platform; only saved searches, matches, and notification preferences are tenant-scoped. Getting this split right is the single highest-leverage decision in this design — getting it wrong means N-times redundant scraping/storage and N-times the anti-bot exposure per portal.
Standard Architecture
System Overview
+---------------------------------------------------------------------------+
| SOURCE ADAPTERS (new) |
| +-----------+ +--------------+ +-----------+ +--------+ +--------------+|
| |DoeOpenData| |AiNetServer | |Cosinex | |Rss | |EmailAlert ||
| |Adapter | |Adapter | |Adapter | |Adapter | |Adapter ||
| |(API,auth- | |(HTML scrape, | |(HTML | |(feed | |(reuses Imap/ ||
| | free) | | public | | scrape, | | parse) | | Exchange ||
| | | | search) | | public) | | | | InboxProvider||
| +-----+-----+ +------+-------+ +-----+-----+ +---+----+ +------+-------+|
| | all implement TenderSourceAdapter -> RawTenderRecord[] | |
+--------+--------------+---------------+-----------+-----------------+---+
`--------------`-------+-------`-----------`-------------'
v
+---------------------------------------------------------------------------+
| TenderIngestionService (new, global -- no tenantId) |
| normalize(raw, sourceType) -> Tender (OCDS-oriented) |
| computeDedupKey() -> upsert by dedupKey -> diff contentHash -> mark changed|
+-------------------------------+-------------------------------------------+
v (only NEW / CHANGED tenders this poll)
+---------------------------------------------------------------------------+
| TenderMatchingService (new) -- DB-query filter evaluation |
| for each active TenderSavedSearch: Prisma `where` over the delta batch |
| (structured filters) + Postgres full-text search (keywords) -> TenderMatch|
+-------------------------------+-------------------------------------------+
v
+---------------------------------------------------------------------------+
| TenderMailService (new) -- reuses SmtpConfig + fresh-transport |
| instant: send on TenderMatch create digest: cron batch per SavedSearch |
+---------------------------------------------------------------------------+
|
+------------------------+---------------------------------------------------+
| TendersController (new) -- GET /tenders (search+filter+paginate), |
| /tenders/:id, /tenders/saved-searches (CRUD, tenant-scoped), |
| /tenders/source-config (admin, global), /tenders/check-now |
+------------------------+---------------------------------------------------+
v
apps/web/.../modules/tender-radar/ (new Next.js module UI)
Component Responsibilities
| Component | Responsibility | Scope | New/Modified |
|---|---|---|---|
TenderSourceAdapter implementations |
Fetch raw records from one source, no normalization | Global | New |
TenderNormalizerService |
Map each source's raw shape → unified Tender fields, compute dedupKey/contentHash |
Global | New |
TenderIngestionService |
Orchestrate poll → normalize → upsert → change-detect per source | Global | New |
TenderSchedulerService |
Dynamic cron per global source + per-tenant cron for email-alert ingestion | Mixed | New |
TenderMatchingService |
Evaluate active saved searches against the new/changed delta | Per-tenant read, global data | New |
TenderMailService |
Send instant/digest notification emails via tenant SMTP | Per-tenant | New |
TendersController |
REST endpoints for list/detail/saved-search CRUD/admin source-config | Mixed | New |
Shared InboxModule (relocated) |
InboxProvider interface + ImapProvider/ExchangeInboxProvider |
Platform-shared | New (extracted from dkv/) |
ModuleRegistryService |
Self-seed tender-radar module row |
Platform | Reused unmodified |
SettingsService / SmtpConfig |
Decrypted per-tenant SMTP for notification sends | Per-tenant | Reused unmodified |
CalendarCryptoService |
AES-256-GCM encryption for any stored credentials | Platform | Reused unmodified |
module-loader.ts MODULE_REGISTRY |
Whitelist entry mapping slug → lazy component | Platform | Modified (one entry) |
AppModule |
Import TendersModule |
Platform | Modified (one import) |
Recommended Project Structure
apps/api/src/
├── inbox/ # NEW -- extracted shared module (was dkv/providers/)
│ ├── inbox.module.ts # exports ImapProvider, ExchangeInboxProvider
│ ├── inbox-provider.interface.ts # InboxProvider contract (moved from dkv.types.ts)
│ ├── imap.provider.ts # moved verbatim from dkv/providers/
│ ├── exchange-inbox.provider.ts # moved verbatim from dkv/providers/
│ └── inbox.types.ts # InboxConfig / InboxEmail / InboxAttachment
│
├── dkv/ # MODIFIED -- imports InboxModule instead of local providers/
│ └── ... # (providers/ folder removed, dkv.types.ts trimmed)
│
├── tenders/ # NEW -- Ausschreibungs-Radar module
│ ├── tenders.module.ts
│ ├── tenders.controller.ts # public list/detail + tenant saved-search CRUD + admin source-config
│ ├── tenders.seed.ts # seeds 'tender-radar' into ModuleRegistry
│ ├── tender-ingestion.service.ts # poll → normalize → upsert → change-detect (per source)
│ ├── tender-matching.service.ts # SavedSearch → Prisma where-clause → TenderMatch
│ ├── tender-mail.service.ts # instant + digest notification sends
│ ├── tender-scheduler.service.ts # SchedulerRegistry cron: 1 per global source + 1 per tenant (email-alert)
│ ├── tender.types.ts # RawTenderRecord, NormalizedTenderFields, SourceType
│ ├── dto/
│ │ ├── saved-search.dto.ts
│ │ ├── source-config.dto.ts
│ │ └── tender-query.dto.ts # pagination + filter query params for GET /tenders
│ └── adapters/
│ ├── tender-source-adapter.interface.ts # fetchTenders(config, since) → RawTenderRecord[]
│ ├── doe-opendata.adapter.ts # Build order Phase A
│ ├── ai-netserver.adapter.ts # Build order Phase B
│ ├── cosinex.adapter.ts # Build order Phase B
│ ├── rss.adapter.ts # Build order Phase C
│ └── email-alert.adapter.ts # Build order Phase C (uses InboxModule)
│
apps/web/src/app/(portal)/modules/
├── tender-radar/ # follow the existing static per-module convention (dkv-fleet, cert-manager)
│ ├── page.tsx # searchable trefferliste + filter sidebar
│ ├── [id]/page.tsx # tender detail view
│ ├── saved-searches/page.tsx # saved search CRUD UI
│ └── settings/page.tsx # admin: source poll config, email-alert inbox config
Structure Rationale
inbox/extraction is a prerequisite, not optional.ImapProvider/ExchangeInboxProviderare already generic overInboxConfig/InboxEmail— nothing in them is DKV-specific. Today they live indkv/providers/and their types live indkv.types.ts, sotenders/would otherwise have to import from inside another feature module's internals (../dkv/providers/imap.provider), which couples two unrelated features and breaks if DKV is ever restructured. Moving them to a sharedinbox/module once, and updatingdkv.module.tsto importInboxModuleinstead, costs one small refactor now and pays for every future module that needs inbox polling (already two: DKV, Tenders).tenders/adapters/mirrorsdkv/providers/— same rationale as DKV: consumers (TenderIngestionService) depend only on theTenderSourceAdapterinterface, never on a concrete adapter. This is what makes "ship DÖE first, add AI-NetServer/cosinex/RSS/email later" possible without touching the ingestion/matching/notification pipeline.- Normalizer is separate from adapters, unlike DKV where
DkvParserServiceis a single PDF parser. Here there are 5 structurally incompatible raw shapes (eForms/OCDS JSON, two flavors of scraped HTML, RSS/Atom XML, free-text alert emails). Each adapter can either normalize inline or delegate to a per-source mapping function insideTenderNormalizerService— either way, the interface boundary isRawTenderRecord[] → Tender[], so the ingestion orchestrator never branches on source type. - Web module UI follows the existing static
modules/<slug>/folder pattern seen indkv-fleet/andcert-manager/(not the dynamic[category]/[moduleSlug]/route also present in the codebase for vehicle/settings sub-pages) — match whichever of the two conventions the team is actively converging on at execution time; both are already present, so this is a phase-planning decision, not an open architectural question.
Architectural Patterns
Pattern 1: Source-Adapter Abstraction (TenderSourceAdapter)
What: One interface, N implementations — directly analogous to InboxProvider (fetchPdfAttachments → fetchTenders).
// tenders/adapters/tender-source-adapter.interface.ts
export interface RawTenderRecord {
sourceType: SourceType; // 'doe-opendata' | 'ai-netserver' | 'cosinex' | 'rss' | 'email-alert'
sourcePortal: string; // 'doe' | 'lhs-vpbw' | 'tender24' | 'vergabe.landbw' | 'dtvp' | 'subreport-elvis' | 'service.bund.de'
sourceRawId: string; // portal-native id/notice number, pre-normalization
sourceUrl: string;
fetchedAt: Date;
payload: unknown; // raw JSON/HTML-extract/RSS-item/email-body — kept for rawPayload + reprocessing
}
export interface TenderSourceAdapter {
readonly sourceType: SourceType;
/** since: only fetch records new/changed after this timestamp (cursor from TenderSourcePollConfig.lastPolledAt) */
fetchTenders(config: TenderSourceConfig, since?: Date): Promise<RawTenderRecord[]>;
testConnection?(config: TenderSourceConfig): Promise<{ success: boolean; message?: string }>;
}
When to use: Any time a pipeline must ingest structurally different sources into one output shape without the orchestrator knowing about each source. Same pattern Tessera already uses for ImapProvider/ExchangeInboxProvider.
Trade-offs: Each adapter owns its own retry/rate-limit/anti-bot logic (AI-NetServer and cosinex adapters need polite scraping delays; DÖE/RSS don't). The interface intentionally does not prescribe HTTP client or scraping library — TenderSourceConfig is adapter-specific (a discriminated union or Json blob per sourceType), same as InboxConfig covers both IMAP and Exchange with one shape only because both providers happen to share fields; here they mostly won't, so TenderSourceConfig should be Json on the Prisma side with adapter-specific Zod/DTO validation, not a single flat interface.
Pattern 2: Normalized OCDS-Oriented Schema + Cross-Source Dedup Key
What: One Tender Prisma model absorbing eForms/OCDS (DÖE), scraped HTML (AI-NetServer, cosinex), RSS, and email-alert text — with a stable dedup key so the same real-world procurement notice appearing on multiple sources (e.g. an AI-NetServer notice that later also appears on DÖE once it crosses the EU threshold, or the same DÖE OCID reappearing on a poll re-run) collapses to one row.
model Tender {
id String @id @default(uuid())
// OCDS-oriented core (see standard.open-contracting.org/latest/en/schema/reference/)
ocid String? // Open Contracting ID, e.g. "ocds-mnwr74-XXXXXXXX" — present when sourced via DÖE/TED
noticeId String? // portal-native notice/procedure number (AI-NetServer, cosinex, RSS items)
title String
description String? @db.Text
buyerName String?
buyerId String? // e.g. Vergabestelle-ID if the source exposes it
procedureType String? // OCDS tender.procurementMethod / procurementMethodDetails
status String @default("active") // 'active' | 'awarded' | 'cancelled' | 'expired'
cpvCodes String[] @default([])
region String?
plz String?
bundesland String?
estimatedValue Decimal? @db.Decimal(14, 2)
currency String? @default("EUR")
publishedAt DateTime?
deadlineAt DateTime?
// Source + dedup
sourceType String // 'doe-opendata' | 'ai-netserver' | 'cosinex' | 'rss' | 'email-alert'
sourcePortal String // 'doe' | 'lhs-vpbw' | 'dtvp' | 'subreport-elvis' | ...
sourceUrl String?
dedupKey String @unique // see dedup strategy below
contentHash String // hash of normalized fields — detects "changed" vs "identical re-poll"
rawPayload Json // original adapter payload — debugging + future re-normalization
firstSeenAt DateTime @default(now())
lastSeenAt DateTime @updatedAt
createdAt DateTime @default(now())
matches TenderMatch[]
@@index([sourceType])
@@index([deadlineAt])
@@index([publishedAt])
@@index([bundesland])
}
Dedup key strategy (priority order, computed by TenderNormalizerService):
ocid— when the source provides an OCDS Open Contracting ID (always true for DÖE/TED; the platform's registered OCDS prefix isocds-mnwr74). OCID is designed exactly for this — joining the same contracting process across publishers.${sourcePortal}:${noticeId}— when the portal exposes a stable native notice/procedure number (AI-NetServer Bietercockpit ID, cosinex Vergabenummer, RSS item guid, email-alert reference number). This is the primary key for scraped/RSS/email sources, since they never carry an OCID.sha256(sourcePortal + normalizedTitle + buyerName + deadlineAt)— last-resort fallback only when a source gives neither an OCID nor a stable ID (should be rare; flag these rows for manual review via adedupConfidence: 'low'marker if this path is hit).
dedupKey is the @unique upsert target: prisma.tender.upsert({ where: { dedupKey }, ... }). contentHash (hash of title+deadline+value+status) is separate from dedupKey — it answers "did anything about this same notice change since we last saw it" (deadline extension, cancellation), which is what should trigger re-matching and potentially a "notice updated" notification, whereas an unchanged re-poll should just bump lastSeenAt and stop.
Trade-offs: A single wide table is simpler to query/filter/index than per-source tables + a union view, and matches how the UI wants to browse ("one trefferliste across all sources"). The cost is that source-specific fields that don't map cleanly (e.g. cosinex-specific metadata) live only in rawPayload (Json, unindexed) — acceptable, since the feasibility research shows the cross-source overlap (title, buyer, deadline, value, CPV, region) covers what filtering/notification actually need.
Pattern 3: Global Data, Per-Tenant Filtering (the key deviation from the DKV template)
What: Tender rows carry no tenantId — they are platform-wide. Per-tenant scoping happens one layer up, in TenderSavedSearch and TenderMatch.
model TenderSavedSearch {
id String @id @default(uuid())
tenantId String
userId String? // null = tenant-wide search, set = personal search
name String
keywords String[] @default([]) // full-text match against title+description
bundeslaender String[] @default([])
plzPrefixes String[] @default([])
cpvCodes String[] @default([])
minValue Decimal? @db.Decimal(14, 2)
maxValue Decimal? @db.Decimal(14, 2)
deadlineWithinDays Int?
notifyMode String @default("digest") // 'none' | 'digest' | 'instant'
digestHour Int? @default(7) // for digest mode: hour-of-day to send
isActive Boolean @default(true)
createdAt DateTime @default(now())
updatedAt DateTime @updatedAt
matches TenderMatch[]
@@index([tenantId])
}
model TenderMatch {
id String @id @default(uuid())
tenderId String
tender Tender @relation(fields: [tenderId], references: [id], onDelete: Cascade)
savedSearchId String
savedSearch TenderSavedSearch @relation(fields: [savedSearchId], references: [id], onDelete: Cascade)
tenantId String // denormalized for fast tenant-scoped queries/RLS
matchedAt DateTime @default(now())
notifiedAt DateTime? // null = not yet sent (instant) or not yet in a digest
@@unique([tenderId, savedSearchId])
@@index([tenantId])
@@index([notifiedAt])
}
Why this beats a tenantId on Tender: DÖE alone publishes thousands of Oberschwelle notices; duplicating that table N times (once per tenant) multiplies storage for zero benefit — every tenant sees the same underlying notice, just filtered differently. It also means the DÖE/AI-NetServer/cosinex/RSS pollers run once for the whole platform, not once per active tenant — critical for the scraping sources, where running the same scrape N times per tenant multiplies anti-bot/ToS exposure on portals that already sit at "Niedrig-Mittel" risk per the feasibility research. TenantModuleActivation still gates whether a tenant sees the module at all (standard Tessera marketplace pattern) — but activation controls visibility, not a second data copy.
When this pattern does NOT apply: email-alert ingestion. A tenant's alert emails arrive in that tenant's own mailbox (their own registered "gespeicherte Suche" on a portal) — so TenderInboxConfig (credentials, mirroring DkvModuleConfig.encryptedInboxCreds) is legitimately per-tenant, even though the Tender rows it produces still land in the same global table (deduped against whatever DÖE/AI-NetServer/cosinex already ingested for the same notice).
Pattern 4: Filter Evaluation — DB Query Against the Delta, Not In-Memory Full-Table Scan
What: Filtering happens as a Postgres query scoped to the just-ingested batch, not (a) a full in-memory scan of all tenders per saved search, nor (b) a full re-scan of the entire Tender table on every poll.
// tender-matching.service.ts (sketch)
async matchDelta(newOrChangedTenderIds: string[]): Promise<void> {
const savedSearches = await this.prisma.tenderSavedSearch.findMany({ where: { isActive: true } });
for (const search of savedSearches) {
const where = this._buildWhereClause(search, newOrChangedTenderIds); // structured filters
const matches = await this.prisma.tender.findMany({ where }); // DB does the heavy lifting
// full-text keyword refinement, if keywords present, folded into the same query via
// a raw `to_tsvector('german', title || ' ' || description) @@ plainto_tsquery(...)`
for (const tender of matches) {
await this.prisma.tenderMatch.upsert({
where: { tenderId_savedSearchId: { tenderId: tender.id, savedSearchId: search.id } },
create: { tenderId: tender.id, savedSearchId: search.id, tenantId: search.tenantId },
update: {}, // matchedAt stays as first-seen; re-match is idempotent
});
}
}
}
Why DB, not in-memory: structured filters (CPV array overlap, numeric value range, date range, region) are exactly what Postgres indexes/arrays/range queries are built for, and the corpus (all German public tenders) will reach tens of thousands of rows within the first year — loading that into Node to filter per saved search per tenant does not scale and duplicates work Postgres already does better. Keyword matching specifically should use a Postgres full-text tsvector GIN index on title || description (German text-search config) rather than ILIKE '%term%' scans — ILIKE on a growing table degrades linearly, GIN full-text does not.
Why "scoped to the delta" and not the whole table every poll: re-evaluating every saved search against the entire Tender table on every poll cycle is O(searches × table-size) repeated hourly — wasteful and re-creates matches that already exist (idempotent upsert hides the waste but not the cost). Instead, TenderIngestionService passes the list of tender IDs that were newly created or had a contentHash change in this poll run to TenderMatchingService.matchDelta(), so the query is WHERE id IN (delta) AND <search filters> — bounded by poll batch size (dozens to low hundreds), not table size.
Trade-off: if a saved search is created or edited after a tender was ingested, that tender won't retroactively appear in the search's matches until the next time it's re-touched (re-poll sees no change → no delta → not re-evaluated). Mitigate with an explicit "backfill" action: when a TenderSavedSearch is created/edited, run matchDelta() once against all tenders from the last N days (bounded, on-demand, not a recurring cost) rather than the full historical table.
Pattern 5: Scheduled Polling — Global Cron Per Source + Per-Tenant Cron Only for Email-Alerts
What: Extends DkvSchedulerService's SchedulerRegistry.addCronJob() pattern, but with two distinct job populations:
- Global source jobs (DÖE, RSS feeds, AI-NetServer, cosinex): one cron job per row in
TenderSourcePollConfig(admin-managed, not tenant-scoped), namedtender-poll-${sourceConfigId}. Interval is source-appropriate — DÖE can poll frequently (auth-free API, cheap), scraping sources should poll less often and with jitter to stay polite. - Per-tenant email-alert jobs: one cron job per tenant with an active
TenderInboxConfig, namedtender-inbox-poll-${tenantId}— this is the one place Tessera's existing per-tenant scheduling gap (DKV's scheduler is documented as "v1 single-tenant,findFirst()") must actually be solved properly, since two tenants could each have their own mailbox subscribed to different portal alerts. LoopTenderInboxConfig.findMany({ where: { isActive: true } })ononModuleInitand register one job per row, mirroringsetInterval(intervalMin, tenantId)but keyed by tenant instead of a single global slot. - Digest cron: one platform-wide cron (e.g. hourly) that queries
TenderMatchrows wherenotifiedAt IS NULLand the owningTenderSavedSearch.notifyMode = 'digest'anddigestHourmatches the current hour, groups by tenant+savedSearch, and hands off toTenderMailService.
Change detection: contentHash (see Pattern 2) is the mechanism — TenderIngestionService.upsert() compares the freshly computed hash against the stored one; identical → touch lastSeenAt only, skip matching; different → update fields, recompute contentHash, add to this poll's "changed" delta so it flows through matching/notification again (a deadline extension should re-surface in a saved search, a brand-new notice obviously should).
Pattern 6: Notification Dispatch Reusing SmtpConfig + Fresh-Transport, Not the Global MailerService
What: TenderMailService is built exactly like DkvMailService — nodemailer.createTransport() freshly per send, using SettingsService.getDecryptedSmtpConfig(tenantId) — not the platform's global MailerService/@nestjs-modules/mailer (which is reserved for system emails: password reset, welcome mail, configured once at bootstrap with a static transport). This distinction already exists in the codebase (DkvMailService vs MailService) and should hold here too: tender notifications are tenant-directed business content, and the tenant may have configured their own outbound SMTP relay that differs from the platform's.
- Instant: triggered synchronously (or via a lightweight in-process queue, matching DKV's direct-call style — no message broker in this stack) right after
TenderMatchingServicecreates aTenderMatchwithsavedSearch.notifyMode === 'instant'. - Digest: triggered by the digest cron (Pattern 5), batching all unnotified matches for a tenant+savedSearch into one summary email, then setting
notifiedAton each includedTenderMatch— same "mark as sent" idempotency DKV uses forDkvInvoiceHistory.
Trade-off: reusing per-send transport creation means every notification email opens/closes its own SMTP connection (as DKV already accepts) — fine at Tessera's realistic tender-volume/tenant-count, and it guarantees an admin's SMTP config change takes effect on the very next send without a service restart (same Pitfall-3 mitigation DKV already documents).
Data Flow
Ingestion → Notification Flow
[Cron tick: TenderSchedulerService]
v
[Adapter.fetchTenders(config, since)] -> RawTenderRecord[]
v
[TenderNormalizerService.normalize()] -> { fields, dedupKey, contentHash }
v
[TenderIngestionService.upsert()] -> prisma.tender.upsert({ where: { dedupKey } })
v (only rows that were newly created OR whose contentHash changed)
[TenderMatchingService.matchDelta(deltaIds)]
v (per active TenderSavedSearch, DB-scoped query)
[prisma.tenderMatch.upsert()]
v
+--------------------------+---------------------------+
| notifyMode='instant' | notifyMode='digest' |
v v
[TenderMailService.sendInstant()] [Digest cron batches unnotified matches -> sendDigest()]
v v
[nodemailer via tenant SmtpConfig, fresh transport per send]
Read Flow (Portal UI)
[User opens Ausschreibungs-Radar]
v
GET /tenders?keywords=&bundesland=&cpv=&minValue=&deadlineBefore= (TendersController)
v
prisma.tender.findMany({ where: <same structured-filter builder as TenderMatchingService> })
v
[Trefferliste UI] -> click -> GET /tenders/:id -> [Detail view, incl. rawPayload debug panel for admins]
[User manages saved searches]
v
POST/PUT/DELETE /tenders/saved-searches (tenant-scoped, req.tenantId)
v
prisma.tenderSavedSearch.upsert({ ..., tenantId })
v (on create/edit) -> one-off matchDelta() backfill against recent tenders (Pattern 4 mitigation)
Scaling Considerations
| Scale | Architecture Adjustments |
|---|---|
| Single tenant (current, internal test phase) | Exactly as designed above — global Tender table, one poller per source, no extra work needed even though only one tenant exists yet, because the schema is already tenant-agnostic at the data layer. |
| Multiple tenants, few saved searches each | matchDelta() cost scales with (poll batch size × active saved searches), both small — no changes needed. |
| Many tenants, many saved searches, high tender volume | Add a GIN full-text index on title/description (Pattern 4) before this becomes necessary, not after. If matchDelta() ever becomes a bottleneck, batch saved-search evaluation into a single SQL query per poll (Tender × SavedSearch cross-join filtered in one statement) instead of one query per search — straightforward migration since the where-clause builder is already centralized. |
Scaling Priorities
- First bottleneck: keyword filtering via
ILIKEif the full-text index is skipped in the first slice — fix before it matters, it's a single migration (CREATE INDEX ... USING GIN (to_tsvector('german', title || ' ' || coalesce(description,'')))). - Second bottleneck: AI-NetServer/cosinex scraping adapters getting rate-limited or blocked as tender volume/poll frequency grows — mitigate with per-adapter jittered intervals and respecting any
Retry-After, not a platform-wide fix.
Anti-Patterns
Anti-Pattern 1: Tenant-Scoping the Tender Table Like DKV Data
What people do: Copy the DKV template literally — add tenantId to Tender, run every adapter poll once per active tenant (matching DkvSchedulerService's per-tenant cron intent).
Why it's wrong: Multiplies scraping requests against AI-NetServer/cosinex by tenant count (worse ToS exposure on portals already flagged "Niedrig-Mittel" risk), multiplies storage for identical public data, and makes cross-tenant dedup impossible (the same DÖE notice would need deduping and tenant-duplicating, which is incoherent).
Do this instead: Global Tender table (Pattern 3); tenant scoping lives one layer up in TenderSavedSearch/TenderMatch. Only the email-alert path (genuinely per-tenant mailbox) needs per-tenant scheduling.
Anti-Pattern 2: One Adapter Per Portal Instead of Per Platform
What people do: Build a LhsVpbwAdapter, Tender24Adapter, VergabeLandbwAdapter as three separate classes because they're three separate portal URLs.
Why it's wrong: The feasibility research already established these three (plus many unlisted others) share the same AI AG NetServer fingerprint (/NetServer/…ControllerServlet) — one HTML/DOM shape. Three adapter classes triple the maintenance burden for zero behavioral difference; only the base URL and possibly a search-form parameter differ, which belongs in TenderSourceConfig, not in three code paths.
Do this instead: One AiNetServerAdapter, config-driven per portal instance (base URL + optional search params), same for a future CosinexAdapter covering DTVP and other cosinex Vergabemarktplatz instances.
Anti-Pattern 3: In-Memory Filtering Across the Whole Table
What people do: prisma.tender.findMany() with no where, then .filter() in TypeScript per saved search.
Why it's wrong: Works fine in a demo with 50 rows, degrades badly once DÖE's Oberschwelle backbone plus scraped Unterschwelle notices accumulate over months, and re-does full-table work on every poll instead of scoping to the delta.
Do this instead: Pattern 4 — structured Prisma where + Postgres full-text index, scoped to the newly-changed batch.
Integration Points
External Services
| Service | Integration Pattern | Notes |
|---|---|---|
DÖE OpenData API (oeffentlichevergabe.de) |
Auth-free HTTP GET against the OpenData/Swagger-documented endpoints; paginate; filter by publishedAt/last-poll cursor |
eForms-DE / OCDS ocds-mnwr74 / CSV formats available — prefer the OCDS export for direct field alignment with the Tender schema; ~75% of market value by € per feasibility doc, zero scraping/ToS risk |
| AI AG NetServer portals (lhs-vpbw, tender24, vergabe.landbw, + others) | HTML scrape of the public search result pages (no login required for search) | One adapter, config-driven per instance; feasibility doc rates ToS risk "Niedrig-Mittel" — implement politely (rate limit, honest UA string, cache ETags if offered) |
| cosinex Vergabemarktplatz (DTVP + other Länder instances) | HTML scrape of public search, structurally incompatible with AI-NetServer — separate adapter | Reusable across NRW/BB/NI/RLP cosinex instances per feasibility doc |
| RSS feeds (subreport-elvis, service.bund.de) | Standard RSS/Atom parse (e.g. rss-parser or fast-xml-parser), config-driven feed URL list |
Lowest ToS risk of the scraped sources; service.bund.de rated "Niedrig" |
| Portal email alerts (Unterschwelle long tail, 8 of 10 portals) | Reuses InboxProvider (extracted ImapProvider/ExchangeInboxProvider) — per-tenant mailbox subscribed to each portal's native "gespeicherte Suche" alert |
Requires manual one-time setup per portal (register a saved search on the portal itself); the module only ingests+parses the resulting alert emails, does not create the portal-side saved search |
| TED API v3 | Optional, deprioritized — keyless, EU-wide, largely redundant to DÖE for DE-only coverage | Build order: only if EU-wide coverage becomes a requirement later |
Internal Boundaries
| Boundary | Communication | Notes |
|---|---|---|
TendersModule ↔ ModuleRegistryModule |
Direct DI import, OnModuleInit self-seed (tenders.seed.ts) — identical to dkv.seed.ts |
New module import, no registry changes needed beyond the self-seed call |
TendersModule ↔ InboxModule (new, extracted) |
Direct DI import; EmailAlertAdapter depends on ImapProvider/ExchangeInboxProvider |
Requires the one-time extraction of inbox/ out of dkv/ (see Structure Rationale) |
TendersModule ↔ SettingsModule |
Direct DI import, reused unmodified — SettingsService.getDecryptedSmtpConfig(tenantId) |
Same pattern as DkvModule |
TendersModule ↔ TenantMiddleware/TenantGuard |
req.tenantId extraction for saved-search/notification endpoints only — not for GET /tenders list/detail, which is platform-global read access gated only by TenantModuleActivation (module licensing), not by tenant-owned data |
This is the one controller where "tenant-scoped" and "tenant-gated" genuinely differ — worth flagging explicitly in the phase plan so it isn't implemented as a blanket where: { tenantId } by habit |
TendersController ↔ module-loader.ts (web) |
New MODULE_REGISTRY['tender-radar'] entry, same as cert-manager/dkv-fleet |
One-line addition, whitelist pattern (T-03-09) — must not be skipped or the module page 404s even if activated |
AppModule ↔ TendersModule |
New import in app.module.ts |
One line |
Build Order — Ships DÖE-First as a Usable Slice
Ordered by dependency; each step after step 8 is additive and doesn't touch the pipeline built before it (the point of the adapter abstraction).
Phase A — DÖE-only usable slice (schema + one source + filter + UI + notification, end-to-end):
- Prisma migration:
Tender,TenderSourcePollConfig,TenderSavedSearch,TenderMatch(+ GIN full-text index ontitle/description). TendersModuleskeleton +tenders.seed.ts(module-registry self-seed, category e.g.procurement).TenderSourceAdapterinterface +DoeOpenDataAdapter(auth-free, OCDS-formatted fetch).TenderNormalizerService(OCDS release →Tenderfields,ocid-based dedup key,contentHash).TenderIngestionService(poll → normalize → upsert → change-detect) +TenderSchedulerService(single global cron for DÖE).TenderMatchingService(DB-query filter evaluation, Pattern 4) +TendersController(GET /tenders,GET /tenders/:id, saved-search CRUD).apps/web/.../modules/tender-radar/— trefferliste + filter UI + saved-search management +MODULE_REGISTRYentry. This alone is already a usable, demoable slice — DÖE covers ~75% of market value by € per feasibility doc.TenderMailService(instant + digest, reusingSmtpConfig) + digest cron. Closes the loop on the milestone's "optional E-Mail-Versand" requirement using only the DÖE source.
Phase B — Scraping adapters for the Unterschwelle long tail (pipeline unchanged, adapters only):
9. AiNetServerAdapter (covers lhs-vpbw, tender24, vergabe.landbw + any future AI AG portal via config).
10. CosinexAdapter (covers DTVP; reusable for other cosinex Länder marketplaces later).
Phase C — RSS + email-alert (lowest ROI per feasibility doc, do last):
11. RssAdapter (subreport-elvis, service.bund.de).
12. Extract inbox/ shared module out of dkv/ (prerequisite refactor).
13. TenderInboxConfig (per-tenant, mirrors DkvModuleConfig credential pattern) + EmailAlertAdapter + per-tenant scheduler jobs.
Explicitly out of this build order: TED API v3 (redundant to DÖE for DE-only), vergabe24/aumass (AGB-prohibited scraping — feasibility doc flags these as avoid).
New vs Modified — Explicit Inventory
New:
apps/api/src/tenders/— entire module (controller, services, scheduler, seed, dto/, adapters/, types)apps/api/src/inbox/— extracted shared inbox module- Prisma models:
Tender,TenderSourcePollConfig,TenderSavedSearch,TenderMatch,TenderInboxConfig apps/web/src/app/(portal)/modules/tender-radar/— list, detail, saved-search, settings pages
Modified:
apps/api/prisma/schema.prisma— add the 5 new models + migrationapps/api/src/dkv/—providers/folder removed, importsInboxModuleinstead;dkv.types.tstrimmed ofInboxConfig/InboxEmail/InboxAttachment(moved toinbox/inbox.types.ts)apps/api/src/app.module.ts— importTendersModule(andInboxModuleif not auto-imported viaTendersModule's own imports)apps/web/src/lib/module-loader.ts— add'tender-radar'entry toMODULE_REGISTRY
Explicitly NOT modified: module-registry.service.ts, prisma-tenant.extension.ts (forTenant), mail.module.ts/mail.service.ts (global system mailer stays untouched — tender notifications use the DKV-style per-tenant transport pattern instead), settings.service.ts.
Sources
- Existing codebase (verified by direct read):
apps/api/src/dkv/*,apps/api/src/module-registry/*,apps/api/src/mail/*,apps/api/src/prisma/prisma-tenant.extension.ts,apps/api/src/tenant/tenant.middleware.ts,apps/api/prisma/schema.prisma,apps/web/src/lib/module-loader.ts,apps/web/src/app/(portal)/modules/dkv-fleet/*— HIGH confidence, ground truth. .planning/research/ausschreibungs-portale-feasibility.md(2026-07-16) — portal platform fingerprints, ToS risk ratings, DÖE/TED coverage estimates — HIGH confidence (project's own prior research).- OCDS Release Reference — Open Contracting Data Standard 1.1.5 — core schema fields (ocid, release, tender, parties, buyer) — MEDIUM confidence (public spec, not project-specific).
- OCDS Building Blocks — OCID composition (registered prefix + publisher-chosen process id) — MEDIUM confidence.
- oeffentlichevergabe.de OpenData Swagger UI — confirms
ocds-mnwr74as the registered German federal OCDS prefix (registered 2023-02-06) and CC-Zero licensing — MEDIUM confidence (page requires JS to render full endpoint/pagination detail; exact pagination parameters remain an open verification point, already flagged in the feasibility doc).
Architecture research for: Tessera Ausschreibungs-Radar module (v1.1 milestone) Researched: 2026-07-17