docs: complete project research
Ausschreibungs-Radar (v1.1) STACK/FEATURES/ARCHITECTURE/PITFALLS research plus SUMMARY.md synthesis.
This commit is contained in:
+410
-272
@@ -1,320 +1,458 @@
|
||||
# Architecture Patterns
|
||||
# Architecture Research — Ausschreibungs-Radar Module Integration
|
||||
|
||||
**Domain:** Modular portal platform with marketplace, multi-tenancy, configurable dashboard, and desktop wrapper
|
||||
**Researched:** 2026-06-18
|
||||
**Domain:** Multi-source tender/procurement-notice ingestion module for an existing NestJS 11 modular-monolith platform (Tessera)
|
||||
**Researched:** 2026-07-17
|
||||
**Confidence:** HIGH (module registration, tenant scoping, inbox/mail reuse — verified against existing `dkv/` and `module-registry/` source) / MEDIUM (OCDS field mapping, DÖE API pagination — verified via OCDS spec + DÖE Swagger listing, not a live API call)
|
||||
|
||||
## Recommended Architecture
|
||||
> **Note:** This supersedes the v1.0 platform-level `ARCHITECTURE.md` (2026-06-18, described a generic Fastify/Traefik shell that predates the stack decisions now recorded in `CLAUDE.md`). This file is scoped to the v1.1 Ausschreibungs-Radar milestone and describes how the new module integrates into the actual, current NestJS 11 + Next.js codebase — not a from-scratch platform design.
|
||||
|
||||
Tessera follows a **modular monolith** pattern: a single deployable backend that internally separates concerns into distinct modules, fronted by a micro-frontend-capable shell. This avoids premature microservice complexity while maintaining clean module boundaries for future extraction.
|
||||
## Summary
|
||||
|
||||
The Ausschreibungs-Radar module ("tenders" module) is a **new, self-contained NestJS module** built strictly on top of existing Tessera infrastructure — module-registry self-seeding, `@nestjs/schedule` dynamic cron jobs, the `ImapProvider`/`ExchangeInboxProvider` inbox abstraction, tenant-scoped SMTP via `SmtpConfig` + fresh-transport-per-send, and the module-loader-driven Next.js portal route. The one architectural decision that **breaks from the DKV template** is deliberate and important: **tender data is platform-global, not tenant-owned.** DKV invoices belong to one tenant; a DÖE/AI-NetServer/cosinex tender notice is a public fact relevant to *every* tenant. Ingestion, normalization, and dedup therefore run once for the whole platform; only **saved searches, matches, and notification preferences** are tenant-scoped. Getting this split right is the single highest-leverage decision in this design — getting it wrong means N-times redundant scraping/storage and N-times the anti-bot exposure per portal.
|
||||
|
||||
## Standard Architecture
|
||||
|
||||
### System Overview
|
||||
|
||||
```
|
||||
+-------------------------------------------------------------------+
|
||||
| Desktop Wrapper (Tauri) |
|
||||
+-------------------------------------------------------------------+
|
||||
| Frontend Shell (React SPA) |
|
||||
| +------------+ +------------+ +------------+ +-------------+ |
|
||||
| | Sidebar | | Dashboard | | Marketplace| | Module UI | |
|
||||
| | Navigation | | (Widgets) | | Browser | | (lazy-load) | |
|
||||
| +------------+ +------------+ +------------+ +-------------+ |
|
||||
+-------------------------------------------------------------------+
|
||||
| API Gateway (Traefik / Nginx) |
|
||||
+-------------------------------------------------------------------+
|
||||
| Backend (Node.js / Fastify) |
|
||||
| +--------+ +--------+ +--------+ +---------+ +----------+ |
|
||||
| | Auth | | Tenant | | Module | | Market- | | Dashboard| |
|
||||
| | Module | | Module | | Loader | | place | | Service | |
|
||||
| +--------+ +--------+ +--------+ +---------+ +----------+ |
|
||||
+-------------------------------------------------------------------+
|
||||
| PostgreSQL (shared, RLS) |
|
||||
+-------------------------------------------------------------------+
|
||||
| Docker Compose (orchestration) |
|
||||
+-------------------------------------------------------------------+
|
||||
+---------------------------------------------------------------------------+
|
||||
| SOURCE ADAPTERS (new) |
|
||||
| +-----------+ +--------------+ +-----------+ +--------+ +--------------+|
|
||||
| |DoeOpenData| |AiNetServer | |Cosinex | |Rss | |EmailAlert ||
|
||||
| |Adapter | |Adapter | |Adapter | |Adapter | |Adapter ||
|
||||
| |(API,auth- | |(HTML scrape, | |(HTML | |(feed | |(reuses Imap/ ||
|
||||
| | free) | | public | | scrape, | | parse) | | Exchange ||
|
||||
| | | | search) | | public) | | | | InboxProvider||
|
||||
| +-----+-----+ +------+-------+ +-----+-----+ +---+----+ +------+-------+|
|
||||
| | all implement TenderSourceAdapter -> RawTenderRecord[] | |
|
||||
+--------+--------------+---------------+-----------+-----------------+---+
|
||||
`--------------`-------+-------`-----------`-------------'
|
||||
v
|
||||
+---------------------------------------------------------------------------+
|
||||
| TenderIngestionService (new, global -- no tenantId) |
|
||||
| normalize(raw, sourceType) -> Tender (OCDS-oriented) |
|
||||
| computeDedupKey() -> upsert by dedupKey -> diff contentHash -> mark changed|
|
||||
+-------------------------------+-------------------------------------------+
|
||||
v (only NEW / CHANGED tenders this poll)
|
||||
+---------------------------------------------------------------------------+
|
||||
| TenderMatchingService (new) -- DB-query filter evaluation |
|
||||
| for each active TenderSavedSearch: Prisma `where` over the delta batch |
|
||||
| (structured filters) + Postgres full-text search (keywords) -> TenderMatch|
|
||||
+-------------------------------+-------------------------------------------+
|
||||
v
|
||||
+---------------------------------------------------------------------------+
|
||||
| TenderMailService (new) -- reuses SmtpConfig + fresh-transport |
|
||||
| instant: send on TenderMatch create digest: cron batch per SavedSearch |
|
||||
+---------------------------------------------------------------------------+
|
||||
|
|
||||
+------------------------+---------------------------------------------------+
|
||||
| TendersController (new) -- GET /tenders (search+filter+paginate), |
|
||||
| /tenders/:id, /tenders/saved-searches (CRUD, tenant-scoped), |
|
||||
| /tenders/source-config (admin, global), /tenders/check-now |
|
||||
+------------------------+---------------------------------------------------+
|
||||
v
|
||||
apps/web/.../modules/tender-radar/ (new Next.js module UI)
|
||||
```
|
||||
|
||||
### Component Boundaries
|
||||
### Component Responsibilities
|
||||
|
||||
| Component | Responsibility | Communicates With |
|
||||
|-----------|---------------|-------------------|
|
||||
| **Frontend Shell** | Application frame (header, sidebar, routing), theme, i18n | Backend API via REST/WebSocket |
|
||||
| **Dashboard Engine** | Widget grid, drag-and-drop, layout persistence | Backend Dashboard Service for saving layouts |
|
||||
| **Marketplace UI** | Browse modules, view details, request activation | Backend Marketplace Service |
|
||||
| **Module UI Slots** | Lazy-loaded UI for activated modules | Module-specific backend endpoints |
|
||||
| **API Gateway** | Reverse proxy, rate limiting, tenant header injection | All backend services |
|
||||
| **Auth Module** | Login, session/JWT, LDAP integration, user management | PostgreSQL, LDAP server |
|
||||
| **Tenant Module** | Tenant CRUD, tenant context resolution, tenant-specific config | PostgreSQL, injected into every request |
|
||||
| **Module Loader** | Plugin lifecycle (discover, validate, activate, deactivate) | Filesystem/registry, PostgreSQL |
|
||||
| **Marketplace Service** | Module catalog, licensing, activation per tenant | PostgreSQL, Module Loader |
|
||||
| **Dashboard Service** | Widget registry, layout CRUD per user per tenant | PostgreSQL |
|
||||
| **PostgreSQL** | Persistent storage, row-level security for tenant isolation | All backend modules |
|
||||
| **Desktop Wrapper (Tauri)** | Native window, system tray, local shortcuts | Frontend Shell (wraps the web app) |
|
||||
| Component | Responsibility | Scope | New/Modified |
|
||||
|-----------|----------------|-------|---------------|
|
||||
| `TenderSourceAdapter` implementations | Fetch raw records from one source, no normalization | Global | New |
|
||||
| `TenderNormalizerService` | Map each source's raw shape → unified `Tender` fields, compute `dedupKey`/`contentHash` | Global | New |
|
||||
| `TenderIngestionService` | Orchestrate poll → normalize → upsert → change-detect per source | Global | New |
|
||||
| `TenderSchedulerService` | Dynamic cron per global source + per-tenant cron for email-alert ingestion | Mixed | New |
|
||||
| `TenderMatchingService` | Evaluate active saved searches against the new/changed delta | Per-tenant read, global data | New |
|
||||
| `TenderMailService` | Send instant/digest notification emails via tenant SMTP | Per-tenant | New |
|
||||
| `TendersController` | REST endpoints for list/detail/saved-search CRUD/admin source-config | Mixed | New |
|
||||
| Shared `InboxModule` (relocated) | `InboxProvider` interface + `ImapProvider`/`ExchangeInboxProvider` | Platform-shared | New (extracted from `dkv/`) |
|
||||
| `ModuleRegistryService` | Self-seed `tender-radar` module row | Platform | Reused unmodified |
|
||||
| `SettingsService` / `SmtpConfig` | Decrypted per-tenant SMTP for notification sends | Per-tenant | Reused unmodified |
|
||||
| `CalendarCryptoService` | AES-256-GCM encryption for any stored credentials | Platform | Reused unmodified |
|
||||
| `module-loader.ts` `MODULE_REGISTRY` | Whitelist entry mapping slug → lazy component | Platform | Modified (one entry) |
|
||||
| `AppModule` | Import `TendersModule` | Platform | Modified (one import) |
|
||||
|
||||
### Data Flow
|
||||
|
||||
**Request flow (authenticated):**
|
||||
## Recommended Project Structure
|
||||
|
||||
```
|
||||
User Action
|
||||
-> Desktop Wrapper / Browser
|
||||
-> Frontend Shell (React Router)
|
||||
-> HTTP Request with JWT + Tenant-ID header
|
||||
-> API Gateway (validates JWT, injects tenant context)
|
||||
-> Backend Route Handler
|
||||
-> Service Layer (business logic)
|
||||
-> PostgreSQL (RLS enforces tenant isolation)
|
||||
<- Response
|
||||
<- JSON Response
|
||||
<- Frontend renders
|
||||
apps/api/src/
|
||||
├── inbox/ # NEW -- extracted shared module (was dkv/providers/)
|
||||
│ ├── inbox.module.ts # exports ImapProvider, ExchangeInboxProvider
|
||||
│ ├── inbox-provider.interface.ts # InboxProvider contract (moved from dkv.types.ts)
|
||||
│ ├── imap.provider.ts # moved verbatim from dkv/providers/
|
||||
│ ├── exchange-inbox.provider.ts # moved verbatim from dkv/providers/
|
||||
│ └── inbox.types.ts # InboxConfig / InboxEmail / InboxAttachment
|
||||
│
|
||||
├── dkv/ # MODIFIED -- imports InboxModule instead of local providers/
|
||||
│ └── ... # (providers/ folder removed, dkv.types.ts trimmed)
|
||||
│
|
||||
├── tenders/ # NEW -- Ausschreibungs-Radar module
|
||||
│ ├── tenders.module.ts
|
||||
│ ├── tenders.controller.ts # public list/detail + tenant saved-search CRUD + admin source-config
|
||||
│ ├── tenders.seed.ts # seeds 'tender-radar' into ModuleRegistry
|
||||
│ ├── tender-ingestion.service.ts # poll → normalize → upsert → change-detect (per source)
|
||||
│ ├── tender-matching.service.ts # SavedSearch → Prisma where-clause → TenderMatch
|
||||
│ ├── tender-mail.service.ts # instant + digest notification sends
|
||||
│ ├── tender-scheduler.service.ts # SchedulerRegistry cron: 1 per global source + 1 per tenant (email-alert)
|
||||
│ ├── tender.types.ts # RawTenderRecord, NormalizedTenderFields, SourceType
|
||||
│ ├── dto/
|
||||
│ │ ├── saved-search.dto.ts
|
||||
│ │ ├── source-config.dto.ts
|
||||
│ │ └── tender-query.dto.ts # pagination + filter query params for GET /tenders
|
||||
│ └── adapters/
|
||||
│ ├── tender-source-adapter.interface.ts # fetchTenders(config, since) → RawTenderRecord[]
|
||||
│ ├── doe-opendata.adapter.ts # Build order Phase A
|
||||
│ ├── ai-netserver.adapter.ts # Build order Phase B
|
||||
│ ├── cosinex.adapter.ts # Build order Phase B
|
||||
│ ├── rss.adapter.ts # Build order Phase C
|
||||
│ └── email-alert.adapter.ts # Build order Phase C (uses InboxModule)
|
||||
│
|
||||
apps/web/src/app/(portal)/modules/
|
||||
├── tender-radar/ # follow the existing static per-module convention (dkv-fleet, cert-manager)
|
||||
│ ├── page.tsx # searchable trefferliste + filter sidebar
|
||||
│ ├── [id]/page.tsx # tender detail view
|
||||
│ ├── saved-searches/page.tsx # saved search CRUD UI
|
||||
│ └── settings/page.tsx # admin: source poll config, email-alert inbox config
|
||||
```
|
||||
|
||||
**Module activation flow:**
|
||||
### Structure Rationale
|
||||
|
||||
```
|
||||
Admin browses Marketplace
|
||||
-> Selects module, clicks "Activate"
|
||||
-> POST /api/marketplace/modules/:id/activate
|
||||
-> Marketplace Service checks license entitlement
|
||||
-> Module Loader registers module for tenant
|
||||
-> INSERT module_activations (tenant_id, module_id, status)
|
||||
-> Frontend sidebar updates (module appears)
|
||||
<- Success response
|
||||
```
|
||||
- **`inbox/` extraction is a prerequisite, not optional.** `ImapProvider`/`ExchangeInboxProvider` are already generic over `InboxConfig`/`InboxEmail` — nothing in them is DKV-specific. Today they live in `dkv/providers/` and their types live in `dkv.types.ts`, so `tenders/` would otherwise have to import from inside another feature module's internals (`../dkv/providers/imap.provider`), which couples two unrelated features and breaks if DKV is ever restructured. Moving them to a shared `inbox/` module once, and updating `dkv.module.ts` to import `InboxModule` instead, costs one small refactor now and pays for every future module that needs inbox polling (already two: DKV, Tenders).
|
||||
- **`tenders/adapters/` mirrors `dkv/providers/`** — same rationale as DKV: consumers (`TenderIngestionService`) depend only on the `TenderSourceAdapter` interface, never on a concrete adapter. This is what makes "ship DÖE first, add AI-NetServer/cosinex/RSS/email later" possible without touching the ingestion/matching/notification pipeline.
|
||||
- **Normalizer is separate from adapters**, unlike DKV where `DkvParserService` is a single PDF parser. Here there are 5 structurally incompatible raw shapes (eForms/OCDS JSON, two flavors of scraped HTML, RSS/Atom XML, free-text alert emails). Each adapter can either normalize inline or delegate to a per-source mapping function inside `TenderNormalizerService` — either way, the *interface boundary* is `RawTenderRecord[] → Tender[]`, so the ingestion orchestrator never branches on source type.
|
||||
- **Web module UI follows the existing static `modules/<slug>/` folder pattern** seen in `dkv-fleet/` and `cert-manager/` (not the dynamic `[category]/[moduleSlug]/` route also present in the codebase for vehicle/settings sub-pages) — match whichever of the two conventions the team is actively converging on at execution time; both are already present, so this is a phase-planning decision, not an open architectural question.
|
||||
|
||||
**Dashboard widget flow:**
|
||||
## Architectural Patterns
|
||||
|
||||
```
|
||||
User opens Dashboard
|
||||
-> GET /api/dashboard/layout
|
||||
-> Returns user's widget layout (positions, sizes)
|
||||
-> Frontend renders react-grid-layout with widget components
|
||||
-> User drags/resizes widget
|
||||
-> PUT /api/dashboard/layout (debounced save)
|
||||
-> Persists to PostgreSQL
|
||||
```
|
||||
### Pattern 1: Source-Adapter Abstraction (`TenderSourceAdapter`)
|
||||
|
||||
## Core Architecture Decisions
|
||||
|
||||
### 1. Modular Monolith over Microservices
|
||||
|
||||
**Why:** Tessera is built by a single developer (with Claude). Microservices add deployment, debugging, and network complexity that provides zero benefit at this scale. A modular monolith gives clean separation with a single deployment unit.
|
||||
|
||||
**Structure:** Each domain (auth, tenant, marketplace, dashboard, modules) lives in its own directory with its own routes, services, and repository files. They communicate through in-process function calls, not HTTP.
|
||||
|
||||
**Future path:** If a module becomes a bottleneck, extract it to a separate service behind the API gateway. The clean boundaries make this straightforward.
|
||||
|
||||
### 2. Shared Database with Row-Level Security (RLS)
|
||||
|
||||
**Why:** Separate databases per tenant adds massive operational overhead. PostgreSQL RLS enforces tenant isolation at the database level, meaning even application bugs cannot leak data across tenants.
|
||||
|
||||
**Implementation:**
|
||||
```sql
|
||||
-- Every tenant-scoped table has a tenant_id column
|
||||
ALTER TABLE modules ENABLE ROW LEVEL SECURITY;
|
||||
CREATE POLICY tenant_isolation ON modules
|
||||
USING (tenant_id = current_setting('app.current_tenant')::uuid);
|
||||
```
|
||||
|
||||
The backend sets `app.current_tenant` on each database connection based on the authenticated user's tenant. RLS handles the rest transparently.
|
||||
|
||||
### 3. Plugin/Module System as Data-Driven Registry
|
||||
|
||||
**Why:** Modules should not require restarting the server to be discovered. A database-driven registry with filesystem-based module code gives hot-activation without runtime code loading risks.
|
||||
|
||||
**How it works:**
|
||||
- Module metadata (name, version, category, routes, permissions) stored in `modules` table
|
||||
- Module code lives in `src/modules/<module-name>/` with a standard interface
|
||||
- Activation is per-tenant: `module_activations` table links tenant to module
|
||||
- Frontend lazy-loads module UI bundles only when the module is active for the current tenant
|
||||
- Backend routes for a module are only registered/accessible when the module is active
|
||||
|
||||
### 4. Frontend Shell with Lazy Module Loading
|
||||
|
||||
**Why:** Loading all module UIs upfront wastes bandwidth and exposes code for modules the tenant has not licensed. Lazy loading (React.lazy + dynamic import) loads module UIs on demand.
|
||||
|
||||
**Pattern:**
|
||||
```typescript
|
||||
// Module registry maps module_id -> lazy component
|
||||
const moduleRegistry: Record<string, () => Promise<{ default: ComponentType }>> = {
|
||||
'domaincheck': () => import('./modules/domaincheck/DomaincheckPage'),
|
||||
'email-tools': () => import('./modules/email-tools/EmailToolsPage'),
|
||||
};
|
||||
```
|
||||
|
||||
The shell only renders module routes that are in the tenant's active module list (fetched from the backend on login).
|
||||
|
||||
### 5. Tauri over Electron for Desktop Wrapper
|
||||
|
||||
**Why:** Tauri produces ~10MB binaries vs Electron's 100MB+. RAM usage is 20-40MB vs 200-400MB. Tauri uses the system WebView (no bundled Chromium), has a Rust backend for native features, and has a smaller attack surface. The non-programmer maintainer benefits from the simpler, lighter deployment.
|
||||
|
||||
**Architecture:** The Tauri wrapper is a thin shell. It loads the same web application served locally or from the server. Native features (system tray, auto-update, window management) are exposed through Tauri commands.
|
||||
|
||||
## Patterns to Follow
|
||||
|
||||
### Pattern 1: Tenant Context Middleware
|
||||
|
||||
**What:** A middleware that extracts tenant identity from the JWT/session, sets it on the request context, and configures the database connection with RLS.
|
||||
|
||||
**When:** Every authenticated request.
|
||||
**What:** One interface, N implementations — directly analogous to `InboxProvider` (`fetchPdfAttachments` → `fetchTenders`).
|
||||
|
||||
```typescript
|
||||
// middleware/tenantContext.ts
|
||||
async function tenantContext(req: FastifyRequest, reply: FastifyReply) {
|
||||
const tenantId = req.user.tenantId;
|
||||
if (!tenantId) return reply.code(403).send({ error: 'No tenant context' });
|
||||
|
||||
// Set RLS context on the database connection
|
||||
await req.db.query(`SET app.current_tenant = '${tenantId}'`);
|
||||
req.tenantId = tenantId;
|
||||
// tenders/adapters/tender-source-adapter.interface.ts
|
||||
export interface RawTenderRecord {
|
||||
sourceType: SourceType; // 'doe-opendata' | 'ai-netserver' | 'cosinex' | 'rss' | 'email-alert'
|
||||
sourcePortal: string; // 'doe' | 'lhs-vpbw' | 'tender24' | 'vergabe.landbw' | 'dtvp' | 'subreport-elvis' | 'service.bund.de'
|
||||
sourceRawId: string; // portal-native id/notice number, pre-normalization
|
||||
sourceUrl: string;
|
||||
fetchedAt: Date;
|
||||
payload: unknown; // raw JSON/HTML-extract/RSS-item/email-body — kept for rawPayload + reprocessing
|
||||
}
|
||||
|
||||
export interface TenderSourceAdapter {
|
||||
readonly sourceType: SourceType;
|
||||
/** since: only fetch records new/changed after this timestamp (cursor from TenderSourcePollConfig.lastPolledAt) */
|
||||
fetchTenders(config: TenderSourceConfig, since?: Date): Promise<RawTenderRecord[]>;
|
||||
testConnection?(config: TenderSourceConfig): Promise<{ success: boolean; message?: string }>;
|
||||
}
|
||||
```
|
||||
|
||||
### Pattern 2: Module Interface Contract
|
||||
**When to use:** Any time a pipeline must ingest structurally different sources into one output shape without the orchestrator knowing about each source. Same pattern Tessera already uses for `ImapProvider`/`ExchangeInboxProvider`.
|
||||
|
||||
**What:** Every module (backend) exports a standard interface so the platform can discover, mount, and manage it uniformly.
|
||||
**Trade-offs:** Each adapter owns its own retry/rate-limit/anti-bot logic (AI-NetServer and cosinex adapters need polite scraping delays; DÖE/RSS don't). The interface intentionally does *not* prescribe HTTP client or scraping library — `TenderSourceConfig` is adapter-specific (a discriminated union or `Json` blob per `sourceType`), same as `InboxConfig` covers both IMAP and Exchange with one shape only because both providers happen to share fields; here they mostly won't, so `TenderSourceConfig` should be `Json` on the Prisma side with adapter-specific Zod/DTO validation, not a single flat interface.
|
||||
|
||||
**When:** Building any new module.
|
||||
### Pattern 2: Normalized OCDS-Oriented Schema + Cross-Source Dedup Key
|
||||
|
||||
**What:** One `Tender` Prisma model absorbing eForms/OCDS (DÖE), scraped HTML (AI-NetServer, cosinex), RSS, and email-alert text — with a **stable dedup key** so the same real-world procurement notice appearing on multiple sources (e.g. an AI-NetServer notice that later also appears on DÖE once it crosses the EU threshold, or the same DÖE OCID reappearing on a poll re-run) collapses to one row.
|
||||
|
||||
```prisma
|
||||
model Tender {
|
||||
id String @id @default(uuid())
|
||||
|
||||
// OCDS-oriented core (see standard.open-contracting.org/latest/en/schema/reference/)
|
||||
ocid String? // Open Contracting ID, e.g. "ocds-mnwr74-XXXXXXXX" — present when sourced via DÖE/TED
|
||||
noticeId String? // portal-native notice/procedure number (AI-NetServer, cosinex, RSS items)
|
||||
title String
|
||||
description String? @db.Text
|
||||
buyerName String?
|
||||
buyerId String? // e.g. Vergabestelle-ID if the source exposes it
|
||||
procedureType String? // OCDS tender.procurementMethod / procurementMethodDetails
|
||||
status String @default("active") // 'active' | 'awarded' | 'cancelled' | 'expired'
|
||||
cpvCodes String[] @default([])
|
||||
region String?
|
||||
plz String?
|
||||
bundesland String?
|
||||
estimatedValue Decimal? @db.Decimal(14, 2)
|
||||
currency String? @default("EUR")
|
||||
publishedAt DateTime?
|
||||
deadlineAt DateTime?
|
||||
|
||||
// Source + dedup
|
||||
sourceType String // 'doe-opendata' | 'ai-netserver' | 'cosinex' | 'rss' | 'email-alert'
|
||||
sourcePortal String // 'doe' | 'lhs-vpbw' | 'dtvp' | 'subreport-elvis' | ...
|
||||
sourceUrl String?
|
||||
dedupKey String @unique // see dedup strategy below
|
||||
contentHash String // hash of normalized fields — detects "changed" vs "identical re-poll"
|
||||
rawPayload Json // original adapter payload — debugging + future re-normalization
|
||||
|
||||
firstSeenAt DateTime @default(now())
|
||||
lastSeenAt DateTime @updatedAt
|
||||
createdAt DateTime @default(now())
|
||||
|
||||
matches TenderMatch[]
|
||||
|
||||
@@index([sourceType])
|
||||
@@index([deadlineAt])
|
||||
@@index([publishedAt])
|
||||
@@index([bundesland])
|
||||
}
|
||||
```
|
||||
|
||||
**Dedup key strategy (priority order, computed by `TenderNormalizerService`):**
|
||||
1. `ocid` — when the source provides an OCDS Open Contracting ID (always true for DÖE/TED; the platform's registered OCDS prefix is `ocds-mnwr74`). OCID is designed exactly for this — joining the same contracting process across publishers.
|
||||
2. `${sourcePortal}:${noticeId}` — when the portal exposes a stable native notice/procedure number (AI-NetServer Bietercockpit ID, cosinex Vergabenummer, RSS item guid, email-alert reference number). This is the *primary* key for scraped/RSS/email sources, since they never carry an OCID.
|
||||
3. `sha256(sourcePortal + normalizedTitle + buyerName + deadlineAt)` — last-resort fallback only when a source gives neither an OCID nor a stable ID (should be rare; flag these rows for manual review via a `dedupConfidence: 'low'` marker if this path is hit).
|
||||
|
||||
`dedupKey` is the `@unique` upsert target: `prisma.tender.upsert({ where: { dedupKey }, ... })`. `contentHash` (hash of title+deadline+value+status) is separate from `dedupKey` — it answers "did anything about this same notice change since we last saw it" (deadline extension, cancellation), which is what should trigger re-matching and potentially a "notice updated" notification, whereas an unchanged re-poll should just bump `lastSeenAt` and stop.
|
||||
|
||||
**Trade-offs:** A single wide table is simpler to query/filter/index than per-source tables + a union view, and matches how the UI wants to browse ("one trefferliste across all sources"). The cost is that source-specific fields that don't map cleanly (e.g. cosinex-specific metadata) live only in `rawPayload` (Json, unindexed) — acceptable, since the feasibility research shows the cross-source overlap (title, buyer, deadline, value, CPV, region) covers what filtering/notification actually need.
|
||||
|
||||
### Pattern 3: Global Data, Per-Tenant Filtering (the key deviation from the DKV template)
|
||||
|
||||
**What:** `Tender` rows carry **no `tenantId`** — they are platform-wide. Per-tenant scoping happens one layer up, in `TenderSavedSearch` and `TenderMatch`.
|
||||
|
||||
```prisma
|
||||
model TenderSavedSearch {
|
||||
id String @id @default(uuid())
|
||||
tenantId String
|
||||
userId String? // null = tenant-wide search, set = personal search
|
||||
name String
|
||||
keywords String[] @default([]) // full-text match against title+description
|
||||
bundeslaender String[] @default([])
|
||||
plzPrefixes String[] @default([])
|
||||
cpvCodes String[] @default([])
|
||||
minValue Decimal? @db.Decimal(14, 2)
|
||||
maxValue Decimal? @db.Decimal(14, 2)
|
||||
deadlineWithinDays Int?
|
||||
notifyMode String @default("digest") // 'none' | 'digest' | 'instant'
|
||||
digestHour Int? @default(7) // for digest mode: hour-of-day to send
|
||||
isActive Boolean @default(true)
|
||||
createdAt DateTime @default(now())
|
||||
updatedAt DateTime @updatedAt
|
||||
matches TenderMatch[]
|
||||
|
||||
@@index([tenantId])
|
||||
}
|
||||
|
||||
model TenderMatch {
|
||||
id String @id @default(uuid())
|
||||
tenderId String
|
||||
tender Tender @relation(fields: [tenderId], references: [id], onDelete: Cascade)
|
||||
savedSearchId String
|
||||
savedSearch TenderSavedSearch @relation(fields: [savedSearchId], references: [id], onDelete: Cascade)
|
||||
tenantId String // denormalized for fast tenant-scoped queries/RLS
|
||||
matchedAt DateTime @default(now())
|
||||
notifiedAt DateTime? // null = not yet sent (instant) or not yet in a digest
|
||||
|
||||
@@unique([tenderId, savedSearchId])
|
||||
@@index([tenantId])
|
||||
@@index([notifiedAt])
|
||||
}
|
||||
```
|
||||
|
||||
**Why this beats a `tenantId` on `Tender`:** DÖE alone publishes thousands of Oberschwelle notices; duplicating that table N times (once per tenant) multiplies storage for zero benefit — every tenant sees the same underlying notice, just filtered differently. It also means the DÖE/AI-NetServer/cosinex/RSS pollers run **once for the whole platform**, not once per active tenant — critical for the scraping sources, where running the same scrape N times per tenant multiplies anti-bot/ToS exposure on portals that already sit at "Niedrig-Mittel" risk per the feasibility research. `TenantModuleActivation` still gates whether a tenant sees the module at all (standard Tessera marketplace pattern) — but activation controls *visibility*, not a second data copy.
|
||||
|
||||
**When this pattern does NOT apply:** email-alert ingestion. A tenant's alert emails arrive in *that tenant's own mailbox* (their own registered "gespeicherte Suche" on a portal) — so `TenderInboxConfig` (credentials, mirroring `DkvModuleConfig.encryptedInboxCreds`) is legitimately per-tenant, even though the `Tender` rows it produces still land in the same global table (deduped against whatever DÖE/AI-NetServer/cosinex already ingested for the same notice).
|
||||
|
||||
### Pattern 4: Filter Evaluation — DB Query Against the Delta, Not In-Memory Full-Table Scan
|
||||
|
||||
**What:** Filtering happens as a **Postgres query scoped to the just-ingested batch**, not (a) a full in-memory scan of all tenders per saved search, nor (b) a full re-scan of the entire `Tender` table on every poll.
|
||||
|
||||
```typescript
|
||||
// modules/<name>/index.ts
|
||||
export interface TesseraModule {
|
||||
id: string;
|
||||
version: string;
|
||||
category: string;
|
||||
routes: (app: FastifyInstance) => void;
|
||||
widgets?: WidgetDefinition[]; // Optional dashboard widgets
|
||||
permissions?: string[]; // Required permissions
|
||||
onActivate?: (tenantId: string) => Promise<void>;
|
||||
onDeactivate?: (tenantId: string) => Promise<void>;
|
||||
// tender-matching.service.ts (sketch)
|
||||
async matchDelta(newOrChangedTenderIds: string[]): Promise<void> {
|
||||
const savedSearches = await this.prisma.tenderSavedSearch.findMany({ where: { isActive: true } });
|
||||
for (const search of savedSearches) {
|
||||
const where = this._buildWhereClause(search, newOrChangedTenderIds); // structured filters
|
||||
const matches = await this.prisma.tender.findMany({ where }); // DB does the heavy lifting
|
||||
// full-text keyword refinement, if keywords present, folded into the same query via
|
||||
// a raw `to_tsvector('german', title || ' ' || description) @@ plainto_tsquery(...)`
|
||||
for (const tender of matches) {
|
||||
await this.prisma.tenderMatch.upsert({
|
||||
where: { tenderId_savedSearchId: { tenderId: tender.id, savedSearchId: search.id } },
|
||||
create: { tenderId: tender.id, savedSearchId: search.id, tenantId: search.tenantId },
|
||||
update: {}, // matchedAt stays as first-seen; re-match is idempotent
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Pattern 3: Widget Component Contract
|
||||
**Why DB, not in-memory:** structured filters (CPV array overlap, numeric value range, date range, region) are exactly what Postgres indexes/arrays/range queries are built for, and the corpus (all German public tenders) will reach tens of thousands of rows within the first year — loading that into Node to filter per saved search per tenant does not scale and duplicates work Postgres already does better. Keyword matching specifically should use a **Postgres full-text `tsvector` GIN index** on `title || description` (German text-search config) rather than `ILIKE '%term%'` scans — `ILIKE` on a growing table degrades linearly, GIN full-text does not.
|
||||
|
||||
**What:** Dashboard widgets follow a standard interface for the grid layout system.
|
||||
**Why "scoped to the delta" and not the whole table every poll:** re-evaluating every saved search against the *entire* `Tender` table on every poll cycle is O(searches × table-size) repeated hourly — wasteful and re-creates matches that already exist (idempotent upsert hides the waste but not the cost). Instead, `TenderIngestionService` passes the list of tender IDs that were newly created or had a `contentHash` change in *this* poll run to `TenderMatchingService.matchDelta()`, so the query is `WHERE id IN (delta) AND <search filters>` — bounded by poll batch size (dozens to low hundreds), not table size.
|
||||
|
||||
**When:** Building any dashboard widget (core or module-provided).
|
||||
**Trade-off:** if a saved search is created or edited *after* a tender was ingested, that tender won't retroactively appear in the search's matches until the next time it's re-touched (re-poll sees no change → no delta → not re-evaluated). Mitigate with an explicit "backfill" action: when a `TenderSavedSearch` is created/edited, run `matchDelta()` once against *all* tenders from the last N days (bounded, on-demand, not a recurring cost) rather than the full historical table.
|
||||
|
||||
```typescript
|
||||
// widgets/types.ts
|
||||
export interface WidgetDefinition {
|
||||
id: string;
|
||||
name: string; // i18n key
|
||||
defaultSize: { w: number; h: number };
|
||||
minSize?: { w: number; h: number };
|
||||
component: () => Promise<{ default: ComponentType<WidgetProps> }>;
|
||||
configSchema?: JSONSchema; // Optional widget settings
|
||||
}
|
||||
### Pattern 5: Scheduled Polling — Global Cron Per Source + Per-Tenant Cron Only for Email-Alerts
|
||||
|
||||
export interface WidgetProps {
|
||||
config: Record<string, unknown>;
|
||||
tenantId: string;
|
||||
userId: string;
|
||||
}
|
||||
```
|
||||
**What:** Extends `DkvSchedulerService`'s `SchedulerRegistry.addCronJob()` pattern, but with two distinct job populations:
|
||||
|
||||
### Pattern 4: Feature Flag via Module Activation
|
||||
- **Global source jobs** (DÖE, RSS feeds, AI-NetServer, cosinex): one cron job per row in `TenderSourcePollConfig` (admin-managed, not tenant-scoped), named `tender-poll-${sourceConfigId}`. Interval is source-appropriate — DÖE can poll frequently (auth-free API, cheap), scraping sources should poll less often and with jitter to stay polite.
|
||||
- **Per-tenant email-alert jobs**: one cron job per tenant with an active `TenderInboxConfig`, named `tender-inbox-poll-${tenantId}` — this is the one place Tessera's existing per-tenant scheduling gap (DKV's scheduler is documented as "v1 single-tenant, `findFirst()`") must actually be solved properly, since two tenants could each have their own mailbox subscribed to different portal alerts. Loop `TenderInboxConfig.findMany({ where: { isActive: true } })` on `onModuleInit` and register one job per row, mirroring `setInterval(intervalMin, tenantId)` but keyed by tenant instead of a single global slot.
|
||||
- **Digest cron**: one platform-wide cron (e.g. hourly) that queries `TenderMatch` rows where `notifiedAt IS NULL` and the owning `TenderSavedSearch.notifyMode = 'digest'` and `digestHour` matches the current hour, groups by tenant+savedSearch, and hands off to `TenderMailService`.
|
||||
|
||||
**What:** Instead of traditional feature flags, Tessera uses module activation as the feature gating mechanism. If a module is not activated for a tenant, its routes return 403, its UI does not load, and its sidebar entry is hidden.
|
||||
**Change detection:** `contentHash` (see Pattern 2) is the mechanism — `TenderIngestionService.upsert()` compares the freshly computed hash against the stored one; identical → touch `lastSeenAt` only, skip matching; different → update fields, recompute `contentHash`, add to this poll's "changed" delta so it flows through matching/notification again (a deadline extension should re-surface in a saved search, a brand-new notice obviously should).
|
||||
|
||||
**When:** Controlling feature access per tenant.
|
||||
### Pattern 6: Notification Dispatch Reusing `SmtpConfig` + Fresh-Transport, Not the Global `MailerService`
|
||||
|
||||
## Anti-Patterns to Avoid
|
||||
**What:** `TenderMailService` is built exactly like `DkvMailService` — `nodemailer.createTransport()` freshly per send, using `SettingsService.getDecryptedSmtpConfig(tenantId)` — **not** the platform's global `MailerService`/`@nestjs-modules/mailer` (which is reserved for system emails: password reset, welcome mail, configured once at bootstrap with a static transport). This distinction already exists in the codebase (`DkvMailService` vs `MailService`) and should hold here too: tender notifications are tenant-directed business content, and the tenant may have configured their own outbound SMTP relay that differs from the platform's.
|
||||
|
||||
### Anti-Pattern 1: Module-to-Module Direct Dependencies
|
||||
- **Instant:** triggered synchronously (or via a lightweight in-process queue, matching DKV's direct-call style — no message broker in this stack) right after `TenderMatchingService` creates a `TenderMatch` with `savedSearch.notifyMode === 'instant'`.
|
||||
- **Digest:** triggered by the digest cron (Pattern 5), batching all unnotified matches for a tenant+savedSearch into one summary email, then setting `notifiedAt` on each included `TenderMatch` — same "mark as sent" idempotency DKV uses for `DkvInvoiceHistory`.
|
||||
|
||||
**What:** Module A directly imports and calls Module B's internal functions.
|
||||
**Trade-off:** reusing per-send transport creation means every notification email opens/closes its own SMTP connection (as DKV already accepts) — fine at Tessera's realistic tender-volume/tenant-count, and it guarantees an admin's SMTP config change takes effect on the very next send without a service restart (same Pitfall-3 mitigation DKV already documents).
|
||||
|
||||
**Why bad:** Creates coupling that makes modules impossible to activate independently. If Module B is deactivated, Module A breaks.
|
||||
## Data Flow
|
||||
|
||||
**Instead:** Use an event bus or shared service layer. Modules publish events; other modules subscribe. If the publisher is missing, subscribers simply receive no events.
|
||||
|
||||
### Anti-Pattern 2: Tenant ID in Application Logic
|
||||
|
||||
**What:** Manually filtering by `tenant_id` in every query throughout the codebase.
|
||||
|
||||
**Why bad:** A single missed filter leaks data across tenants. Hundreds of places to maintain.
|
||||
|
||||
**Instead:** Use PostgreSQL RLS. Set tenant context once per request at the middleware level. All queries are automatically filtered. Defense in depth: the application layer still passes tenant_id, but RLS is the safety net.
|
||||
|
||||
### Anti-Pattern 3: Monolithic Frontend Bundle
|
||||
|
||||
**What:** Bundling all module UIs into a single JavaScript bundle.
|
||||
|
||||
**Why bad:** Users download code for modules they cannot access. Bundle size grows linearly with module count. Exposes unlicensed module code.
|
||||
|
||||
**Instead:** Code-split per module. Use dynamic imports. Only load module bundles when the user navigates to an active module.
|
||||
|
||||
### Anti-Pattern 4: Per-Tenant Database/Schema
|
||||
|
||||
**What:** Creating a separate PostgreSQL database or schema for each tenant.
|
||||
|
||||
**Why bad:** Operational nightmare at scale — migrations must run N times, connection pooling is per-tenant, monitoring multiplies. Overkill for a portal where tenants share the same data model.
|
||||
|
||||
**Instead:** Shared schema with RLS. Single migration path. Single connection pool. Isolation enforced at row level.
|
||||
|
||||
## Scalability Considerations
|
||||
|
||||
| Concern | At 10 users (internal) | At 100 tenants | At 1000+ tenants |
|
||||
|---------|------------------------|----------------|------------------|
|
||||
| **Database** | Single PostgreSQL, no pooler needed | PgBouncer for connection pooling | Read replicas, consider partitioning large tables by tenant_id |
|
||||
| **Backend** | Single container | Horizontal scaling behind gateway (2-4 replicas) | Auto-scaling, consider extracting hot modules to separate services |
|
||||
| **Frontend** | Single static bundle | CDN for static assets | CDN + edge caching, consider module federation for team-developed modules |
|
||||
| **Module isolation** | Shared process, trust all modules | Same, but add resource limits per module route | Consider containerized module backends for untrusted/third-party modules |
|
||||
| **File storage** | Local volume | S3-compatible object storage | Same + lifecycle policies |
|
||||
|
||||
## Suggested Build Order
|
||||
|
||||
Based on component dependencies, the recommended build order is:
|
||||
### Ingestion → Notification Flow
|
||||
|
||||
```
|
||||
Phase 1: Foundation
|
||||
├── Docker Compose setup (PostgreSQL + backend + frontend containers)
|
||||
├── PostgreSQL schema with tenant_id columns + RLS policies
|
||||
├── Backend skeleton (Fastify + request lifecycle)
|
||||
└── Frontend shell (React + routing + layout frame)
|
||||
|
||||
Phase 2: Authentication & Tenancy
|
||||
├── Auth module (login, JWT, session management)
|
||||
├── Tenant middleware (context injection, RLS activation)
|
||||
├── User management (CRUD, roles)
|
||||
└── LDAP integration
|
||||
|
||||
Phase 3: Module System
|
||||
├── Module interface contract
|
||||
├── Module registry (database-driven)
|
||||
├── Module loader (route mounting, activation/deactivation)
|
||||
└── First example module (Domaincheck)
|
||||
|
||||
Phase 4: Marketplace & Dashboard
|
||||
├── Marketplace service (catalog, categories, licensing)
|
||||
├── Marketplace UI (browse, activate, manage)
|
||||
├── Dashboard service (layout persistence)
|
||||
└── Dashboard UI (react-grid-layout, core widgets)
|
||||
|
||||
Phase 5: Polish & Desktop
|
||||
├── i18n (DE + EN)
|
||||
├── Light/Dark theme
|
||||
├── Tauri desktop wrapper
|
||||
└── Gitea CI/CD integration
|
||||
[Cron tick: TenderSchedulerService]
|
||||
v
|
||||
[Adapter.fetchTenders(config, since)] -> RawTenderRecord[]
|
||||
v
|
||||
[TenderNormalizerService.normalize()] -> { fields, dedupKey, contentHash }
|
||||
v
|
||||
[TenderIngestionService.upsert()] -> prisma.tender.upsert({ where: { dedupKey } })
|
||||
v (only rows that were newly created OR whose contentHash changed)
|
||||
[TenderMatchingService.matchDelta(deltaIds)]
|
||||
v (per active TenderSavedSearch, DB-scoped query)
|
||||
[prisma.tenderMatch.upsert()]
|
||||
v
|
||||
+--------------------------+---------------------------+
|
||||
| notifyMode='instant' | notifyMode='digest' |
|
||||
v v
|
||||
[TenderMailService.sendInstant()] [Digest cron batches unnotified matches -> sendDigest()]
|
||||
v v
|
||||
[nodemailer via tenant SmtpConfig, fresh transport per send]
|
||||
```
|
||||
|
||||
**Dependency rationale:**
|
||||
- Phase 1 first because everything depends on the database, backend framework, and frontend shell
|
||||
- Phase 2 before modules because module activation requires knowing WHO is asking and WHICH tenant they belong to
|
||||
- Phase 3 before marketplace because the marketplace manages modules — the module system must exist first
|
||||
- Phase 4 can partially parallelize (dashboard is independent of marketplace) but both need the module system
|
||||
- Phase 5 is pure enhancement — i18n/theme are cross-cutting but easier to retrofit than to block on
|
||||
### Read Flow (Portal UI)
|
||||
|
||||
```
|
||||
[User opens Ausschreibungs-Radar]
|
||||
v
|
||||
GET /tenders?keywords=&bundesland=&cpv=&minValue=&deadlineBefore= (TendersController)
|
||||
v
|
||||
prisma.tender.findMany({ where: <same structured-filter builder as TenderMatchingService> })
|
||||
v
|
||||
[Trefferliste UI] -> click -> GET /tenders/:id -> [Detail view, incl. rawPayload debug panel for admins]
|
||||
|
||||
[User manages saved searches]
|
||||
v
|
||||
POST/PUT/DELETE /tenders/saved-searches (tenant-scoped, req.tenantId)
|
||||
v
|
||||
prisma.tenderSavedSearch.upsert({ ..., tenantId })
|
||||
v (on create/edit) -> one-off matchDelta() backfill against recent tenders (Pattern 4 mitigation)
|
||||
```
|
||||
|
||||
## Scaling Considerations
|
||||
|
||||
| Scale | Architecture Adjustments |
|
||||
|-------|---------------------------|
|
||||
| Single tenant (current, internal test phase) | Exactly as designed above — global `Tender` table, one poller per source, no extra work needed even though only one tenant exists yet, because the schema is already tenant-agnostic at the data layer. |
|
||||
| Multiple tenants, few saved searches each | `matchDelta()` cost scales with (poll batch size × active saved searches), both small — no changes needed. |
|
||||
| Many tenants, many saved searches, high tender volume | Add a GIN full-text index on `title`/`description` (Pattern 4) before this becomes necessary, not after. If `matchDelta()` ever becomes a bottleneck, batch saved-search evaluation into a single SQL query per poll (`Tender × SavedSearch` cross-join filtered in one statement) instead of one query per search — straightforward migration since the where-clause builder is already centralized. |
|
||||
|
||||
### Scaling Priorities
|
||||
|
||||
1. **First bottleneck:** keyword filtering via `ILIKE` if the full-text index is skipped in the first slice — fix before it matters, it's a single migration (`CREATE INDEX ... USING GIN (to_tsvector('german', title || ' ' || coalesce(description,'')))`).
|
||||
2. **Second bottleneck:** AI-NetServer/cosinex scraping adapters getting rate-limited or blocked as tender volume/poll frequency grows — mitigate with per-adapter jittered intervals and respecting any `Retry-After`, not a platform-wide fix.
|
||||
|
||||
## Anti-Patterns
|
||||
|
||||
### Anti-Pattern 1: Tenant-Scoping the `Tender` Table Like DKV Data
|
||||
|
||||
**What people do:** Copy the DKV template literally — add `tenantId` to `Tender`, run every adapter poll once per active tenant (matching `DkvSchedulerService`'s per-tenant cron intent).
|
||||
**Why it's wrong:** Multiplies scraping requests against AI-NetServer/cosinex by tenant count (worse ToS exposure on portals already flagged "Niedrig-Mittel" risk), multiplies storage for identical public data, and makes cross-tenant dedup impossible (the same DÖE notice would need deduping *and* tenant-duplicating, which is incoherent).
|
||||
**Do this instead:** Global `Tender` table (Pattern 3); tenant scoping lives one layer up in `TenderSavedSearch`/`TenderMatch`. Only the email-alert path (genuinely per-tenant mailbox) needs per-tenant scheduling.
|
||||
|
||||
### Anti-Pattern 2: One Adapter Per Portal Instead of Per Platform
|
||||
|
||||
**What people do:** Build a `LhsVpbwAdapter`, `Tender24Adapter`, `VergabeLandbwAdapter` as three separate classes because they're three separate portal URLs.
|
||||
**Why it's wrong:** The feasibility research already established these three (plus many unlisted others) share the same AI AG NetServer fingerprint (`/NetServer/…ControllerServlet`) — one HTML/DOM shape. Three adapter classes triple the maintenance burden for zero behavioral difference; only the base URL and possibly a search-form parameter differ, which belongs in `TenderSourceConfig`, not in three code paths.
|
||||
**Do this instead:** One `AiNetServerAdapter`, config-driven per portal instance (base URL + optional search params), same for a future `CosinexAdapter` covering DTVP and other cosinex Vergabemarktplatz instances.
|
||||
|
||||
### Anti-Pattern 3: In-Memory Filtering Across the Whole Table
|
||||
|
||||
**What people do:** `prisma.tender.findMany()` with no `where`, then `.filter()` in TypeScript per saved search.
|
||||
**Why it's wrong:** Works fine in a demo with 50 rows, degrades badly once DÖE's Oberschwelle backbone plus scraped Unterschwelle notices accumulate over months, and re-does full-table work on every poll instead of scoping to the delta.
|
||||
**Do this instead:** Pattern 4 — structured Prisma `where` + Postgres full-text index, scoped to the newly-changed batch.
|
||||
|
||||
## Integration Points
|
||||
|
||||
### External Services
|
||||
|
||||
| Service | Integration Pattern | Notes |
|
||||
|---------|----------------------|-------|
|
||||
| DÖE OpenData API (`oeffentlichevergabe.de`) | Auth-free HTTP GET against the OpenData/Swagger-documented endpoints; paginate; filter by `publishedAt`/last-poll cursor | eForms-DE / **OCDS `ocds-mnwr74`** / CSV formats available — prefer the OCDS export for direct field alignment with the `Tender` schema; ~75% of market value by € per feasibility doc, zero scraping/ToS risk |
|
||||
| AI AG NetServer portals (lhs-vpbw, tender24, vergabe.landbw, + others) | HTML scrape of the public search result pages (no login required for search) | One adapter, config-driven per instance; feasibility doc rates ToS risk "Niedrig-Mittel" — implement politely (rate limit, honest UA string, cache ETags if offered) |
|
||||
| cosinex Vergabemarktplatz (DTVP + other Länder instances) | HTML scrape of public search, structurally incompatible with AI-NetServer — separate adapter | Reusable across NRW/BB/NI/RLP cosinex instances per feasibility doc |
|
||||
| RSS feeds (subreport-elvis, service.bund.de) | Standard RSS/Atom parse (e.g. `rss-parser` or `fast-xml-parser`), config-driven feed URL list | Lowest ToS risk of the scraped sources; service.bund.de rated "Niedrig" |
|
||||
| Portal email alerts (Unterschwelle long tail, 8 of 10 portals) | Reuses `InboxProvider` (extracted `ImapProvider`/`ExchangeInboxProvider`) — per-tenant mailbox subscribed to each portal's native "gespeicherte Suche" alert | Requires manual one-time setup per portal (register a saved search on the portal itself); the module only ingests+parses the resulting alert emails, does not create the portal-side saved search |
|
||||
| TED API v3 | Optional, deprioritized — keyless, EU-wide, largely redundant to DÖE for DE-only coverage | Build order: only if EU-wide coverage becomes a requirement later |
|
||||
|
||||
### Internal Boundaries
|
||||
|
||||
| Boundary | Communication | Notes |
|
||||
|----------|----------------|-------|
|
||||
| `TendersModule` ↔ `ModuleRegistryModule` | Direct DI import, `OnModuleInit` self-seed (`tenders.seed.ts`) — identical to `dkv.seed.ts` | New module import, no registry changes needed beyond the self-seed call |
|
||||
| `TendersModule` ↔ `InboxModule` (new, extracted) | Direct DI import; `EmailAlertAdapter` depends on `ImapProvider`/`ExchangeInboxProvider` | Requires the one-time extraction of `inbox/` out of `dkv/` (see Structure Rationale) |
|
||||
| `TendersModule` ↔ `SettingsModule` | Direct DI import, reused unmodified — `SettingsService.getDecryptedSmtpConfig(tenantId)` | Same pattern as `DkvModule` |
|
||||
| `TendersModule` ↔ `TenantMiddleware`/`TenantGuard` | `req.tenantId` extraction for saved-search/notification endpoints only — **not** for `GET /tenders` list/detail, which is platform-global read access gated only by `TenantModuleActivation` (module licensing), not by tenant-owned data | This is the one controller where "tenant-scoped" and "tenant-gated" genuinely differ — worth flagging explicitly in the phase plan so it isn't implemented as a blanket `where: { tenantId }` by habit |
|
||||
| `TendersController` ↔ `module-loader.ts` (web) | New `MODULE_REGISTRY['tender-radar']` entry, same as `cert-manager`/`dkv-fleet` | One-line addition, whitelist pattern (`T-03-09`) — must not be skipped or the module page 404s even if activated |
|
||||
| `AppModule` ↔ `TendersModule` | New import in `app.module.ts` | One line |
|
||||
|
||||
## Build Order — Ships DÖE-First as a Usable Slice
|
||||
|
||||
Ordered by dependency; each step after step 8 is additive and doesn't touch the pipeline built before it (the point of the adapter abstraction).
|
||||
|
||||
**Phase A — DÖE-only usable slice (schema + one source + filter + UI + notification, end-to-end):**
|
||||
1. Prisma migration: `Tender`, `TenderSourcePollConfig`, `TenderSavedSearch`, `TenderMatch` (+ GIN full-text index on `title`/`description`).
|
||||
2. `TendersModule` skeleton + `tenders.seed.ts` (module-registry self-seed, category e.g. `procurement`).
|
||||
3. `TenderSourceAdapter` interface + `DoeOpenDataAdapter` (auth-free, OCDS-formatted fetch).
|
||||
4. `TenderNormalizerService` (OCDS release → `Tender` fields, `ocid`-based dedup key, `contentHash`).
|
||||
5. `TenderIngestionService` (poll → normalize → upsert → change-detect) + `TenderSchedulerService` (single global cron for DÖE).
|
||||
6. `TenderMatchingService` (DB-query filter evaluation, Pattern 4) + `TendersController` (`GET /tenders`, `GET /tenders/:id`, saved-search CRUD).
|
||||
7. `apps/web/.../modules/tender-radar/` — trefferliste + filter UI + saved-search management + `MODULE_REGISTRY` entry. **This alone is already a usable, demoable slice** — DÖE covers ~75% of market value by € per feasibility doc.
|
||||
8. `TenderMailService` (instant + digest, reusing `SmtpConfig`) + digest cron. Closes the loop on the milestone's "optional E-Mail-Versand" requirement using only the DÖE source.
|
||||
|
||||
**Phase B — Scraping adapters for the Unterschwelle long tail (pipeline unchanged, adapters only):**
|
||||
9. `AiNetServerAdapter` (covers lhs-vpbw, tender24, vergabe.landbw + any future AI AG portal via config).
|
||||
10. `CosinexAdapter` (covers DTVP; reusable for other cosinex Länder marketplaces later).
|
||||
|
||||
**Phase C — RSS + email-alert (lowest ROI per feasibility doc, do last):**
|
||||
11. `RssAdapter` (subreport-elvis, service.bund.de).
|
||||
12. Extract `inbox/` shared module out of `dkv/` (prerequisite refactor).
|
||||
13. `TenderInboxConfig` (per-tenant, mirrors `DkvModuleConfig` credential pattern) + `EmailAlertAdapter` + per-tenant scheduler jobs.
|
||||
|
||||
**Explicitly out of this build order:** TED API v3 (redundant to DÖE for DE-only), vergabe24/aumass (AGB-prohibited scraping — feasibility doc flags these as avoid).
|
||||
|
||||
## New vs Modified — Explicit Inventory
|
||||
|
||||
**New:**
|
||||
- `apps/api/src/tenders/` — entire module (controller, services, scheduler, seed, dto/, adapters/, types)
|
||||
- `apps/api/src/inbox/` — extracted shared inbox module
|
||||
- Prisma models: `Tender`, `TenderSourcePollConfig`, `TenderSavedSearch`, `TenderMatch`, `TenderInboxConfig`
|
||||
- `apps/web/src/app/(portal)/modules/tender-radar/` — list, detail, saved-search, settings pages
|
||||
|
||||
**Modified:**
|
||||
- `apps/api/prisma/schema.prisma` — add the 5 new models + migration
|
||||
- `apps/api/src/dkv/` — `providers/` folder removed, imports `InboxModule` instead; `dkv.types.ts` trimmed of `InboxConfig`/`InboxEmail`/`InboxAttachment` (moved to `inbox/inbox.types.ts`)
|
||||
- `apps/api/src/app.module.ts` — import `TendersModule` (and `InboxModule` if not auto-imported via `TendersModule`'s own imports)
|
||||
- `apps/web/src/lib/module-loader.ts` — add `'tender-radar'` entry to `MODULE_REGISTRY`
|
||||
|
||||
**Explicitly NOT modified:** `module-registry.service.ts`, `prisma-tenant.extension.ts` (`forTenant`), `mail.module.ts`/`mail.service.ts` (global system mailer stays untouched — tender notifications use the DKV-style per-tenant transport pattern instead), `settings.service.ts`.
|
||||
|
||||
## Sources
|
||||
|
||||
- [Multi-Tenant Databases with Postgres Row-Level Security](https://www.midnytecity.com.au/blogs/multi-tenant-databases-with-postgres-row-level-security)
|
||||
- [AWS: Multi-tenant data isolation with PostgreSQL Row Level Security](https://aws.amazon.com/blogs/database/multi-tenant-data-isolation-with-postgresql-row-level-security/)
|
||||
- [Approaches to implementing multi-tenancy in SaaS applications - Red Hat](https://developers.redhat.com/articles/2022/05/09/approaches-implementing-multi-tenancy-saas-applications)
|
||||
- [Node.js Plugin Architecture: Build Your Own Plugin System](https://medium.com/codeelevation/node-js-plugin-architecture-build-your-own-plugin-system-with-es-modules-5b9a5df19884)
|
||||
- [How to Build Plugin Architecture in Node.js](https://oneuptime.com/blog/post/2026-01-26-nodejs-plugin-architecture/view)
|
||||
- [Building Customizable Dashboard Widgets Using React Grid Layout](https://www.antstack.com/blog/building-customizable-dashboard-widgets-using-react-grid-layout/)
|
||||
- [react-grid-layout - GitHub](https://github.com/react-grid-layout/react-grid-layout)
|
||||
- [Micro-Frontend Architecture with Module Federation](https://module-federation.io/)
|
||||
- [Tauri vs Electron: The Complete Developer's Guide (2026)](https://blog.nishikanta.in/tauri-vs-electron-the-complete-developers-guide-2026)
|
||||
- [Tauri in 2026: Build Cross-Platform Desktop Apps](https://dev.to/ottoaria/tauri-in-2026-build-cross-platform-desktop-apps-with-web-technologies-better-than-electron-11mo)
|
||||
- [Using a Reverse Proxy to Expose Multiple Microservices Through a Single Port in Docker Compose](https://dev.to/syed_omair/using-a-reverse-proxy-to-expose-multiple-microservices-through-a-single-port-in-docker-compose-4h9e)
|
||||
- [Developing a Multi-Tenant SaaS Application: The 2026 Architecture Guide](https://apipilot.com/developing-a-multi-tenant-saas-application-the-2026-architecture-guide/)
|
||||
- Existing codebase (verified by direct read): `apps/api/src/dkv/*`, `apps/api/src/module-registry/*`, `apps/api/src/mail/*`, `apps/api/src/prisma/prisma-tenant.extension.ts`, `apps/api/src/tenant/tenant.middleware.ts`, `apps/api/prisma/schema.prisma`, `apps/web/src/lib/module-loader.ts`, `apps/web/src/app/(portal)/modules/dkv-fleet/*` — HIGH confidence, ground truth.
|
||||
- `.planning/research/ausschreibungs-portale-feasibility.md` (2026-07-16) — portal platform fingerprints, ToS risk ratings, DÖE/TED coverage estimates — HIGH confidence (project's own prior research).
|
||||
- [OCDS Release Reference — Open Contracting Data Standard 1.1.5](https://standard.open-contracting.org/latest/en/schema/reference/) — core schema fields (ocid, release, tender, parties, buyer) — MEDIUM confidence (public spec, not project-specific).
|
||||
- [OCDS Building Blocks](https://standard.open-contracting.org/latest/en/getting_started/building_blocks/) — OCID composition (registered prefix + publisher-chosen process id) — MEDIUM confidence.
|
||||
- [oeffentlichevergabe.de OpenData Swagger UI](https://oeffentlichevergabe.de/documentation/swagger-ui/opendata/index.html) — confirms `ocds-mnwr74` as the registered German federal OCDS prefix (registered 2023-02-06) and CC-Zero licensing — MEDIUM confidence (page requires JS to render full endpoint/pagination detail; exact pagination parameters remain an open verification point, already flagged in the feasibility doc).
|
||||
|
||||
---
|
||||
*Architecture research for: Tessera Ausschreibungs-Radar module (v1.1 milestone)*
|
||||
*Researched: 2026-07-17*
|
||||
|
||||
Reference in New Issue
Block a user