diff --git a/.planning/phases/07-dkv-fleet-module/07-RESEARCH.md b/.planning/phases/07-dkv-fleet-module/07-RESEARCH.md new file mode 100644 index 0000000..bc7470d --- /dev/null +++ b/.planning/phases/07-dkv-fleet-module/07-RESEARCH.md @@ -0,0 +1,972 @@ +# Phase 7: DKV Fleet Module - Research + +**Researched:** 2026-06-26 +**Domain:** Email inbox monitoring, PDF parsing, Excel export, NestJS scheduling, SMTP configuration +**Confidence:** MEDIUM (stack verified; DKV PDF regex patterns LOW — only structure description available, no live parse test) + +--- + + +## User Constraints (from CONTEXT.md) + +### Locked Decisions + +**D-01:** Support both IMAP (imapflow) and Exchange (EWS via ews-javascript-api) — selectable per module config. +**D-02:** Polling via configurable cron job (interval in minutes) + manual "Jetzt prüfen" button. +**D-03:** Inbox config fields: Protocol (IMAP/Exchange), Host, Port, Username (optional), Password (optional), Encryption (None/STARTTLS/SSL-TLS), Folder, Sender filter. +**D-04:** Credentials stored encrypted (AES-256-GCM via existing `crypto.service.ts`). +**D-05:** SMTP settings live in **general settings** (shared, not per-module): Host, Port, Username (optional), Password (optional), Encryption, Sender address. +**D-06:** Existing `mail` module must be extended to read SMTP config from DB instead of hardcoded env vars. +**D-07:** Export recipient address configured per module (not in general settings). +**D-08:** Parse DKV E-Rechnung PDF using `pdf-parse` (text extraction + regex). +**D-09:** Parsed data per vehicle block: Kennzeichen (from `VEHICLE: ...` header), per-transaction rows with Lieferdatum, Servicestation Ort, Kilometerstand, Produkt, Menge, Einheit. +**D-10:** On parse failure: 3 retries, then mark as failed in processing history with error message. +**D-11:** Output format: `.xlsx` (Excel), generated with `xlsx` (SheetJS). +**D-12:** Filename: `DKV_YYYY-MM_.xlsx`. +**D-13:** Exactly 5 columns: Lieferdatum, Fahrzeug, Fahrer, Ort, Kilometerstand. +**D-14:** One Excel file per processed invoice. +**D-15:** Last 10 export files stored in `user-files/`, downloadable from module UI. +**D-16:** On SMTP failure: 3 retries with exponential backoff, then mark as "Versand fehlgeschlagen"; file stays available for manual download. +**D-17:** Vehicle master data in DB per tenant: Kennzeichen, Marke, Modell, Fahrer. +**D-18:** Importable as CSV; also manually editable in module UI (CRUD table). +**D-19:** Fahrzeug column format string configurable (default: `{Marke}/{Modell}/{Kennzeichen}`). +**D-20:** Processing history table in UI: Datum/Zeit, Rechnungsnummer, Anzahl Fahrzeuge, Anzahl Transaktionen, Status, Export-Dateiname. +**D-21:** History stored in DB; entries kept indefinitely (no auto-purge in v1). + +### Claude's Discretion + +- DB schema design (DkvModuleConfig, DkvVehicleMaster, DkvInvoiceHistory, SmtpConfig tables) +- Cron job implementation (`@nestjs/schedule` ScheduleModule, dynamic SchedulerRegistry pattern) +- EWS reuse from existing `ews-javascript-api` for inbox email access (mirroring ExchangeProvider calendar pattern) +- IMAP library: `imapflow` (confirmed, modern Promise-based) +- PDF parsing robustness (handle multi-page, multi-vehicle DKV format) +- Frontend component patterns (reuse existing admin table / settings patterns) + +### Deferred Ideas (OUT OF SCOPE) + +None — discussion stayed within phase scope. + + +--- + + +## Phase Requirements + +| ID | Description | Research Support | +|----|-------------|------------------| +| DKV-01 | System automatically detects DKV invoice PDFs from a configured email inbox (IMAP or Exchange) | imapflow + ews-javascript-api patterns, ScheduleModule cron job, sender filter search | +| DKV-02 | All transactions parsed per vehicle: date, service station, km, product, quantity, unit, net/gross | pdf-parse v2 text extraction + DKV-specific regex over vehicle blocks | +| DKV-03 | License plate → driver mapping importable as CSV and manually editable in UI | Prisma DkvVehicleMaster table, CSV import via multipart/form-data, CRUD controller + frontend table | +| DKV-04 | Processed data exported as Excel and sent to configurable recipient via SMTP | xlsx (SheetJS) buffer generation, nodemailer createTransport() with DB config, retry logic | +| DKV-05 | SMTP settings in general settings; module config (inbox, sender filter, folder, recipient) editable in module settings UI | SmtpConfig Prisma table, new /settings/general/smtp route, DkvModuleConfig table | + + +--- + +## Summary + +Phase 7 builds a NestJS module (`dkv`) that polls an email inbox (IMAP or Exchange), downloads PDF attachments from DKV, extracts vehicle/transaction data with regex, maps license plates to drivers via a configurable DB table, generates an Excel file, and delivers it via SMTP. The module is self-contained within the Tessera NestJS monorepo but touches three cross-cutting concerns: the mail module (SMTP config from DB), the settings subsystem (new "General" category for SMTP), and the portal frontend (new module route + settings route). + +All three new npm packages (`imapflow`, `pdf-parse`, `xlsx`) are vetted: `pdf-parse` and `xlsx` are OK per package legitimacy check; `imapflow` is flagged SUS due to a version published today (false positive — the package itself has existed since 2019 with 1.1M weekly downloads). The Exchange inbox access reuses the existing `ews-javascript-api@0.15.3` and follows the `ExchangeProvider` calendar pattern already in the codebase. + +The critical implementation detail is the **mail service extension strategy**: `@nestjs-modules/mailer` cannot change its SMTP transport at runtime (it's configured once at startup). The correct approach per D-06 is to add a `DkvMailService` that calls `nodemailer.createTransport()` directly with config loaded from DB at send time, while leaving the existing `MailService`/`MailerModule` untouched for system emails (password reset, welcome). + +**Primary recommendation:** Build the DKV module as a standard NestJS module following the CalendarModule provider-abstraction pattern. Handle SMTP separately via direct nodemailer transport creation. Use `SchedulerRegistry` for dynamic cron interval updates. + +--- + +## Architectural Responsibility Map + +| Capability | Primary Tier | Secondary Tier | Rationale | +|------------|-------------|----------------|-----------| +| Email inbox polling (IMAP/EWS) | API / Backend | — | Server-side background job; credentials never touch browser | +| PDF attachment download | API / Backend | — | Binary stream from inbox, processed server-side | +| DKV PDF text extraction + regex | API / Backend | — | CPU-bound parsing, no browser involvement | +| Excel file generation | API / Backend | — | xlsx library runs server-side; file written to user-files/ | +| SMTP send with attachment | API / Backend | — | Outbound mail is always server-to-server | +| Vehicle master CRUD | API / Backend | Frontend | REST endpoints + CRUD UI table | +| CSV vehicle import | API / Backend | Frontend | Multipart upload to API; parsed server-side | +| Processing history | Database / Storage | API | Prisma table queried by API, displayed in frontend | +| Module config (inbox settings) | API / Backend | Frontend | Config stored in DB, UI form to edit | +| SMTP config (general settings) | API / Backend | Frontend | SmtpConfig DB table, new settings page | +| Export file download | API / Backend | Frontend | File served from user-files/ via API endpoint | +| Module UI (history table, manual trigger) | Frontend Server (SSR) | Browser / Client | Next.js page with client components for state | + +--- + +## Standard Stack + +### Core (new packages for this phase) + +| Library | Version | Purpose | Why Standard | +|---------|---------|---------|--------------| +| `imapflow` | 1.4.3 | IMAP inbox polling, message search, attachment download | Modern Promise/async-iterator API, TypeScript types included, replaces older node-imap [ASSUMED based on CONTEXT.md D-01 decision] | +| `pdf-parse` | 2.4.5 | PDF text extraction from Buffer | Class-based API (v2), TypeScript types built-in, 5.9M weekly downloads [VERIFIED: npm registry] | +| `xlsx` | 0.18.5 | Excel file generation | SheetJS Community Edition, 11.5M weekly downloads, buffer output API, MIT license [VERIFIED: npm registry] | + +### Already installed (no new install needed) + +| Library | Version | Purpose | Note | +|---------|---------|---------|------| +| `ews-javascript-api` | 0.15.3 | Exchange inbox access | Follow existing `ExchangeProvider` pattern [ASSUMED: same API works for email folders] | +| `nodemailer` | ^9.0.1 | SMTP send | Use `createTransport()` directly (NOT via @nestjs-modules/mailer) [ASSUMED] | +| `@nestjs/schedule` | ^6.1.3 | Cron job scheduling | Already in package.json, NOT yet imported in AppModule — Wave 0 must add `ScheduleModule.forRoot()` [VERIFIED: codebase grep] | +| `@nestjs-modules/mailer` | ^2.3.7 | System emails (password reset) | Keep unchanged; DKV bypasses it for dynamic SMTP | + +### Installation (new packages only) + +```bash +pnpm --filter @tessera/api add imapflow pdf-parse xlsx +``` + +```bash +# TypeScript types — pdf-parse and imapflow ship their own types +# xlsx also ships types; no @types/* needed for these packages +pnpm --filter @tessera/api add -D @types/xlsx 2>/dev/null || true +``` + +**Version verification (run before installing):** +```bash +npm view imapflow version # 1.4.3 +npm view pdf-parse version # 2.4.5 +npm view xlsx version # 0.18.5 +``` + +--- + +## Package Legitimacy Audit + +> Packages verified via `gsd-tools query package-legitimacy check --ecosystem npm imapflow pdf-parse xlsx` + +| Package | Registry | Age | Downloads | Source Repo | Verdict | Disposition | +|---------|----------|-----|-----------|-------------|---------|-------------| +| `imapflow` | npm | Since 2019 | ~1.1M/wk | github.com/postalsys/imapflow | SUS (too-new flag) | **Approved** — flag is false positive (new *version* published today, not new package; 1.1M/wk and legitimate repo) | +| `pdf-parse` | npm | Since 2018 (v2: 2025) | ~5.9M/wk | github.com/mehmet-kozan/pdf-parse | OK | Approved | +| `xlsx` | npm | Since 2012 | ~11.5M/wk | github.com/SheetJS/sheetjs | OK | Approved | + +**Packages removed due to SLOP verdict:** none + +**Packages flagged as suspicious SUS:** `imapflow` — planner should note context (false positive: version released today, package established since 2019 with 1.1M/wk downloads from postalsys/imapflow). No `checkpoint:human-verify` needed given evidence. + +--- + +## Architecture Patterns + +### System Architecture Diagram + +``` +[Email Inbox: IMAP/EWS] + │ (poll every N min OR manual trigger) + ▼ +[DkvSchedulerService] ──── @nestjs/schedule SchedulerRegistry + │ + ▼ +[InboxProvider interface] + ├── ImapProvider (imapflow) + └── ExchangeInboxProvider (ews-javascript-api) + │ (search by sender filter, list unseen emails) + ▼ +[PDF Attachment Buffer] + │ + ▼ +[DkvParserService] + └── pdf-parse v2: PDFParse({ data: buffer }).getText() + └── Regex extraction: vehicle blocks → transaction rows + │ + ▼ +[DkvVehicleMaster lookup] ◄── Prisma: DkvVehicleMaster + └── Kennzeichen → Marke, Modell, Fahrer + │ + ▼ +[DkvExportService] + └── xlsx: build 5-column worksheet → Buffer + └── Write to user-files/DKV_YYYY-MM_.xlsx + └── Prune: keep last 10 files + │ + ▼ +[DkvMailService] + └── nodemailer.createTransport(smtpConfigFromDB) + └── sendMail({ to: recipient, attachments: [xlsx buffer] }) + └── 3 retries with exponential backoff + │ + ▼ +[DkvInvoiceHistory record saved] ──► Prisma: DkvInvoiceHistory + │ + ▼ +[Frontend: /modules/dkv-fleet/page.tsx] + ├── History table (GET /dkv/history) + ├── Manual "Jetzt prüfen" button (POST /dkv/check-now) + ├── Export file download (GET /dkv/exports/:filename) + └── Settings tab → /modules/dkv-fleet/settings/ + └── Inbox config form (GET/PUT /dkv/config) + └── Vehicle CRUD table (GET/POST/PUT/DELETE /dkv/vehicles) + └── CSV import button (POST /dkv/vehicles/import) + +[/settings/general/smtp] ──► GET/PUT /settings/smtp ──► SmtpConfig (Prisma) +``` + +### Recommended Project Structure + +``` +apps/api/src/dkv/ +├── dkv.module.ts # Module: imports ScheduleModule, CalendarCryptoService +├── dkv.controller.ts # REST: /dkv/* routes (history, config, vehicles, check-now, exports) +├── dkv.service.ts # Orchestration: coordinates scheduler, parser, export, mail +├── dkv-scheduler.service.ts # SchedulerRegistry wrapper, cron lifecycle management +├── dkv-parser.service.ts # pdf-parse + DKV regex → structured data +├── dkv-export.service.ts # xlsx buffer generation, user-files/ management (10-file limit) +├── dkv-mail.service.ts # nodemailer.createTransport() with SmtpConfig from DB +├── providers/ +│ ├── inbox-provider.interface.ts # InboxProvider: connect(), searchSender(), downloadAttachments() +│ ├── imap.provider.ts # imapflow implementation +│ └── exchange-inbox.provider.ts # ews-javascript-api implementation (mirrors ExchangeProvider) +└── dto/ + ├── dkv-config.dto.ts + ├── dkv-vehicle.dto.ts + └── dkv-history.dto.ts + +apps/api/src/settings/ +├── settings.module.ts # NEW: SMTP config management +├── settings.controller.ts # GET/PUT /settings/smtp +├── settings.service.ts # SmtpConfig CRUD + decrypt +└── dto/smtp-config.dto.ts + +apps/web/src/app/(portal)/modules/dkv-fleet/ +├── page.tsx # Main view: invoice history table +├── settings/ +│ ├── page.tsx # Module settings: inbox config + vehicle master +│ └── components/ +│ ├── InboxConfigForm.tsx +│ ├── VehicleTable.tsx +│ └── CsvImportButton.tsx +└── components/ + └── InvoiceHistoryTable.tsx + +apps/web/src/app/(portal)/settings/general/ +└── smtp/ + └── page.tsx # SMTP settings page (new settings category) +``` + +### Prisma Schema (new tables) + +```prisma +model DkvModuleConfig { + id String @id @default(uuid()) + tenantId String @unique + protocol String @default("imap") // 'imap' | 'exchange' + host String? + port Int? + encryption String @default("ssl-tls") // 'none' | 'starttls' | 'ssl-tls' + folder String @default("INBOX") + senderFilter String? + pollIntervalMin Int @default(60) + isActive Boolean @default(false) + exportRecipient String? + vehicleFormatString String @default("{Marke}/{Modell}/{Kennzeichen}") + encryptedInboxCreds String? // AES-256-GCM: JSON { username, password } encrypted + createdAt DateTime @default(now()) + updatedAt DateTime @updatedAt + @@index([tenantId]) +} + +model DkvVehicleMaster { + id String @id @default(uuid()) + tenantId String + kennzeichen String + marke String + modell String + fahrer String // "Vorname Nachname" + createdAt DateTime @default(now()) + updatedAt DateTime @updatedAt + @@unique([tenantId, kennzeichen]) + @@index([tenantId]) +} + +model DkvInvoiceHistory { + id String @id @default(uuid()) + tenantId String + datumZeit DateTime @default(now()) + rechnungsnummer String + anzahlFahrzeuge Int @default(0) + anzahlTransaktionen Int @default(0) + status String // 'Verarbeitet' | 'Fehler' | 'Versand fehlgeschlagen' + errorMessage String? + exportFilename String? + createdAt DateTime @default(now()) + @@index([tenantId]) + @@index([datumZeit]) +} + +model SmtpConfig { + id String @id @default(uuid()) + tenantId String @unique + host String + port Int @default(587) + encryption String @default("starttls") // 'none' | 'starttls' | 'ssl-tls' + username String? + encryptedPassword String? // AES-256-GCM via CalendarCryptoService + fromAddress String + createdAt DateTime @default(now()) + updatedAt DateTime @updatedAt +} +``` + +### Pattern 1: InboxProvider Interface (mirrors CalendarProvider) + +**What:** Abstract inbox provider so IMAP and Exchange are interchangeable. +**When to use:** Any service that needs to list and download email attachments. + +```typescript +// Source: mirrors apps/api/src/calendar/calendar.service.ts CalendarProvider pattern +export interface InboxEmail { + uid: number | string; + messageId: string; + subject: string; + from: string; + date: Date; + attachments: InboxAttachment[]; +} + +export interface InboxAttachment { + filename: string; + contentType: string; + buffer: Buffer; +} + +export interface InboxProvider { + fetchPdfAttachments( + config: InboxConfig, + ): Promise; + + testConnection(config: InboxConfig): Promise; +} +``` + +### Pattern 2: imapflow IMAP Provider + +**What:** Fetch emails from a specific folder filtered by sender, download PDF attachments. +**When to use:** When `DkvModuleConfig.protocol === 'imap'`. + +```typescript +// Source: [CITED: imapflow.com/docs/examples/fetching-messages/] +import { ImapFlow } from 'imapflow'; + +async function fetchPdfAttachments(config: InboxConfig): Promise { + const client = new ImapFlow({ + host: config.host, + port: config.port, + secure: config.encryption === 'ssl-tls', // port 993 + requireTLS: config.encryption === 'starttls', // port 587 + auth: { user: config.username, pass: config.password }, + logger: false, // suppress verbose imapflow logs + }); + + await client.connect(); + const lock = await client.getMailboxLock(config.folder); + const results: InboxEmail[] = []; + + try { + // Search by sender — returns UIDs + const uids = await client.search({ from: config.senderFilter }, { uid: true }); + if (uids.length === 0) return results; + + // Fetch all first, then process (avoids IMAP deadlock — see Pitfall 1) + const messages = await client.fetchAll( + uids.join(','), + { envelope: true, bodyStructure: true }, + { uid: true }, + ); + + for (const msg of messages) { + const pdfs = collectPdfParts(msg.bodyStructure); + const attachments: InboxAttachment[] = []; + for (const partId of pdfs) { + const { content } = await client.download( + String(msg.uid), partId, { uid: true } + ); + const buf = await streamToBuffer(content); + attachments.push({ filename: `attachment-${partId}.pdf`, contentType: 'application/pdf', buffer: buf }); + } + results.push({ uid: msg.uid!, messageId: msg.envelope.messageId ?? '', subject: msg.envelope.subject ?? '', from: msg.envelope.from?.[0]?.address ?? '', date: msg.envelope.date ?? new Date(), attachments }); + } + } finally { + lock.release(); + await client.logout(); + } + return results; +} +``` + +### Pattern 3: pdf-parse v2 Text Extraction + +**What:** Extract raw text from a PDF Buffer using the v2 class-based API. +**When to use:** After downloading a PDF attachment from the inbox. + +```typescript +// Source: [CITED: mehmet-kozan.github.io/pdf-parse/typedoc/] +// IMPORTANT: pdf-parse v2 uses a class-based API — NOT the v1 function call pdfParse(buffer) +import { PDFParse } from 'pdf-parse'; + +async function extractPdfText(buffer: Buffer): Promise { + const parser = new PDFParse({ data: buffer }); // 'data' key accepts Buffer [ASSUMED] + const result = await parser.getText(); + await parser.destroy(); // always call destroy() to free memory + return result.text; // full text from all pages concatenated +} +``` + +> **Assumption flag:** The `{ data: buffer }` LoadParameters key is inferred from the TypeDoc page — confirmed the interface accepts both URL and buffer, but exact property name tagged [ASSUMED]. Planner should add a Wave 0 verify step that runs a quick test parse against `user-files/invoice.pdf`. + +### Pattern 4: DKV PDF Regex — Vehicle Block Extraction + +**What:** Parse DKV E-Rechnung text into structured vehicle/transaction data. +**When to use:** After extracting raw text from the PDF. + +```typescript +// Source: [ASSUMED based on CONTEXT.md D-09 + D-specifics section] +// DKV PDF structure (from CONTEXT.md specifics): +// VEHICLE: {Kennzeichen} CARD NO.: {CardNo} +// ... transaction rows ... +// TOTAL: ... + +interface DkvTransaction { + lieferdatum: string; // "DD.MM.YYYY" + ort: string; // Servicestation Ort + kilometerstand: number; + produkt: string; + menge: number; + einheit: string; + netto: number; + brutto: number; +} + +interface DkvVehicleBlock { + kennzeichen: string; + cardNumber: string; + transactions: DkvTransaction[]; +} + +function parseDkvText(text: string): DkvVehicleBlock[] { + const vehicles: DkvVehicleBlock[] = []; + + // Split text into vehicle blocks — anchor on VEHICLE: marker + // Pattern: "VEHICLE: GP-JL 728E CARD NO.: 7020261..." + const vehicleBlockPattern = /VEHICLE:\s+(\S+)\s+CARD NO\.:\s+(\S+)([\s\S]*?)(?=VEHICLE:|$)/g; + let match: RegExpExecArray | null; + + while ((match = vehicleBlockPattern.exec(text)) !== null) { + const kennzeichen = match[1]; + const cardNumber = match[2]; + const blockText = match[3]; + + const transactions = parseTransactionRows(blockText, kennzeichen); + vehicles.push({ kennzeichen, cardNumber, transactions }); + } + return vehicles; +} + +// Transaction row regex — [ASSUMED] based on DKV invoice structure description +// Row format: "DD.MM.YYYY Station City NNN.NNN km Diesel NN.NNN L" +function parseTransactionRows(blockText: string, _plate: string): DkvTransaction[] { + const txPattern = + /(\d{2}\.\d{2}\.\d{4})\s+(.+?)\s{2,}([\d.,]+)\s+km\s+(\S+)\s+([\d.,]+)\s+(\w+)/g; + const transactions: DkvTransaction[] = []; + let m: RegExpExecArray | null; + while ((m = txPattern.exec(blockText)) !== null) { + transactions.push({ + lieferdatum: m[1], + ort: m[2].trim(), + kilometerstand: parseFloat(m[3].replace('.', '').replace(',', '.')), + produkt: m[4], + menge: parseFloat(m[5].replace('.', '').replace(',', '.')), + einheit: m[6], + netto: 0, // TODO: extract from block if present + brutto: 0, + }); + } + return transactions; +} +``` + +> **Warning:** The transaction row regex is [ASSUMED] from the CONTEXT.md structure description. The actual DKV PDF format must be validated against `user-files/invoice.pdf` in Wave 0 before the parser service is written. This is the highest-risk regex in the system. + +### Pattern 5: SheetJS Excel Generation + +**What:** Build a 5-column worksheet and return it as a Buffer. +**When to use:** After assembling all vehicle/transaction rows for one invoice. + +```typescript +// Source: [CITED: docs.sheetjs.com/docs/api/write-options/] +import * as XLSX from 'xlsx'; + +function buildExcelBuffer(rows: ExportRow[], vehicleFormat: string): Buffer { + const headers = ['Lieferdatum', 'Fahrzeug', 'Fahrer', 'Ort', 'Kilometerstand']; + const data = rows.map(r => [ + r.lieferdatum, // string "DD.MM.YYYY" — write as-is, no Date conversion + r.fahrzeug, // "Marke/Modell/Kennzeichen" — from format string + r.fahrer, // "Vorname Nachname" + r.ort, // string + r.kilometerstand // number — no unit + ]); + + const ws = XLSX.utils.aoa_to_sheet([headers, ...data]); + const wb = XLSX.utils.book_new(); + XLSX.utils.book_append_sheet(wb, ws, 'DKV Export'); + + return XLSX.write(wb, { type: 'buffer', bookType: 'xlsx' }) as Buffer; +} +``` + +### Pattern 6: Dynamic SMTP via Nodemailer (bypasses @nestjs-modules/mailer) + +**What:** Create a nodemailer transport from DB config at send time. +**When to use:** Every DKV export send — config may change between runs. + +```typescript +// Source: [CITED: nodemailer.com/smtp] [ASSUMED: dynamic createTransport pattern] +import * as nodemailer from 'nodemailer'; + +async function sendExportEmail( + smtpConfig: SmtpConfig, + recipient: string, + attachmentBuffer: Buffer, + filename: string, +): Promise { + const secure = smtpConfig.encryption === 'ssl-tls'; // port 465 + const requireTLS = smtpConfig.encryption === 'starttls'; // port 587 + + const transport = nodemailer.createTransport({ + host: smtpConfig.host, + port: smtpConfig.port, + secure, + requireTLS, + auth: smtpConfig.username + ? { user: smtpConfig.username, pass: smtpConfig.decryptedPassword } + : undefined, + }); + + await transport.sendMail({ + from: smtpConfig.fromAddress, + to: recipient, + subject: `DKV Flottenabrechnung: ${filename}`, + text: 'DKV Flottenabrechnung im Anhang.', + attachments: [ + { filename, content: attachmentBuffer, contentType: 'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet' }, + ], + }); +} +``` + +### Pattern 7: Dynamic Cron Job (SchedulerRegistry) + +**What:** Create/update a cron job from a DB-sourced interval in minutes. +**When to use:** On module init and after config change. + +```typescript +// Source: [CITED: oneuptime.com/blog/post/2026-02-02-nestjs-task-scheduling/view] +import { Injectable, OnModuleInit } from '@nestjs/common'; +import { SchedulerRegistry } from '@nestjs/schedule'; +import { CronJob } from 'cron'; + +const JOB_NAME = 'dkv-inbox-poll'; + +@Injectable() +export class DkvSchedulerService implements OnModuleInit { + constructor( + private readonly schedulerRegistry: SchedulerRegistry, + private readonly dkvService: DkvService, + ) {} + + onModuleInit() { + // Load interval from DB config on startup + this.dkvService.loadConfig().then(config => { + if (config?.isActive) { + this.setInterval(config.pollIntervalMin); + } + }); + } + + setInterval(intervalMin: number): void { + // Remove existing job if any + try { + const existing = this.schedulerRegistry.getCronJob(JOB_NAME); + existing.stop(); + this.schedulerRegistry.deleteCronJob(JOB_NAME); + } catch { /* not registered yet */ } + + // Create new job with computed cron expression + const cronExpr = `*/${intervalMin} * * * *`; // every N minutes + const job = new CronJob(cronExpr, () => { + this.dkvService.processInbox().catch(err => + this.logger.error('DKV inbox poll failed', err) + ); + }); + this.schedulerRegistry.addCronJob(JOB_NAME, job); + job.start(); + } +} +``` + +### Pattern 8: ScheduleModule Registration (AppModule update) + +`@nestjs/schedule` is installed but NOT yet imported in `app.module.ts`. Must be added in Wave 0. + +```typescript +// apps/api/src/app.module.ts — add to imports array +import { ScheduleModule } from '@nestjs/schedule'; + +@Module({ + imports: [ + // ... existing imports ... + ScheduleModule.forRoot(), // ADD THIS — required for cron/interval decorators + DkvModule, // ADD THIS — new DKV fleet module + ], +}) +``` + +### Settings Sidebar Extension + +Add "Allgemein" category to `SettingsSidebar` with SMTP link: + +```typescript +// apps/web/src/components/settings/settings-sidebar.tsx — extend items array +const generalItems = [ + { label: t('categorySmtp'), href: '/settings/general/smtp' }, +]; +``` + +### Anti-Patterns to Avoid + +- **Mixing fetch + download in imapflow async iterator:** The imapflow docs explicitly warn against it — causes IMAP connection deadlock. Always call `fetchAll()` first to collect all message metadata, then loop and call `download()` separately. +- **Using @nestjs-modules/mailer for dynamic SMTP:** The mailer transport is configured once at startup via `forRootAsync`. Attempting to change it at runtime breaks all system emails. Use `nodemailer.createTransport()` directly in `DkvMailService`. +- **Calling `pdfParse(buffer)` v1 API:** pdf-parse v2 is NOT backward-compatible. The old `import pdfParse from 'pdf-parse'; const data = await pdfParse(buffer);` call will fail. Use `new PDFParse({ data: buffer })` instead. +- **Writing Lieferdatum as a JS Date in xlsx:** The DKV dates are already formatted German strings ("13.12.2020"). Writing them as JS Date objects causes Excel to serialize them as numeric serials (or reformat them). Write as strings directly. +- **Storing decrypted credentials in logs:** NEVER log the result of `crypto.decrypt()` — follow T-05-13 pattern from CalendarService (generic error messages only). +- **@Interval() decorator with hardcoded value:** The poll interval is configurable from DB — use `SchedulerRegistry.addCronJob()` instead of the static `@Interval(ms)` decorator, so the interval can be updated when config changes. + +--- + +## Don't Hand-Roll + +| Problem | Don't Build | Use Instead | Why | +|---------|-------------|-------------|-----| +| PDF text extraction | Custom pdfjs integration | `pdf-parse` v2 | pdfjs-dist requires worker setup, much heavier | +| Excel file generation | CSV with .xlsx extension | `xlsx` (SheetJS) | Excel requires OOXML zip format; CSV named as xlsx corrupts in Excel | +| IMAP protocol | Raw TCP/IMAP commands | `imapflow` | IMAP has 30+ capabilities, extension negotiation, IDLE mode — do not hand-roll | +| AES encryption | Custom cipher | `CalendarCryptoService` | Already implemented, tested, consistent with calendar credentials | +| Cron scheduling | setInterval() in a Node.js service | `@nestjs/schedule` + `SchedulerRegistry` | Restart-safe, injectable, monitorable | +| CSV parsing | String split by comma | (inline split is fine for simple CSVs) | DKV vehicle CSV is simple (Kennzeichen;Marke;Modell;Fahrer) — regex or split is acceptable; don't add papaparse just for this | + +**Key insight:** PDF-to-Excel pipelines look simple but have many format edge cases (multi-page PDFs, Unicode station names, German number formats with dot-as-thousands-separator). Trust the established libraries for the binary format layers; only write custom regex for DKV-specific structure extraction. + +--- + +## Common Pitfalls + +### Pitfall 1: imapflow IMAP Connection Deadlock + +**What goes wrong:** Calling `client.download()` inside a `fetch()` async iterator causes the IMAP connection to deadlock — the iterator holds the connection open while `download()` tries to issue another command on the same connection. + +**Why it happens:** imapflow uses a single IMAP connection. The `fetch()` method returns an async iterator that streams messages; issuing another command while the iterator is active queues on the same connection and never resolves. + +**How to avoid:** Always use `fetchAll()` (which resolves once, fully) to collect message metadata first. Then loop over the results and call `download()` separately. This is explicitly documented in imapflow examples. + +**Warning signs:** `await client.download(...)` never resolves when called inside a `for await ... of client.fetch(...)` loop. + +### Pitfall 2: pdf-parse v1 vs v2 API Break + +**What goes wrong:** Calling `pdfParse(buffer)` (v1 API) throws `TypeError: pdfParse is not a function` or similar because pdf-parse v2 is a class-based API. + +**Why it happens:** The npm package `pdf-parse` at v2.x is a complete rewrite by mehmet-kozan with a new class-based API. Any tutorial from before 2025 shows the v1 function-call API. + +**How to avoid:** Use `new PDFParse({ data: buffer }); await parser.getText(); await parser.destroy();`. Write a Wave 0 integration test parsing `user-files/invoice.pdf` to confirm the API before writing the production parser. + +**Warning signs:** LangChain community issue #22 confirms this exact breakage: "PDFLoader declares pdf-parse v2 but only supports v1". + +### Pitfall 3: @nestjs-modules/mailer Cannot Change Transport at Runtime + +**What goes wrong:** Extending `MailModule` to load SMTP config from DB at startup seems straightforward, but if the user updates SMTP config in the UI, the running transport still uses the old settings until a restart. + +**Why it happens:** `MailerModule.forRootAsync()` constructs the nodemailer transport once when the NestJS DI container initializes. There's no API to replace the transport object afterward. + +**How to avoid:** Keep `MailerModule` for system emails (env-based, rarely changed). For DKV sends, use `nodemailer.createTransport(await loadSmtpConfig())` at send time in `DkvMailService`. Config changes take effect immediately on the next send. + +### Pitfall 4: ScheduleModule Not Registered in AppModule + +**What goes wrong:** `@Cron()` / `@Interval()` decorators silently do nothing; `SchedulerRegistry` throws "SchedulerRegistry is not injectable" or jobs never fire. + +**Why it happens:** `@nestjs/schedule` is installed but `ScheduleModule.forRoot()` is NOT imported in `AppModule` (confirmed by grep — absent from app.module.ts). + +**How to avoid:** Wave 0 task must add `ScheduleModule.forRoot()` to AppModule imports. This is a prerequisite for DkvSchedulerService to function. + +**Warning signs:** CronJob registered via `schedulerRegistry.addCronJob()` but callback never fires. + +### Pitfall 5: German Number Format in km/Menge Parsing + +**What goes wrong:** DKV PDFs use German number formatting: `19.234,56` (dot as thousands separator, comma as decimal). `parseFloat("19.234,56")` returns 19.234 (truncated at the comma). + +**Why it happens:** JavaScript `parseFloat` expects English format (comma as thousands separator, dot as decimal). + +**How to avoid:** Strip dots then replace comma with dot before parsing: `parseFloat(raw.replace(/\./g, '').replace(',', '.'))`. Apply this to Kilometerstand and Menge fields. + +### Pitfall 6: EWS Email vs Calendar Folder Access + +**What goes wrong:** Copying `ExchangeProvider` directly and calling `FindAppointments` returns no results because email folders use `FindItems` / `EmailMessage.Bind`, not `FindAppointments`. + +**Why it happens:** EWS has separate item types — `Appointment` for calendar, `EmailMessage` for inbox. The existing ExchangeProvider uses the Calendar well-known folder name. + +**How to avoid:** In `ExchangeInboxProvider`, use `ews.WellKnownFolderName.Inbox` with `service.FindItems()` and bind to `EmailMessage`. Follow EWS docs for email vs appointment distinction. [ASSUMED — not verified against ews-javascript-api 0.15.3 API docs] + +### Pitfall 7: user-files/ 10-File Pruning Race Condition + +**What goes wrong:** If two invoices are processed simultaneously (concurrent cron + manual trigger), both may pass the "10 files exist" check and write an 11th file before either runs the prune. + +**Why it happens:** File count check and write are not atomic. + +**How to avoid:** Use a mutex (simple in-memory lock in `DkvExportService`) or route all inbox processing through a queue so only one invoice processes at a time. Simplest: add a `private processing = false` guard in `DkvService.processInbox()` — if already processing, return early. + +--- + +## Code Examples + +### Inversion of Control: Credential Encryption Pattern + +```typescript +// Source: apps/api/src/calendar/crypto.service.ts [VERIFIED: codebase] +// Reuse CalendarCryptoService directly — it's not calendar-specific despite its name. +// Encrypt both inbox credentials and SMTP password with the same CALENDAR_ENCRYPTION_KEY. + +const encryptedInboxCreds = this.crypto.encrypt( + JSON.stringify({ username: dto.username, password: dto.password }) +); +// Store as single encrypted JSON blob in DkvModuleConfig.encryptedInboxCreds + +// Decrypt at use time: +const { username, password } = JSON.parse(this.crypto.decrypt(config.encryptedInboxCreds!)); +``` + +### CSV Vehicle Import Pattern + +```typescript +// Simple CSV parse for DKV vehicle master — semicolon-delimited +// Expected format: Kennzeichen;Marke;Modell;Fahrer +// Source: [ASSUMED] +function parseVehicleCsv(csvText: string): CreateVehicleDto[] { + const lines = csvText.trim().split('\n'); + const headers = lines[0].split(';').map(h => h.trim().toLowerCase()); + return lines.slice(1).map(line => { + const cols = line.split(';').map(c => c.trim()); + return { + kennzeichen: cols[headers.indexOf('kennzeichen')] ?? '', + marke: cols[headers.indexOf('marke')] ?? '', + modell: cols[headers.indexOf('modell')] ?? '', + fahrer: cols[headers.indexOf('fahrer')] ?? '', + }; + }).filter(v => v.kennzeichen); +} +``` + +### Fahrzeug Format String Resolution + +```typescript +// Source: [ASSUMED based on D-19 format template] +// Default: "{Marke}/{Modell}/{Kennzeichen}" +function resolveFahrzeug( + vehicle: DkvVehicleMaster, + formatString: string, +): string { + return formatString + .replace('{Marke}', vehicle.marke) + .replace('{Modell}', vehicle.modell) + .replace('{Kennzeichen}', vehicle.kennzeichen) + .replace('{Fahrer}', vehicle.fahrer); +} +``` + +--- + +## Runtime State Inventory + +> Greenfield module — no existing runtime state to migrate. All tables are new. + +| Category | Items Found | Action Required | +|----------|-------------|-----------------| +| Stored data | None — DkvModuleConfig, DkvVehicleMaster, DkvInvoiceHistory, SmtpConfig are new tables | Prisma migration only | +| Live service config | None | — | +| OS-registered state | None | — | +| Secrets/env vars | `CALENDAR_ENCRYPTION_KEY` is reused for DKV credential encryption — no new env var needed | Ensure env var is set in Docker Compose (already required by CalendarModule) | +| Build artifacts | None | — | + +--- + +## Environment Availability + +| Dependency | Required By | Available | Version | Fallback | +|------------|------------|-----------|---------|----------| +| Node.js | imapflow, pdf-parse, xlsx | Yes | v24.16.0 | — | +| Docker | DB + full stack | Yes | 29.5.3 | — | +| pnpm | package install | Yes | 9.15.0 | — | +| `CALENDAR_ENCRYPTION_KEY` env var | CalendarCryptoService (shared) | Assumed set | — | Module init throws if missing | +| `user-files/` directory | Export file storage | Yes (exists in repo) | — | Create if missing | +| `user-files/invoice.pdf` | Wave 0 parser validation | Yes (in repo) | — | Required for regex dev | + +**Missing dependencies with no fallback:** None identified. + +--- + +## State of the Art + +| Old Approach | Current Approach | When Changed | Impact | +|--------------|------------------|--------------|--------| +| pdf-parse v1: `pdfParse(buffer)` function | pdf-parse v2: `new PDFParse({data}).getText()` | v2.x (2025) | Breaking API change — all v1 tutorials are wrong | +| @nestjs/schedule: static `@Interval()` | Dynamic `SchedulerRegistry.addCronJob()` | Always supported | Required for DB-configurable intervals | +| @nestjs-modules/mailer dynamic transport | nodemailer.createTransport() at send time | Pattern choice | Only way to support runtime SMTP config changes | + +**Deprecated/outdated:** +- `node-imap`: Outdated callback-based IMAP library. Do not use. imapflow is the modern replacement. +- `pdf-parse` v1 API (`pdfParse(buffer)` function): Removed in v2.x. + +--- + +## Validation Architecture + +### Test Framework + +| Property | Value | +|----------|-------| +| Framework | Vitest 4.1.9 (apps/web only) | +| Config file | apps/web/vitest.config.ts | +| Quick run command | `pnpm --filter @tessera/web test` | +| Full suite command | `pnpm --filter @tessera/web test` | +| API test framework | None configured — no vitest/jest in apps/api | + +> **Note:** The API has no test framework configured. Phase 7 API logic (PDF parser, Excel builder, SMTP sender) is business-critical and stateless — ideal for unit tests. However, adding a full test framework to apps/api is out of scope for this phase. API logic should be validated manually using the Wave 0 parser validation task (parse `user-files/invoice.pdf`, inspect output). Frontend components follow the existing Vitest + Testing Library pattern. + +### Phase Requirements → Test Map + +| Req ID | Behavior | Test Type | Automated Command | File Exists? | +|--------|----------|-----------|-------------------|-------------| +| DKV-01 | Inbox polling detects DKV emails | Manual (requires live inbox) | — | — | +| DKV-02 | PDF parser extracts correct vehicle/transaction data | Manual integration (parse invoice.pdf in Wave 0) | — | ❌ Wave 0 | +| DKV-03 | Vehicle CRUD table renders and submits correctly | Unit (Vitest + Testing Library) | `pnpm --filter @tessera/web test` | ❌ Wave 0 | +| DKV-04 | Excel export has correct 5 columns and data | Manual (open generated .xlsx in Excel) | — | — | +| DKV-05 | SMTP settings form saves and reads back | Unit (Vitest + Testing Library) | `pnpm --filter @tessera/web test` | ❌ Wave 0 | + +### Sampling Rate + +- **Per task commit:** `pnpm --filter @tessera/web test` (frontend tests only) +- **Per wave merge:** `pnpm --filter @tessera/web test && pnpm --filter @tessera/api type-check` +- **Phase gate:** All frontend tests green + manual validation of PDF parse output + manual Excel inspection before `/gsd-verify-work` + +### Wave 0 Gaps + +- [ ] `apps/api/src/dkv/dkv-parser.service.spec.ts` — manual validation script parsing `user-files/invoice.pdf` and printing structured output (not automated test, but required before writing production regex) +- [ ] `apps/web/src/app/(portal)/modules/dkv-fleet/settings/components/VehicleTable.test.tsx` — covers DKV-03 +- [ ] `apps/web/src/app/(portal)/settings/general/smtp/smtp-settings.test.tsx` — covers DKV-05 + +--- + +## Security Domain + +> `security_enforcement: true` in config.json, ASVS level 1. + +### Applicable ASVS Categories + +| ASVS Category | Applies | Standard Control | +|---------------|---------|-----------------| +| V2 Authentication | No | Module is admin-only, gated by existing JwtAuthGuard + RolesGuard | +| V3 Session Management | No | Handled by existing auth layer | +| V4 Access Control | Yes | ADMIN role required for all /dkv/* endpoints — use `@Roles(Role.ADMIN)` | +| V5 Input Validation | Yes | class-validator DTOs for all inputs; sanitize senderFilter (email format), folder (alphanumeric/slash) | +| V6 Cryptography | Yes | CalendarCryptoService (AES-256-GCM) for credentials — never hand-roll | +| V7 Error Handling | Yes | Never expose decrypted credentials in error messages — follow T-05-13 generic error pattern | + +### Known Threat Patterns for this Stack + +| Pattern | STRIDE | Standard Mitigation | +|---------|--------|---------------------| +| IMAP credential exposure in logs | Info Disclosure | Never log decrypted passwords; use `logger: false` in ImapFlow constructor | +| SMTP open relay via user-supplied config | Tampering | Validate host/port inputs; SMTP config is admin-only | +| Path traversal in export filename | Tampering | Filename is generated server-side (`DKV_YYYY-MM_.xlsx`), never user-supplied | +| PDF bomb / decompression bomb | Denial of Service | Set max file size limit for email attachments before passing to pdf-parse | +| Credential injection via senderFilter | Tampering | Validate senderFilter as email address format (IsEmail() validator) | +| Excessive history accumulation | Denial of Service | Pagination on GET /dkv/history (page + limit params) | + +--- + +## Assumptions Log + +| # | Claim | Section | Risk if Wrong | +|---|-------|---------|---------------| +| A1 | `pdf-parse` v2 accepts `{ data: Buffer }` as LoadParameters | Pattern 3 / Code Examples | Parser fails at runtime; need to test against invoice.pdf in Wave 0 | +| A2 | DKV PDF regex pattern (VEHICLE: marker, transaction row format) matches actual invoice.pdf | Pattern 4 / Code Examples | Parser produces no vehicles; all invoices fail with parse error | +| A3 | `ews-javascript-api` v0.15.3 supports `FindItems` on Inbox folder for email messages (not just calendar) | Architecture Patterns | Exchange inbox polling broken; Exchange users cannot use the module | +| A4 | imapflow STARTTLS mode: `secure: false` connects without TLS then STARTTLS upgrades automatically | Pattern 2 | STARTTLS connections fail or use wrong port | +| A5 | nodemailer `createTransport()` `requireTLS: true` triggers STARTTLS negotiation | Pattern 6 | STARTTLS SMTP send fails | +| A6 | German date strings from DKV ("13.12.2020") written as strings in xlsx display correctly in Excel | Pattern 5 | Excel treats them as text — may cause sorting issues, but functionally correct | + +--- + +## Open Questions + +1. **pdf-parse v2 LoadParameters Buffer key** + - What we know: Documentation shows `{ url: '...' }` and `{ data: buffer }` loading modes + - What's unclear: Exact property name for Buffer loading (`data` vs `buffer` vs `content`) + - Recommendation: Wave 0 must include a 10-line test script: `new PDFParse({ data: fs.readFileSync('user-files/invoice.pdf') })` — if it fails, try `{ url: 'file://...' }` as fallback + +2. **DKV PDF exact regex for transaction rows** + - What we know: VEHICLE: marker anchors vehicle blocks; each row has date, station, km, product, quantity, unit + - What's unclear: Exact column spacing, whether amounts (Netto/Brutto) are on the same row or summary rows + - Recommendation: Wave 0 must print the raw text from invoice.pdf and examine it before writing regex + +3. **ScheduleModule in AppModule** + - What we know: `@nestjs/schedule@^6.1.3` is installed, NOT in AppModule imports + - What's unclear: Whether any other planned module (future phase) also needs it — no conflict either way + - Recommendation: Add `ScheduleModule.forRoot()` to AppModule in Wave 0, then DkvModule uses `SchedulerRegistry` + +4. **Settings sidebar restructuring** + - What we know: Settings currently has only "Dashboard" category (Widgets + Calendar) + - What's unclear: Whether to add "Allgemein > SMTP" as a new top-level sidebar category or embed it in an existing one + - Recommendation: Add a "Allgemein" category section to `SettingsSidebar` with SMTP as the first item — clean separation from dashboard settings + +--- + +## Sources + +### Primary (MEDIUM confidence — context7 not available, using official docs) + +- [ImapFlow Documentation — Fetching Messages Examples](https://imapflow.com/docs/examples/fetching-messages/) — fetch/search/download patterns +- [ImapFlow Client API](https://imapflow.com/docs/api/imapflow-client/) — API reference +- [pdf-parse TypeDoc](https://mehmet-kozan.github.io/pdf-parse/typedoc/index.html) — LoadParameters interface, Buffer loading +- [SheetJS Community Edition — Write Options](https://docs.sheetjs.com/docs/api/write-options/) — buffer write pattern +- [NestJS Task Scheduling — oneuptime.com](https://oneuptime.com/blog/post/2026-02-02-nestjs-task-scheduling/view) — SchedulerRegistry.addCronJob pattern + +### Secondary (LOW confidence — WebSearch with official source cross-check) + +- [npm: imapflow@1.4.3](https://www.npmjs.com/package/imapflow) — version + repository verified +- [npm: pdf-parse@2.4.5](https://www.npmjs.com/package/pdf-parse) — version + repository verified +- [npm: xlsx@0.18.5](https://www.npmjs.com/package/xlsx) — version + repository verified +- [Nodemailer SMTP docs](https://nodemailer.com/smtp/) — SMTP transport options + +### Codebase (VERIFIED: grep) + +- `apps/api/src/calendar/crypto.service.ts` — AES-256-GCM encryption service, reused for DKV credentials +- `apps/api/src/calendar/providers/exchange.provider.ts` — EWS pattern to mirror for inbox access +- `apps/api/src/calendar/calendar.service.ts` — Provider abstraction pattern +- `apps/api/src/mail/mail.module.ts` — Static SMTP config; confirms @nestjs-modules/mailer is startup-configured +- `apps/api/src/app.module.ts` — Confirmed `ScheduleModule` is NOT imported +- `apps/api/package.json` — Confirmed `@nestjs/schedule@^6.1.3`, `nodemailer@^9.0.1`, `ews-javascript-api@0.15.3` installed + +--- + +## Metadata + +**Confidence breakdown:** +- Standard Stack: MEDIUM — npm versions verified, API patterns from official docs but context7 unavailable +- Architecture: MEDIUM — mirrors established CalendarModule patterns; DB schema is Claude's discretion +- PDF Regex: LOW — DKV PDF structure from CONTEXT.md description only; must validate against invoice.pdf in Wave 0 +- Pitfalls: MEDIUM — pdf-parse v2 break and imapflow deadlock are documented; EWS email API is [ASSUMED] +- ScheduleModule: HIGH — absence confirmed by grep; setup pattern from official NestJS docs + +**Research date:** 2026-06-26 +**Valid until:** 2026-07-26 (stable libraries; DKV PDF format unlikely to change)