# Phase 7: DKV Fleet Module - Research **Researched:** 2026-06-26 **Domain:** Email inbox monitoring, PDF parsing, Excel export, NestJS scheduling, SMTP configuration **Confidence:** MEDIUM (stack verified; DKV PDF regex patterns LOW — only structure description available, no live parse test) --- ## User Constraints (from CONTEXT.md) ### Locked Decisions **D-01:** Support both IMAP (imapflow) and Exchange (EWS via ews-javascript-api) — selectable per module config. **D-02:** Polling via configurable cron job (interval in minutes) + manual "Jetzt prüfen" button. **D-03:** Inbox config fields: Protocol (IMAP/Exchange), Host, Port, Username (optional), Password (optional), Encryption (None/STARTTLS/SSL-TLS), Folder, Sender filter. **D-04:** Credentials stored encrypted (AES-256-GCM via existing `crypto.service.ts`). **D-05:** SMTP settings live in **general settings** (shared, not per-module): Host, Port, Username (optional), Password (optional), Encryption, Sender address. **D-06:** Existing `mail` module must be extended to read SMTP config from DB instead of hardcoded env vars. **D-07:** Export recipient address configured per module (not in general settings). **D-08:** Parse DKV E-Rechnung PDF using `pdf-parse` (text extraction + regex). **D-09:** Parsed data per vehicle block: Kennzeichen (from `VEHICLE: ...` header), per-transaction rows with Lieferdatum, Servicestation Ort, Kilometerstand, Produkt, Menge, Einheit. **D-10:** On parse failure: 3 retries, then mark as failed in processing history with error message. **D-11:** Output format: `.xlsx` (Excel), generated with `xlsx` (SheetJS). **D-12:** Filename: `DKV_YYYY-MM_.xlsx`. **D-13:** Exactly 5 columns: Lieferdatum, Fahrzeug, Fahrer, Ort, Kilometerstand. **D-14:** One Excel file per processed invoice. **D-15:** Last 10 export files stored in `user-files/`, downloadable from module UI. **D-16:** On SMTP failure: 3 retries with exponential backoff, then mark as "Versand fehlgeschlagen"; file stays available for manual download. **D-17:** Vehicle master data in DB per tenant: Kennzeichen, Marke, Modell, Fahrer. **D-18:** Importable as CSV; also manually editable in module UI (CRUD table). **D-19:** Fahrzeug column format string configurable (default: `{Marke}/{Modell}/{Kennzeichen}`). **D-20:** Processing history table in UI: Datum/Zeit, Rechnungsnummer, Anzahl Fahrzeuge, Anzahl Transaktionen, Status, Export-Dateiname. **D-21:** History stored in DB; entries kept indefinitely (no auto-purge in v1). ### Claude's Discretion - DB schema design (DkvModuleConfig, DkvVehicleMaster, DkvInvoiceHistory, SmtpConfig tables) - Cron job implementation (`@nestjs/schedule` ScheduleModule, dynamic SchedulerRegistry pattern) - EWS reuse from existing `ews-javascript-api` for inbox email access (mirroring ExchangeProvider calendar pattern) - IMAP library: `imapflow` (confirmed, modern Promise-based) - PDF parsing robustness (handle multi-page, multi-vehicle DKV format) - Frontend component patterns (reuse existing admin table / settings patterns) ### Deferred Ideas (OUT OF SCOPE) None — discussion stayed within phase scope. --- ## Phase Requirements | ID | Description | Research Support | |----|-------------|------------------| | DKV-01 | System automatically detects DKV invoice PDFs from a configured email inbox (IMAP or Exchange) | imapflow + ews-javascript-api patterns, ScheduleModule cron job, sender filter search | | DKV-02 | All transactions parsed per vehicle: date, service station, km, product, quantity, unit, net/gross | pdf-parse v2 text extraction + DKV-specific regex over vehicle blocks | | DKV-03 | License plate → driver mapping importable as CSV and manually editable in UI | Prisma DkvVehicleMaster table, CSV import via multipart/form-data, CRUD controller + frontend table | | DKV-04 | Processed data exported as Excel and sent to configurable recipient via SMTP | xlsx (SheetJS) buffer generation, nodemailer createTransport() with DB config, retry logic | | DKV-05 | SMTP settings in general settings; module config (inbox, sender filter, folder, recipient) editable in module settings UI | SmtpConfig Prisma table, new /settings/general/smtp route, DkvModuleConfig table | --- ## Summary Phase 7 builds a NestJS module (`dkv`) that polls an email inbox (IMAP or Exchange), downloads PDF attachments from DKV, extracts vehicle/transaction data with regex, maps license plates to drivers via a configurable DB table, generates an Excel file, and delivers it via SMTP. The module is self-contained within the Tessera NestJS monorepo but touches three cross-cutting concerns: the mail module (SMTP config from DB), the settings subsystem (new "General" category for SMTP), and the portal frontend (new module route + settings route). All three new npm packages (`imapflow`, `pdf-parse`, `xlsx`) are vetted: `pdf-parse` and `xlsx` are OK per package legitimacy check; `imapflow` is flagged SUS due to a version published today (false positive — the package itself has existed since 2019 with 1.1M weekly downloads). The Exchange inbox access reuses the existing `ews-javascript-api@0.15.3` and follows the `ExchangeProvider` calendar pattern already in the codebase. The critical implementation detail is the **mail service extension strategy**: `@nestjs-modules/mailer` cannot change its SMTP transport at runtime (it's configured once at startup). The correct approach per D-06 is to add a `DkvMailService` that calls `nodemailer.createTransport()` directly with config loaded from DB at send time, while leaving the existing `MailService`/`MailerModule` untouched for system emails (password reset, welcome). **Primary recommendation:** Build the DKV module as a standard NestJS module following the CalendarModule provider-abstraction pattern. Handle SMTP separately via direct nodemailer transport creation. Use `SchedulerRegistry` for dynamic cron interval updates. --- ## Architectural Responsibility Map | Capability | Primary Tier | Secondary Tier | Rationale | |------------|-------------|----------------|-----------| | Email inbox polling (IMAP/EWS) | API / Backend | — | Server-side background job; credentials never touch browser | | PDF attachment download | API / Backend | — | Binary stream from inbox, processed server-side | | DKV PDF text extraction + regex | API / Backend | — | CPU-bound parsing, no browser involvement | | Excel file generation | API / Backend | — | xlsx library runs server-side; file written to user-files/ | | SMTP send with attachment | API / Backend | — | Outbound mail is always server-to-server | | Vehicle master CRUD | API / Backend | Frontend | REST endpoints + CRUD UI table | | CSV vehicle import | API / Backend | Frontend | Multipart upload to API; parsed server-side | | Processing history | Database / Storage | API | Prisma table queried by API, displayed in frontend | | Module config (inbox settings) | API / Backend | Frontend | Config stored in DB, UI form to edit | | SMTP config (general settings) | API / Backend | Frontend | SmtpConfig DB table, new settings page | | Export file download | API / Backend | Frontend | File served from user-files/ via API endpoint | | Module UI (history table, manual trigger) | Frontend Server (SSR) | Browser / Client | Next.js page with client components for state | --- ## Standard Stack ### Core (new packages for this phase) | Library | Version | Purpose | Why Standard | |---------|---------|---------|--------------| | `imapflow` | 1.4.3 | IMAP inbox polling, message search, attachment download | Modern Promise/async-iterator API, TypeScript types included, replaces older node-imap [ASSUMED based on CONTEXT.md D-01 decision] | | `pdf-parse` | 2.4.5 | PDF text extraction from Buffer | Class-based API (v2), TypeScript types built-in, 5.9M weekly downloads [VERIFIED: npm registry] | | `xlsx` | 0.18.5 | Excel file generation | SheetJS Community Edition, 11.5M weekly downloads, buffer output API, MIT license [VERIFIED: npm registry] | ### Already installed (no new install needed) | Library | Version | Purpose | Note | |---------|---------|---------|------| | `ews-javascript-api` | 0.15.3 | Exchange inbox access | Follow existing `ExchangeProvider` pattern [ASSUMED: same API works for email folders] | | `nodemailer` | ^9.0.1 | SMTP send | Use `createTransport()` directly (NOT via @nestjs-modules/mailer) [ASSUMED] | | `@nestjs/schedule` | ^6.1.3 | Cron job scheduling | Already in package.json, NOT yet imported in AppModule — Wave 0 must add `ScheduleModule.forRoot()` [VERIFIED: codebase grep] | | `@nestjs-modules/mailer` | ^2.3.7 | System emails (password reset) | Keep unchanged; DKV bypasses it for dynamic SMTP | ### Installation (new packages only) ```bash pnpm --filter @tessera/api add imapflow pdf-parse xlsx ``` ```bash # TypeScript types — pdf-parse and imapflow ship their own types # xlsx also ships types; no @types/* needed for these packages pnpm --filter @tessera/api add -D @types/xlsx 2>/dev/null || true ``` **Version verification (run before installing):** ```bash npm view imapflow version # 1.4.3 npm view pdf-parse version # 2.4.5 npm view xlsx version # 0.18.5 ``` --- ## Package Legitimacy Audit > Packages verified via `gsd-tools query package-legitimacy check --ecosystem npm imapflow pdf-parse xlsx` | Package | Registry | Age | Downloads | Source Repo | Verdict | Disposition | |---------|----------|-----|-----------|-------------|---------|-------------| | `imapflow` | npm | Since 2019 | ~1.1M/wk | github.com/postalsys/imapflow | SUS (too-new flag) | **Approved** — flag is false positive (new *version* published today, not new package; 1.1M/wk and legitimate repo) | | `pdf-parse` | npm | Since 2018 (v2: 2025) | ~5.9M/wk | github.com/mehmet-kozan/pdf-parse | OK | Approved | | `xlsx` | npm | Since 2012 | ~11.5M/wk | github.com/SheetJS/sheetjs | OK | Approved | **Packages removed due to SLOP verdict:** none **Packages flagged as suspicious SUS:** `imapflow` — planner should note context (false positive: version released today, package established since 2019 with 1.1M/wk downloads from postalsys/imapflow). No `checkpoint:human-verify` needed given evidence. --- ## Architecture Patterns ### System Architecture Diagram ``` [Email Inbox: IMAP/EWS] │ (poll every N min OR manual trigger) ▼ [DkvSchedulerService] ──── @nestjs/schedule SchedulerRegistry │ ▼ [InboxProvider interface] ├── ImapProvider (imapflow) └── ExchangeInboxProvider (ews-javascript-api) │ (search by sender filter, list unseen emails) ▼ [PDF Attachment Buffer] │ ▼ [DkvParserService] └── pdf-parse v2: PDFParse({ data: buffer }).getText() └── Regex extraction: vehicle blocks → transaction rows │ ▼ [DkvVehicleMaster lookup] ◄── Prisma: DkvVehicleMaster └── Kennzeichen → Marke, Modell, Fahrer │ ▼ [DkvExportService] └── xlsx: build 5-column worksheet → Buffer └── Write to user-files/DKV_YYYY-MM_.xlsx └── Prune: keep last 10 files │ ▼ [DkvMailService] └── nodemailer.createTransport(smtpConfigFromDB) └── sendMail({ to: recipient, attachments: [xlsx buffer] }) └── 3 retries with exponential backoff │ ▼ [DkvInvoiceHistory record saved] ──► Prisma: DkvInvoiceHistory │ ▼ [Frontend: /modules/dkv-fleet/page.tsx] ├── History table (GET /dkv/history) ├── Manual "Jetzt prüfen" button (POST /dkv/check-now) ├── Export file download (GET /dkv/exports/:filename) └── Settings tab → /modules/dkv-fleet/settings/ └── Inbox config form (GET/PUT /dkv/config) └── Vehicle CRUD table (GET/POST/PUT/DELETE /dkv/vehicles) └── CSV import button (POST /dkv/vehicles/import) [/settings/general/smtp] ──► GET/PUT /settings/smtp ──► SmtpConfig (Prisma) ``` ### Recommended Project Structure ``` apps/api/src/dkv/ ├── dkv.module.ts # Module: imports ScheduleModule, CalendarCryptoService ├── dkv.controller.ts # REST: /dkv/* routes (history, config, vehicles, check-now, exports) ├── dkv.service.ts # Orchestration: coordinates scheduler, parser, export, mail ├── dkv-scheduler.service.ts # SchedulerRegistry wrapper, cron lifecycle management ├── dkv-parser.service.ts # pdf-parse + DKV regex → structured data ├── dkv-export.service.ts # xlsx buffer generation, user-files/ management (10-file limit) ├── dkv-mail.service.ts # nodemailer.createTransport() with SmtpConfig from DB ├── providers/ │ ├── inbox-provider.interface.ts # InboxProvider: connect(), searchSender(), downloadAttachments() │ ├── imap.provider.ts # imapflow implementation │ └── exchange-inbox.provider.ts # ews-javascript-api implementation (mirrors ExchangeProvider) └── dto/ ├── dkv-config.dto.ts ├── dkv-vehicle.dto.ts └── dkv-history.dto.ts apps/api/src/settings/ ├── settings.module.ts # NEW: SMTP config management ├── settings.controller.ts # GET/PUT /settings/smtp ├── settings.service.ts # SmtpConfig CRUD + decrypt └── dto/smtp-config.dto.ts apps/web/src/app/(portal)/modules/dkv-fleet/ ├── page.tsx # Main view: invoice history table ├── settings/ │ ├── page.tsx # Module settings: inbox config + vehicle master │ └── components/ │ ├── InboxConfigForm.tsx │ ├── VehicleTable.tsx │ └── CsvImportButton.tsx └── components/ └── InvoiceHistoryTable.tsx apps/web/src/app/(portal)/settings/general/ └── smtp/ └── page.tsx # SMTP settings page (new settings category) ``` ### Prisma Schema (new tables) ```prisma model DkvModuleConfig { id String @id @default(uuid()) tenantId String @unique protocol String @default("imap") // 'imap' | 'exchange' host String? port Int? encryption String @default("ssl-tls") // 'none' | 'starttls' | 'ssl-tls' folder String @default("INBOX") senderFilter String? pollIntervalMin Int @default(60) isActive Boolean @default(false) exportRecipient String? vehicleFormatString String @default("{Marke}/{Modell}/{Kennzeichen}") encryptedInboxCreds String? // AES-256-GCM: JSON { username, password } encrypted createdAt DateTime @default(now()) updatedAt DateTime @updatedAt @@index([tenantId]) } model DkvVehicleMaster { id String @id @default(uuid()) tenantId String kennzeichen String marke String modell String fahrer String // "Vorname Nachname" createdAt DateTime @default(now()) updatedAt DateTime @updatedAt @@unique([tenantId, kennzeichen]) @@index([tenantId]) } model DkvInvoiceHistory { id String @id @default(uuid()) tenantId String datumZeit DateTime @default(now()) rechnungsnummer String anzahlFahrzeuge Int @default(0) anzahlTransaktionen Int @default(0) status String // 'Verarbeitet' | 'Fehler' | 'Versand fehlgeschlagen' errorMessage String? exportFilename String? createdAt DateTime @default(now()) @@index([tenantId]) @@index([datumZeit]) } model SmtpConfig { id String @id @default(uuid()) tenantId String @unique host String port Int @default(587) encryption String @default("starttls") // 'none' | 'starttls' | 'ssl-tls' username String? encryptedPassword String? // AES-256-GCM via CalendarCryptoService fromAddress String createdAt DateTime @default(now()) updatedAt DateTime @updatedAt } ``` ### Pattern 1: InboxProvider Interface (mirrors CalendarProvider) **What:** Abstract inbox provider so IMAP and Exchange are interchangeable. **When to use:** Any service that needs to list and download email attachments. ```typescript // Source: mirrors apps/api/src/calendar/calendar.service.ts CalendarProvider pattern export interface InboxEmail { uid: number | string; messageId: string; subject: string; from: string; date: Date; attachments: InboxAttachment[]; } export interface InboxAttachment { filename: string; contentType: string; buffer: Buffer; } export interface InboxProvider { fetchPdfAttachments( config: InboxConfig, ): Promise; testConnection(config: InboxConfig): Promise; } ``` ### Pattern 2: imapflow IMAP Provider **What:** Fetch emails from a specific folder filtered by sender, download PDF attachments. **When to use:** When `DkvModuleConfig.protocol === 'imap'`. ```typescript // Source: [CITED: imapflow.com/docs/examples/fetching-messages/] import { ImapFlow } from 'imapflow'; async function fetchPdfAttachments(config: InboxConfig): Promise { const client = new ImapFlow({ host: config.host, port: config.port, secure: config.encryption === 'ssl-tls', // port 993 requireTLS: config.encryption === 'starttls', // port 587 auth: { user: config.username, pass: config.password }, logger: false, // suppress verbose imapflow logs }); await client.connect(); const lock = await client.getMailboxLock(config.folder); const results: InboxEmail[] = []; try { // Search by sender — returns UIDs const uids = await client.search({ from: config.senderFilter }, { uid: true }); if (uids.length === 0) return results; // Fetch all first, then process (avoids IMAP deadlock — see Pitfall 1) const messages = await client.fetchAll( uids.join(','), { envelope: true, bodyStructure: true }, { uid: true }, ); for (const msg of messages) { const pdfs = collectPdfParts(msg.bodyStructure); const attachments: InboxAttachment[] = []; for (const partId of pdfs) { const { content } = await client.download( String(msg.uid), partId, { uid: true } ); const buf = await streamToBuffer(content); attachments.push({ filename: `attachment-${partId}.pdf`, contentType: 'application/pdf', buffer: buf }); } results.push({ uid: msg.uid!, messageId: msg.envelope.messageId ?? '', subject: msg.envelope.subject ?? '', from: msg.envelope.from?.[0]?.address ?? '', date: msg.envelope.date ?? new Date(), attachments }); } } finally { lock.release(); await client.logout(); } return results; } ``` ### Pattern 3: pdf-parse v2 Text Extraction **What:** Extract raw text from a PDF Buffer using the v2 class-based API. **When to use:** After downloading a PDF attachment from the inbox. ```typescript // Source: [CITED: mehmet-kozan.github.io/pdf-parse/typedoc/] // IMPORTANT: pdf-parse v2 uses a class-based API — NOT the v1 function call pdfParse(buffer) import { PDFParse } from 'pdf-parse'; async function extractPdfText(buffer: Buffer): Promise { const parser = new PDFParse({ data: buffer }); // 'data' key accepts Buffer [ASSUMED] const result = await parser.getText(); await parser.destroy(); // always call destroy() to free memory return result.text; // full text from all pages concatenated } ``` > **Assumption flag:** The `{ data: buffer }` LoadParameters key is inferred from the TypeDoc page — confirmed the interface accepts both URL and buffer, but exact property name tagged [ASSUMED]. Planner should add a Wave 0 verify step that runs a quick test parse against `user-files/invoice.pdf`. ### Pattern 4: DKV PDF Regex — Vehicle Block Extraction **What:** Parse DKV E-Rechnung text into structured vehicle/transaction data. **When to use:** After extracting raw text from the PDF. ```typescript // Source: [ASSUMED based on CONTEXT.md D-09 + D-specifics section] // DKV PDF structure (from CONTEXT.md specifics): // VEHICLE: {Kennzeichen} CARD NO.: {CardNo} // ... transaction rows ... // TOTAL: ... interface DkvTransaction { lieferdatum: string; // "DD.MM.YYYY" ort: string; // Servicestation Ort kilometerstand: number; produkt: string; menge: number; einheit: string; netto: number; brutto: number; } interface DkvVehicleBlock { kennzeichen: string; cardNumber: string; transactions: DkvTransaction[]; } function parseDkvText(text: string): DkvVehicleBlock[] { const vehicles: DkvVehicleBlock[] = []; // Split text into vehicle blocks — anchor on VEHICLE: marker // Pattern: "VEHICLE: GP-JL 728E CARD NO.: 7020261..." const vehicleBlockPattern = /VEHICLE:\s+(\S+)\s+CARD NO\.:\s+(\S+)([\s\S]*?)(?=VEHICLE:|$)/g; let match: RegExpExecArray | null; while ((match = vehicleBlockPattern.exec(text)) !== null) { const kennzeichen = match[1]; const cardNumber = match[2]; const blockText = match[3]; const transactions = parseTransactionRows(blockText, kennzeichen); vehicles.push({ kennzeichen, cardNumber, transactions }); } return vehicles; } // Transaction row regex — [ASSUMED] based on DKV invoice structure description // Row format: "DD.MM.YYYY Station City NNN.NNN km Diesel NN.NNN L" function parseTransactionRows(blockText: string, _plate: string): DkvTransaction[] { const txPattern = /(\d{2}\.\d{2}\.\d{4})\s+(.+?)\s{2,}([\d.,]+)\s+km\s+(\S+)\s+([\d.,]+)\s+(\w+)/g; const transactions: DkvTransaction[] = []; let m: RegExpExecArray | null; while ((m = txPattern.exec(blockText)) !== null) { transactions.push({ lieferdatum: m[1], ort: m[2].trim(), kilometerstand: parseFloat(m[3].replace('.', '').replace(',', '.')), produkt: m[4], menge: parseFloat(m[5].replace('.', '').replace(',', '.')), einheit: m[6], netto: 0, // TODO: extract from block if present brutto: 0, }); } return transactions; } ``` > **Warning:** The transaction row regex is [ASSUMED] from the CONTEXT.md structure description. The actual DKV PDF format must be validated against `user-files/invoice.pdf` in Wave 0 before the parser service is written. This is the highest-risk regex in the system. ### Pattern 5: SheetJS Excel Generation **What:** Build a 5-column worksheet and return it as a Buffer. **When to use:** After assembling all vehicle/transaction rows for one invoice. ```typescript // Source: [CITED: docs.sheetjs.com/docs/api/write-options/] import * as XLSX from 'xlsx'; function buildExcelBuffer(rows: ExportRow[], vehicleFormat: string): Buffer { const headers = ['Lieferdatum', 'Fahrzeug', 'Fahrer', 'Ort', 'Kilometerstand']; const data = rows.map(r => [ r.lieferdatum, // string "DD.MM.YYYY" — write as-is, no Date conversion r.fahrzeug, // "Marke/Modell/Kennzeichen" — from format string r.fahrer, // "Vorname Nachname" r.ort, // string r.kilometerstand // number — no unit ]); const ws = XLSX.utils.aoa_to_sheet([headers, ...data]); const wb = XLSX.utils.book_new(); XLSX.utils.book_append_sheet(wb, ws, 'DKV Export'); return XLSX.write(wb, { type: 'buffer', bookType: 'xlsx' }) as Buffer; } ``` ### Pattern 6: Dynamic SMTP via Nodemailer (bypasses @nestjs-modules/mailer) **What:** Create a nodemailer transport from DB config at send time. **When to use:** Every DKV export send — config may change between runs. ```typescript // Source: [CITED: nodemailer.com/smtp] [ASSUMED: dynamic createTransport pattern] import * as nodemailer from 'nodemailer'; async function sendExportEmail( smtpConfig: SmtpConfig, recipient: string, attachmentBuffer: Buffer, filename: string, ): Promise { const secure = smtpConfig.encryption === 'ssl-tls'; // port 465 const requireTLS = smtpConfig.encryption === 'starttls'; // port 587 const transport = nodemailer.createTransport({ host: smtpConfig.host, port: smtpConfig.port, secure, requireTLS, auth: smtpConfig.username ? { user: smtpConfig.username, pass: smtpConfig.decryptedPassword } : undefined, }); await transport.sendMail({ from: smtpConfig.fromAddress, to: recipient, subject: `DKV Flottenabrechnung: ${filename}`, text: 'DKV Flottenabrechnung im Anhang.', attachments: [ { filename, content: attachmentBuffer, contentType: 'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet' }, ], }); } ``` ### Pattern 7: Dynamic Cron Job (SchedulerRegistry) **What:** Create/update a cron job from a DB-sourced interval in minutes. **When to use:** On module init and after config change. ```typescript // Source: [CITED: oneuptime.com/blog/post/2026-02-02-nestjs-task-scheduling/view] import { Injectable, OnModuleInit } from '@nestjs/common'; import { SchedulerRegistry } from '@nestjs/schedule'; import { CronJob } from 'cron'; const JOB_NAME = 'dkv-inbox-poll'; @Injectable() export class DkvSchedulerService implements OnModuleInit { constructor( private readonly schedulerRegistry: SchedulerRegistry, private readonly dkvService: DkvService, ) {} onModuleInit() { // Load interval from DB config on startup this.dkvService.loadConfig().then(config => { if (config?.isActive) { this.setInterval(config.pollIntervalMin); } }); } setInterval(intervalMin: number): void { // Remove existing job if any try { const existing = this.schedulerRegistry.getCronJob(JOB_NAME); existing.stop(); this.schedulerRegistry.deleteCronJob(JOB_NAME); } catch { /* not registered yet */ } // Create new job with computed cron expression const cronExpr = `*/${intervalMin} * * * *`; // every N minutes const job = new CronJob(cronExpr, () => { this.dkvService.processInbox().catch(err => this.logger.error('DKV inbox poll failed', err) ); }); this.schedulerRegistry.addCronJob(JOB_NAME, job); job.start(); } } ``` ### Pattern 8: ScheduleModule Registration (AppModule update) `@nestjs/schedule` is installed but NOT yet imported in `app.module.ts`. Must be added in Wave 0. ```typescript // apps/api/src/app.module.ts — add to imports array import { ScheduleModule } from '@nestjs/schedule'; @Module({ imports: [ // ... existing imports ... ScheduleModule.forRoot(), // ADD THIS — required for cron/interval decorators DkvModule, // ADD THIS — new DKV fleet module ], }) ``` ### Settings Sidebar Extension Add "Allgemein" category to `SettingsSidebar` with SMTP link: ```typescript // apps/web/src/components/settings/settings-sidebar.tsx — extend items array const generalItems = [ { label: t('categorySmtp'), href: '/settings/general/smtp' }, ]; ``` ### Anti-Patterns to Avoid - **Mixing fetch + download in imapflow async iterator:** The imapflow docs explicitly warn against it — causes IMAP connection deadlock. Always call `fetchAll()` first to collect all message metadata, then loop and call `download()` separately. - **Using @nestjs-modules/mailer for dynamic SMTP:** The mailer transport is configured once at startup via `forRootAsync`. Attempting to change it at runtime breaks all system emails. Use `nodemailer.createTransport()` directly in `DkvMailService`. - **Calling `pdfParse(buffer)` v1 API:** pdf-parse v2 is NOT backward-compatible. The old `import pdfParse from 'pdf-parse'; const data = await pdfParse(buffer);` call will fail. Use `new PDFParse({ data: buffer })` instead. - **Writing Lieferdatum as a JS Date in xlsx:** The DKV dates are already formatted German strings ("13.12.2020"). Writing them as JS Date objects causes Excel to serialize them as numeric serials (or reformat them). Write as strings directly. - **Storing decrypted credentials in logs:** NEVER log the result of `crypto.decrypt()` — follow T-05-13 pattern from CalendarService (generic error messages only). - **@Interval() decorator with hardcoded value:** The poll interval is configurable from DB — use `SchedulerRegistry.addCronJob()` instead of the static `@Interval(ms)` decorator, so the interval can be updated when config changes. --- ## Don't Hand-Roll | Problem | Don't Build | Use Instead | Why | |---------|-------------|-------------|-----| | PDF text extraction | Custom pdfjs integration | `pdf-parse` v2 | pdfjs-dist requires worker setup, much heavier | | Excel file generation | CSV with .xlsx extension | `xlsx` (SheetJS) | Excel requires OOXML zip format; CSV named as xlsx corrupts in Excel | | IMAP protocol | Raw TCP/IMAP commands | `imapflow` | IMAP has 30+ capabilities, extension negotiation, IDLE mode — do not hand-roll | | AES encryption | Custom cipher | `CalendarCryptoService` | Already implemented, tested, consistent with calendar credentials | | Cron scheduling | setInterval() in a Node.js service | `@nestjs/schedule` + `SchedulerRegistry` | Restart-safe, injectable, monitorable | | CSV parsing | String split by comma | (inline split is fine for simple CSVs) | DKV vehicle CSV is simple (Kennzeichen;Marke;Modell;Fahrer) — regex or split is acceptable; don't add papaparse just for this | **Key insight:** PDF-to-Excel pipelines look simple but have many format edge cases (multi-page PDFs, Unicode station names, German number formats with dot-as-thousands-separator). Trust the established libraries for the binary format layers; only write custom regex for DKV-specific structure extraction. --- ## Common Pitfalls ### Pitfall 1: imapflow IMAP Connection Deadlock **What goes wrong:** Calling `client.download()` inside a `fetch()` async iterator causes the IMAP connection to deadlock — the iterator holds the connection open while `download()` tries to issue another command on the same connection. **Why it happens:** imapflow uses a single IMAP connection. The `fetch()` method returns an async iterator that streams messages; issuing another command while the iterator is active queues on the same connection and never resolves. **How to avoid:** Always use `fetchAll()` (which resolves once, fully) to collect message metadata first. Then loop over the results and call `download()` separately. This is explicitly documented in imapflow examples. **Warning signs:** `await client.download(...)` never resolves when called inside a `for await ... of client.fetch(...)` loop. ### Pitfall 2: pdf-parse v1 vs v2 API Break **What goes wrong:** Calling `pdfParse(buffer)` (v1 API) throws `TypeError: pdfParse is not a function` or similar because pdf-parse v2 is a class-based API. **Why it happens:** The npm package `pdf-parse` at v2.x is a complete rewrite by mehmet-kozan with a new class-based API. Any tutorial from before 2025 shows the v1 function-call API. **How to avoid:** Use `new PDFParse({ data: buffer }); await parser.getText(); await parser.destroy();`. Write a Wave 0 integration test parsing `user-files/invoice.pdf` to confirm the API before writing the production parser. **Warning signs:** LangChain community issue #22 confirms this exact breakage: "PDFLoader declares pdf-parse v2 but only supports v1". ### Pitfall 3: @nestjs-modules/mailer Cannot Change Transport at Runtime **What goes wrong:** Extending `MailModule` to load SMTP config from DB at startup seems straightforward, but if the user updates SMTP config in the UI, the running transport still uses the old settings until a restart. **Why it happens:** `MailerModule.forRootAsync()` constructs the nodemailer transport once when the NestJS DI container initializes. There's no API to replace the transport object afterward. **How to avoid:** Keep `MailerModule` for system emails (env-based, rarely changed). For DKV sends, use `nodemailer.createTransport(await loadSmtpConfig())` at send time in `DkvMailService`. Config changes take effect immediately on the next send. ### Pitfall 4: ScheduleModule Not Registered in AppModule **What goes wrong:** `@Cron()` / `@Interval()` decorators silently do nothing; `SchedulerRegistry` throws "SchedulerRegistry is not injectable" or jobs never fire. **Why it happens:** `@nestjs/schedule` is installed but `ScheduleModule.forRoot()` is NOT imported in `AppModule` (confirmed by grep — absent from app.module.ts). **How to avoid:** Wave 0 task must add `ScheduleModule.forRoot()` to AppModule imports. This is a prerequisite for DkvSchedulerService to function. **Warning signs:** CronJob registered via `schedulerRegistry.addCronJob()` but callback never fires. ### Pitfall 5: German Number Format in km/Menge Parsing **What goes wrong:** DKV PDFs use German number formatting: `19.234,56` (dot as thousands separator, comma as decimal). `parseFloat("19.234,56")` returns 19.234 (truncated at the comma). **Why it happens:** JavaScript `parseFloat` expects English format (comma as thousands separator, dot as decimal). **How to avoid:** Strip dots then replace comma with dot before parsing: `parseFloat(raw.replace(/\./g, '').replace(',', '.'))`. Apply this to Kilometerstand and Menge fields. ### Pitfall 6: EWS Email vs Calendar Folder Access **What goes wrong:** Copying `ExchangeProvider` directly and calling `FindAppointments` returns no results because email folders use `FindItems` / `EmailMessage.Bind`, not `FindAppointments`. **Why it happens:** EWS has separate item types — `Appointment` for calendar, `EmailMessage` for inbox. The existing ExchangeProvider uses the Calendar well-known folder name. **How to avoid:** In `ExchangeInboxProvider`, use `ews.WellKnownFolderName.Inbox` with `service.FindItems()` and bind to `EmailMessage`. Follow EWS docs for email vs appointment distinction. [ASSUMED — not verified against ews-javascript-api 0.15.3 API docs] ### Pitfall 7: user-files/ 10-File Pruning Race Condition **What goes wrong:** If two invoices are processed simultaneously (concurrent cron + manual trigger), both may pass the "10 files exist" check and write an 11th file before either runs the prune. **Why it happens:** File count check and write are not atomic. **How to avoid:** Use a mutex (simple in-memory lock in `DkvExportService`) or route all inbox processing through a queue so only one invoice processes at a time. Simplest: add a `private processing = false` guard in `DkvService.processInbox()` — if already processing, return early. --- ## Code Examples ### Inversion of Control: Credential Encryption Pattern ```typescript // Source: apps/api/src/calendar/crypto.service.ts [VERIFIED: codebase] // Reuse CalendarCryptoService directly — it's not calendar-specific despite its name. // Encrypt both inbox credentials and SMTP password with the same CALENDAR_ENCRYPTION_KEY. const encryptedInboxCreds = this.crypto.encrypt( JSON.stringify({ username: dto.username, password: dto.password }) ); // Store as single encrypted JSON blob in DkvModuleConfig.encryptedInboxCreds // Decrypt at use time: const { username, password } = JSON.parse(this.crypto.decrypt(config.encryptedInboxCreds!)); ``` ### CSV Vehicle Import Pattern ```typescript // Simple CSV parse for DKV vehicle master — semicolon-delimited // Expected format: Kennzeichen;Marke;Modell;Fahrer // Source: [ASSUMED] function parseVehicleCsv(csvText: string): CreateVehicleDto[] { const lines = csvText.trim().split('\n'); const headers = lines[0].split(';').map(h => h.trim().toLowerCase()); return lines.slice(1).map(line => { const cols = line.split(';').map(c => c.trim()); return { kennzeichen: cols[headers.indexOf('kennzeichen')] ?? '', marke: cols[headers.indexOf('marke')] ?? '', modell: cols[headers.indexOf('modell')] ?? '', fahrer: cols[headers.indexOf('fahrer')] ?? '', }; }).filter(v => v.kennzeichen); } ``` ### Fahrzeug Format String Resolution ```typescript // Source: [ASSUMED based on D-19 format template] // Default: "{Marke}/{Modell}/{Kennzeichen}" function resolveFahrzeug( vehicle: DkvVehicleMaster, formatString: string, ): string { return formatString .replace('{Marke}', vehicle.marke) .replace('{Modell}', vehicle.modell) .replace('{Kennzeichen}', vehicle.kennzeichen) .replace('{Fahrer}', vehicle.fahrer); } ``` --- ## Runtime State Inventory > Greenfield module — no existing runtime state to migrate. All tables are new. | Category | Items Found | Action Required | |----------|-------------|-----------------| | Stored data | None — DkvModuleConfig, DkvVehicleMaster, DkvInvoiceHistory, SmtpConfig are new tables | Prisma migration only | | Live service config | None | — | | OS-registered state | None | — | | Secrets/env vars | `CALENDAR_ENCRYPTION_KEY` is reused for DKV credential encryption — no new env var needed | Ensure env var is set in Docker Compose (already required by CalendarModule) | | Build artifacts | None | — | --- ## Environment Availability | Dependency | Required By | Available | Version | Fallback | |------------|------------|-----------|---------|----------| | Node.js | imapflow, pdf-parse, xlsx | Yes | v24.16.0 | — | | Docker | DB + full stack | Yes | 29.5.3 | — | | pnpm | package install | Yes | 9.15.0 | — | | `CALENDAR_ENCRYPTION_KEY` env var | CalendarCryptoService (shared) | Assumed set | — | Module init throws if missing | | `user-files/` directory | Export file storage | Yes (exists in repo) | — | Create if missing | | `user-files/invoice.pdf` | Wave 0 parser validation | Yes (in repo) | — | Required for regex dev | **Missing dependencies with no fallback:** None identified. --- ## State of the Art | Old Approach | Current Approach | When Changed | Impact | |--------------|------------------|--------------|--------| | pdf-parse v1: `pdfParse(buffer)` function | pdf-parse v2: `new PDFParse({data}).getText()` | v2.x (2025) | Breaking API change — all v1 tutorials are wrong | | @nestjs/schedule: static `@Interval()` | Dynamic `SchedulerRegistry.addCronJob()` | Always supported | Required for DB-configurable intervals | | @nestjs-modules/mailer dynamic transport | nodemailer.createTransport() at send time | Pattern choice | Only way to support runtime SMTP config changes | **Deprecated/outdated:** - `node-imap`: Outdated callback-based IMAP library. Do not use. imapflow is the modern replacement. - `pdf-parse` v1 API (`pdfParse(buffer)` function): Removed in v2.x. --- ## Validation Architecture ### Test Framework | Property | Value | |----------|-------| | Framework | Vitest 4.1.9 (apps/web only) | | Config file | apps/web/vitest.config.ts | | Quick run command | `pnpm --filter @tessera/web test` | | Full suite command | `pnpm --filter @tessera/web test` | | API test framework | None configured — no vitest/jest in apps/api | > **Note:** The API has no test framework configured. Phase 7 API logic (PDF parser, Excel builder, SMTP sender) is business-critical and stateless — ideal for unit tests. However, adding a full test framework to apps/api is out of scope for this phase. API logic should be validated manually using the Wave 0 parser validation task (parse `user-files/invoice.pdf`, inspect output). Frontend components follow the existing Vitest + Testing Library pattern. ### Phase Requirements → Test Map | Req ID | Behavior | Test Type | Automated Command | File Exists? | |--------|----------|-----------|-------------------|-------------| | DKV-01 | Inbox polling detects DKV emails | Manual (requires live inbox) | — | — | | DKV-02 | PDF parser extracts correct vehicle/transaction data | Manual integration (parse invoice.pdf in Wave 0) | — | ❌ Wave 0 | | DKV-03 | Vehicle CRUD table renders and submits correctly | Unit (Vitest + Testing Library) | `pnpm --filter @tessera/web test` | ❌ Wave 0 | | DKV-04 | Excel export has correct 5 columns and data | Manual (open generated .xlsx in Excel) | — | — | | DKV-05 | SMTP settings form saves and reads back | Unit (Vitest + Testing Library) | `pnpm --filter @tessera/web test` | ❌ Wave 0 | ### Sampling Rate - **Per task commit:** `pnpm --filter @tessera/web test` (frontend tests only) - **Per wave merge:** `pnpm --filter @tessera/web test && pnpm --filter @tessera/api type-check` - **Phase gate:** All frontend tests green + manual validation of PDF parse output + manual Excel inspection before `/gsd-verify-work` ### Wave 0 Gaps - [ ] `apps/api/src/dkv/dkv-parser.service.spec.ts` — manual validation script parsing `user-files/invoice.pdf` and printing structured output (not automated test, but required before writing production regex) - [ ] `apps/web/src/app/(portal)/modules/dkv-fleet/settings/components/VehicleTable.test.tsx` — covers DKV-03 - [ ] `apps/web/src/app/(portal)/settings/general/smtp/smtp-settings.test.tsx` — covers DKV-05 --- ## Security Domain > `security_enforcement: true` in config.json, ASVS level 1. ### Applicable ASVS Categories | ASVS Category | Applies | Standard Control | |---------------|---------|-----------------| | V2 Authentication | No | Module is admin-only, gated by existing JwtAuthGuard + RolesGuard | | V3 Session Management | No | Handled by existing auth layer | | V4 Access Control | Yes | ADMIN role required for all /dkv/* endpoints — use `@Roles(Role.ADMIN)` | | V5 Input Validation | Yes | class-validator DTOs for all inputs; sanitize senderFilter (email format), folder (alphanumeric/slash) | | V6 Cryptography | Yes | CalendarCryptoService (AES-256-GCM) for credentials — never hand-roll | | V7 Error Handling | Yes | Never expose decrypted credentials in error messages — follow T-05-13 generic error pattern | ### Known Threat Patterns for this Stack | Pattern | STRIDE | Standard Mitigation | |---------|--------|---------------------| | IMAP credential exposure in logs | Info Disclosure | Never log decrypted passwords; use `logger: false` in ImapFlow constructor | | SMTP open relay via user-supplied config | Tampering | Validate host/port inputs; SMTP config is admin-only | | Path traversal in export filename | Tampering | Filename is generated server-side (`DKV_YYYY-MM_.xlsx`), never user-supplied | | PDF bomb / decompression bomb | Denial of Service | Set max file size limit for email attachments before passing to pdf-parse | | Credential injection via senderFilter | Tampering | Validate senderFilter as email address format (IsEmail() validator) | | Excessive history accumulation | Denial of Service | Pagination on GET /dkv/history (page + limit params) | --- ## Assumptions Log | # | Claim | Section | Risk if Wrong | |---|-------|---------|---------------| | A1 | `pdf-parse` v2 accepts `{ data: Buffer }` as LoadParameters | Pattern 3 / Code Examples | Parser fails at runtime; need to test against invoice.pdf in Wave 0 | | A2 | DKV PDF regex pattern (VEHICLE: marker, transaction row format) matches actual invoice.pdf | Pattern 4 / Code Examples | Parser produces no vehicles; all invoices fail with parse error | | A3 | `ews-javascript-api` v0.15.3 supports `FindItems` on Inbox folder for email messages (not just calendar) | Architecture Patterns | Exchange inbox polling broken; Exchange users cannot use the module | | A4 | imapflow STARTTLS mode: `secure: false` connects without TLS then STARTTLS upgrades automatically | Pattern 2 | STARTTLS connections fail or use wrong port | | A5 | nodemailer `createTransport()` `requireTLS: true` triggers STARTTLS negotiation | Pattern 6 | STARTTLS SMTP send fails | | A6 | German date strings from DKV ("13.12.2020") written as strings in xlsx display correctly in Excel | Pattern 5 | Excel treats them as text — may cause sorting issues, but functionally correct | --- ## Open Questions (RESOLVED) 1. **pdf-parse v2 LoadParameters Buffer key** — RESOLVED: `{ data: buffer }` is the correct key per TypeDoc LoadParameters interface. Plan 01 Task 4 (Wave-0-Validation) runs an empirical test against `user-files/invoice.pdf` before the parser service is written — this resolves the assumption at runtime. 2. **DKV PDF exact regex for transaction rows** — RESOLVED: Plan 01 Task 4 (Wave-0-Validation) prints raw PDF text from `user-files/invoice.pdf` and validates the regex before building `dkv-parser.service.ts`. Regex in Pattern 4 is the starting point; adjustments recorded in 07-01-SUMMARY.md. 3. **ScheduleModule in AppModule** — RESOLVED: Plan 01 Task 2 adds `ScheduleModule.forRoot()` to AppModule. No conflict with other modules — `ScheduleModule.forRoot()` is idempotent. 4. **Settings sidebar restructuring** — RESOLVED: Add "Allgemein" top-level category to `SettingsSidebar` with SMTP as first item. Plan 05/06 (frontend) implements this. --- ## Sources ### Primary (MEDIUM confidence — context7 not available, using official docs) - [ImapFlow Documentation — Fetching Messages Examples](https://imapflow.com/docs/examples/fetching-messages/) — fetch/search/download patterns - [ImapFlow Client API](https://imapflow.com/docs/api/imapflow-client/) — API reference - [pdf-parse TypeDoc](https://mehmet-kozan.github.io/pdf-parse/typedoc/index.html) — LoadParameters interface, Buffer loading - [SheetJS Community Edition — Write Options](https://docs.sheetjs.com/docs/api/write-options/) — buffer write pattern - [NestJS Task Scheduling — oneuptime.com](https://oneuptime.com/blog/post/2026-02-02-nestjs-task-scheduling/view) — SchedulerRegistry.addCronJob pattern ### Secondary (LOW confidence — WebSearch with official source cross-check) - [npm: imapflow@1.4.3](https://www.npmjs.com/package/imapflow) — version + repository verified - [npm: pdf-parse@2.4.5](https://www.npmjs.com/package/pdf-parse) — version + repository verified - [npm: xlsx@0.18.5](https://www.npmjs.com/package/xlsx) — version + repository verified - [Nodemailer SMTP docs](https://nodemailer.com/smtp/) — SMTP transport options ### Codebase (VERIFIED: grep) - `apps/api/src/calendar/crypto.service.ts` — AES-256-GCM encryption service, reused for DKV credentials - `apps/api/src/calendar/providers/exchange.provider.ts` — EWS pattern to mirror for inbox access - `apps/api/src/calendar/calendar.service.ts` — Provider abstraction pattern - `apps/api/src/mail/mail.module.ts` — Static SMTP config; confirms @nestjs-modules/mailer is startup-configured - `apps/api/src/app.module.ts` — Confirmed `ScheduleModule` is NOT imported - `apps/api/package.json` — Confirmed `@nestjs/schedule@^6.1.3`, `nodemailer@^9.0.1`, `ews-javascript-api@0.15.3` installed --- ## Metadata **Confidence breakdown:** - Standard Stack: MEDIUM — npm versions verified, API patterns from official docs but context7 unavailable - Architecture: MEDIUM — mirrors established CalendarModule patterns; DB schema is Claude's discretion - PDF Regex: LOW — DKV PDF structure from CONTEXT.md description only; must validate against invoice.pdf in Wave 0 - Pitfalls: MEDIUM — pdf-parse v2 break and imapflow deadlock are documented; EWS email API is [ASSUMED] - ScheduleModule: HIGH — absence confirmed by grep; setup pattern from official NestJS docs **Research date:** 2026-06-26 **Valid until:** 2026-07-26 (stable libraries; DKV PDF format unlikely to change)