docs(07): research phase DKV fleet module
This commit is contained in:
@@ -0,0 +1,972 @@
|
||||
# Phase 7: DKV Fleet Module - Research
|
||||
|
||||
**Researched:** 2026-06-26
|
||||
**Domain:** Email inbox monitoring, PDF parsing, Excel export, NestJS scheduling, SMTP configuration
|
||||
**Confidence:** MEDIUM (stack verified; DKV PDF regex patterns LOW — only structure description available, no live parse test)
|
||||
|
||||
---
|
||||
|
||||
<user_constraints>
|
||||
## User Constraints (from CONTEXT.md)
|
||||
|
||||
### Locked Decisions
|
||||
|
||||
**D-01:** Support both IMAP (imapflow) and Exchange (EWS via ews-javascript-api) — selectable per module config.
|
||||
**D-02:** Polling via configurable cron job (interval in minutes) + manual "Jetzt prüfen" button.
|
||||
**D-03:** Inbox config fields: Protocol (IMAP/Exchange), Host, Port, Username (optional), Password (optional), Encryption (None/STARTTLS/SSL-TLS), Folder, Sender filter.
|
||||
**D-04:** Credentials stored encrypted (AES-256-GCM via existing `crypto.service.ts`).
|
||||
**D-05:** SMTP settings live in **general settings** (shared, not per-module): Host, Port, Username (optional), Password (optional), Encryption, Sender address.
|
||||
**D-06:** Existing `mail` module must be extended to read SMTP config from DB instead of hardcoded env vars.
|
||||
**D-07:** Export recipient address configured per module (not in general settings).
|
||||
**D-08:** Parse DKV E-Rechnung PDF using `pdf-parse` (text extraction + regex).
|
||||
**D-09:** Parsed data per vehicle block: Kennzeichen (from `VEHICLE: ...` header), per-transaction rows with Lieferdatum, Servicestation Ort, Kilometerstand, Produkt, Menge, Einheit.
|
||||
**D-10:** On parse failure: 3 retries, then mark as failed in processing history with error message.
|
||||
**D-11:** Output format: `.xlsx` (Excel), generated with `xlsx` (SheetJS).
|
||||
**D-12:** Filename: `DKV_YYYY-MM_<Rechnungsnummer>.xlsx`.
|
||||
**D-13:** Exactly 5 columns: Lieferdatum, Fahrzeug, Fahrer, Ort, Kilometerstand.
|
||||
**D-14:** One Excel file per processed invoice.
|
||||
**D-15:** Last 10 export files stored in `user-files/`, downloadable from module UI.
|
||||
**D-16:** On SMTP failure: 3 retries with exponential backoff, then mark as "Versand fehlgeschlagen"; file stays available for manual download.
|
||||
**D-17:** Vehicle master data in DB per tenant: Kennzeichen, Marke, Modell, Fahrer.
|
||||
**D-18:** Importable as CSV; also manually editable in module UI (CRUD table).
|
||||
**D-19:** Fahrzeug column format string configurable (default: `{Marke}/{Modell}/{Kennzeichen}`).
|
||||
**D-20:** Processing history table in UI: Datum/Zeit, Rechnungsnummer, Anzahl Fahrzeuge, Anzahl Transaktionen, Status, Export-Dateiname.
|
||||
**D-21:** History stored in DB; entries kept indefinitely (no auto-purge in v1).
|
||||
|
||||
### Claude's Discretion
|
||||
|
||||
- DB schema design (DkvModuleConfig, DkvVehicleMaster, DkvInvoiceHistory, SmtpConfig tables)
|
||||
- Cron job implementation (`@nestjs/schedule` ScheduleModule, dynamic SchedulerRegistry pattern)
|
||||
- EWS reuse from existing `ews-javascript-api` for inbox email access (mirroring ExchangeProvider calendar pattern)
|
||||
- IMAP library: `imapflow` (confirmed, modern Promise-based)
|
||||
- PDF parsing robustness (handle multi-page, multi-vehicle DKV format)
|
||||
- Frontend component patterns (reuse existing admin table / settings patterns)
|
||||
|
||||
### Deferred Ideas (OUT OF SCOPE)
|
||||
|
||||
None — discussion stayed within phase scope.
|
||||
</user_constraints>
|
||||
|
||||
---
|
||||
|
||||
<phase_requirements>
|
||||
## Phase Requirements
|
||||
|
||||
| ID | Description | Research Support |
|
||||
|----|-------------|------------------|
|
||||
| DKV-01 | System automatically detects DKV invoice PDFs from a configured email inbox (IMAP or Exchange) | imapflow + ews-javascript-api patterns, ScheduleModule cron job, sender filter search |
|
||||
| DKV-02 | All transactions parsed per vehicle: date, service station, km, product, quantity, unit, net/gross | pdf-parse v2 text extraction + DKV-specific regex over vehicle blocks |
|
||||
| DKV-03 | License plate → driver mapping importable as CSV and manually editable in UI | Prisma DkvVehicleMaster table, CSV import via multipart/form-data, CRUD controller + frontend table |
|
||||
| DKV-04 | Processed data exported as Excel and sent to configurable recipient via SMTP | xlsx (SheetJS) buffer generation, nodemailer createTransport() with DB config, retry logic |
|
||||
| DKV-05 | SMTP settings in general settings; module config (inbox, sender filter, folder, recipient) editable in module settings UI | SmtpConfig Prisma table, new /settings/general/smtp route, DkvModuleConfig table |
|
||||
</phase_requirements>
|
||||
|
||||
---
|
||||
|
||||
## Summary
|
||||
|
||||
Phase 7 builds a NestJS module (`dkv`) that polls an email inbox (IMAP or Exchange), downloads PDF attachments from DKV, extracts vehicle/transaction data with regex, maps license plates to drivers via a configurable DB table, generates an Excel file, and delivers it via SMTP. The module is self-contained within the Tessera NestJS monorepo but touches three cross-cutting concerns: the mail module (SMTP config from DB), the settings subsystem (new "General" category for SMTP), and the portal frontend (new module route + settings route).
|
||||
|
||||
All three new npm packages (`imapflow`, `pdf-parse`, `xlsx`) are vetted: `pdf-parse` and `xlsx` are OK per package legitimacy check; `imapflow` is flagged SUS due to a version published today (false positive — the package itself has existed since 2019 with 1.1M weekly downloads). The Exchange inbox access reuses the existing `ews-javascript-api@0.15.3` and follows the `ExchangeProvider` calendar pattern already in the codebase.
|
||||
|
||||
The critical implementation detail is the **mail service extension strategy**: `@nestjs-modules/mailer` cannot change its SMTP transport at runtime (it's configured once at startup). The correct approach per D-06 is to add a `DkvMailService` that calls `nodemailer.createTransport()` directly with config loaded from DB at send time, while leaving the existing `MailService`/`MailerModule` untouched for system emails (password reset, welcome).
|
||||
|
||||
**Primary recommendation:** Build the DKV module as a standard NestJS module following the CalendarModule provider-abstraction pattern. Handle SMTP separately via direct nodemailer transport creation. Use `SchedulerRegistry` for dynamic cron interval updates.
|
||||
|
||||
---
|
||||
|
||||
## Architectural Responsibility Map
|
||||
|
||||
| Capability | Primary Tier | Secondary Tier | Rationale |
|
||||
|------------|-------------|----------------|-----------|
|
||||
| Email inbox polling (IMAP/EWS) | API / Backend | — | Server-side background job; credentials never touch browser |
|
||||
| PDF attachment download | API / Backend | — | Binary stream from inbox, processed server-side |
|
||||
| DKV PDF text extraction + regex | API / Backend | — | CPU-bound parsing, no browser involvement |
|
||||
| Excel file generation | API / Backend | — | xlsx library runs server-side; file written to user-files/ |
|
||||
| SMTP send with attachment | API / Backend | — | Outbound mail is always server-to-server |
|
||||
| Vehicle master CRUD | API / Backend | Frontend | REST endpoints + CRUD UI table |
|
||||
| CSV vehicle import | API / Backend | Frontend | Multipart upload to API; parsed server-side |
|
||||
| Processing history | Database / Storage | API | Prisma table queried by API, displayed in frontend |
|
||||
| Module config (inbox settings) | API / Backend | Frontend | Config stored in DB, UI form to edit |
|
||||
| SMTP config (general settings) | API / Backend | Frontend | SmtpConfig DB table, new settings page |
|
||||
| Export file download | API / Backend | Frontend | File served from user-files/ via API endpoint |
|
||||
| Module UI (history table, manual trigger) | Frontend Server (SSR) | Browser / Client | Next.js page with client components for state |
|
||||
|
||||
---
|
||||
|
||||
## Standard Stack
|
||||
|
||||
### Core (new packages for this phase)
|
||||
|
||||
| Library | Version | Purpose | Why Standard |
|
||||
|---------|---------|---------|--------------|
|
||||
| `imapflow` | 1.4.3 | IMAP inbox polling, message search, attachment download | Modern Promise/async-iterator API, TypeScript types included, replaces older node-imap [ASSUMED based on CONTEXT.md D-01 decision] |
|
||||
| `pdf-parse` | 2.4.5 | PDF text extraction from Buffer | Class-based API (v2), TypeScript types built-in, 5.9M weekly downloads [VERIFIED: npm registry] |
|
||||
| `xlsx` | 0.18.5 | Excel file generation | SheetJS Community Edition, 11.5M weekly downloads, buffer output API, MIT license [VERIFIED: npm registry] |
|
||||
|
||||
### Already installed (no new install needed)
|
||||
|
||||
| Library | Version | Purpose | Note |
|
||||
|---------|---------|---------|------|
|
||||
| `ews-javascript-api` | 0.15.3 | Exchange inbox access | Follow existing `ExchangeProvider` pattern [ASSUMED: same API works for email folders] |
|
||||
| `nodemailer` | ^9.0.1 | SMTP send | Use `createTransport()` directly (NOT via @nestjs-modules/mailer) [ASSUMED] |
|
||||
| `@nestjs/schedule` | ^6.1.3 | Cron job scheduling | Already in package.json, NOT yet imported in AppModule — Wave 0 must add `ScheduleModule.forRoot()` [VERIFIED: codebase grep] |
|
||||
| `@nestjs-modules/mailer` | ^2.3.7 | System emails (password reset) | Keep unchanged; DKV bypasses it for dynamic SMTP |
|
||||
|
||||
### Installation (new packages only)
|
||||
|
||||
```bash
|
||||
pnpm --filter @tessera/api add imapflow pdf-parse xlsx
|
||||
```
|
||||
|
||||
```bash
|
||||
# TypeScript types — pdf-parse and imapflow ship their own types
|
||||
# xlsx also ships types; no @types/* needed for these packages
|
||||
pnpm --filter @tessera/api add -D @types/xlsx 2>/dev/null || true
|
||||
```
|
||||
|
||||
**Version verification (run before installing):**
|
||||
```bash
|
||||
npm view imapflow version # 1.4.3
|
||||
npm view pdf-parse version # 2.4.5
|
||||
npm view xlsx version # 0.18.5
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Package Legitimacy Audit
|
||||
|
||||
> Packages verified via `gsd-tools query package-legitimacy check --ecosystem npm imapflow pdf-parse xlsx`
|
||||
|
||||
| Package | Registry | Age | Downloads | Source Repo | Verdict | Disposition |
|
||||
|---------|----------|-----|-----------|-------------|---------|-------------|
|
||||
| `imapflow` | npm | Since 2019 | ~1.1M/wk | github.com/postalsys/imapflow | SUS (too-new flag) | **Approved** — flag is false positive (new *version* published today, not new package; 1.1M/wk and legitimate repo) |
|
||||
| `pdf-parse` | npm | Since 2018 (v2: 2025) | ~5.9M/wk | github.com/mehmet-kozan/pdf-parse | OK | Approved |
|
||||
| `xlsx` | npm | Since 2012 | ~11.5M/wk | github.com/SheetJS/sheetjs | OK | Approved |
|
||||
|
||||
**Packages removed due to SLOP verdict:** none
|
||||
|
||||
**Packages flagged as suspicious SUS:** `imapflow` — planner should note context (false positive: version released today, package established since 2019 with 1.1M/wk downloads from postalsys/imapflow). No `checkpoint:human-verify` needed given evidence.
|
||||
|
||||
---
|
||||
|
||||
## Architecture Patterns
|
||||
|
||||
### System Architecture Diagram
|
||||
|
||||
```
|
||||
[Email Inbox: IMAP/EWS]
|
||||
│ (poll every N min OR manual trigger)
|
||||
▼
|
||||
[DkvSchedulerService] ──── @nestjs/schedule SchedulerRegistry
|
||||
│
|
||||
▼
|
||||
[InboxProvider interface]
|
||||
├── ImapProvider (imapflow)
|
||||
└── ExchangeInboxProvider (ews-javascript-api)
|
||||
│ (search by sender filter, list unseen emails)
|
||||
▼
|
||||
[PDF Attachment Buffer]
|
||||
│
|
||||
▼
|
||||
[DkvParserService]
|
||||
└── pdf-parse v2: PDFParse({ data: buffer }).getText()
|
||||
└── Regex extraction: vehicle blocks → transaction rows
|
||||
│
|
||||
▼
|
||||
[DkvVehicleMaster lookup] ◄── Prisma: DkvVehicleMaster
|
||||
└── Kennzeichen → Marke, Modell, Fahrer
|
||||
│
|
||||
▼
|
||||
[DkvExportService]
|
||||
└── xlsx: build 5-column worksheet → Buffer
|
||||
└── Write to user-files/DKV_YYYY-MM_<nr>.xlsx
|
||||
└── Prune: keep last 10 files
|
||||
│
|
||||
▼
|
||||
[DkvMailService]
|
||||
└── nodemailer.createTransport(smtpConfigFromDB)
|
||||
└── sendMail({ to: recipient, attachments: [xlsx buffer] })
|
||||
└── 3 retries with exponential backoff
|
||||
│
|
||||
▼
|
||||
[DkvInvoiceHistory record saved] ──► Prisma: DkvInvoiceHistory
|
||||
│
|
||||
▼
|
||||
[Frontend: /modules/dkv-fleet/page.tsx]
|
||||
├── History table (GET /dkv/history)
|
||||
├── Manual "Jetzt prüfen" button (POST /dkv/check-now)
|
||||
├── Export file download (GET /dkv/exports/:filename)
|
||||
└── Settings tab → /modules/dkv-fleet/settings/
|
||||
└── Inbox config form (GET/PUT /dkv/config)
|
||||
└── Vehicle CRUD table (GET/POST/PUT/DELETE /dkv/vehicles)
|
||||
└── CSV import button (POST /dkv/vehicles/import)
|
||||
|
||||
[/settings/general/smtp] ──► GET/PUT /settings/smtp ──► SmtpConfig (Prisma)
|
||||
```
|
||||
|
||||
### Recommended Project Structure
|
||||
|
||||
```
|
||||
apps/api/src/dkv/
|
||||
├── dkv.module.ts # Module: imports ScheduleModule, CalendarCryptoService
|
||||
├── dkv.controller.ts # REST: /dkv/* routes (history, config, vehicles, check-now, exports)
|
||||
├── dkv.service.ts # Orchestration: coordinates scheduler, parser, export, mail
|
||||
├── dkv-scheduler.service.ts # SchedulerRegistry wrapper, cron lifecycle management
|
||||
├── dkv-parser.service.ts # pdf-parse + DKV regex → structured data
|
||||
├── dkv-export.service.ts # xlsx buffer generation, user-files/ management (10-file limit)
|
||||
├── dkv-mail.service.ts # nodemailer.createTransport() with SmtpConfig from DB
|
||||
├── providers/
|
||||
│ ├── inbox-provider.interface.ts # InboxProvider: connect(), searchSender(), downloadAttachments()
|
||||
│ ├── imap.provider.ts # imapflow implementation
|
||||
│ └── exchange-inbox.provider.ts # ews-javascript-api implementation (mirrors ExchangeProvider)
|
||||
└── dto/
|
||||
├── dkv-config.dto.ts
|
||||
├── dkv-vehicle.dto.ts
|
||||
└── dkv-history.dto.ts
|
||||
|
||||
apps/api/src/settings/
|
||||
├── settings.module.ts # NEW: SMTP config management
|
||||
├── settings.controller.ts # GET/PUT /settings/smtp
|
||||
├── settings.service.ts # SmtpConfig CRUD + decrypt
|
||||
└── dto/smtp-config.dto.ts
|
||||
|
||||
apps/web/src/app/(portal)/modules/dkv-fleet/
|
||||
├── page.tsx # Main view: invoice history table
|
||||
├── settings/
|
||||
│ ├── page.tsx # Module settings: inbox config + vehicle master
|
||||
│ └── components/
|
||||
│ ├── InboxConfigForm.tsx
|
||||
│ ├── VehicleTable.tsx
|
||||
│ └── CsvImportButton.tsx
|
||||
└── components/
|
||||
└── InvoiceHistoryTable.tsx
|
||||
|
||||
apps/web/src/app/(portal)/settings/general/
|
||||
└── smtp/
|
||||
└── page.tsx # SMTP settings page (new settings category)
|
||||
```
|
||||
|
||||
### Prisma Schema (new tables)
|
||||
|
||||
```prisma
|
||||
model DkvModuleConfig {
|
||||
id String @id @default(uuid())
|
||||
tenantId String @unique
|
||||
protocol String @default("imap") // 'imap' | 'exchange'
|
||||
host String?
|
||||
port Int?
|
||||
encryption String @default("ssl-tls") // 'none' | 'starttls' | 'ssl-tls'
|
||||
folder String @default("INBOX")
|
||||
senderFilter String?
|
||||
pollIntervalMin Int @default(60)
|
||||
isActive Boolean @default(false)
|
||||
exportRecipient String?
|
||||
vehicleFormatString String @default("{Marke}/{Modell}/{Kennzeichen}")
|
||||
encryptedInboxCreds String? // AES-256-GCM: JSON { username, password } encrypted
|
||||
createdAt DateTime @default(now())
|
||||
updatedAt DateTime @updatedAt
|
||||
@@index([tenantId])
|
||||
}
|
||||
|
||||
model DkvVehicleMaster {
|
||||
id String @id @default(uuid())
|
||||
tenantId String
|
||||
kennzeichen String
|
||||
marke String
|
||||
modell String
|
||||
fahrer String // "Vorname Nachname"
|
||||
createdAt DateTime @default(now())
|
||||
updatedAt DateTime @updatedAt
|
||||
@@unique([tenantId, kennzeichen])
|
||||
@@index([tenantId])
|
||||
}
|
||||
|
||||
model DkvInvoiceHistory {
|
||||
id String @id @default(uuid())
|
||||
tenantId String
|
||||
datumZeit DateTime @default(now())
|
||||
rechnungsnummer String
|
||||
anzahlFahrzeuge Int @default(0)
|
||||
anzahlTransaktionen Int @default(0)
|
||||
status String // 'Verarbeitet' | 'Fehler' | 'Versand fehlgeschlagen'
|
||||
errorMessage String?
|
||||
exportFilename String?
|
||||
createdAt DateTime @default(now())
|
||||
@@index([tenantId])
|
||||
@@index([datumZeit])
|
||||
}
|
||||
|
||||
model SmtpConfig {
|
||||
id String @id @default(uuid())
|
||||
tenantId String @unique
|
||||
host String
|
||||
port Int @default(587)
|
||||
encryption String @default("starttls") // 'none' | 'starttls' | 'ssl-tls'
|
||||
username String?
|
||||
encryptedPassword String? // AES-256-GCM via CalendarCryptoService
|
||||
fromAddress String
|
||||
createdAt DateTime @default(now())
|
||||
updatedAt DateTime @updatedAt
|
||||
}
|
||||
```
|
||||
|
||||
### Pattern 1: InboxProvider Interface (mirrors CalendarProvider)
|
||||
|
||||
**What:** Abstract inbox provider so IMAP and Exchange are interchangeable.
|
||||
**When to use:** Any service that needs to list and download email attachments.
|
||||
|
||||
```typescript
|
||||
// Source: mirrors apps/api/src/calendar/calendar.service.ts CalendarProvider pattern
|
||||
export interface InboxEmail {
|
||||
uid: number | string;
|
||||
messageId: string;
|
||||
subject: string;
|
||||
from: string;
|
||||
date: Date;
|
||||
attachments: InboxAttachment[];
|
||||
}
|
||||
|
||||
export interface InboxAttachment {
|
||||
filename: string;
|
||||
contentType: string;
|
||||
buffer: Buffer;
|
||||
}
|
||||
|
||||
export interface InboxProvider {
|
||||
fetchPdfAttachments(
|
||||
config: InboxConfig,
|
||||
): Promise<InboxEmail[]>;
|
||||
|
||||
testConnection(config: InboxConfig): Promise<boolean>;
|
||||
}
|
||||
```
|
||||
|
||||
### Pattern 2: imapflow IMAP Provider
|
||||
|
||||
**What:** Fetch emails from a specific folder filtered by sender, download PDF attachments.
|
||||
**When to use:** When `DkvModuleConfig.protocol === 'imap'`.
|
||||
|
||||
```typescript
|
||||
// Source: [CITED: imapflow.com/docs/examples/fetching-messages/]
|
||||
import { ImapFlow } from 'imapflow';
|
||||
|
||||
async function fetchPdfAttachments(config: InboxConfig): Promise<InboxEmail[]> {
|
||||
const client = new ImapFlow({
|
||||
host: config.host,
|
||||
port: config.port,
|
||||
secure: config.encryption === 'ssl-tls', // port 993
|
||||
requireTLS: config.encryption === 'starttls', // port 587
|
||||
auth: { user: config.username, pass: config.password },
|
||||
logger: false, // suppress verbose imapflow logs
|
||||
});
|
||||
|
||||
await client.connect();
|
||||
const lock = await client.getMailboxLock(config.folder);
|
||||
const results: InboxEmail[] = [];
|
||||
|
||||
try {
|
||||
// Search by sender — returns UIDs
|
||||
const uids = await client.search({ from: config.senderFilter }, { uid: true });
|
||||
if (uids.length === 0) return results;
|
||||
|
||||
// Fetch all first, then process (avoids IMAP deadlock — see Pitfall 1)
|
||||
const messages = await client.fetchAll(
|
||||
uids.join(','),
|
||||
{ envelope: true, bodyStructure: true },
|
||||
{ uid: true },
|
||||
);
|
||||
|
||||
for (const msg of messages) {
|
||||
const pdfs = collectPdfParts(msg.bodyStructure);
|
||||
const attachments: InboxAttachment[] = [];
|
||||
for (const partId of pdfs) {
|
||||
const { content } = await client.download(
|
||||
String(msg.uid), partId, { uid: true }
|
||||
);
|
||||
const buf = await streamToBuffer(content);
|
||||
attachments.push({ filename: `attachment-${partId}.pdf`, contentType: 'application/pdf', buffer: buf });
|
||||
}
|
||||
results.push({ uid: msg.uid!, messageId: msg.envelope.messageId ?? '', subject: msg.envelope.subject ?? '', from: msg.envelope.from?.[0]?.address ?? '', date: msg.envelope.date ?? new Date(), attachments });
|
||||
}
|
||||
} finally {
|
||||
lock.release();
|
||||
await client.logout();
|
||||
}
|
||||
return results;
|
||||
}
|
||||
```
|
||||
|
||||
### Pattern 3: pdf-parse v2 Text Extraction
|
||||
|
||||
**What:** Extract raw text from a PDF Buffer using the v2 class-based API.
|
||||
**When to use:** After downloading a PDF attachment from the inbox.
|
||||
|
||||
```typescript
|
||||
// Source: [CITED: mehmet-kozan.github.io/pdf-parse/typedoc/]
|
||||
// IMPORTANT: pdf-parse v2 uses a class-based API — NOT the v1 function call pdfParse(buffer)
|
||||
import { PDFParse } from 'pdf-parse';
|
||||
|
||||
async function extractPdfText(buffer: Buffer): Promise<string> {
|
||||
const parser = new PDFParse({ data: buffer }); // 'data' key accepts Buffer [ASSUMED]
|
||||
const result = await parser.getText();
|
||||
await parser.destroy(); // always call destroy() to free memory
|
||||
return result.text; // full text from all pages concatenated
|
||||
}
|
||||
```
|
||||
|
||||
> **Assumption flag:** The `{ data: buffer }` LoadParameters key is inferred from the TypeDoc page — confirmed the interface accepts both URL and buffer, but exact property name tagged [ASSUMED]. Planner should add a Wave 0 verify step that runs a quick test parse against `user-files/invoice.pdf`.
|
||||
|
||||
### Pattern 4: DKV PDF Regex — Vehicle Block Extraction
|
||||
|
||||
**What:** Parse DKV E-Rechnung text into structured vehicle/transaction data.
|
||||
**When to use:** After extracting raw text from the PDF.
|
||||
|
||||
```typescript
|
||||
// Source: [ASSUMED based on CONTEXT.md D-09 + D-specifics section]
|
||||
// DKV PDF structure (from CONTEXT.md specifics):
|
||||
// VEHICLE: {Kennzeichen} CARD NO.: {CardNo}
|
||||
// ... transaction rows ...
|
||||
// TOTAL: ...
|
||||
|
||||
interface DkvTransaction {
|
||||
lieferdatum: string; // "DD.MM.YYYY"
|
||||
ort: string; // Servicestation Ort
|
||||
kilometerstand: number;
|
||||
produkt: string;
|
||||
menge: number;
|
||||
einheit: string;
|
||||
netto: number;
|
||||
brutto: number;
|
||||
}
|
||||
|
||||
interface DkvVehicleBlock {
|
||||
kennzeichen: string;
|
||||
cardNumber: string;
|
||||
transactions: DkvTransaction[];
|
||||
}
|
||||
|
||||
function parseDkvText(text: string): DkvVehicleBlock[] {
|
||||
const vehicles: DkvVehicleBlock[] = [];
|
||||
|
||||
// Split text into vehicle blocks — anchor on VEHICLE: marker
|
||||
// Pattern: "VEHICLE: GP-JL 728E CARD NO.: 7020261..."
|
||||
const vehicleBlockPattern = /VEHICLE:\s+(\S+)\s+CARD NO\.:\s+(\S+)([\s\S]*?)(?=VEHICLE:|$)/g;
|
||||
let match: RegExpExecArray | null;
|
||||
|
||||
while ((match = vehicleBlockPattern.exec(text)) !== null) {
|
||||
const kennzeichen = match[1];
|
||||
const cardNumber = match[2];
|
||||
const blockText = match[3];
|
||||
|
||||
const transactions = parseTransactionRows(blockText, kennzeichen);
|
||||
vehicles.push({ kennzeichen, cardNumber, transactions });
|
||||
}
|
||||
return vehicles;
|
||||
}
|
||||
|
||||
// Transaction row regex — [ASSUMED] based on DKV invoice structure description
|
||||
// Row format: "DD.MM.YYYY Station City NNN.NNN km Diesel NN.NNN L"
|
||||
function parseTransactionRows(blockText: string, _plate: string): DkvTransaction[] {
|
||||
const txPattern =
|
||||
/(\d{2}\.\d{2}\.\d{4})\s+(.+?)\s{2,}([\d.,]+)\s+km\s+(\S+)\s+([\d.,]+)\s+(\w+)/g;
|
||||
const transactions: DkvTransaction[] = [];
|
||||
let m: RegExpExecArray | null;
|
||||
while ((m = txPattern.exec(blockText)) !== null) {
|
||||
transactions.push({
|
||||
lieferdatum: m[1],
|
||||
ort: m[2].trim(),
|
||||
kilometerstand: parseFloat(m[3].replace('.', '').replace(',', '.')),
|
||||
produkt: m[4],
|
||||
menge: parseFloat(m[5].replace('.', '').replace(',', '.')),
|
||||
einheit: m[6],
|
||||
netto: 0, // TODO: extract from block if present
|
||||
brutto: 0,
|
||||
});
|
||||
}
|
||||
return transactions;
|
||||
}
|
||||
```
|
||||
|
||||
> **Warning:** The transaction row regex is [ASSUMED] from the CONTEXT.md structure description. The actual DKV PDF format must be validated against `user-files/invoice.pdf` in Wave 0 before the parser service is written. This is the highest-risk regex in the system.
|
||||
|
||||
### Pattern 5: SheetJS Excel Generation
|
||||
|
||||
**What:** Build a 5-column worksheet and return it as a Buffer.
|
||||
**When to use:** After assembling all vehicle/transaction rows for one invoice.
|
||||
|
||||
```typescript
|
||||
// Source: [CITED: docs.sheetjs.com/docs/api/write-options/]
|
||||
import * as XLSX from 'xlsx';
|
||||
|
||||
function buildExcelBuffer(rows: ExportRow[], vehicleFormat: string): Buffer {
|
||||
const headers = ['Lieferdatum', 'Fahrzeug', 'Fahrer', 'Ort', 'Kilometerstand'];
|
||||
const data = rows.map(r => [
|
||||
r.lieferdatum, // string "DD.MM.YYYY" — write as-is, no Date conversion
|
||||
r.fahrzeug, // "Marke/Modell/Kennzeichen" — from format string
|
||||
r.fahrer, // "Vorname Nachname"
|
||||
r.ort, // string
|
||||
r.kilometerstand // number — no unit
|
||||
]);
|
||||
|
||||
const ws = XLSX.utils.aoa_to_sheet([headers, ...data]);
|
||||
const wb = XLSX.utils.book_new();
|
||||
XLSX.utils.book_append_sheet(wb, ws, 'DKV Export');
|
||||
|
||||
return XLSX.write(wb, { type: 'buffer', bookType: 'xlsx' }) as Buffer;
|
||||
}
|
||||
```
|
||||
|
||||
### Pattern 6: Dynamic SMTP via Nodemailer (bypasses @nestjs-modules/mailer)
|
||||
|
||||
**What:** Create a nodemailer transport from DB config at send time.
|
||||
**When to use:** Every DKV export send — config may change between runs.
|
||||
|
||||
```typescript
|
||||
// Source: [CITED: nodemailer.com/smtp] [ASSUMED: dynamic createTransport pattern]
|
||||
import * as nodemailer from 'nodemailer';
|
||||
|
||||
async function sendExportEmail(
|
||||
smtpConfig: SmtpConfig,
|
||||
recipient: string,
|
||||
attachmentBuffer: Buffer,
|
||||
filename: string,
|
||||
): Promise<void> {
|
||||
const secure = smtpConfig.encryption === 'ssl-tls'; // port 465
|
||||
const requireTLS = smtpConfig.encryption === 'starttls'; // port 587
|
||||
|
||||
const transport = nodemailer.createTransport({
|
||||
host: smtpConfig.host,
|
||||
port: smtpConfig.port,
|
||||
secure,
|
||||
requireTLS,
|
||||
auth: smtpConfig.username
|
||||
? { user: smtpConfig.username, pass: smtpConfig.decryptedPassword }
|
||||
: undefined,
|
||||
});
|
||||
|
||||
await transport.sendMail({
|
||||
from: smtpConfig.fromAddress,
|
||||
to: recipient,
|
||||
subject: `DKV Flottenabrechnung: ${filename}`,
|
||||
text: 'DKV Flottenabrechnung im Anhang.',
|
||||
attachments: [
|
||||
{ filename, content: attachmentBuffer, contentType: 'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet' },
|
||||
],
|
||||
});
|
||||
}
|
||||
```
|
||||
|
||||
### Pattern 7: Dynamic Cron Job (SchedulerRegistry)
|
||||
|
||||
**What:** Create/update a cron job from a DB-sourced interval in minutes.
|
||||
**When to use:** On module init and after config change.
|
||||
|
||||
```typescript
|
||||
// Source: [CITED: oneuptime.com/blog/post/2026-02-02-nestjs-task-scheduling/view]
|
||||
import { Injectable, OnModuleInit } from '@nestjs/common';
|
||||
import { SchedulerRegistry } from '@nestjs/schedule';
|
||||
import { CronJob } from 'cron';
|
||||
|
||||
const JOB_NAME = 'dkv-inbox-poll';
|
||||
|
||||
@Injectable()
|
||||
export class DkvSchedulerService implements OnModuleInit {
|
||||
constructor(
|
||||
private readonly schedulerRegistry: SchedulerRegistry,
|
||||
private readonly dkvService: DkvService,
|
||||
) {}
|
||||
|
||||
onModuleInit() {
|
||||
// Load interval from DB config on startup
|
||||
this.dkvService.loadConfig().then(config => {
|
||||
if (config?.isActive) {
|
||||
this.setInterval(config.pollIntervalMin);
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
setInterval(intervalMin: number): void {
|
||||
// Remove existing job if any
|
||||
try {
|
||||
const existing = this.schedulerRegistry.getCronJob(JOB_NAME);
|
||||
existing.stop();
|
||||
this.schedulerRegistry.deleteCronJob(JOB_NAME);
|
||||
} catch { /* not registered yet */ }
|
||||
|
||||
// Create new job with computed cron expression
|
||||
const cronExpr = `*/${intervalMin} * * * *`; // every N minutes
|
||||
const job = new CronJob(cronExpr, () => {
|
||||
this.dkvService.processInbox().catch(err =>
|
||||
this.logger.error('DKV inbox poll failed', err)
|
||||
);
|
||||
});
|
||||
this.schedulerRegistry.addCronJob(JOB_NAME, job);
|
||||
job.start();
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Pattern 8: ScheduleModule Registration (AppModule update)
|
||||
|
||||
`@nestjs/schedule` is installed but NOT yet imported in `app.module.ts`. Must be added in Wave 0.
|
||||
|
||||
```typescript
|
||||
// apps/api/src/app.module.ts — add to imports array
|
||||
import { ScheduleModule } from '@nestjs/schedule';
|
||||
|
||||
@Module({
|
||||
imports: [
|
||||
// ... existing imports ...
|
||||
ScheduleModule.forRoot(), // ADD THIS — required for cron/interval decorators
|
||||
DkvModule, // ADD THIS — new DKV fleet module
|
||||
],
|
||||
})
|
||||
```
|
||||
|
||||
### Settings Sidebar Extension
|
||||
|
||||
Add "Allgemein" category to `SettingsSidebar` with SMTP link:
|
||||
|
||||
```typescript
|
||||
// apps/web/src/components/settings/settings-sidebar.tsx — extend items array
|
||||
const generalItems = [
|
||||
{ label: t('categorySmtp'), href: '/settings/general/smtp' },
|
||||
];
|
||||
```
|
||||
|
||||
### Anti-Patterns to Avoid
|
||||
|
||||
- **Mixing fetch + download in imapflow async iterator:** The imapflow docs explicitly warn against it — causes IMAP connection deadlock. Always call `fetchAll()` first to collect all message metadata, then loop and call `download()` separately.
|
||||
- **Using @nestjs-modules/mailer for dynamic SMTP:** The mailer transport is configured once at startup via `forRootAsync`. Attempting to change it at runtime breaks all system emails. Use `nodemailer.createTransport()` directly in `DkvMailService`.
|
||||
- **Calling `pdfParse(buffer)` v1 API:** pdf-parse v2 is NOT backward-compatible. The old `import pdfParse from 'pdf-parse'; const data = await pdfParse(buffer);` call will fail. Use `new PDFParse({ data: buffer })` instead.
|
||||
- **Writing Lieferdatum as a JS Date in xlsx:** The DKV dates are already formatted German strings ("13.12.2020"). Writing them as JS Date objects causes Excel to serialize them as numeric serials (or reformat them). Write as strings directly.
|
||||
- **Storing decrypted credentials in logs:** NEVER log the result of `crypto.decrypt()` — follow T-05-13 pattern from CalendarService (generic error messages only).
|
||||
- **@Interval() decorator with hardcoded value:** The poll interval is configurable from DB — use `SchedulerRegistry.addCronJob()` instead of the static `@Interval(ms)` decorator, so the interval can be updated when config changes.
|
||||
|
||||
---
|
||||
|
||||
## Don't Hand-Roll
|
||||
|
||||
| Problem | Don't Build | Use Instead | Why |
|
||||
|---------|-------------|-------------|-----|
|
||||
| PDF text extraction | Custom pdfjs integration | `pdf-parse` v2 | pdfjs-dist requires worker setup, much heavier |
|
||||
| Excel file generation | CSV with .xlsx extension | `xlsx` (SheetJS) | Excel requires OOXML zip format; CSV named as xlsx corrupts in Excel |
|
||||
| IMAP protocol | Raw TCP/IMAP commands | `imapflow` | IMAP has 30+ capabilities, extension negotiation, IDLE mode — do not hand-roll |
|
||||
| AES encryption | Custom cipher | `CalendarCryptoService` | Already implemented, tested, consistent with calendar credentials |
|
||||
| Cron scheduling | setInterval() in a Node.js service | `@nestjs/schedule` + `SchedulerRegistry` | Restart-safe, injectable, monitorable |
|
||||
| CSV parsing | String split by comma | (inline split is fine for simple CSVs) | DKV vehicle CSV is simple (Kennzeichen;Marke;Modell;Fahrer) — regex or split is acceptable; don't add papaparse just for this |
|
||||
|
||||
**Key insight:** PDF-to-Excel pipelines look simple but have many format edge cases (multi-page PDFs, Unicode station names, German number formats with dot-as-thousands-separator). Trust the established libraries for the binary format layers; only write custom regex for DKV-specific structure extraction.
|
||||
|
||||
---
|
||||
|
||||
## Common Pitfalls
|
||||
|
||||
### Pitfall 1: imapflow IMAP Connection Deadlock
|
||||
|
||||
**What goes wrong:** Calling `client.download()` inside a `fetch()` async iterator causes the IMAP connection to deadlock — the iterator holds the connection open while `download()` tries to issue another command on the same connection.
|
||||
|
||||
**Why it happens:** imapflow uses a single IMAP connection. The `fetch()` method returns an async iterator that streams messages; issuing another command while the iterator is active queues on the same connection and never resolves.
|
||||
|
||||
**How to avoid:** Always use `fetchAll()` (which resolves once, fully) to collect message metadata first. Then loop over the results and call `download()` separately. This is explicitly documented in imapflow examples.
|
||||
|
||||
**Warning signs:** `await client.download(...)` never resolves when called inside a `for await ... of client.fetch(...)` loop.
|
||||
|
||||
### Pitfall 2: pdf-parse v1 vs v2 API Break
|
||||
|
||||
**What goes wrong:** Calling `pdfParse(buffer)` (v1 API) throws `TypeError: pdfParse is not a function` or similar because pdf-parse v2 is a class-based API.
|
||||
|
||||
**Why it happens:** The npm package `pdf-parse` at v2.x is a complete rewrite by mehmet-kozan with a new class-based API. Any tutorial from before 2025 shows the v1 function-call API.
|
||||
|
||||
**How to avoid:** Use `new PDFParse({ data: buffer }); await parser.getText(); await parser.destroy();`. Write a Wave 0 integration test parsing `user-files/invoice.pdf` to confirm the API before writing the production parser.
|
||||
|
||||
**Warning signs:** LangChain community issue #22 confirms this exact breakage: "PDFLoader declares pdf-parse v2 but only supports v1".
|
||||
|
||||
### Pitfall 3: @nestjs-modules/mailer Cannot Change Transport at Runtime
|
||||
|
||||
**What goes wrong:** Extending `MailModule` to load SMTP config from DB at startup seems straightforward, but if the user updates SMTP config in the UI, the running transport still uses the old settings until a restart.
|
||||
|
||||
**Why it happens:** `MailerModule.forRootAsync()` constructs the nodemailer transport once when the NestJS DI container initializes. There's no API to replace the transport object afterward.
|
||||
|
||||
**How to avoid:** Keep `MailerModule` for system emails (env-based, rarely changed). For DKV sends, use `nodemailer.createTransport(await loadSmtpConfig())` at send time in `DkvMailService`. Config changes take effect immediately on the next send.
|
||||
|
||||
### Pitfall 4: ScheduleModule Not Registered in AppModule
|
||||
|
||||
**What goes wrong:** `@Cron()` / `@Interval()` decorators silently do nothing; `SchedulerRegistry` throws "SchedulerRegistry is not injectable" or jobs never fire.
|
||||
|
||||
**Why it happens:** `@nestjs/schedule` is installed but `ScheduleModule.forRoot()` is NOT imported in `AppModule` (confirmed by grep — absent from app.module.ts).
|
||||
|
||||
**How to avoid:** Wave 0 task must add `ScheduleModule.forRoot()` to AppModule imports. This is a prerequisite for DkvSchedulerService to function.
|
||||
|
||||
**Warning signs:** CronJob registered via `schedulerRegistry.addCronJob()` but callback never fires.
|
||||
|
||||
### Pitfall 5: German Number Format in km/Menge Parsing
|
||||
|
||||
**What goes wrong:** DKV PDFs use German number formatting: `19.234,56` (dot as thousands separator, comma as decimal). `parseFloat("19.234,56")` returns 19.234 (truncated at the comma).
|
||||
|
||||
**Why it happens:** JavaScript `parseFloat` expects English format (comma as thousands separator, dot as decimal).
|
||||
|
||||
**How to avoid:** Strip dots then replace comma with dot before parsing: `parseFloat(raw.replace(/\./g, '').replace(',', '.'))`. Apply this to Kilometerstand and Menge fields.
|
||||
|
||||
### Pitfall 6: EWS Email vs Calendar Folder Access
|
||||
|
||||
**What goes wrong:** Copying `ExchangeProvider` directly and calling `FindAppointments` returns no results because email folders use `FindItems` / `EmailMessage.Bind`, not `FindAppointments`.
|
||||
|
||||
**Why it happens:** EWS has separate item types — `Appointment` for calendar, `EmailMessage` for inbox. The existing ExchangeProvider uses the Calendar well-known folder name.
|
||||
|
||||
**How to avoid:** In `ExchangeInboxProvider`, use `ews.WellKnownFolderName.Inbox` with `service.FindItems()` and bind to `EmailMessage`. Follow EWS docs for email vs appointment distinction. [ASSUMED — not verified against ews-javascript-api 0.15.3 API docs]
|
||||
|
||||
### Pitfall 7: user-files/ 10-File Pruning Race Condition
|
||||
|
||||
**What goes wrong:** If two invoices are processed simultaneously (concurrent cron + manual trigger), both may pass the "10 files exist" check and write an 11th file before either runs the prune.
|
||||
|
||||
**Why it happens:** File count check and write are not atomic.
|
||||
|
||||
**How to avoid:** Use a mutex (simple in-memory lock in `DkvExportService`) or route all inbox processing through a queue so only one invoice processes at a time. Simplest: add a `private processing = false` guard in `DkvService.processInbox()` — if already processing, return early.
|
||||
|
||||
---
|
||||
|
||||
## Code Examples
|
||||
|
||||
### Inversion of Control: Credential Encryption Pattern
|
||||
|
||||
```typescript
|
||||
// Source: apps/api/src/calendar/crypto.service.ts [VERIFIED: codebase]
|
||||
// Reuse CalendarCryptoService directly — it's not calendar-specific despite its name.
|
||||
// Encrypt both inbox credentials and SMTP password with the same CALENDAR_ENCRYPTION_KEY.
|
||||
|
||||
const encryptedInboxCreds = this.crypto.encrypt(
|
||||
JSON.stringify({ username: dto.username, password: dto.password })
|
||||
);
|
||||
// Store as single encrypted JSON blob in DkvModuleConfig.encryptedInboxCreds
|
||||
|
||||
// Decrypt at use time:
|
||||
const { username, password } = JSON.parse(this.crypto.decrypt(config.encryptedInboxCreds!));
|
||||
```
|
||||
|
||||
### CSV Vehicle Import Pattern
|
||||
|
||||
```typescript
|
||||
// Simple CSV parse for DKV vehicle master — semicolon-delimited
|
||||
// Expected format: Kennzeichen;Marke;Modell;Fahrer
|
||||
// Source: [ASSUMED]
|
||||
function parseVehicleCsv(csvText: string): CreateVehicleDto[] {
|
||||
const lines = csvText.trim().split('\n');
|
||||
const headers = lines[0].split(';').map(h => h.trim().toLowerCase());
|
||||
return lines.slice(1).map(line => {
|
||||
const cols = line.split(';').map(c => c.trim());
|
||||
return {
|
||||
kennzeichen: cols[headers.indexOf('kennzeichen')] ?? '',
|
||||
marke: cols[headers.indexOf('marke')] ?? '',
|
||||
modell: cols[headers.indexOf('modell')] ?? '',
|
||||
fahrer: cols[headers.indexOf('fahrer')] ?? '',
|
||||
};
|
||||
}).filter(v => v.kennzeichen);
|
||||
}
|
||||
```
|
||||
|
||||
### Fahrzeug Format String Resolution
|
||||
|
||||
```typescript
|
||||
// Source: [ASSUMED based on D-19 format template]
|
||||
// Default: "{Marke}/{Modell}/{Kennzeichen}"
|
||||
function resolveFahrzeug(
|
||||
vehicle: DkvVehicleMaster,
|
||||
formatString: string,
|
||||
): string {
|
||||
return formatString
|
||||
.replace('{Marke}', vehicle.marke)
|
||||
.replace('{Modell}', vehicle.modell)
|
||||
.replace('{Kennzeichen}', vehicle.kennzeichen)
|
||||
.replace('{Fahrer}', vehicle.fahrer);
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Runtime State Inventory
|
||||
|
||||
> Greenfield module — no existing runtime state to migrate. All tables are new.
|
||||
|
||||
| Category | Items Found | Action Required |
|
||||
|----------|-------------|-----------------|
|
||||
| Stored data | None — DkvModuleConfig, DkvVehicleMaster, DkvInvoiceHistory, SmtpConfig are new tables | Prisma migration only |
|
||||
| Live service config | None | — |
|
||||
| OS-registered state | None | — |
|
||||
| Secrets/env vars | `CALENDAR_ENCRYPTION_KEY` is reused for DKV credential encryption — no new env var needed | Ensure env var is set in Docker Compose (already required by CalendarModule) |
|
||||
| Build artifacts | None | — |
|
||||
|
||||
---
|
||||
|
||||
## Environment Availability
|
||||
|
||||
| Dependency | Required By | Available | Version | Fallback |
|
||||
|------------|------------|-----------|---------|----------|
|
||||
| Node.js | imapflow, pdf-parse, xlsx | Yes | v24.16.0 | — |
|
||||
| Docker | DB + full stack | Yes | 29.5.3 | — |
|
||||
| pnpm | package install | Yes | 9.15.0 | — |
|
||||
| `CALENDAR_ENCRYPTION_KEY` env var | CalendarCryptoService (shared) | Assumed set | — | Module init throws if missing |
|
||||
| `user-files/` directory | Export file storage | Yes (exists in repo) | — | Create if missing |
|
||||
| `user-files/invoice.pdf` | Wave 0 parser validation | Yes (in repo) | — | Required for regex dev |
|
||||
|
||||
**Missing dependencies with no fallback:** None identified.
|
||||
|
||||
---
|
||||
|
||||
## State of the Art
|
||||
|
||||
| Old Approach | Current Approach | When Changed | Impact |
|
||||
|--------------|------------------|--------------|--------|
|
||||
| pdf-parse v1: `pdfParse(buffer)` function | pdf-parse v2: `new PDFParse({data}).getText()` | v2.x (2025) | Breaking API change — all v1 tutorials are wrong |
|
||||
| @nestjs/schedule: static `@Interval()` | Dynamic `SchedulerRegistry.addCronJob()` | Always supported | Required for DB-configurable intervals |
|
||||
| @nestjs-modules/mailer dynamic transport | nodemailer.createTransport() at send time | Pattern choice | Only way to support runtime SMTP config changes |
|
||||
|
||||
**Deprecated/outdated:**
|
||||
- `node-imap`: Outdated callback-based IMAP library. Do not use. imapflow is the modern replacement.
|
||||
- `pdf-parse` v1 API (`pdfParse(buffer)` function): Removed in v2.x.
|
||||
|
||||
---
|
||||
|
||||
## Validation Architecture
|
||||
|
||||
### Test Framework
|
||||
|
||||
| Property | Value |
|
||||
|----------|-------|
|
||||
| Framework | Vitest 4.1.9 (apps/web only) |
|
||||
| Config file | apps/web/vitest.config.ts |
|
||||
| Quick run command | `pnpm --filter @tessera/web test` |
|
||||
| Full suite command | `pnpm --filter @tessera/web test` |
|
||||
| API test framework | None configured — no vitest/jest in apps/api |
|
||||
|
||||
> **Note:** The API has no test framework configured. Phase 7 API logic (PDF parser, Excel builder, SMTP sender) is business-critical and stateless — ideal for unit tests. However, adding a full test framework to apps/api is out of scope for this phase. API logic should be validated manually using the Wave 0 parser validation task (parse `user-files/invoice.pdf`, inspect output). Frontend components follow the existing Vitest + Testing Library pattern.
|
||||
|
||||
### Phase Requirements → Test Map
|
||||
|
||||
| Req ID | Behavior | Test Type | Automated Command | File Exists? |
|
||||
|--------|----------|-----------|-------------------|-------------|
|
||||
| DKV-01 | Inbox polling detects DKV emails | Manual (requires live inbox) | — | — |
|
||||
| DKV-02 | PDF parser extracts correct vehicle/transaction data | Manual integration (parse invoice.pdf in Wave 0) | — | ❌ Wave 0 |
|
||||
| DKV-03 | Vehicle CRUD table renders and submits correctly | Unit (Vitest + Testing Library) | `pnpm --filter @tessera/web test` | ❌ Wave 0 |
|
||||
| DKV-04 | Excel export has correct 5 columns and data | Manual (open generated .xlsx in Excel) | — | — |
|
||||
| DKV-05 | SMTP settings form saves and reads back | Unit (Vitest + Testing Library) | `pnpm --filter @tessera/web test` | ❌ Wave 0 |
|
||||
|
||||
### Sampling Rate
|
||||
|
||||
- **Per task commit:** `pnpm --filter @tessera/web test` (frontend tests only)
|
||||
- **Per wave merge:** `pnpm --filter @tessera/web test && pnpm --filter @tessera/api type-check`
|
||||
- **Phase gate:** All frontend tests green + manual validation of PDF parse output + manual Excel inspection before `/gsd-verify-work`
|
||||
|
||||
### Wave 0 Gaps
|
||||
|
||||
- [ ] `apps/api/src/dkv/dkv-parser.service.spec.ts` — manual validation script parsing `user-files/invoice.pdf` and printing structured output (not automated test, but required before writing production regex)
|
||||
- [ ] `apps/web/src/app/(portal)/modules/dkv-fleet/settings/components/VehicleTable.test.tsx` — covers DKV-03
|
||||
- [ ] `apps/web/src/app/(portal)/settings/general/smtp/smtp-settings.test.tsx` — covers DKV-05
|
||||
|
||||
---
|
||||
|
||||
## Security Domain
|
||||
|
||||
> `security_enforcement: true` in config.json, ASVS level 1.
|
||||
|
||||
### Applicable ASVS Categories
|
||||
|
||||
| ASVS Category | Applies | Standard Control |
|
||||
|---------------|---------|-----------------|
|
||||
| V2 Authentication | No | Module is admin-only, gated by existing JwtAuthGuard + RolesGuard |
|
||||
| V3 Session Management | No | Handled by existing auth layer |
|
||||
| V4 Access Control | Yes | ADMIN role required for all /dkv/* endpoints — use `@Roles(Role.ADMIN)` |
|
||||
| V5 Input Validation | Yes | class-validator DTOs for all inputs; sanitize senderFilter (email format), folder (alphanumeric/slash) |
|
||||
| V6 Cryptography | Yes | CalendarCryptoService (AES-256-GCM) for credentials — never hand-roll |
|
||||
| V7 Error Handling | Yes | Never expose decrypted credentials in error messages — follow T-05-13 generic error pattern |
|
||||
|
||||
### Known Threat Patterns for this Stack
|
||||
|
||||
| Pattern | STRIDE | Standard Mitigation |
|
||||
|---------|--------|---------------------|
|
||||
| IMAP credential exposure in logs | Info Disclosure | Never log decrypted passwords; use `logger: false` in ImapFlow constructor |
|
||||
| SMTP open relay via user-supplied config | Tampering | Validate host/port inputs; SMTP config is admin-only |
|
||||
| Path traversal in export filename | Tampering | Filename is generated server-side (`DKV_YYYY-MM_<nr>.xlsx`), never user-supplied |
|
||||
| PDF bomb / decompression bomb | Denial of Service | Set max file size limit for email attachments before passing to pdf-parse |
|
||||
| Credential injection via senderFilter | Tampering | Validate senderFilter as email address format (IsEmail() validator) |
|
||||
| Excessive history accumulation | Denial of Service | Pagination on GET /dkv/history (page + limit params) |
|
||||
|
||||
---
|
||||
|
||||
## Assumptions Log
|
||||
|
||||
| # | Claim | Section | Risk if Wrong |
|
||||
|---|-------|---------|---------------|
|
||||
| A1 | `pdf-parse` v2 accepts `{ data: Buffer }` as LoadParameters | Pattern 3 / Code Examples | Parser fails at runtime; need to test against invoice.pdf in Wave 0 |
|
||||
| A2 | DKV PDF regex pattern (VEHICLE: marker, transaction row format) matches actual invoice.pdf | Pattern 4 / Code Examples | Parser produces no vehicles; all invoices fail with parse error |
|
||||
| A3 | `ews-javascript-api` v0.15.3 supports `FindItems` on Inbox folder for email messages (not just calendar) | Architecture Patterns | Exchange inbox polling broken; Exchange users cannot use the module |
|
||||
| A4 | imapflow STARTTLS mode: `secure: false` connects without TLS then STARTTLS upgrades automatically | Pattern 2 | STARTTLS connections fail or use wrong port |
|
||||
| A5 | nodemailer `createTransport()` `requireTLS: true` triggers STARTTLS negotiation | Pattern 6 | STARTTLS SMTP send fails |
|
||||
| A6 | German date strings from DKV ("13.12.2020") written as strings in xlsx display correctly in Excel | Pattern 5 | Excel treats them as text — may cause sorting issues, but functionally correct |
|
||||
|
||||
---
|
||||
|
||||
## Open Questions
|
||||
|
||||
1. **pdf-parse v2 LoadParameters Buffer key**
|
||||
- What we know: Documentation shows `{ url: '...' }` and `{ data: buffer }` loading modes
|
||||
- What's unclear: Exact property name for Buffer loading (`data` vs `buffer` vs `content`)
|
||||
- Recommendation: Wave 0 must include a 10-line test script: `new PDFParse({ data: fs.readFileSync('user-files/invoice.pdf') })` — if it fails, try `{ url: 'file://...' }` as fallback
|
||||
|
||||
2. **DKV PDF exact regex for transaction rows**
|
||||
- What we know: VEHICLE: marker anchors vehicle blocks; each row has date, station, km, product, quantity, unit
|
||||
- What's unclear: Exact column spacing, whether amounts (Netto/Brutto) are on the same row or summary rows
|
||||
- Recommendation: Wave 0 must print the raw text from invoice.pdf and examine it before writing regex
|
||||
|
||||
3. **ScheduleModule in AppModule**
|
||||
- What we know: `@nestjs/schedule@^6.1.3` is installed, NOT in AppModule imports
|
||||
- What's unclear: Whether any other planned module (future phase) also needs it — no conflict either way
|
||||
- Recommendation: Add `ScheduleModule.forRoot()` to AppModule in Wave 0, then DkvModule uses `SchedulerRegistry`
|
||||
|
||||
4. **Settings sidebar restructuring**
|
||||
- What we know: Settings currently has only "Dashboard" category (Widgets + Calendar)
|
||||
- What's unclear: Whether to add "Allgemein > SMTP" as a new top-level sidebar category or embed it in an existing one
|
||||
- Recommendation: Add a "Allgemein" category section to `SettingsSidebar` with SMTP as the first item — clean separation from dashboard settings
|
||||
|
||||
---
|
||||
|
||||
## Sources
|
||||
|
||||
### Primary (MEDIUM confidence — context7 not available, using official docs)
|
||||
|
||||
- [ImapFlow Documentation — Fetching Messages Examples](https://imapflow.com/docs/examples/fetching-messages/) — fetch/search/download patterns
|
||||
- [ImapFlow Client API](https://imapflow.com/docs/api/imapflow-client/) — API reference
|
||||
- [pdf-parse TypeDoc](https://mehmet-kozan.github.io/pdf-parse/typedoc/index.html) — LoadParameters interface, Buffer loading
|
||||
- [SheetJS Community Edition — Write Options](https://docs.sheetjs.com/docs/api/write-options/) — buffer write pattern
|
||||
- [NestJS Task Scheduling — oneuptime.com](https://oneuptime.com/blog/post/2026-02-02-nestjs-task-scheduling/view) — SchedulerRegistry.addCronJob pattern
|
||||
|
||||
### Secondary (LOW confidence — WebSearch with official source cross-check)
|
||||
|
||||
- [npm: imapflow@1.4.3](https://www.npmjs.com/package/imapflow) — version + repository verified
|
||||
- [npm: pdf-parse@2.4.5](https://www.npmjs.com/package/pdf-parse) — version + repository verified
|
||||
- [npm: xlsx@0.18.5](https://www.npmjs.com/package/xlsx) — version + repository verified
|
||||
- [Nodemailer SMTP docs](https://nodemailer.com/smtp/) — SMTP transport options
|
||||
|
||||
### Codebase (VERIFIED: grep)
|
||||
|
||||
- `apps/api/src/calendar/crypto.service.ts` — AES-256-GCM encryption service, reused for DKV credentials
|
||||
- `apps/api/src/calendar/providers/exchange.provider.ts` — EWS pattern to mirror for inbox access
|
||||
- `apps/api/src/calendar/calendar.service.ts` — Provider abstraction pattern
|
||||
- `apps/api/src/mail/mail.module.ts` — Static SMTP config; confirms @nestjs-modules/mailer is startup-configured
|
||||
- `apps/api/src/app.module.ts` — Confirmed `ScheduleModule` is NOT imported
|
||||
- `apps/api/package.json` — Confirmed `@nestjs/schedule@^6.1.3`, `nodemailer@^9.0.1`, `ews-javascript-api@0.15.3` installed
|
||||
|
||||
---
|
||||
|
||||
## Metadata
|
||||
|
||||
**Confidence breakdown:**
|
||||
- Standard Stack: MEDIUM — npm versions verified, API patterns from official docs but context7 unavailable
|
||||
- Architecture: MEDIUM — mirrors established CalendarModule patterns; DB schema is Claude's discretion
|
||||
- PDF Regex: LOW — DKV PDF structure from CONTEXT.md description only; must validate against invoice.pdf in Wave 0
|
||||
- Pitfalls: MEDIUM — pdf-parse v2 break and imapflow deadlock are documented; EWS email API is [ASSUMED]
|
||||
- ScheduleModule: HIGH — absence confirmed by grep; setup pattern from official NestJS docs
|
||||
|
||||
**Research date:** 2026-06-26
|
||||
**Valid until:** 2026-07-26 (stable libraries; DKV PDF format unlikely to change)
|
||||
Reference in New Issue
Block a user