schalli/tessera-ctl

Fork 0

Files

T

schalli 741feb5946

Tessera CI/CD / Build & Deploy (push) Blocked by required conditions

Details

Tessera CI/CD / Lint & Type Check (push) Successful in 39s

Details

Tessera CI/CD / Tests (push) Waiting to run

Details

docs(07): research phase DKV fleet module

2026-06-26 14:37:51 +02:00

47 KiB

Raw Blame History

Phase 7: DKV Fleet Module - Research

Researched: 2026-06-26 Domain: Email inbox monitoring, PDF parsing, Excel export, NestJS scheduling, SMTP configuration Confidence: MEDIUM (stack verified; DKV PDF regex patterns LOW — only structure description available, no live parse test)

<user_constraints>

User Constraints (from CONTEXT.md)

Locked Decisions

D-01: Support both IMAP (imapflow) and Exchange (EWS via ews-javascript-api) — selectable per module config. D-02: Polling via configurable cron job (interval in minutes) + manual "Jetzt prüfen" button. D-03: Inbox config fields: Protocol (IMAP/Exchange), Host, Port, Username (optional), Password (optional), Encryption (None/STARTTLS/SSL-TLS), Folder, Sender filter. D-04: Credentials stored encrypted (AES-256-GCM via existing crypto.service.ts). D-05: SMTP settings live in general settings (shared, not per-module): Host, Port, Username (optional), Password (optional), Encryption, Sender address. D-06: Existing mail module must be extended to read SMTP config from DB instead of hardcoded env vars. D-07: Export recipient address configured per module (not in general settings). D-08: Parse DKV E-Rechnung PDF using pdf-parse (text extraction + regex). D-09: Parsed data per vehicle block: Kennzeichen (from VEHICLE: ... header), per-transaction rows with Lieferdatum, Servicestation Ort, Kilometerstand, Produkt, Menge, Einheit. D-10: On parse failure: 3 retries, then mark as failed in processing history with error message. D-11: Output format: .xlsx (Excel), generated with xlsx (SheetJS). D-12: Filename: DKV_YYYY-MM_<Rechnungsnummer>.xlsx. D-13: Exactly 5 columns: Lieferdatum, Fahrzeug, Fahrer, Ort, Kilometerstand. D-14: One Excel file per processed invoice. D-15: Last 10 export files stored in user-files/, downloadable from module UI. D-16: On SMTP failure: 3 retries with exponential backoff, then mark as "Versand fehlgeschlagen"; file stays available for manual download. D-17: Vehicle master data in DB per tenant: Kennzeichen, Marke, Modell, Fahrer. D-18: Importable as CSV; also manually editable in module UI (CRUD table). D-19: Fahrzeug column format string configurable (default: {Marke}/{Modell}/{Kennzeichen}). D-20: Processing history table in UI: Datum/Zeit, Rechnungsnummer, Anzahl Fahrzeuge, Anzahl Transaktionen, Status, Export-Dateiname. D-21: History stored in DB; entries kept indefinitely (no auto-purge in v1).

Claude's Discretion

DB schema design (DkvModuleConfig, DkvVehicleMaster, DkvInvoiceHistory, SmtpConfig tables)
Cron job implementation (@nestjs/schedule ScheduleModule, dynamic SchedulerRegistry pattern)
EWS reuse from existing ews-javascript-api for inbox email access (mirroring ExchangeProvider calendar pattern)
IMAP library: imapflow (confirmed, modern Promise-based)
PDF parsing robustness (handle multi-page, multi-vehicle DKV format)
Frontend component patterns (reuse existing admin table / settings patterns)

Deferred Ideas (OUT OF SCOPE)

None — discussion stayed within phase scope. </user_constraints>

<phase_requirements>

Phase Requirements

ID	Description	Research Support
DKV-01	System automatically detects DKV invoice PDFs from a configured email inbox (IMAP or Exchange)	imapflow + ews-javascript-api patterns, ScheduleModule cron job, sender filter search
DKV-02	All transactions parsed per vehicle: date, service station, km, product, quantity, unit, net/gross	pdf-parse v2 text extraction + DKV-specific regex over vehicle blocks
DKV-03	License plate → driver mapping importable as CSV and manually editable in UI	Prisma DkvVehicleMaster table, CSV import via multipart/form-data, CRUD controller + frontend table
DKV-04	Processed data exported as Excel and sent to configurable recipient via SMTP	xlsx (SheetJS) buffer generation, nodemailer createTransport() with DB config, retry logic
DKV-05	SMTP settings in general settings; module config (inbox, sender filter, folder, recipient) editable in module settings UI	SmtpConfig Prisma table, new /settings/general/smtp route, DkvModuleConfig table
</phase_requirements>

Summary

Phase 7 builds a NestJS module (dkv) that polls an email inbox (IMAP or Exchange), downloads PDF attachments from DKV, extracts vehicle/transaction data with regex, maps license plates to drivers via a configurable DB table, generates an Excel file, and delivers it via SMTP. The module is self-contained within the Tessera NestJS monorepo but touches three cross-cutting concerns: the mail module (SMTP config from DB), the settings subsystem (new "General" category for SMTP), and the portal frontend (new module route + settings route).

All three new npm packages (imapflow, pdf-parse, xlsx) are vetted: pdf-parse and xlsx are OK per package legitimacy check; imapflow is flagged SUS due to a version published today (false positive — the package itself has existed since 2019 with 1.1M weekly downloads). The Exchange inbox access reuses the existing ews-javascript-api@0.15.3 and follows the ExchangeProvider calendar pattern already in the codebase.

The critical implementation detail is the mail service extension strategy: @nestjs-modules/mailer cannot change its SMTP transport at runtime (it's configured once at startup). The correct approach per D-06 is to add a DkvMailService that calls nodemailer.createTransport() directly with config loaded from DB at send time, while leaving the existing MailService/MailerModule untouched for system emails (password reset, welcome).

Primary recommendation: Build the DKV module as a standard NestJS module following the CalendarModule provider-abstraction pattern. Handle SMTP separately via direct nodemailer transport creation. Use SchedulerRegistry for dynamic cron interval updates.

Architectural Responsibility Map

Capability	Primary Tier	Secondary Tier	Rationale
Email inbox polling (IMAP/EWS)	API / Backend	—	Server-side background job; credentials never touch browser
PDF attachment download	API / Backend	—	Binary stream from inbox, processed server-side
DKV PDF text extraction + regex	API / Backend	—	CPU-bound parsing, no browser involvement
Excel file generation	API / Backend	—	xlsx library runs server-side; file written to user-files/
SMTP send with attachment	API / Backend	—	Outbound mail is always server-to-server
Vehicle master CRUD	API / Backend	Frontend	REST endpoints + CRUD UI table
CSV vehicle import	API / Backend	Frontend	Multipart upload to API; parsed server-side
Processing history	Database / Storage	API	Prisma table queried by API, displayed in frontend
Module config (inbox settings)	API / Backend	Frontend	Config stored in DB, UI form to edit
SMTP config (general settings)	API / Backend	Frontend	SmtpConfig DB table, new settings page
Export file download	API / Backend	Frontend	File served from user-files/ via API endpoint
Module UI (history table, manual trigger)	Frontend Server (SSR)	Browser / Client	Next.js page with client components for state

Standard Stack

Core (new packages for this phase)

Library	Version	Purpose	Why Standard
`imapflow`	1.4.3	IMAP inbox polling, message search, attachment download	Modern Promise/async-iterator API, TypeScript types included, replaces older node-imap [ASSUMED based on CONTEXT.md D-01 decision]
`pdf-parse`	2.4.5	PDF text extraction from Buffer	Class-based API (v2), TypeScript types built-in, 5.9M weekly downloads [VERIFIED: npm registry]
`xlsx`	0.18.5	Excel file generation	SheetJS Community Edition, 11.5M weekly downloads, buffer output API, MIT license [VERIFIED: npm registry]

Already installed (no new install needed)

Library	Version	Purpose	Note
`ews-javascript-api`	0.15.3	Exchange inbox access	Follow existing `ExchangeProvider` pattern [ASSUMED: same API works for email folders]
`nodemailer`	^9.0.1	SMTP send	Use `createTransport()` directly (NOT via @nestjs-modules/mailer) [ASSUMED]
`@nestjs/schedule`	^6.1.3	Cron job scheduling	Already in package.json, NOT yet imported in AppModule — Wave 0 must add `ScheduleModule.forRoot()` [VERIFIED: codebase grep]
`@nestjs-modules/mailer`	^2.3.7	System emails (password reset)	Keep unchanged; DKV bypasses it for dynamic SMTP

Installation (new packages only)

pnpm --filter @tessera/api add imapflow pdf-parse xlsx

# TypeScript types — pdf-parse and imapflow ship their own types
# xlsx also ships types; no @types/* needed for these packages
pnpm --filter @tessera/api add -D @types/xlsx 2>/dev/null || true

Version verification (run before installing):

npm view imapflow version     # 1.4.3
npm view pdf-parse version    # 2.4.5
npm view xlsx version         # 0.18.5

Package Legitimacy Audit

Packages verified via gsd-tools query package-legitimacy check --ecosystem npm imapflow pdf-parse xlsx

Package	Registry	Age	Downloads	Source Repo	Verdict	Disposition
`imapflow`	npm	Since 2019	~1.1M/wk	github.com/postalsys/imapflow	SUS (too-new flag)	Approved — flag is false positive (new version published today, not new package; 1.1M/wk and legitimate repo)
`pdf-parse`	npm	Since 2018 (v2: 2025)	~5.9M/wk	github.com/mehmet-kozan/pdf-parse	OK	Approved
`xlsx`	npm	Since 2012	~11.5M/wk	github.com/SheetJS/sheetjs	OK	Approved

Packages removed due to SLOP verdict: none

Packages flagged as suspicious SUS: imapflow — planner should note context (false positive: version released today, package established since 2019 with 1.1M/wk downloads from postalsys/imapflow). No checkpoint:human-verify needed given evidence.

Architecture Patterns

System Architecture Diagram

[Email Inbox: IMAP/EWS]
         │  (poll every N min OR manual trigger)
         ▼
[DkvSchedulerService] ──── @nestjs/schedule SchedulerRegistry
         │
         ▼
[InboxProvider interface]
    ├── ImapProvider (imapflow)
    └── ExchangeInboxProvider (ews-javascript-api)
         │  (search by sender filter, list unseen emails)
         ▼
[PDF Attachment Buffer]
         │
         ▼
[DkvParserService]
    └── pdf-parse v2: PDFParse({ data: buffer }).getText()
    └── Regex extraction: vehicle blocks → transaction rows
         │
         ▼
[DkvVehicleMaster lookup]  ◄── Prisma: DkvVehicleMaster
    └── Kennzeichen → Marke, Modell, Fahrer
         │
         ▼
[DkvExportService]
    └── xlsx: build 5-column worksheet → Buffer
    └── Write to user-files/DKV_YYYY-MM_<nr>.xlsx
    └── Prune: keep last 10 files
         │
         ▼
[DkvMailService]
    └── nodemailer.createTransport(smtpConfigFromDB)
    └── sendMail({ to: recipient, attachments: [xlsx buffer] })
    └── 3 retries with exponential backoff
         │
         ▼
[DkvInvoiceHistory record saved]  ──► Prisma: DkvInvoiceHistory
         │
         ▼
[Frontend: /modules/dkv-fleet/page.tsx]
    ├── History table (GET /dkv/history)
    ├── Manual "Jetzt prüfen" button (POST /dkv/check-now)
    ├── Export file download (GET /dkv/exports/:filename)
    └── Settings tab → /modules/dkv-fleet/settings/
         └── Inbox config form (GET/PUT /dkv/config)
         └── Vehicle CRUD table (GET/POST/PUT/DELETE /dkv/vehicles)
         └── CSV import button (POST /dkv/vehicles/import)

[/settings/general/smtp]  ──► GET/PUT /settings/smtp  ──► SmtpConfig (Prisma)

Recommended Project Structure

apps/api/src/dkv/
├── dkv.module.ts             # Module: imports ScheduleModule, CalendarCryptoService
├── dkv.controller.ts         # REST: /dkv/* routes (history, config, vehicles, check-now, exports)
├── dkv.service.ts            # Orchestration: coordinates scheduler, parser, export, mail
├── dkv-scheduler.service.ts  # SchedulerRegistry wrapper, cron lifecycle management
├── dkv-parser.service.ts     # pdf-parse + DKV regex → structured data
├── dkv-export.service.ts     # xlsx buffer generation, user-files/ management (10-file limit)
├── dkv-mail.service.ts       # nodemailer.createTransport() with SmtpConfig from DB
├── providers/
│   ├── inbox-provider.interface.ts    # InboxProvider: connect(), searchSender(), downloadAttachments()
│   ├── imap.provider.ts               # imapflow implementation
│   └── exchange-inbox.provider.ts     # ews-javascript-api implementation (mirrors ExchangeProvider)
└── dto/
    ├── dkv-config.dto.ts
    ├── dkv-vehicle.dto.ts
    └── dkv-history.dto.ts

apps/api/src/settings/
├── settings.module.ts        # NEW: SMTP config management
├── settings.controller.ts    # GET/PUT /settings/smtp
├── settings.service.ts       # SmtpConfig CRUD + decrypt
└── dto/smtp-config.dto.ts

apps/web/src/app/(portal)/modules/dkv-fleet/
├── page.tsx                  # Main view: invoice history table
├── settings/
│   ├── page.tsx              # Module settings: inbox config + vehicle master
│   └── components/
│       ├── InboxConfigForm.tsx
│       ├── VehicleTable.tsx
│       └── CsvImportButton.tsx
└── components/
    └── InvoiceHistoryTable.tsx

apps/web/src/app/(portal)/settings/general/
└── smtp/
    └── page.tsx              # SMTP settings page (new settings category)

Prisma Schema (new tables)

model DkvModuleConfig {
  id                  String   @id @default(uuid())
  tenantId            String   @unique
  protocol            String   @default("imap")   // 'imap' | 'exchange'
  host                String?
  port                Int?
  encryption          String   @default("ssl-tls") // 'none' | 'starttls' | 'ssl-tls'
  folder              String   @default("INBOX")
  senderFilter        String?
  pollIntervalMin     Int      @default(60)
  isActive            Boolean  @default(false)
  exportRecipient     String?
  vehicleFormatString String   @default("{Marke}/{Modell}/{Kennzeichen}")
  encryptedInboxCreds String?  // AES-256-GCM: JSON { username, password } encrypted
  createdAt           DateTime @default(now())
  updatedAt           DateTime @updatedAt
  @@index([tenantId])
}

model DkvVehicleMaster {
  id          String   @id @default(uuid())
  tenantId    String
  kennzeichen String
  marke       String
  modell      String
  fahrer      String   // "Vorname Nachname"
  createdAt   DateTime @default(now())
  updatedAt   DateTime @updatedAt
  @@unique([tenantId, kennzeichen])
  @@index([tenantId])
}

model DkvInvoiceHistory {
  id                  String   @id @default(uuid())
  tenantId            String
  datumZeit           DateTime @default(now())
  rechnungsnummer     String
  anzahlFahrzeuge     Int      @default(0)
  anzahlTransaktionen Int      @default(0)
  status              String   // 'Verarbeitet' | 'Fehler' | 'Versand fehlgeschlagen'
  errorMessage        String?
  exportFilename      String?
  createdAt           DateTime @default(now())
  @@index([tenantId])
  @@index([datumZeit])
}

model SmtpConfig {
  id                String   @id @default(uuid())
  tenantId          String   @unique
  host              String
  port              Int      @default(587)
  encryption        String   @default("starttls") // 'none' | 'starttls' | 'ssl-tls'
  username          String?
  encryptedPassword String?  // AES-256-GCM via CalendarCryptoService
  fromAddress       String
  createdAt         DateTime @default(now())
  updatedAt         DateTime @updatedAt
}

Pattern 1: InboxProvider Interface (mirrors CalendarProvider)

What: Abstract inbox provider so IMAP and Exchange are interchangeable. When to use: Any service that needs to list and download email attachments.

// Source: mirrors apps/api/src/calendar/calendar.service.ts CalendarProvider pattern
export interface InboxEmail {
  uid: number | string;
  messageId: string;
  subject: string;
  from: string;
  date: Date;
  attachments: InboxAttachment[];
}

export interface InboxAttachment {
  filename: string;
  contentType: string;
  buffer: Buffer;
}

export interface InboxProvider {
  fetchPdfAttachments(
    config: InboxConfig,
  ): Promise<InboxEmail[]>;

  testConnection(config: InboxConfig): Promise<boolean>;
}

Pattern 2: imapflow IMAP Provider

What: Fetch emails from a specific folder filtered by sender, download PDF attachments. When to use: When DkvModuleConfig.protocol === 'imap'.

// Source: [CITED: imapflow.com/docs/examples/fetching-messages/]
import { ImapFlow } from 'imapflow';

async function fetchPdfAttachments(config: InboxConfig): Promise<InboxEmail[]> {
  const client = new ImapFlow({
    host: config.host,
    port: config.port,
    secure: config.encryption === 'ssl-tls',      // port 993
    requireTLS: config.encryption === 'starttls', // port 587
    auth: { user: config.username, pass: config.password },
    logger: false, // suppress verbose imapflow logs
  });

  await client.connect();
  const lock = await client.getMailboxLock(config.folder);
  const results: InboxEmail[] = [];

  try {
    // Search by sender — returns UIDs
    const uids = await client.search({ from: config.senderFilter }, { uid: true });
    if (uids.length === 0) return results;

    // Fetch all first, then process (avoids IMAP deadlock — see Pitfall 1)
    const messages = await client.fetchAll(
      uids.join(','),
      { envelope: true, bodyStructure: true },
      { uid: true },
    );

    for (const msg of messages) {
      const pdfs = collectPdfParts(msg.bodyStructure);
      const attachments: InboxAttachment[] = [];
      for (const partId of pdfs) {
        const { content } = await client.download(
          String(msg.uid), partId, { uid: true }
        );
        const buf = await streamToBuffer(content);
        attachments.push({ filename: `attachment-${partId}.pdf`, contentType: 'application/pdf', buffer: buf });
      }
      results.push({ uid: msg.uid!, messageId: msg.envelope.messageId ?? '', subject: msg.envelope.subject ?? '', from: msg.envelope.from?.[0]?.address ?? '', date: msg.envelope.date ?? new Date(), attachments });
    }
  } finally {
    lock.release();
    await client.logout();
  }
  return results;
}

Pattern 3: pdf-parse v2 Text Extraction

What: Extract raw text from a PDF Buffer using the v2 class-based API. When to use: After downloading a PDF attachment from the inbox.

// Source: [CITED: mehmet-kozan.github.io/pdf-parse/typedoc/]
// IMPORTANT: pdf-parse v2 uses a class-based API — NOT the v1 function call pdfParse(buffer)
import { PDFParse } from 'pdf-parse';

async function extractPdfText(buffer: Buffer): Promise<string> {
  const parser = new PDFParse({ data: buffer }); // 'data' key accepts Buffer [ASSUMED]
  const result = await parser.getText();
  await parser.destroy(); // always call destroy() to free memory
  return result.text;     // full text from all pages concatenated
}

Assumption flag: The { data: buffer } LoadParameters key is inferred from the TypeDoc page — confirmed the interface accepts both URL and buffer, but exact property name tagged [ASSUMED]. Planner should add a Wave 0 verify step that runs a quick test parse against user-files/invoice.pdf.

Pattern 4: DKV PDF Regex — Vehicle Block Extraction

What: Parse DKV E-Rechnung text into structured vehicle/transaction data. When to use: After extracting raw text from the PDF.

// Source: [ASSUMED based on CONTEXT.md D-09 + D-specifics section]
// DKV PDF structure (from CONTEXT.md specifics):
//   VEHICLE: {Kennzeichen} CARD NO.: {CardNo}
//   ... transaction rows ...
//   TOTAL: ...

interface DkvTransaction {
  lieferdatum: string;     // "DD.MM.YYYY"
  ort: string;             // Servicestation Ort
  kilometerstand: number;
  produkt: string;
  menge: number;
  einheit: string;
  netto: number;
  brutto: number;
}

interface DkvVehicleBlock {
  kennzeichen: string;
  cardNumber: string;
  transactions: DkvTransaction[];
}

function parseDkvText(text: string): DkvVehicleBlock[] {
  const vehicles: DkvVehicleBlock[] = [];

  // Split text into vehicle blocks — anchor on VEHICLE: marker
  // Pattern: "VEHICLE: GP-JL 728E CARD NO.: 7020261..."
  const vehicleBlockPattern = /VEHICLE:\s+(\S+)\s+CARD NO\.:\s+(\S+)([\s\S]*?)(?=VEHICLE:|$)/g;
  let match: RegExpExecArray | null;

  while ((match = vehicleBlockPattern.exec(text)) !== null) {
    const kennzeichen = match[1];
    const cardNumber = match[2];
    const blockText = match[3];

    const transactions = parseTransactionRows(blockText, kennzeichen);
    vehicles.push({ kennzeichen, cardNumber, transactions });
  }
  return vehicles;
}

// Transaction row regex — [ASSUMED] based on DKV invoice structure description
// Row format: "DD.MM.YYYY  Station City  NNN.NNN km  Diesel  NN.NNN  L"
function parseTransactionRows(blockText: string, _plate: string): DkvTransaction[] {
  const txPattern =
    /(\d{2}\.\d{2}\.\d{4})\s+(.+?)\s{2,}([\d.,]+)\s+km\s+(\S+)\s+([\d.,]+)\s+(\w+)/g;
  const transactions: DkvTransaction[] = [];
  let m: RegExpExecArray | null;
  while ((m = txPattern.exec(blockText)) !== null) {
    transactions.push({
      lieferdatum: m[1],
      ort: m[2].trim(),
      kilometerstand: parseFloat(m[3].replace('.', '').replace(',', '.')),
      produkt: m[4],
      menge: parseFloat(m[5].replace('.', '').replace(',', '.')),
      einheit: m[6],
      netto: 0,   // TODO: extract from block if present
      brutto: 0,
    });
  }
  return transactions;
}

Warning: The transaction row regex is [ASSUMED] from the CONTEXT.md structure description. The actual DKV PDF format must be validated against user-files/invoice.pdf in Wave 0 before the parser service is written. This is the highest-risk regex in the system.

Pattern 5: SheetJS Excel Generation

What: Build a 5-column worksheet and return it as a Buffer. When to use: After assembling all vehicle/transaction rows for one invoice.

// Source: [CITED: docs.sheetjs.com/docs/api/write-options/]
import * as XLSX from 'xlsx';

function buildExcelBuffer(rows: ExportRow[], vehicleFormat: string): Buffer {
  const headers = ['Lieferdatum', 'Fahrzeug', 'Fahrer', 'Ort', 'Kilometerstand'];
  const data = rows.map(r => [
    r.lieferdatum,   // string "DD.MM.YYYY" — write as-is, no Date conversion
    r.fahrzeug,      // "Marke/Modell/Kennzeichen" — from format string
    r.fahrer,        // "Vorname Nachname"
    r.ort,           // string
    r.kilometerstand // number — no unit
  ]);

  const ws = XLSX.utils.aoa_to_sheet([headers, ...data]);
  const wb = XLSX.utils.book_new();
  XLSX.utils.book_append_sheet(wb, ws, 'DKV Export');

  return XLSX.write(wb, { type: 'buffer', bookType: 'xlsx' }) as Buffer;
}

Pattern 6: Dynamic SMTP via Nodemailer (bypasses @nestjs-modules/mailer)

What: Create a nodemailer transport from DB config at send time. When to use: Every DKV export send — config may change between runs.

// Source: [CITED: nodemailer.com/smtp]  [ASSUMED: dynamic createTransport pattern]
import * as nodemailer from 'nodemailer';

async function sendExportEmail(
  smtpConfig: SmtpConfig,
  recipient: string,
  attachmentBuffer: Buffer,
  filename: string,
): Promise<void> {
  const secure = smtpConfig.encryption === 'ssl-tls'; // port 465
  const requireTLS = smtpConfig.encryption === 'starttls'; // port 587

  const transport = nodemailer.createTransport({
    host: smtpConfig.host,
    port: smtpConfig.port,
    secure,
    requireTLS,
    auth: smtpConfig.username
      ? { user: smtpConfig.username, pass: smtpConfig.decryptedPassword }
      : undefined,
  });

  await transport.sendMail({
    from: smtpConfig.fromAddress,
    to: recipient,
    subject: `DKV Flottenabrechnung: ${filename}`,
    text: 'DKV Flottenabrechnung im Anhang.',
    attachments: [
      { filename, content: attachmentBuffer, contentType: 'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet' },
    ],
  });
}

Pattern 7: Dynamic Cron Job (SchedulerRegistry)

What: Create/update a cron job from a DB-sourced interval in minutes. When to use: On module init and after config change.

// Source: [CITED: oneuptime.com/blog/post/2026-02-02-nestjs-task-scheduling/view]
import { Injectable, OnModuleInit } from '@nestjs/common';
import { SchedulerRegistry } from '@nestjs/schedule';
import { CronJob } from 'cron';

const JOB_NAME = 'dkv-inbox-poll';

@Injectable()
export class DkvSchedulerService implements OnModuleInit {
  constructor(
    private readonly schedulerRegistry: SchedulerRegistry,
    private readonly dkvService: DkvService,
  ) {}

  onModuleInit() {
    // Load interval from DB config on startup
    this.dkvService.loadConfig().then(config => {
      if (config?.isActive) {
        this.setInterval(config.pollIntervalMin);
      }
    });
  }

  setInterval(intervalMin: number): void {
    // Remove existing job if any
    try {
      const existing = this.schedulerRegistry.getCronJob(JOB_NAME);
      existing.stop();
      this.schedulerRegistry.deleteCronJob(JOB_NAME);
    } catch { /* not registered yet */ }

    // Create new job with computed cron expression
    const cronExpr = `*/${intervalMin} * * * *`; // every N minutes
    const job = new CronJob(cronExpr, () => {
      this.dkvService.processInbox().catch(err =>
        this.logger.error('DKV inbox poll failed', err)
      );
    });
    this.schedulerRegistry.addCronJob(JOB_NAME, job);
    job.start();
  }
}

Pattern 8: ScheduleModule Registration (AppModule update)

@nestjs/schedule is installed but NOT yet imported in app.module.ts. Must be added in Wave 0.

// apps/api/src/app.module.ts — add to imports array
import { ScheduleModule } from '@nestjs/schedule';

@Module({
  imports: [
    // ... existing imports ...
    ScheduleModule.forRoot(),   // ADD THIS — required for cron/interval decorators
    DkvModule,                  // ADD THIS — new DKV fleet module
  ],
})

Settings Sidebar Extension

Add "Allgemein" category to SettingsSidebar with SMTP link:

// apps/web/src/components/settings/settings-sidebar.tsx — extend items array
const generalItems = [
  { label: t('categorySmtp'), href: '/settings/general/smtp' },
];

Anti-Patterns to Avoid

Mixing fetch + download in imapflow async iterator: The imapflow docs explicitly warn against it — causes IMAP connection deadlock. Always call fetchAll() first to collect all message metadata, then loop and call download() separately.
Using @nestjs-modules/mailer for dynamic SMTP: The mailer transport is configured once at startup via forRootAsync. Attempting to change it at runtime breaks all system emails. Use nodemailer.createTransport() directly in DkvMailService.
Calling pdfParse(buffer) v1 API: pdf-parse v2 is NOT backward-compatible. The old import pdfParse from 'pdf-parse'; const data = await pdfParse(buffer); call will fail. Use new PDFParse({ data: buffer }) instead.
Writing Lieferdatum as a JS Date in xlsx: The DKV dates are already formatted German strings ("13.12.2020"). Writing them as JS Date objects causes Excel to serialize them as numeric serials (or reformat them). Write as strings directly.
Storing decrypted credentials in logs: NEVER log the result of crypto.decrypt() — follow T-05-13 pattern from CalendarService (generic error messages only).
@Interval() decorator with hardcoded value: The poll interval is configurable from DB — use SchedulerRegistry.addCronJob() instead of the static @Interval(ms) decorator, so the interval can be updated when config changes.

Don't Hand-Roll

Problem	Don't Build	Use Instead	Why
PDF text extraction	Custom pdfjs integration	`pdf-parse` v2	pdfjs-dist requires worker setup, much heavier
Excel file generation	CSV with .xlsx extension	`xlsx` (SheetJS)	Excel requires OOXML zip format; CSV named as xlsx corrupts in Excel
IMAP protocol	Raw TCP/IMAP commands	`imapflow`	IMAP has 30+ capabilities, extension negotiation, IDLE mode — do not hand-roll
AES encryption	Custom cipher	`CalendarCryptoService`	Already implemented, tested, consistent with calendar credentials
Cron scheduling	setInterval() in a Node.js service	`@nestjs/schedule` + `SchedulerRegistry`	Restart-safe, injectable, monitorable
CSV parsing	String split by comma	(inline split is fine for simple CSVs)	DKV vehicle CSV is simple (Kennzeichen;Marke;Modell;Fahrer) — regex or split is acceptable; don't add papaparse just for this

Key insight: PDF-to-Excel pipelines look simple but have many format edge cases (multi-page PDFs, Unicode station names, German number formats with dot-as-thousands-separator). Trust the established libraries for the binary format layers; only write custom regex for DKV-specific structure extraction.

Common Pitfalls

Pitfall 1: imapflow IMAP Connection Deadlock

What goes wrong: Calling client.download() inside a fetch() async iterator causes the IMAP connection to deadlock — the iterator holds the connection open while download() tries to issue another command on the same connection.

Why it happens: imapflow uses a single IMAP connection. The fetch() method returns an async iterator that streams messages; issuing another command while the iterator is active queues on the same connection and never resolves.

How to avoid: Always use fetchAll() (which resolves once, fully) to collect message metadata first. Then loop over the results and call download() separately. This is explicitly documented in imapflow examples.

Warning signs: await client.download(...) never resolves when called inside a for await ... of client.fetch(...) loop.

Pitfall 2: pdf-parse v1 vs v2 API Break

What goes wrong: Calling pdfParse(buffer) (v1 API) throws TypeError: pdfParse is not a function or similar because pdf-parse v2 is a class-based API.

Why it happens: The npm package pdf-parse at v2.x is a complete rewrite by mehmet-kozan with a new class-based API. Any tutorial from before 2025 shows the v1 function-call API.

How to avoid: Use new PDFParse({ data: buffer }); await parser.getText(); await parser.destroy();. Write a Wave 0 integration test parsing user-files/invoice.pdf to confirm the API before writing the production parser.

Warning signs: LangChain community issue #22 confirms this exact breakage: "PDFLoader declares pdf-parse v2 but only supports v1".

Pitfall 3: @nestjs-modules/mailer Cannot Change Transport at Runtime

What goes wrong: Extending MailModule to load SMTP config from DB at startup seems straightforward, but if the user updates SMTP config in the UI, the running transport still uses the old settings until a restart.

Why it happens: MailerModule.forRootAsync() constructs the nodemailer transport once when the NestJS DI container initializes. There's no API to replace the transport object afterward.

How to avoid: Keep MailerModule for system emails (env-based, rarely changed). For DKV sends, use nodemailer.createTransport(await loadSmtpConfig()) at send time in DkvMailService. Config changes take effect immediately on the next send.

Pitfall 4: ScheduleModule Not Registered in AppModule

What goes wrong: @Cron() / @Interval() decorators silently do nothing; SchedulerRegistry throws "SchedulerRegistry is not injectable" or jobs never fire.

Why it happens: @nestjs/schedule is installed but ScheduleModule.forRoot() is NOT imported in AppModule (confirmed by grep — absent from app.module.ts).

How to avoid: Wave 0 task must add ScheduleModule.forRoot() to AppModule imports. This is a prerequisite for DkvSchedulerService to function.

Warning signs: CronJob registered via schedulerRegistry.addCronJob() but callback never fires.

Pitfall 5: German Number Format in km/Menge Parsing

What goes wrong: DKV PDFs use German number formatting: 19.234,56 (dot as thousands separator, comma as decimal). parseFloat("19.234,56") returns 19.234 (truncated at the comma).

Why it happens: JavaScript parseFloat expects English format (comma as thousands separator, dot as decimal).

How to avoid: Strip dots then replace comma with dot before parsing: parseFloat(raw.replace(/\./g, '').replace(',', '.')). Apply this to Kilometerstand and Menge fields.

Pitfall 6: EWS Email vs Calendar Folder Access

What goes wrong: Copying ExchangeProvider directly and calling FindAppointments returns no results because email folders use FindItems / EmailMessage.Bind, not FindAppointments.

Why it happens: EWS has separate item types — Appointment for calendar, EmailMessage for inbox. The existing ExchangeProvider uses the Calendar well-known folder name.

How to avoid: In ExchangeInboxProvider, use ews.WellKnownFolderName.Inbox with service.FindItems() and bind to EmailMessage. Follow EWS docs for email vs appointment distinction. [ASSUMED — not verified against ews-javascript-api 0.15.3 API docs]

Pitfall 7: user-files/ 10-File Pruning Race Condition

What goes wrong: If two invoices are processed simultaneously (concurrent cron + manual trigger), both may pass the "10 files exist" check and write an 11th file before either runs the prune.

Why it happens: File count check and write are not atomic.

How to avoid: Use a mutex (simple in-memory lock in DkvExportService) or route all inbox processing through a queue so only one invoice processes at a time. Simplest: add a private processing = false guard in DkvService.processInbox() — if already processing, return early.

Code Examples

Inversion of Control: Credential Encryption Pattern

// Source: apps/api/src/calendar/crypto.service.ts [VERIFIED: codebase]
// Reuse CalendarCryptoService directly — it's not calendar-specific despite its name.
// Encrypt both inbox credentials and SMTP password with the same CALENDAR_ENCRYPTION_KEY.

const encryptedInboxCreds = this.crypto.encrypt(
  JSON.stringify({ username: dto.username, password: dto.password })
);
// Store as single encrypted JSON blob in DkvModuleConfig.encryptedInboxCreds

// Decrypt at use time:
const { username, password } = JSON.parse(this.crypto.decrypt(config.encryptedInboxCreds!));

CSV Vehicle Import Pattern

// Simple CSV parse for DKV vehicle master — semicolon-delimited
// Expected format: Kennzeichen;Marke;Modell;Fahrer
// Source: [ASSUMED]
function parseVehicleCsv(csvText: string): CreateVehicleDto[] {
  const lines = csvText.trim().split('\n');
  const headers = lines[0].split(';').map(h => h.trim().toLowerCase());
  return lines.slice(1).map(line => {
    const cols = line.split(';').map(c => c.trim());
    return {
      kennzeichen: cols[headers.indexOf('kennzeichen')] ?? '',
      marke: cols[headers.indexOf('marke')] ?? '',
      modell: cols[headers.indexOf('modell')] ?? '',
      fahrer: cols[headers.indexOf('fahrer')] ?? '',
    };
  }).filter(v => v.kennzeichen);
}

Fahrzeug Format String Resolution

// Source: [ASSUMED based on D-19 format template]
// Default: "{Marke}/{Modell}/{Kennzeichen}"
function resolveFahrzeug(
  vehicle: DkvVehicleMaster,
  formatString: string,
): string {
  return formatString
    .replace('{Marke}', vehicle.marke)
    .replace('{Modell}', vehicle.modell)
    .replace('{Kennzeichen}', vehicle.kennzeichen)
    .replace('{Fahrer}', vehicle.fahrer);
}

Runtime State Inventory

Greenfield module — no existing runtime state to migrate. All tables are new.

Category	Items Found	Action Required
Stored data	None — DkvModuleConfig, DkvVehicleMaster, DkvInvoiceHistory, SmtpConfig are new tables	Prisma migration only
Live service config	None	—
OS-registered state	None	—
Secrets/env vars	`CALENDAR_ENCRYPTION_KEY` is reused for DKV credential encryption — no new env var needed	Ensure env var is set in Docker Compose (already required by CalendarModule)
Build artifacts	None	—

Environment Availability

Dependency	Required By	Available	Version	Fallback
Node.js	imapflow, pdf-parse, xlsx	Yes	v24.16.0	—
Docker	DB + full stack	Yes	29.5.3	—
pnpm	package install	Yes	9.15.0	—
`CALENDAR_ENCRYPTION_KEY` env var	CalendarCryptoService (shared)	Assumed set	—	Module init throws if missing
`user-files/` directory	Export file storage	Yes (exists in repo)	—	Create if missing
`user-files/invoice.pdf`	Wave 0 parser validation	Yes (in repo)	—	Required for regex dev

Missing dependencies with no fallback: None identified.

State of the Art

Old Approach	Current Approach	When Changed	Impact
pdf-parse v1: `pdfParse(buffer)` function	pdf-parse v2: `new PDFParse({data}).getText()`	v2.x (2025)	Breaking API change — all v1 tutorials are wrong
@nestjs/schedule: static `@Interval()`	Dynamic `SchedulerRegistry.addCronJob()`	Always supported	Required for DB-configurable intervals
@nestjs-modules/mailer dynamic transport	nodemailer.createTransport() at send time	Pattern choice	Only way to support runtime SMTP config changes

Deprecated/outdated:

node-imap: Outdated callback-based IMAP library. Do not use. imapflow is the modern replacement.
pdf-parse v1 API (pdfParse(buffer) function): Removed in v2.x.

Validation Architecture

Test Framework

Property	Value
Framework	Vitest 4.1.9 (apps/web only)
Config file	apps/web/vitest.config.ts
Quick run command	`pnpm --filter @tessera/web test`
Full suite command	`pnpm --filter @tessera/web test`
API test framework	None configured — no vitest/jest in apps/api

Note: The API has no test framework configured. Phase 7 API logic (PDF parser, Excel builder, SMTP sender) is business-critical and stateless — ideal for unit tests. However, adding a full test framework to apps/api is out of scope for this phase. API logic should be validated manually using the Wave 0 parser validation task (parse user-files/invoice.pdf, inspect output). Frontend components follow the existing Vitest + Testing Library pattern.

Phase Requirements → Test Map

Req ID	Behavior	Test Type	Automated Command	File Exists?
DKV-01	Inbox polling detects DKV emails	Manual (requires live inbox)	—	—
DKV-02	PDF parser extracts correct vehicle/transaction data	Manual integration (parse invoice.pdf in Wave 0)	—	❌ Wave 0
DKV-03	Vehicle CRUD table renders and submits correctly	Unit (Vitest + Testing Library)	`pnpm --filter @tessera/web test`	❌ Wave 0
DKV-04	Excel export has correct 5 columns and data	Manual (open generated .xlsx in Excel)	—	—
DKV-05	SMTP settings form saves and reads back	Unit (Vitest + Testing Library)	`pnpm --filter @tessera/web test`	❌ Wave 0

Sampling Rate

Per task commit: pnpm --filter @tessera/web test (frontend tests only)
Per wave merge: pnpm --filter @tessera/web test && pnpm --filter @tessera/api type-check
Phase gate: All frontend tests green + manual validation of PDF parse output + manual Excel inspection before /gsd-verify-work

Wave 0 Gaps

apps/api/src/dkv/dkv-parser.service.spec.ts — manual validation script parsing user-files/invoice.pdf and printing structured output (not automated test, but required before writing production regex)
apps/web/src/app/(portal)/modules/dkv-fleet/settings/components/VehicleTable.test.tsx — covers DKV-03
apps/web/src/app/(portal)/settings/general/smtp/smtp-settings.test.tsx — covers DKV-05

Security Domain

security_enforcement: true in config.json, ASVS level 1.

Applicable ASVS Categories

ASVS Category	Applies	Standard Control
V2 Authentication	No	Module is admin-only, gated by existing JwtAuthGuard + RolesGuard
V3 Session Management	No	Handled by existing auth layer
V4 Access Control	Yes	ADMIN role required for all /dkv/* endpoints — use `@Roles(Role.ADMIN)`
V5 Input Validation	Yes	class-validator DTOs for all inputs; sanitize senderFilter (email format), folder (alphanumeric/slash)
V6 Cryptography	Yes	CalendarCryptoService (AES-256-GCM) for credentials — never hand-roll
V7 Error Handling	Yes	Never expose decrypted credentials in error messages — follow T-05-13 generic error pattern

Known Threat Patterns for this Stack

Pattern	STRIDE	Standard Mitigation
IMAP credential exposure in logs	Info Disclosure	Never log decrypted passwords; use `logger: false` in ImapFlow constructor
SMTP open relay via user-supplied config	Tampering	Validate host/port inputs; SMTP config is admin-only
Path traversal in export filename	Tampering	Filename is generated server-side (`DKV_YYYY-MM_<nr>.xlsx`), never user-supplied
PDF bomb / decompression bomb	Denial of Service	Set max file size limit for email attachments before passing to pdf-parse
Credential injection via senderFilter	Tampering	Validate senderFilter as email address format (IsEmail() validator)
Excessive history accumulation	Denial of Service	Pagination on GET /dkv/history (page + limit params)

Assumptions Log

#	Claim	Section	Risk if Wrong
A1	`pdf-parse` v2 accepts `{ data: Buffer }` as LoadParameters	Pattern 3 / Code Examples	Parser fails at runtime; need to test against invoice.pdf in Wave 0
A2	DKV PDF regex pattern (VEHICLE: marker, transaction row format) matches actual invoice.pdf	Pattern 4 / Code Examples	Parser produces no vehicles; all invoices fail with parse error
A3	`ews-javascript-api` v0.15.3 supports `FindItems` on Inbox folder for email messages (not just calendar)	Architecture Patterns	Exchange inbox polling broken; Exchange users cannot use the module
A4	imapflow STARTTLS mode: `secure: false` connects without TLS then STARTTLS upgrades automatically	Pattern 2	STARTTLS connections fail or use wrong port
A5	nodemailer `createTransport()` `requireTLS: true` triggers STARTTLS negotiation	Pattern 6	STARTTLS SMTP send fails
A6	German date strings from DKV ("13.12.2020") written as strings in xlsx display correctly in Excel	Pattern 5	Excel treats them as text — may cause sorting issues, but functionally correct

Open Questions

pdf-parse v2 LoadParameters Buffer key
- What we know: Documentation shows { url: '...' } and { data: buffer } loading modes
- What's unclear: Exact property name for Buffer loading (data vs buffer vs content)
- Recommendation: Wave 0 must include a 10-line test script: new PDFParse({ data: fs.readFileSync('user-files/invoice.pdf') }) — if it fails, try { url: 'file://...' } as fallback
DKV PDF exact regex for transaction rows
- What we know: VEHICLE: marker anchors vehicle blocks; each row has date, station, km, product, quantity, unit
- What's unclear: Exact column spacing, whether amounts (Netto/Brutto) are on the same row or summary rows
- Recommendation: Wave 0 must print the raw text from invoice.pdf and examine it before writing regex
ScheduleModule in AppModule
- What we know: @nestjs/schedule@^6.1.3 is installed, NOT in AppModule imports
- What's unclear: Whether any other planned module (future phase) also needs it — no conflict either way
- Recommendation: Add ScheduleModule.forRoot() to AppModule in Wave 0, then DkvModule uses SchedulerRegistry
Settings sidebar restructuring
- What we know: Settings currently has only "Dashboard" category (Widgets + Calendar)
- What's unclear: Whether to add "Allgemein > SMTP" as a new top-level sidebar category or embed it in an existing one
- Recommendation: Add a "Allgemein" category section to SettingsSidebar with SMTP as the first item — clean separation from dashboard settings

Sources

Primary (MEDIUM confidence — context7 not available, using official docs)

ImapFlow Documentation — Fetching Messages Examples — fetch/search/download patterns
ImapFlow Client API — API reference
pdf-parse TypeDoc — LoadParameters interface, Buffer loading
SheetJS Community Edition — Write Options — buffer write pattern
NestJS Task Scheduling — oneuptime.com — SchedulerRegistry.addCronJob pattern

Secondary (LOW confidence — WebSearch with official source cross-check)

npm: imapflow@1.4.3 — version + repository verified
npm: pdf-parse@2.4.5 — version + repository verified
npm: xlsx@0.18.5 — version + repository verified
Nodemailer SMTP docs — SMTP transport options

Codebase (VERIFIED: grep)

apps/api/src/calendar/crypto.service.ts — AES-256-GCM encryption service, reused for DKV credentials
apps/api/src/calendar/providers/exchange.provider.ts — EWS pattern to mirror for inbox access
apps/api/src/calendar/calendar.service.ts — Provider abstraction pattern
apps/api/src/mail/mail.module.ts — Static SMTP config; confirms @nestjs-modules/mailer is startup-configured
apps/api/src/app.module.ts — Confirmed ScheduleModule is NOT imported
apps/api/package.json — Confirmed @nestjs/schedule@^6.1.3, nodemailer@^9.0.1, ews-javascript-api@0.15.3 installed

Metadata

Confidence breakdown:

Standard Stack: MEDIUM — npm versions verified, API patterns from official docs but context7 unavailable
Architecture: MEDIUM — mirrors established CalendarModule patterns; DB schema is Claude's discretion
PDF Regex: LOW — DKV PDF structure from CONTEXT.md description only; must validate against invoice.pdf in Wave 0
Pitfalls: MEDIUM — pdf-parse v2 break and imapflow deadlock are documented; EWS email API is [ASSUMED]
ScheduleModule: HIGH — absence confirmed by grep; setup pattern from official NestJS docs

Research date: 2026-06-26 Valid until: 2026-07-26 (stable libraries; DKV PDF format unlikely to change)

47 KiB Raw Blame History

Phase 7: DKV Fleet Module - Research

User Constraints (from CONTEXT.md)

Locked Decisions

Claude's Discretion

Deferred Ideas (OUT OF SCOPE)

Phase Requirements

Summary

Architectural Responsibility Map

Standard Stack

Core (new packages for this phase)

Already installed (no new install needed)

Installation (new packages only)

Package Legitimacy Audit

Architecture Patterns

System Architecture Diagram

Recommended Project Structure

Prisma Schema (new tables)

Pattern 1: InboxProvider Interface (mirrors CalendarProvider)

Pattern 2: imapflow IMAP Provider

Pattern 3: pdf-parse v2 Text Extraction

Pattern 4: DKV PDF Regex — Vehicle Block Extraction

Pattern 5: SheetJS Excel Generation

Pattern 6: Dynamic SMTP via Nodemailer (bypasses @nestjs-modules/mailer)

Pattern 7: Dynamic Cron Job (SchedulerRegistry)

Pattern 8: ScheduleModule Registration (AppModule update)

Settings Sidebar Extension

Anti-Patterns to Avoid

Don't Hand-Roll

Common Pitfalls

Pitfall 1: imapflow IMAP Connection Deadlock

Pitfall 2: pdf-parse v1 vs v2 API Break

Pitfall 3: @nestjs-modules/mailer Cannot Change Transport at Runtime

Pitfall 4: ScheduleModule Not Registered in AppModule

Pitfall 5: German Number Format in km/Menge Parsing

Pitfall 6: EWS Email vs Calendar Folder Access

Pitfall 7: user-files/ 10-File Pruning Race Condition

Code Examples

Inversion of Control: Credential Encryption Pattern

CSV Vehicle Import Pattern

Fahrzeug Format String Resolution

Runtime State Inventory

Environment Availability

State of the Art

Validation Architecture

Test Framework

Phase Requirements → Test Map

Sampling Rate

Wave 0 Gaps

Security Domain

Applicable ASVS Categories

Known Threat Patterns for this Stack

Assumptions Log

Open Questions

Sources

Primary (MEDIUM confidence — context7 not available, using official docs)

Secondary (LOW confidence — WebSearch with official source cross-check)

Codebase (VERIFIED: grep)

Metadata

47 KiB

Raw Blame History