Commit Graph

15 Commits

Author SHA1 Message Date
schalli 636fe0df8f refactor(quick-260921-bi2): maschinelle Lint-Fixe und toten Code abbauen
- Aufgabe 2: vier sichere Biome-Regeln (useImportType pfadgebunden auf
  apps/web+packages, noUselessEscapeInRegex, useConst,
  useExponentiationOperator) sowie fuenf ungesicherte Regeln
  (useNodejsImportProtocol, useLiteralKeys, useOptionalChain, useTemplate,
  useParseIntRadix) angewendet und den gesamten Diff von Hand gelesen
  (ldap.service.ts zeichenweise gegen Gross-/Kleinschreibung der
  AD-Merkmale, auth.service.ts/jwt.strategy.ts gegen Durchwinken bei
  fehlender Sitzung geprueft)
- noUselessSwitchCase bleibt bewusst stehen (tender-normalizer.service.ts:60,
  die Fallmarke dokumentiert Absicht)
- Toter Code (D-03): fuenf folgenlose Auffangvariablen entfernt, eine
  nicht benutzte Funktion (forSystemQuery, Pruefskript) entfernt, ein
  positionsgebundener Dekoratorparameter umbenannt (current-user.decorator.ts),
  fuenf Symptomfunde entfernt und als Folgeaufgaben zu melden (siehe unten)
- Sechs weitere, im Plan nicht namentlich gelistete aber
  gleich-kategorische Dead-Code-Fundstellen in Testdateien zusaetzlich
  bereinigt (groups.service.spec.ts, cert-manager.test.tsx,
  ldap.service.spec.ts, prisma-tenant.extension.spec.ts x3) — noetig, um
  die vom Plan selbst verlangten Nullstaende bei noUnusedVariables/
  noUnusedImports/noUnusedFunctionParameters zu erreichen

Dekoratordaten aus apps/api unveraendert (593 Zeilen, sha256 6e1583f1...).
Endstand 620 Befunde (541 echt, 79 Test) statt der im Plan geschaetzten
621/542 — eine Differenz von 1, weil das Streichen des Namens aus
`catch (e: any)` in calendar.service.ts (Symptom-Fix) den dort ebenfalls
gemeldeten noExplicitAny-Befund miteliminiert; das ist eine erwuenschte
Nebenwirkung, keine Regression. Fehlerstufe 0, beide Testlaeufe
punktgleich gruen (69/1124, 66/459), pnpm type-check 4/4, pnpm lint
--force 5/5.

Folgeaufgaben aus D-03 (nicht in diesem Vorgang behoben):
- force-password-change.interceptor.ts: Freigabeliste prueft nur den Pfad,
  nicht die HTTP-Methode
- change-password/page.tsx: nach erzwungenem Wechsel bleibt die Person auf
  der Seite stehen (keine Weiterleitung, keine Aktualisierung der
  Benutzerablage)
- VehicleTable.tsx: Loeschschaltflaeche hat keinen Besetztzustand, laesst
  sich doppelt ausloesen
- SplitTab.tsx: downloadAllAsZip erhielt eine ungenutzte
  Uebersetzungsfunktion, Hinweis auf fest verdrahtete Texte im Zip-Pfad

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TPPB4ApQxzSU1rwV2Ffj9J
2026-09-21 08:58:32 +02:00
schalli 7dee116254 test(17-02): D-06 ownerTenantId tagging for RSS ingestion + seed idempotency
- RssAdapter.fetchTenders tags every record from a feed with a tenantId
  with the same D-13 ownerTenantId origin marking email-alert records
  carry since Phase 14; platform-wide feeds (no tenantId) stay unmarked.
  The pure parseFeed mapping is untouched — tagging happens in the
  fan-out loop that knows which row a batch came from
- Extracted the service.bund.de seed out of TendersModule.onModuleInit
  into seedServiceBundRssFeed() (tenders.seed.ts, same pattern as the
  existing seedTendersModule), so the find-then-create idempotency added
  in Task 1 is unit-tested directly instead of only via a Nest bootstrap
- New rss-feed-migration-sql.spec.ts: text-only check of the Task 1
  migration file (nullable columns, dropped/created indexes, no
  existing-row mutation, correct ordering)

- Files modified: apps/api/src/tenders/adapters/rss.adapter.ts, apps/api/src/tenders/tenders.module.ts, apps/api/src/tenders/tenders.seed.ts, apps/api/src/tenders/adapters/rss.adapter.spec.ts, apps/api/src/tenders/tenders.seed.spec.ts, apps/api/src/tenders/rss-feed-migration-sql.spec.ts
2026-08-12 11:45:27 +02:00
schalli 55ceb24ee9 test(17-01): prove multi-mailbox fan-out and the userId migration's SQL order
Der mandantenuebergreifende Sammelabruf in email-alert.adapter.ts bleibt
mechanisch unveraendert (findMany({isActive:true}) in einem Zug,
Fehlerbehandlung je Zeile) — geaendert wird nur die Warnmeldung (Zeilen-id
+ Besitzer statt Mandant, T-17-03) und die Klassendoku.

- Neue Tests: zwei aktive Postfaecher DESSELBEN Mandanten werden beide mit
  ihren jeweils eigenen Zugangsdaten abgeholt; ein kaputtes Postfach
  blockiert das andere nicht und protokolliert eine Warnung ohne
  Zugangsdaten/Adresse; die Herkunftsmarkierung folgt dem Mandantenfeld
  der jeweiligen Zeile (zwei Mandanten -> zwei Werte). Erwartungswerte von
  Hand geschrieben, nicht ueber die Produktivfunktion erzeugt.
- tender-email-config.service.spec.ts (bereits in der Task-2-Migration
  mitgeliefert) deckt zusaetzlich: tenantId wird beim Anlegen mitgeschrieben,
  zwei Nutzer desselben Mandanten erzeugen zwei Zeilen statt eine zu
  ueberschreiben.
- email-config-migration-sql.spec.ts (neu, Vorbild
  doe-url-migration-sql.spec.ts): prueft die Reihenfolge der
  Hand-Migration textuell — Zuordnung vor Loeschung, Pflicht erst nach
  Befuellung, alte Eindeutigkeit runter/neue rauf, gewoehnlicher
  tenantId-Index bleibt stehen.

src/tenders: 335/335 gruen. API gesamt: 603/603. Web gesamt: 192/192.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-12 11:24:12 +02:00
schalli f574884b32 refactor: rename the encryption key to what it actually protects
Tessera CI/CD / Lint & Type Check (push) Successful in 54s
Tessera CI/CD / Tests (push) Successful in 51s
Tessera CI/CD / Build & Publish Images (push) Successful in 25s
CALENDAR_ENCRYPTION_KEY was named after the calendar module because that
module needed encryption first, in Phase 5. Every feature since has shared the
same key -- SMTP, the DKV and tender mailboxes, and as of today the LDAP bind
password -- so the name has been describing one of five users rather than the
thing itself, and each new feature inherited the confusion.

TESSERA_ENCRYPTION_KEY is the name now. The old one is still read, because
renaming outright would stop every existing installation at the next start:
their .env carries the old name, and compose was just made to fail hard on a
missing key. When only the old name is present the API logs a deprecation
warning naming both, and when both are set the new one wins -- otherwise a
half-migrated .env would encrypt with one key and decrypt with the other.

CalendarCryptoService becomes CryptoService in its own global CryptoModule.
Four modules used to import CalendarModule purely to reach the provider, which
read as a dependency on calendars where there was none; that import is gone.

Compose keeps the hard failure: without either name the stack refuses to
start. Verified in both files for all three cases -- neither name set (abort),
only the old name (starts), only the new name (starts).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-11 14:30:46 +02:00
schalli ecf7872730 fix(tenders): link DOE notices to the notice page, not the API
The doe-opendata adapter stored the OCDS document's own `uri` as sourceUrl.
That is the API address of the record and serves OCDS JSON by design, so
anyone following the link from the results list, the detail view, or an alert
mail landed on raw JSON instead of the notice.

Build the human-readable page from the notice id the row already carries
instead. `/ui/de/search/details?noticeId=...` is the redirect target of
`/ui/de/notices/...`, so it needs no redirect. Verified in a browser for both
id shapes the feed uses -- numeric (25673764 -> "Feuerwehr-Geraetehaus Miehlen
Fliesenarbeiten") and UUID (7085ba12-... -> "Holzfassade"). The page is a
single-page app that answers 200 with an identical shell for any id, so this
had to be checked on rendered content; a status code proves nothing.

The adapter alone only fixes new ingests, so a backfill migration rewrites the
rows already stored -- in Tender and in TenderSource, since the detail view
lists per-source links separately. It touches only rows still pointing at
/api/notices/ and only ids of a shape that was actually verified, which makes
it idempotent and keeps an unexpected id from being pasted into a URL. Counted
read-only against the live database beforehand: 2846 DOE rows affected, none
skipped.

Closes the 2026-08-05 backlog item, which was deliberately held until Phase 16
was done.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-11 13:45:04 +02:00
schalli 1be6b15249 feat(14-03): add per-tenant encrypted TenderEmailConfig + ownerTenantId write-side (D-13)
Prisma: new TenderEmailConfig model (per-tenant, tenantId @unique, mirrors
DkvModuleConfig) + Tender.ownerTenantId nullable column + index (D-13:
null = global/platform-wide, unchanged for all existing rows and every
public source; set = visible only to that tenant). Migration
20260723113917_tender_email_config_owner_tenant_id applied locally.

TenderEmailConfigService: safe-select admin CRUD (GET never returns the
password, only hasPassword — T-07-12) with DkvService's encrypt-preserve-
empty semantics, via CalendarCryptoService (AES-256-GCM).

RawTenderRecord/NormalizedTenderFields gain optional ownerTenantId,
threaded through TenderNormalizerService.assemble() unchanged.
TenderDedupService's CREATE branch writes ownerTenantId (defaulting to
null); the UPDATE branch deliberately never references it, so a tender
later also seen on a public source is never retroactively hidden.

EmailAlertAdapter.fetchTenders() now does the real per-tenant fan-out:
findMany({isActive:true}) across ALL tenants (deliberate, documented
cross-tenant platform-scheduler read, never forTenant()/RLS), decrypts
each tenant's credentials, picks imap/exchange provider, and tags every
extracted candidate with ownerTenantId — catch-per-tenant so one broken
mailbox never blocks the others.

tenders.module.ts: imports CalendarModule/InboxModule, registers
EmailAlertAdapter + TenderEmailConfigService, seeds an 'email-alert'
TenderSourcePollConfig row (pollGranularity='tick', isActive=false —
no default mailbox to activate yet, D-02 framework-ready stance).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-23 13:45:11 +02:00
schalli 8983231196 feat(14-03): add EmailAlertAdapter generic extraction + 'email-alert' normalizer dispatch
GREEN phase (TDD) for Task 1: extractCandidateLinks (cheerio a[href] +
footer-noise filter + MAX_LINKS_PER_EMAIL cap, plaintext regex fallback),
titleFromEmail (subject -> first body line -> fallback), and
sourceNoticeIdFor (sha256 link hash) implement D-04's generic, no-portal-
specific-parser evaluation of alert emails.

SourceType gains 'email-alert'; TenderNormalizerService routes it through
the existing normalizeBag() path (same as ai-netserver/cosinex-dtvp/rss).
EmailAlertAdapter.fetchTenders() is a Task-1 placeholder — Task 2 wires the
real per-tenant fan-out.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-23 13:38:36 +02:00
schalli 4d6fbb136e test(14-03): add failing spec for email-alert generic link/subject extraction
RED phase (TDD) for Task 1: pure-function tests for extractCandidateLinks,
titleFromEmail, sourceNoticeIdFor — covers HTML + plaintext bodies,
footer-noise removal, link cap, and D-04 no-portal-specific-parser restraint.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-23 13:37:37 +02:00
schalli e812738c3a feat(14-02): add TenderRssFeedSource CRUD, tick poll gate, and wire RssAdapter
Global admin-managed RSS feed list (TenderRssFeedSource, D-08/D-14) with
a save-time hostname/SSRF guard (TenderRssFeedSourceService) — RSS feed
URLs are runtime admin input, so the code-level SourceRegistry denylist
gate does not cover them; a separate check rejects DENYLISTED_PORTALS
hostnames, non-http(s) schemes, and private/loopback hosts.

Adds TenderSourcePollConfig.pollGranularity ('day' | 'tick', D-15):
pollDueSources() branches per source — 'day' sources keep the existing
lastIngestedDay gate byte-unchanged, 'tick' sources (rss) fetch on every
active scheduler tick regardless of lastIngestedDay, since the day-cursor
gate was built for a genuine daily batch-export API and would otherwise
silently cap RSS to one fetch per calendar day.

Wires RssAdapter.fetchTenders() to fan out over active feed rows (native
fetch + AbortController 15s + response-size ceiling, catch-per-feed),
registers it in tenders.module.ts, and seeds the 'rss' poll config
active with pollGranularity='tick' plus a default-active service.bund.de
feed row (subreport-elvis has no single canonical URL — zero rows seeded,
admin adds relevant municipality feeds).

Migration applied locally per project convention (host -> container IP).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-23 13:24:36 +02:00
schalli 3a96cbbbe6 feat(14-02): add RssAdapter.parseFeed + 'rss' normalizer dispatch
Fixture-first RSS parsing (INGEST-04): parses live-captured
service.bund.de (pubDate present, numeric-HTML-entity titles) and
subreport-elvis (pubDate absent, CDATA titles) feed shapes into
RawTenderRecord[] via fast-xml-parser, mirroring the DoeOpenDataAdapter
config. SourceType extended with 'rss'; normalize() dispatches 'rss'
through the existing normalizeBag() path unchanged (D-04/D-05).

Rule 1 fix: fast-xml-parser only decodes the 5 predefined XML entities,
not numeric character references — added an explicit decode step so
service.bund.de titles ("&#220;bermittlung...") render correctly
instead of leaking raw entity syntax.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-23 13:17:02 +02:00
schalli 12fc5ac02d feat(13-05): add cosinex/DTVP source adapter with fixture tests
Separate HTML adapter for the cosinex Vergabemarktplatz (DTVP) satellite
(sourceType='cosinex-dtvp'), distinct from the NetServer adapter since
cosinex markup differs structurally. Live inspection (2026-07-23) found
the "Aktuelle Bekanntmachungen" results table is fully server-rendered
(not JS-dependent as D-01 anticipated), so selectors are fully populated
rather than falling back to a needs-JS stub — parses publish date,
deadline (or "nv"), title, legal framework/procedure type, buyer name,
and a real per-notice deep link (pid) into RawTenderRecord[].

Rule 1 fix: cosinex serves charset=ISO-8859-1 with raw Latin-1 bytes for
umlauts (not HTML entities); Response.text() always UTF-8-decodes per
the Fetch spec, so the adapter reads arrayBuffer() and decodes explicitly
via TextDecoder('iso-8859-1') to avoid mojibake.

16 spec tests pass against a live-captured fixture (20 rows, transcoded
to UTF-8 on disk): full-fixture parse, deadline/publish date parsing,
nested-<abbr> procedure-type extraction, umlaut decoding, empty/broken
HTML and missing-pid row fallback, fetch-throw/non-2xx fallback, no-axios
and no-input-interpolated-URL guards.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-23 09:48:21 +02:00
schalli 043ada9438 feat(13-04): NetServer-Adapter fuer tender24/lhs-vpbw/vergabe.landbw (INGEST-02)
- Package-Legitimacy-Checkpoint bestaetigt: cheerio (sauberer Gate-Pass,
  27M Downloads/Woche) statt node-html-parser installiert
- EIN config-getriebener NetServerAdapter bedient alle 3 AI-AG-Portale
  via PublicationSearchControllerServlet-Trefferliste, parst <table> mit
  cheerio zu RawTenderRecord[], sourcePortal je Zeile korrekt
- sourceNoticeId = data-oid (stabil, per Zeile eindeutig); sourceUrl =
  Such-URL als dokumentierter Best-Effort-Fallback (kein Deep-Link ohne
  JS-Ausfuehrung ermittelbar, Open Question 2)
- Fehlertoleranz: try/catch pro Zeile + pro Portal, []-Fallback bei
  Totalausfall, AbortController 15s-Timeout, hardcodierte Portal-URLs
  (SSRF-Guard T-13-04-01)
- Live-gecapturte Fixture (tender24.de, 2026-07-23) mit Umlaut-/&amp;-
  Buyer, gefuellter und leerer Deadline; 14 Spec-Tests gruen

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-23 09:13:01 +02:00
schalli 1b11ada112 feat(13-02): generalize adapter contract with portals[] array
TenderSourceAdapter gains a readonly portals: readonly string[] field
so one adapter can serve multiple portals (NetServer: 3, Plan 13-04)
and so SourceRegistry can gate registration per-portal (INGEST-07).
DoeOpenDataAdapter declares portals = ['doe-opendata'] additively,
no behavior change.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-23 08:42:23 +02:00
schalli 3764feb50b feat(10-03): implement DoeOpenDataAdapter — fetch, extract, D-02 filter
- Native fetch + AbortController 15s timeout (icon-discovery idiom, no
  axios); URL host hardcoded, only the internally-computed dayCursor is
  interpolated (T-10-06)
- HTTP 400 treated as an expected no-op (pubDay today/future) -> []
- adm-zip extraction with a pre-extraction decompression-bomb ceiling
  check (sum entry.header.size vs ~50MB) before any entry buffer is read
  (T-10-07); entries are never written to disk
- D-02 open-tender filter: positive tag.includes('tender') match only —
  award/planning/untagged-with-awards excluded (Pitfall C)
- Register DoeOpenDataAdapter in TendersModule.providers
- All doe-opendata.adapter.spec.ts tests green (6/6)
2026-07-21 10:57:29 +02:00
schalli f88e2a8141 test(10-03): add failing specs + real DÖE fixtures for adapter/normalizer
- Capture real, trimmed DÖE day-export fixtures (pubDay=2026-07-19, 8
  notices spanning tender/award/planning/untagged-with-awards tag classes)
  under tenders/__fixtures__/, live-downloaded from oeffentlichevergabe.de
- Define RawTenderRecord/NormalizedTenderFields/SourceType (tender.types.ts)
  and TenderSourceAdapter (day-cursor fetchTenders signature per RESEARCH
  Pattern 1)
- RED: doe-opendata.adapter.spec.ts asserts D-02 exact-count filtering,
  HTTP-400 no-op, and a zip-bomb ceiling guard (T-10-07)
- RED: tender-normalizer.service.spec.ts asserts eForms-primary deadline
  recovery where OCDS is null (RESEARCH Pattern 3), nullable deadline/value,
  dedupKey priority (ocid -> sourcePortal:sourceNoticeId), and a stable
  sha256 contentHash
2026-07-21 10:56:13 +02:00