GREEN phase (TDD) for Task 1: extractCandidateLinks (cheerio a[href] +
footer-noise filter + MAX_LINKS_PER_EMAIL cap, plaintext regex fallback),
titleFromEmail (subject -> first body line -> fallback), and
sourceNoticeIdFor (sha256 link hash) implement D-04's generic, no-portal-
specific-parser evaluation of alert emails.
SourceType gains 'email-alert'; TenderNormalizerService routes it through
the existing normalizeBag() path (same as ai-netserver/cosinex-dtvp/rss).
EmailAlertAdapter.fetchTenders() is a Task-1 placeholder — Task 2 wires the
real per-tenant fan-out.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
RED phase (TDD) for Task 1: pure-function tests for extractCandidateLinks,
titleFromEmail, sourceNoticeIdFor — covers HTML + plaintext bodies,
footer-noise removal, link cap, and D-04 no-portal-specific-parser restraint.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds GET/POST/DELETE /modules/tender-radar/rss-feeds (Roles-guarded
ADMIN/SUPER_ADMIN), declared before the existing @Get(':id') handler to
avoid NestJS route-order shadowing. Delegates to
TenderRssFeedSourceService; the denylist/SSRF rejection (D-14) surfaces
as a 400 unchanged.
Web: tender-radar-api.ts gains listRssFeeds/createRssFeed/deleteRssFeed
(relaying the backend's specific rejection message via
extractErrorMessage), and a new RssFeedListForm client component renders
an "RSS-Feeds" section on the tender-radar settings page (D-09) — list,
add (with inline denylist error), and remove global feed URLs.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Global admin-managed RSS feed list (TenderRssFeedSource, D-08/D-14) with
a save-time hostname/SSRF guard (TenderRssFeedSourceService) — RSS feed
URLs are runtime admin input, so the code-level SourceRegistry denylist
gate does not cover them; a separate check rejects DENYLISTED_PORTALS
hostnames, non-http(s) schemes, and private/loopback hosts.
Adds TenderSourcePollConfig.pollGranularity ('day' | 'tick', D-15):
pollDueSources() branches per source — 'day' sources keep the existing
lastIngestedDay gate byte-unchanged, 'tick' sources (rss) fetch on every
active scheduler tick regardless of lastIngestedDay, since the day-cursor
gate was built for a genuine daily batch-export API and would otherwise
silently cap RSS to one fetch per calendar day.
Wires RssAdapter.fetchTenders() to fan out over active feed rows (native
fetch + AbortController 15s + response-size ceiling, catch-per-feed),
registers it in tenders.module.ts, and seeds the 'rss' poll config
active with pollGranularity='tick' plus a default-active service.bund.de
feed row (subreport-elvis has no single canonical URL — zero rows seeded,
admin adds relevant municipality feeds).
Migration applied locally per project convention (host -> container IP).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Fixture-first RSS parsing (INGEST-04): parses live-captured
service.bund.de (pubDate present, numeric-HTML-entity titles) and
subreport-elvis (pubDate absent, CDATA titles) feed shapes into
RawTenderRecord[] via fast-xml-parser, mirroring the DoeOpenDataAdapter
config. SourceType extended with 'rss'; normalize() dispatches 'rss'
through the existing normalizeBag() path unchanged (D-04/D-05).
Rule 1 fix: fast-xml-parser only decodes the 5 predefined XML entities,
not numeric character references — added an explicit decode step so
service.bund.de titles ("Übermittlung...") render correctly
instead of leaking raw entity syntax.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Add InboxMessage type (subject + html/text body) and fetchMessages() to
the InboxProvider interface, ImapProvider, and ExchangeInboxProvider
- IMAP: findBodyParts() walks the MIME tree for first text/html + text/plain
parts, reusing the connect/lock/search/fetchAll skeleton; marks \Seen
- EWS: new getItemBodySoap() requests item:Body, extracts BodyType via the
existing extractAttr/extractAll helpers, marks IsRead via markReadSoap
- Net-new spec coverage (imap.provider.spec.ts, exchange-inbox.provider.spec.ts)
mocking ImapFlow and httpntlm.post (via require.cache stub, since httpntlm
is loaded with a raw require() that vi.mock cannot intercept)
- fetchPdfAttachments untouched in both providers (D-02); full API suite
(301 tests) + tsc --noEmit stay green
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Move ImapProvider, ExchangeInboxProvider, InboxProvider into apps/api/src/inbox/
- Move InboxConfig/InboxAttachment/InboxEmail into new inbox.types.ts
- dkv.types.ts re-exports the moved types so existing DKV imports keep compiling
- DKV switches import paths to ../inbox/... and imports InboxModule
- Pure move + import-path swap: fetchPdfAttachments and all DKV logic unchanged (D-01/D-02)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Normalizer-Gap (quick 260723-e7i) closed criteria 1/2; re-verification
lifts Phase 13 from 1/4 to 3/4. Truth 3 (cross-source fingerprint dedup)
stays PARTIAL: newly-found cpvDivisions asymmetry (scrapers always [],
~79% of DÖE tenders populated) makes exact-match fingerprint miss most
real overlaps. User decision: accept as dormant deferral — dedup is inert
(both scrapers isActive=false), decide with live data once a 2nd source
is activated.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- bagRecord() helper builds inline RawTenderRecords for the flat ocdsPayload
bag shape (no fixtures exist for NetServer/cosinex-DTVP)
- ai-netserver and cosinex-dtvp full-bag mapping, null/empty-field fallback,
and contentHash-format assertions
- Existing DOE fixture-based assertions untouched, confirming the refactor
didn't change DOE-path behavior
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Extract shared assemble() tail (status/dedupKey/contentHash/publishedAt)
so it is computed identically across all sources, not duplicated
- Move existing DOE eForms/OCDS extraction into normalizeDoe() (byte-identical
behavior, regression guard)
- Add normalizeBag() for the flat ocdsPayload bag shared by the NetServer and
cosinex/DTVP scraper adapters ({title, buyerName, procedureType,
legalFramework, deadlineAt}); legalFramework deliberately not mapped
- normalize() dispatches on raw.sourceType, defaulting to the DOE path so the
additive SourceType union never throws
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds CosinexAdapter to TendersModule's providers and registers it with
SourceRegistry at DI boot, alongside DoeOpenDataAdapter/NetServerAdapter
(cosinex-dtvp is not AGB-denylisted, so registration succeeds). Seeds a
cosinex-dtvp TenderSourcePollConfig row with isActive: false, matching
the ai-netserver "framework ready, activation deferred" stance (D-02).
tsc --noEmit clean; src/tenders slice: 21 files, 221/221 tests pass
(205 pre-existing + 16 new cosinex.adapter.spec.ts).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Separate HTML adapter for the cosinex Vergabemarktplatz (DTVP) satellite
(sourceType='cosinex-dtvp'), distinct from the NetServer adapter since
cosinex markup differs structurally. Live inspection (2026-07-23) found
the "Aktuelle Bekanntmachungen" results table is fully server-rendered
(not JS-dependent as D-01 anticipated), so selectors are fully populated
rather than falling back to a needs-JS stub — parses publish date,
deadline (or "nv"), title, legal framework/procedure type, buyer name,
and a real per-notice deep link (pid) into RawTenderRecord[].
Rule 1 fix: cosinex serves charset=ISO-8859-1 with raw Latin-1 bytes for
umlauts (not HTML entities); Response.text() always UTF-8-decodes per
the Fetch spec, so the adapter reads arrayBuffer() and decodes explicitly
via TextDecoder('iso-8859-1') to avoid mojibake.
16 spec tests pass against a live-captured fixture (20 rows, transcoded
to UTF-8 on disk): full-fixture parse, deadline/publish date parsing,
nested-<abbr> procedure-type extraction, umlaut decoding, empty/broken
HTML and missing-pid row fallback, fetch-throw/non-2xx fallback, no-axios
and no-input-interpolated-URL guards.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Tender.sources[] added to the API client type (sourcePortal, sourceUrl,
sourceNoticeId). TenderDetail now renders one link per TenderSource
with a German portal label (DÖE/tender24/DTVP/...), falling back to
the existing single sourceUrl block when sources is missing or empty
(older responses, single-source tenders).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
GET /modules/tender-radar/:id now includes the TenderSource relation
(sourcePortal, sourceUrl, sourceNoticeId) so a cross-source-deduped
tender's detail response carries links to all its source portals, not
just the single primary sourceUrl column. Route order unchanged (:id
stays after all static routes).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Register SourceRegistry and TenderDedupService as providers.
onModuleInit registers DoeOpenDataAdapter with the registry before the
scheduler's first tick — the DI-boot-time enforcement point for the
INGEST-07 denylist gate (D-06). This is Wave 2's sole writer of
tenders.module.ts; 13-04 (NetServer) and 13-05 (cosinex) add their
own registry.register(...) calls additively in later waves.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replace the DÖE-only findUnique with findMany({isActive:true}) fan-out
(poll-once-fan-out-many, D-01). Each active TenderSourcePollConfig is
resolved through SourceRegistry.get(sourceType) and processed inside
its own try/catch (catch-per-source, D-01) — one broken/blocking source
no longer aborts the tick for the others. dedupActive =
activePortalCount >= 2 (D-05) is computed once per tick and passed to
TenderDedupService.resolve(), which now replaces the direct
tender.upsert call. Delta-only matchDelta boundary (D-07) preserved:
only genuinely-created tender IDs across all sources are collected.
Extended tender-ingestion.service.spec.ts: multi-config fan-out,
catch-per-source isolation, dedupActive gate assertion, adapter-missing
skip, plus the existing SCHEMA-02/D-07/retention/day-cursor suites
updated to the new registry+dedup constructor shape (all green).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
resolve(n, {dedupActive}) matches OCID -> source:noticeId -> fingerprint
(fingerprint tier hard-gated by dedupActive, D-05). On any match the
existing Tender gets an additional TenderSource attached (D-03 merge)
instead of a new Tender row; SCHEMA-02 change-detection is preserved
inline (matched Tender's mutable fields refresh when contentHash
differs, exactly as the old direct tender.upsert UPDATE branch did).
No match -> tender.create (with computed fingerprint) + tenderSource.create.
Plain PrismaService, no forTenant()/RLS (T-10-09).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
GREEN — SourceRegistry.register() throws DeniedPortalError when any
of an adapter's declared portals is in DENYLISTED_PORTALS
(vergabe24, aumass), enforced at DI-registration time (INGEST-07/
D-06), not just documented. get()/activeAdapters() support the
Plan 13-03 poll-once-fan-out-many scheduler. 6/6 tests pass, no
Prisma/scraping import.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
RED — proves Erfolgskriterium 4 (INGEST-07): registering an adapter
whose portals include vergabe24 or aumass must throw DeniedPortalError,
including a mixed portals array with one denylisted entry. Also covers
legitimate register/get/activeAdapters happy paths. Fake adapter stub,
no real scraping.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
TenderSourceAdapter gains a readonly portals: readonly string[] field
so one adapter can serve multiple portals (NetServer: 3, Plan 13-04)
and so SourceRegistry can gate registration per-portal (INGEST-07).
DoeOpenDataAdapter declares portals = ['doe-opendata'] additively,
no behavior change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
SourceType now covers 'doe-opendata' | 'ai-netserver' | 'cosinex-dtvp'
(13-RESEARCH Pattern 1) so the Plan 13-04/05 adapters can register
without further type-contract changes. NormalizedTenderFields gains an
optional fingerprint field for the SCHEMA-03 dedup resolver (Plan
13-03) to populate later. tsc --noEmit clean; full API suite green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Additive schema change (SCHEMA-03/D-03/D-04): new model TenderSource
(1:n Tender, @@unique[sourcePortal, sourceNoticeId], onDelete Cascade)
and a nullable Tender.fingerprint column + index. dedupKey stays
unchanged as the SCHEMA-02 upsert target.
Migration 20260723120000_add_tender_source applies in strict order
(Pitfall 5): table+column create, then one TenderSource row per
pre-existing Tender via SQL INSERT/SELECT, then the unique constraint.
Applied locally against the tessera dev DB (container IP, no host
port) — verified via psql: TenderSource count == Tender count == 2851.
backfill-tender-source.ts is a one-time script that computes
Tender.fingerprint via the Task-1 tenderFingerprint() function
(Decimal->number conversion for estimatedValue, T-13-01-03) — run via
the compiled dist/ output (source uses standard extensionless TS
imports for tsc compatibility). Confirmed: 2851/2851 rows backfilled,
idempotent re-run verified.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
GREEN: title+buyer dominant, CPV division (order-independent, dedup'd),
value bucketed by order-of-magnitude, deadline truncated to day-grain.
sha256 hex, deterministic, no I/O — foundation for the Task-2 backfill
and the Plan 13-03 dedup resolver's fingerprint tier.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
6 plans (INGEST-02/03/07, SCHEMA-03) + Nyquist validation.
Core (registry, fingerprint, dedup, TenderSource, backfill) lands
first and is green independent of live scraping (D-01).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
LDAP-imported users have no local passwordHash, and validateUser only checked
the local password, so they could never log in. Now a passwordless user with
an ldapDn is authenticated by binding as their OWN DN with the entered
password against the tenant's active LDAP config (reusing the ldaps TLS-skip
option). Empty passwords are rejected before binding to avoid AD's
unauthenticated-bind bypass. Local-password users are unchanged.
LdapService.verifyUserCredentials added; LdapModule now exports
LdapConfigService; AuthModule imports LdapModule (no circular dep). 8 new
specs (bind success/fail, empty-password guard, login via bind, wrong pw, no
config, no ldapDn, inactive). API 226 green, tsc clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
schalli