Commit Graph

6 Commits

Author SHA1 Message Date
schalli 8983231196 feat(14-03): add EmailAlertAdapter generic extraction + 'email-alert' normalizer dispatch
GREEN phase (TDD) for Task 1: extractCandidateLinks (cheerio a[href] +
footer-noise filter + MAX_LINKS_PER_EMAIL cap, plaintext regex fallback),
titleFromEmail (subject -> first body line -> fallback), and
sourceNoticeIdFor (sha256 link hash) implement D-04's generic, no-portal-
specific-parser evaluation of alert emails.

SourceType gains 'email-alert'; TenderNormalizerService routes it through
the existing normalizeBag() path (same as ai-netserver/cosinex-dtvp/rss).
EmailAlertAdapter.fetchTenders() is a Task-1 placeholder — Task 2 wires the
real per-tenant fan-out.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-23 13:38:36 +02:00
schalli 3a96cbbbe6 feat(14-02): add RssAdapter.parseFeed + 'rss' normalizer dispatch
Fixture-first RSS parsing (INGEST-04): parses live-captured
service.bund.de (pubDate present, numeric-HTML-entity titles) and
subreport-elvis (pubDate absent, CDATA titles) feed shapes into
RawTenderRecord[] via fast-xml-parser, mirroring the DoeOpenDataAdapter
config. SourceType extended with 'rss'; normalize() dispatches 'rss'
through the existing normalizeBag() path unchanged (D-04/D-05).

Rule 1 fix: fast-xml-parser only decodes the 5 predefined XML entities,
not numeric character references — added an explicit decode step so
service.bund.de titles ("&#220;bermittlung...") render correctly
instead of leaking raw entity syntax.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-23 13:17:02 +02:00
schalli 988100576f test(quick-260723-e7i): cover ai-netserver/cosinex bag normalization + DOE regression
- bagRecord() helper builds inline RawTenderRecords for the flat ocdsPayload
  bag shape (no fixtures exist for NetServer/cosinex-DTVP)
- ai-netserver and cosinex-dtvp full-bag mapping, null/empty-field fallback,
  and contentHash-format assertions
- Existing DOE fixture-based assertions untouched, confirming the refactor
  didn't change DOE-path behavior

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-23 10:19:00 +02:00
schalli f9591dc491 feat(11-03): cpvDivisions column — normalizer + backfill migration
Adds Tender.cpvDivisions String[] (@@index Gin) derived at ingestion time
via cpv-catalog.ts's divisionOf() — replaces exact-match cpvCodes
comparison with a typesafe, GIN-indexable hasSome target (FILTER-03,
Pitfall 2). Backfill migration 20260721150000_tender_cpv_divisions_backfill
applied locally: 1612/1671 rows populated across all observed divisions
(741 rows carry division '45' — Bauarbeiten); idempotent (second run:
UPDATE 0, ADD COLUMN IF NOT EXISTS / CREATE INDEX IF NOT EXISTS both skip
cleanly). Applied via docker exec psql + `prisma migrate resolve
--applied` + `prisma generate` against the local dev DB only — no
Docker deploy on the test server.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-21 16:11:40 +02:00
schalli e4db602597 feat(11-02): normalizer derives bundesland + backfill migration for existing rows
Replace the Phase-10-deferred `bundesland = null` assignment with
bundeslandFromRegion(region) so new ingests are Bundesland-filterable
immediately. Add @@index([bundesland]) for filter performance. New
handwritten migration 20260721140000_tender_bundesland_backfill backfills
the ~1671 pre-existing rows (idempotent UPDATE, only where bundesland IS
NULL AND region IS NOT NULL) — applied locally via
`docker exec tessera-ctl-db-1 psql`, resolved as applied in
_prisma_migrations, and prisma generate re-run.

Verified on the local dev DB: 933/1671 rows now have bundesland set
across all 16 Länder (738 remain NULL where region itself is NULL).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-21 16:01:32 +02:00
schalli f88e2a8141 test(10-03): add failing specs + real DÖE fixtures for adapter/normalizer
- Capture real, trimmed DÖE day-export fixtures (pubDay=2026-07-19, 8
  notices spanning tender/award/planning/untagged-with-awards tag classes)
  under tenders/__fixtures__/, live-downloaded from oeffentlichevergabe.de
- Define RawTenderRecord/NormalizedTenderFields/SourceType (tender.types.ts)
  and TenderSourceAdapter (day-cursor fetchTenders signature per RESEARCH
  Pattern 1)
- RED: doe-opendata.adapter.spec.ts asserts D-02 exact-count filtering,
  HTTP-400 no-op, and a zip-bomb ceiling guard (T-10-07)
- RED: tender-normalizer.service.spec.ts asserts eForms-primary deadline
  recovery where OCDS is null (RESEARCH Pattern 3), nullable deadline/value,
  dedupKey priority (ocid -> sourcePortal:sourceNoticeId), and a stable
  sha256 contentHash
2026-07-21 10:56:13 +02:00