docs(quick-260723-e7i): complete normalizer-gap-phase-13-schliessen-tende quick task
This commit is contained in:
+6
-4
@@ -5,8 +5,8 @@ milestone_name: Ausschreibungs-Radar
|
|||||||
current_phase: 13
|
current_phase: 13
|
||||||
current_phase_name: scraping-adapters-cross-source-dedup
|
current_phase_name: scraping-adapters-cross-source-dedup
|
||||||
status: verifying
|
status: verifying
|
||||||
stopped_at: Completed 13-05-PLAN.md
|
stopped_at: Completed quick task 260723-e7i (normalizer-gap-phase-13-schliessen)
|
||||||
last_updated: "2026-07-23T07:51:24.572Z"
|
last_updated: "2026-07-23T08:20:44.183Z"
|
||||||
last_activity: 2026-07-23
|
last_activity: 2026-07-23
|
||||||
last_activity_desc: Phase 13 execution started
|
last_activity_desc: Phase 13 execution started
|
||||||
progress:
|
progress:
|
||||||
@@ -206,6 +206,7 @@ Recent decisions affecting current work:
|
|||||||
- [Phase ?]: 13-04: cheerio (nicht node-html-parser) fuer NetServer-HTML-Parsing gewaehlt — Legitimacy-Gate flaggte node-html-parser als 'too-new' (Fehlmessung: Latest-Version-Datum statt Package-Alter); cheerio bestand das Gate sauber, Human-Approval eingeholt
|
- [Phase ?]: 13-04: cheerio (nicht node-html-parser) fuer NetServer-HTML-Parsing gewaehlt — Legitimacy-Gate flaggte node-html-parser als 'too-new' (Fehlmessung: Latest-Version-Datum statt Package-Alter); cheerio bestand das Gate sauber, Human-Approval eingeholt
|
||||||
- [Phase ?]: cosinex/DTVP-Selektoren voll befuellt statt needs-JS-deferred (D-01) — Trefferliste ist server-gerendert, live geprueft 2026-07-23
|
- [Phase ?]: cosinex/DTVP-Selektoren voll befuellt statt needs-JS-deferred (D-01) — Trefferliste ist server-gerendert, live geprueft 2026-07-23
|
||||||
- [Phase ?]: cosinex/DTVP liefert echte Notice-Deep-Links (pid=) — besser als NetServer-Fallback (Such-URL); Rule-1-Fix: arrayBuffer()+TextDecoder('iso-8859-1') statt res.text(), da Portal ISO-8859-1 mit rohen Latin-1-Bytes sendet
|
- [Phase ?]: cosinex/DTVP liefert echte Notice-Deep-Links (pid=) — besser als NetServer-Fallback (Such-URL); Rule-1-Fix: arrayBuffer()+TextDecoder('iso-8859-1') statt res.text(), da Portal ISO-8859-1 mit rohen Latin-1-Bytes sendet
|
||||||
|
- [Phase ?]: [quick-260723-e7i]: TenderNormalizerService dispatcht per sourceType (normalizeDoe/normalizeBag) mit geteiltem assemble()-Tail — schliesst den Normalizer-Gap aus 13-VERIFICATION.md fuer NetServer/Cosinex-Bag-Records
|
||||||
|
|
||||||
### Pending Todos
|
### Pending Todos
|
||||||
|
|
||||||
@@ -232,6 +233,7 @@ None yet.
|
|||||||
| 260709-lda | LDAP: ldapts empty-array-Attribut-Bug (E-Mail-Kollision auf Unique-Constraint) | 2026-07-09 | 246dc89 | (direct) |
|
| 260709-lda | LDAP: ldapts empty-array-Attribut-Bug (E-Mail-Kollision auf Unique-Constraint) | 2026-07-09 | 246dc89 | (direct) |
|
||||||
| 260709-sbx | LDAP: Suchbox fuer die entdeckten Gruppen/OUs-Liste | 2026-07-09 | aaa2922 | (direct) |
|
| 260709-sbx | LDAP: Suchbox fuer die entdeckten Gruppen/OUs-Liste | 2026-07-09 | aaa2922 | (direct) |
|
||||||
| 260714-lex | LDAP: Per-User Exclude/Denylist-Filter (Service-Accounts vom Sync ausschliessen) — live verifiziert: deaktiviert 4 Accounts, 2 echte User aktiv | 2026-07-14 | 9d1323f | (direct) |
|
| 260714-lex | LDAP: Per-User Exclude/Denylist-Filter (Service-Accounts vom Sync ausschliessen) — live verifiziert: deaktiviert 4 Accounts, 2 echte User aktiv | 2026-07-14 | 9d1323f | (direct) |
|
||||||
|
| 13 | Normalizer-Gap Phase 13 schliessen: NetServer/Cosinex-Bag-Dispatch (TenderNormalizerService) | 2026-07-23 | 9881005 | — |
|
||||||
|
|
||||||
## Deferred Items
|
## Deferred Items
|
||||||
|
|
||||||
@@ -243,7 +245,7 @@ Items acknowledged and carried forward from previous milestone close:
|
|||||||
|
|
||||||
## Session Continuity
|
## Session Continuity
|
||||||
|
|
||||||
Last session: 2026-07-23T07:51:24.556Z
|
Last session: 2026-07-23T08:20:40.093Z
|
||||||
Stopped at: Completed 13-05-PLAN.md
|
Stopped at: Completed quick task 260723-e7i (normalizer-gap-phase-13-schliessen)
|
||||||
Resume file: None
|
Resume file: None
|
||||||
Last activity: 2026-07-14 - Built LDAP per-user exclude/denylist filter (9d1323f), migration applied on live DB, verified via Playwright: sync deactivated 4 excluded service accounts (administrator/krbtgt/guest/dns-ldap), 2 real LDAP users stay active, 0 wrongly created
|
Last activity: 2026-07-14 - Built LDAP per-user exclude/denylist filter (9d1323f), migration applied on live DB, verified via Playwright: sync deactivated 4 excluded service accounts (administrator/krbtgt/guest/dns-ldap), 2 real LDAP users stay active, 0 wrongly created
|
||||||
|
|||||||
+77
@@ -0,0 +1,77 @@
|
|||||||
|
---
|
||||||
|
phase: quick-260723-e7i
|
||||||
|
plan: 01
|
||||||
|
status: complete
|
||||||
|
subsystem: tenders
|
||||||
|
tags: [normalizer, cross-source-dedup, gap-closure, phase-13]
|
||||||
|
requirements:
|
||||||
|
- INGEST-02
|
||||||
|
- INGEST-03
|
||||||
|
- SCHEMA-03
|
||||||
|
dependency-graph:
|
||||||
|
requires:
|
||||||
|
- apps/api/src/tenders/tender.types.ts
|
||||||
|
- apps/api/src/tenders/adapters/netserver.adapter.ts
|
||||||
|
- apps/api/src/tenders/adapters/cosinex.adapter.ts
|
||||||
|
provides:
|
||||||
|
- TenderNormalizerService per-sourceType dispatch (ai-netserver/cosinex-dtvp bag mapping)
|
||||||
|
affects:
|
||||||
|
- TenderFingerprint cross-source dedup (13-01) — NetServer/Cosinex records now carry real title/buyer/deadline instead of defaults
|
||||||
|
tech-stack:
|
||||||
|
added: []
|
||||||
|
patterns:
|
||||||
|
- "Shared assemble() tail extracted from normalize() so status/dedupKey/contentHash/publishedAt are computed identically across all sourceTypes"
|
||||||
|
key-files:
|
||||||
|
created: []
|
||||||
|
modified:
|
||||||
|
- apps/api/src/tenders/tender-normalizer.service.ts
|
||||||
|
- apps/api/src/tenders/tender-normalizer.service.spec.ts
|
||||||
|
decisions:
|
||||||
|
- "normalize() dispatches on raw.sourceType via switch; default case falls through to the DOE path (never throws) so future additive SourceType union members can't break existing callers"
|
||||||
|
- "legalFramework from the scraper bag is deliberately not mapped — NormalizedTenderFields has no target field for it"
|
||||||
|
- "CoreTenderFields (Omit type) defines the 10 source-specific fields each extraction path must produce; assemble() owns everything else"
|
||||||
|
metrics:
|
||||||
|
duration: "~25 min"
|
||||||
|
completed: "2026-07-23"
|
||||||
|
tasks_completed: 3
|
||||||
|
tasks_total: 3
|
||||||
|
files_modified: 2
|
||||||
|
---
|
||||||
|
|
||||||
|
# Quick Task 260723-e7i: Normalizer-Gap Phase 13 schliessen (NetServer/Cosinex-Bag-Dispatch) Summary
|
||||||
|
|
||||||
|
Fixed `TenderNormalizerService.normalize()` to dispatch per `sourceType`, so `ai-netserver` and `cosinex-dtvp` records (which carry a flat `ocdsPayload` bag instead of eForms/OCDS structure) map their real `title`/`buyerName`/`procedureType`/`deadlineAt` instead of falling through to defaults/nulls — unblocking cross-source fingerprint dedup between DÖE and the scraper adapters.
|
||||||
|
|
||||||
|
## What Was Built
|
||||||
|
|
||||||
|
**Task 1 — Per-sourceType dispatch + shared `assemble()`:**
|
||||||
|
- Extracted a private `assemble(raw, core)` method that computes `status`, `dedupKey`, `contentHash` (via the existing `computeContentHash`, not duplicated), and `publishedAt`, plus the pass-through identity fields (`sourcePortal`, `sourceNoticeId`, `ocid`, `sourceUrl`).
|
||||||
|
- Moved the existing DÖE eForms/OCDS extraction logic into `normalizeDoe(raw)` — byte-identical behavior, confirmed by the full pre-existing test suite staying green.
|
||||||
|
- Added `normalizeBag(raw)` for the flat bag shape shared by NetServer and cosinex/DTVP: reads `raw.ocdsPayload` defensively, maps `title` (fallback `'Unbenannte Ausschreibung'` when empty/missing), `buyerName`/`procedureType` (null-tolerant, no invented fallback strings), `deadlineAt` via the existing `parseOcdsDate()`. `cpvCodes`/`cpvDivisions` = `[]`, `region`/`plz`/`bundesland`/`estimatedValue` = `null`. `legalFramework` is read but intentionally not mapped (no target field).
|
||||||
|
- `normalize()` now does `switch (raw.sourceType)`: `ai-netserver`/`cosinex-dtvp` → `normalizeBag`; `doe-opendata`/`default` → `normalizeDoe` (default never throws, protecting the additive `SourceType` union).
|
||||||
|
|
||||||
|
**Task 2 — Unit tests:**
|
||||||
|
- Added `bagRecord(sourceType, bag)` helper to construct inline `RawTenderRecord`s for the bag sources (no fixture ZIPs exist for these).
|
||||||
|
- New tests: full-bag mapping for `ai-netserver` and `cosinex-dtvp` (identical shape → identical mapping), null/empty-field fallback (`buyerName`/`procedureType`/`deadlineAt` stay `null`, empty `title` falls back), and a `contentHash` sha256-format assertion proving `assemble()` reuse.
|
||||||
|
- Existing DÖE fixture-based regression tests (10 pre-existing tests) untouched and still green.
|
||||||
|
|
||||||
|
**Task 3 — Non-regression gate:**
|
||||||
|
- `npx tsc --noEmit -p tsconfig.json` — clean.
|
||||||
|
- `npx vitest run` — full API suite: **288/288 tests passed** across 25 test files (up from the pre-existing baseline + 4 new bag tests), including `tender-dedup.service.spec.ts`, `tender-ingestion.service.spec.ts`, and `tender-fingerprint.spec.ts`. No code change needed in this task.
|
||||||
|
|
||||||
|
## Deviations from Plan
|
||||||
|
|
||||||
|
None — plan executed exactly as written.
|
||||||
|
|
||||||
|
## Test Results
|
||||||
|
|
||||||
|
- `npx vitest run src/tenders/tender-normalizer.service.spec.ts` — **14/14 passed** (10 pre-existing DÖE tests + 4 new bag tests)
|
||||||
|
- `npx tsc --noEmit -p tsconfig.json` — clean, no errors
|
||||||
|
- `npx vitest run` (full API suite) — **288/288 passed**, 25 test files, 3.06s
|
||||||
|
|
||||||
|
## Self-Check: PASSED
|
||||||
|
|
||||||
|
- FOUND: apps/api/src/tenders/tender-normalizer.service.ts
|
||||||
|
- FOUND: apps/api/src/tenders/tender-normalizer.service.spec.ts
|
||||||
|
- FOUND commit 82c9a48 (Task 1)
|
||||||
|
- FOUND commit 9881005 (Task 2)
|
||||||
Reference in New Issue
Block a user