docs(13-04): complete NetServer adapter plan
This commit is contained in:
@@ -11,7 +11,7 @@
|
||||
### INGEST — Datenquellen
|
||||
|
||||
- [x] **INGEST-01**: Das System ruft Ausschreibungen zeitgesteuert über die DÖE OpenData-API (oeffentlichevergabe.de, eForms/OCDS, auth-frei) ab.
|
||||
- [ ] **INGEST-02**: Das System importiert Ausschreibungen von Administration-Intelligence-NetServer-Portalen (lhs-vpbw, tender24, vergabe.landbw) über einen konfigurierbaren Adapter.
|
||||
- [x] **INGEST-02**: Das System importiert Ausschreibungen von Administration-Intelligence-NetServer-Portalen (lhs-vpbw, tender24, vergabe.landbw) über einen konfigurierbaren Adapter.
|
||||
- [ ] **INGEST-03**: Das System importiert Ausschreibungen vom cosinex-Vergabemarktplatz (DTVP) über einen Adapter.
|
||||
- [ ] **INGEST-04**: Das System importiert Ausschreibungen aus RSS-Feeds (subreport-elvis, service.bund.de).
|
||||
- [ ] **INGEST-05**: Das System liest Portal-Benachrichtigungs-E-Mails aus einem konfigurierten Postfach ein (nutzt bestehende DKV-Inbox-Infrastruktur) und extrahiert daraus Ausschreibungen.
|
||||
@@ -96,7 +96,7 @@
|
||||
| NOTIFY-02 | Phase 12 | Complete |
|
||||
| NOTIFY-03 | Phase 12 | Complete |
|
||||
| NOTIFY-04 | Phase 12 | Complete |
|
||||
| INGEST-02 | Phase 13 | Pending |
|
||||
| INGEST-02 | Phase 13 | Complete |
|
||||
| INGEST-03 | Phase 13 | Pending |
|
||||
| INGEST-07 | Phase 13 | Complete |
|
||||
| SCHEMA-03 | Phase 13 | Complete |
|
||||
|
||||
@@ -439,14 +439,14 @@ Plans:
|
||||
3. A tender that appears via both DÖE and a scraping adapter shows up once in the results list (fuzzy fingerprint dedup on buyer+title+CPV+deadline+value), with links to all of its source portals -- dedup logic only activates once a second source is live
|
||||
4. Attempting to register vergabe24 or aumass as a poll source is refused by the system itself (adapter registry denylist enforced in code), not just documented as forbidden
|
||||
|
||||
**Plans**: 4/6 plans executed
|
||||
**Plans**: 5/6 plans executed
|
||||
|
||||
Plans:
|
||||
|
||||
- [x] 13-01-PLAN.md — TenderSource-Schema + Backfill + NULL-tolerante Fingerprint-Fn + SourceType-Union (SCHEMA-03 Datenschicht)
|
||||
- [x] 13-02-PLAN.md — SourceRegistry + harte Denylist-Gate (vergabe24/aumass) + Adapter-Interface-Generalisierung (INGEST-07)
|
||||
- [x] 13-03-PLAN.md — 3-Stufen-Dedup-Resolver + pollDueSources Fan-out (catch-per-source) + Modul-Wiring (SCHEMA-03)
|
||||
- [ ] 13-04-PLAN.md — NetServer-Adapter (config-getrieben, 3 Portale) + Package-Legitimacy-Checkpoint (INGEST-02)
|
||||
- [x] 13-04-PLAN.md — NetServer-Adapter (config-getrieben, 3 Portale) + Package-Legitimacy-Checkpoint (INGEST-02)
|
||||
- [ ] 13-05-PLAN.md — cosinex/DTVP-Adapter (separat, best-effort) (INGEST-03)
|
||||
- [x] 13-06-PLAN.md — Read-Endpoint include sources[] + TenderDetail Multi-Source-Links (SCHEMA-03 Anzeige)
|
||||
|
||||
@@ -486,5 +486,5 @@ Phases execute in numeric order: 1 -> 2 -> 3 -> 4 -> 5 -> 6 -> 7 -> 8 -> 9 -> 10
|
||||
| 10. Ausschreibungs-Radar Foundation & DÖE Ingestion | 6/6 | Complete | 2026-07-21 |
|
||||
| 11. Filter Engine, Results UI & Saved Searches | 6/6 | In Progress| |
|
||||
| 12. Tender Notifications | 4/4 | In Progress| |
|
||||
| 13. Scraping Adapters & Cross-Source Deduplication | 4/6 | In Progress| |
|
||||
| 13. Scraping Adapters & Cross-Source Deduplication | 5/6 | In Progress| |
|
||||
| 14. RSS, Email-Alert Ingestion & Module Rollout | 0/TBD | Not started | - |
|
||||
|
||||
+9
-7
@@ -5,15 +5,15 @@ milestone_name: Ausschreibungs-Radar
|
||||
current_phase: 13
|
||||
current_phase_name: scraping-adapters-cross-source-dedup
|
||||
status: executing
|
||||
stopped_at: Completed 13-06-PLAN.md
|
||||
last_updated: "2026-07-23T06:57:20.973Z"
|
||||
stopped_at: Completed 13-04-PLAN.md
|
||||
last_updated: "2026-07-23T07:15:59.330Z"
|
||||
last_activity: 2026-07-23
|
||||
last_activity_desc: Phase 13 execution started
|
||||
progress:
|
||||
total_phases: 13
|
||||
completed_phases: 11
|
||||
total_plans: 62
|
||||
completed_plans: 59
|
||||
completed_plans: 60
|
||||
---
|
||||
|
||||
# Project State
|
||||
@@ -28,11 +28,11 @@ See: .planning/PROJECT.md (updated 2026-07-17)
|
||||
## Current Position
|
||||
|
||||
Phase: 13 (scraping-adapters-cross-source-dedup) — EXECUTING
|
||||
Plan: 5 of 6
|
||||
Plan: 6 of 6
|
||||
Status: Ready to execute
|
||||
Last activity: 2026-07-23 — Phase 13 execution started
|
||||
|
||||
Progress: [██████████] 95%
|
||||
Progress: [██████████] 97%
|
||||
|
||||
## Performance Metrics
|
||||
|
||||
@@ -94,6 +94,7 @@ Progress: [██████████] 95%
|
||||
| Phase 13 P02 | 20min | 2 tasks | 4 files |
|
||||
| Phase 13 P03 | 45min | 3 tasks | 5 files |
|
||||
| Phase 13 P06 | 15min | 2 tasks | 5 files |
|
||||
| Phase 13 P04 | 55min | 3 tasks | 6 files |
|
||||
|
||||
## Accumulated Context
|
||||
|
||||
@@ -201,6 +202,7 @@ Recent decisions affecting current work:
|
||||
- [Phase ?]: TenderDedupService tier-2 (source:noticeId) match is unconditional (not gated by dedupActive) — idempotent same-source re-poll must work even with a single active source
|
||||
- [Phase ?]: Fingerprint always computed on tender.create regardless of dedupActive, so DÖE-only tenders become fingerprint-matchable the instant a 2nd source activates without further backfill
|
||||
- [Phase ?]: 13-06: portalLabel()-Map bleibt lokal in TenderDetail.tsx (nicht geteilt) — einziger Consumer bisher; Fallback auf sourceUrl-Block bei fehlendem/leerem sources[].
|
||||
- [Phase ?]: 13-04: cheerio (nicht node-html-parser) fuer NetServer-HTML-Parsing gewaehlt — Legitimacy-Gate flaggte node-html-parser als 'too-new' (Fehlmessung: Latest-Version-Datum statt Package-Alter); cheerio bestand das Gate sauber, Human-Approval eingeholt
|
||||
|
||||
### Pending Todos
|
||||
|
||||
@@ -238,7 +240,7 @@ Items acknowledged and carried forward from previous milestone close:
|
||||
|
||||
## Session Continuity
|
||||
|
||||
Last session: 2026-07-23T06:57:20.960Z
|
||||
Stopped at: Completed 13-06-PLAN.md
|
||||
Last session: 2026-07-23T07:15:59.317Z
|
||||
Stopped at: Completed 13-04-PLAN.md
|
||||
Resume file: None
|
||||
Last activity: 2026-07-14 - Built LDAP per-user exclude/denylist filter (9d1323f), migration applied on live DB, verified via Playwright: sync deactivated 4 excluded service accounts (administrator/krbtgt/guest/dns-ldap), 2 real LDAP users stay active, 0 wrongly created
|
||||
|
||||
@@ -0,0 +1,178 @@
|
||||
---
|
||||
phase: 13-scraping-adapters-cross-source-dedup
|
||||
plan: 04
|
||||
subsystem: backend
|
||||
tags: [cheerio, html-scraping, adapter-pattern, netserver, package-legitimacy-gate]
|
||||
|
||||
requires:
|
||||
- phase: 13-scraping-adapters-cross-source-dedup
|
||||
plan: 01
|
||||
provides: SourceType open union (includes 'ai-netserver'), RawTenderRecord contract
|
||||
- phase: 13-scraping-adapters-cross-source-dedup
|
||||
plan: 02
|
||||
provides: TenderSourceAdapter.portals[], SourceRegistry with DeniedPortalError denylist gate
|
||||
- phase: 13-scraping-adapters-cross-source-dedup
|
||||
plan: 03
|
||||
provides: pollDueSources fan-out + TenderDedupService, TendersModule onModuleInit registration hook
|
||||
provides:
|
||||
- "NetServerAdapter — ONE config-driven adapter serving all 3 AI-AG Vergabe@Net portals (tender24, lhs-vpbw, vergabe.landbw)"
|
||||
- "cheerio (^1.2.0) as the project's HTML-parsing dependency (package-legitimacy-gated, human-approved)"
|
||||
- "Live-captured netserver-search.html fixture for stable, network-free adapter tests"
|
||||
- "NetServerAdapter registered as a TendersModule provider + at DI boot via SourceRegistry.register(); ai-netserver TenderSourcePollConfig seeded (isActive: false)"
|
||||
affects: [13-05-cosinex-adapter, phase-14-admin-source-activation-ui]
|
||||
|
||||
tech-stack:
|
||||
added:
|
||||
- "cheerio ^1.2.0 (HTML parsing, package-legitimacy-gated: human approved over node-html-parser after gate flagged the latter's most-recent-version-publish-date as a 'too-new' false positive)"
|
||||
patterns:
|
||||
- "Config-driven multi-portal adapter: one class + a hardcoded portal-to-baseUrl Record, iterated per fetchTenders() call, each portal wrapped in its own try/catch so one blocked/broken portal never prevents the other two from being polled in the same tick"
|
||||
- "Per-row try/catch inside cheerio .each() — a single malformed table row is skipped (logger.warn) rather than aborting the whole portal's parse; cheerio.load() itself is also wrapped, so a totally unparsable page returns [] instead of throwing"
|
||||
- "RawTenderRecord.ocdsPayload used as a generic field bag (title/buyerName/procedureType/legalFramework/deadlineAt) for a non-OCDS source — deliberately NOT reshaping the raw eForms/OCDS-shaped RawTenderRecord interface; the shared TenderNormalizerService does not yet understand this shape (out of scope for this plan, see Known Stubs)"
|
||||
|
||||
key-files:
|
||||
created:
|
||||
- apps/api/src/tenders/adapters/netserver.adapter.ts
|
||||
- apps/api/src/tenders/adapters/netserver.adapter.spec.ts
|
||||
- apps/api/src/tenders/__fixtures__/netserver-search.html
|
||||
modified:
|
||||
- apps/api/package.json
|
||||
- pnpm-lock.yaml
|
||||
- apps/api/src/tenders/tenders.module.ts
|
||||
|
||||
key-decisions:
|
||||
- "cheerio chosen over node-html-parser at the blocking-human package-legitimacy checkpoint — the automated gate flagged node-html-parser 'SUS'/'too-new' purely because it measures the latest-*version*-publish date (2026-07-06), not true package age (first published 2017-06-14, 129 versions, v9.0.0); cheerio passed the gate cleanly (OK, 27M weekly downloads, published since 2011) so the human approver picked the unambiguous option"
|
||||
- "sourceUrl falls back to the portal's search-results URL, NOT a per-notice deep link — live inspection of the captured HTML found NO <a href> per result row; navigation to a notice's detail page is client-side-JS-only (data-oid + an external click handler script), with no statically-derivable detail URL anywhere in the search response. This resolves 13-RESEARCH.md Open Question 2 with a documented limitation rather than a guessed/unverified URL pattern (avoiding a fabricated, unverifiable servlet path)."
|
||||
- "sourceNoticeId = the row's data-oid attribute (e.g. '54321-NetTender-19f800ce3b1-...') — stable and unique per notice, confirmed unique across all rows in the live fixture"
|
||||
- "dayCursor parameter is accepted (interface conformance) but not used as a query filter — the NetServer search servlet has no documented incremental/date-range parameter (unlike DÖE's day-batch export); first-page-only fetch is accepted as sufficient for the MVP proof (13-RESEARCH.md Open Question 2 recommendation), pagination is out of scope for this plan"
|
||||
- "NetServer's '24:00' deadline convention (meaning midnight at the END of the stated day) is normalized to 00:00 of the next day, rather than left as an invalid Date"
|
||||
- "TenderSourcePollConfig row for 'ai-netserver' is seeded with isActive: false — this plan builds, registers, and tests the adapter; it does not force-activate live polling of the 3 real portals (D-02: activation is a later admin/seed decision, Phase 14 UI)"
|
||||
|
||||
requirements-completed: [INGEST-02]
|
||||
|
||||
coverage:
|
||||
- id: D1
|
||||
description: "ONE config-driven NetServerAdapter declares portals=['tender24','lhs-vpbw','vergabe.landbw'] and a hardcoded per-portal baseUrl map; fetchTenders() fetches and parses all 3 portals independently, sourcePortal set correctly per parsed row"
|
||||
requirement: INGEST-02
|
||||
verification:
|
||||
- kind: unit
|
||||
ref: "apps/api/src/tenders/adapters/netserver.adapter.spec.ts — 'declares sourceType ai-netserver and the 3 AI-AG NetServer portals', 'fetches all 3 portals and aggregates their parsed records' (14/14 tests pass)"
|
||||
status: pass
|
||||
human_judgment: false
|
||||
- id: D2
|
||||
description: "Adapter parses the live-captured NetServer <table> results into RawTenderRecord[] with correct sourceNoticeId (data-oid), umlaut/&-entity-decoded buyer names, and deadline mapped to a Date or null (never throwing on empty deadlines)"
|
||||
requirement: INGEST-02
|
||||
verification:
|
||||
- kind: unit
|
||||
ref: "apps/api/src/tenders/adapters/netserver.adapter.spec.ts — 'decodes an umlaut + &-entity buyer name correctly', 'uses the row data-oid attribute as sourceNoticeId', 'maps a populated deadline row to a Date and an empty-deadline row to null'"
|
||||
status: pass
|
||||
human_judgment: false
|
||||
- id: D3
|
||||
description: "Parse-Totalausfall (empty HTML, no matching rows, malformed row, portal fetch throwing, portal responding non-2xx) returns [] instead of throwing, per portal and in aggregate (D-01 error tolerance)"
|
||||
requirement: INGEST-02
|
||||
verification:
|
||||
- kind: unit
|
||||
ref: "apps/api/src/tenders/adapters/netserver.adapter.spec.ts — 'returns [] for empty HTML', 'returns [] for HTML with no matching result rows', 'skips a row missing data-oid without throwing', 'returns [] for a portal whose fetch throws, without aborting the other portals', 'returns [] for every portal when fetch always throws', 'returns [] without throwing when a portal responds non-2xx'"
|
||||
status: pass
|
||||
human_judgment: false
|
||||
- id: D4
|
||||
description: "NetServerAdapter is a TendersModule provider and is registered with SourceRegistry at DI boot alongside DoeOpenDataAdapter; the INGEST-07 denylist gate structurally applies (tender24/lhs-vpbw/vergabe.landbw are not denylisted, so registration succeeds); project-wide typecheck and the full tenders test slice remain green"
|
||||
requirement: INGEST-02
|
||||
verification:
|
||||
- kind: unit
|
||||
ref: "cd apps/api && npx tsc --noEmit -p tsconfig.json (clean)"
|
||||
status: pass
|
||||
- kind: unit
|
||||
ref: "cd apps/api && npx vitest run src/tenders (20 files, 205/205 pass, includes source-registry.spec.ts 6/6 unchanged and netserver.adapter.spec.ts 14/14 new)"
|
||||
status: pass
|
||||
human_judgment: false
|
||||
- id: D5
|
||||
description: "Package-legitimacy checkpoint (blocking-human) ran BEFORE any install; cheerio was human-approved and installed only after explicit confirmation, never auto-approved"
|
||||
requirement: INGEST-02
|
||||
verification:
|
||||
- kind: manual_gate
|
||||
ref: "gsd-tools query package-legitimacy check --ecosystem npm node-html-parser cheerio (run pre-install) + npm view node-html-parser time.created / npm view cheerio time.created (cross-checked); human approved 'cheerio' via explicit resume instruction"
|
||||
status: pass
|
||||
human_judgment: true
|
||||
|
||||
duration: 55min
|
||||
completed: 2026-07-23
|
||||
status: complete
|
||||
---
|
||||
|
||||
# Phase 13 Plan 04: NetServer-Adapter (tender24/lhs-vpbw/vergabe.landbw) Summary
|
||||
|
||||
**ONE config-driven `NetServerAdapter` (cheerio-based, human-approved at a blocking package-legitimacy checkpoint) now serves all three AI-AG "Vergabe@Net" portals from a single class, parsing the live public `PublicationSearchControllerServlet` results table into `RawTenderRecord[]` with full row/portal error isolation — the second real ingestion source that makes the Phase-13 dedup core (13-03) live-observable (`activePortalCount >= 2` once activated).**
|
||||
|
||||
## Performance
|
||||
|
||||
- **Duration:** ~55 min (including the blocking package-legitimacy checkpoint pause)
|
||||
- **Tasks:** 3/3 completed
|
||||
- **Files modified:** 6 (3 created, 3 modified)
|
||||
|
||||
## Accomplishments
|
||||
- **Blocking package-legitimacy checkpoint honored correctly**: paused BEFORE any install, ran `gsd-tools query package-legitimacy check` against both `node-html-parser` and `cheerio`, cross-verified the flagged `node-html-parser` "too-new" verdict against `npm view ... time.created` (found it to be a false positive — the gate measures latest-version-publish date, not package age; the package is actually 9+ years old). Reported both findings and waited for explicit human approval. Human selected `cheerio` — installed only after that confirmation.
|
||||
- `NetServerAdapter`: ONE class, `portals = ['tender24', 'lhs-vpbw', 'vergabe.landbw']`, hardcoded per-portal `baseUrl` constants (SSRF guard, T-13-04-01), fetches each portal's `PublicationSearchControllerServlet?function=SearchPublications` via native `fetch` + `AbortController` 15s timeout + `redirect: 'follow'`, parses the `<table>` results with cheerio.
|
||||
- Fault tolerance exactly per D-01/Pitfall 2: try/catch per row (skip + `logger.warn`), try/catch per portal (one blocked/broken portal never prevents the other two from being polled), `cheerio.load()` itself wrapped so a totally unparsable page returns `[]` for that portal instead of throwing.
|
||||
- Live-captured fixture (`__fixtures__/netserver-search.html`, 6 real rows from a live `tender24.de` GET, 2026-07-23) exercises umlaut + `&`-entity buyer decoding, a populated deadline, and multiple empty-deadline rows.
|
||||
- 14 spec tests, all green: sourceType/portals declaration, full-fixture parse with correct `sourcePortal`/`sourceNoticeId`/`sourceUrl`, umlaut/entity decoding, `data-oid`-uniqueness, deadline-to-Date-or-null mapping (including NetServer's `24:00` = next-day-midnight convention), empty/broken-HTML `[]`-fallback, missing-`data-oid`-row skip, per-portal fetch-fail isolation, total-fetch-failure `[]`-fallback, non-2xx `[]`-fallback, no-axios guard, no-input-interpolated-URL guard.
|
||||
- `NetServerAdapter` registered as a `TendersModule` provider and at DI boot via `SourceRegistry.register()` (denylist gate structurally applies — none of the 3 portals are denylisted); `ai-netserver` `TenderSourcePollConfig` seeded with `isActive: false` (activation deferred to admin/Phase 14, per D-02).
|
||||
- `npx tsc --noEmit` clean; `src/tenders` slice: 20 files, 205/205 tests pass (191 pre-existing + 14 new).
|
||||
|
||||
## Task Commits
|
||||
|
||||
Each task was committed atomically:
|
||||
|
||||
1. **Task 1: Package-legitimacy checkpoint (blocking-human)** — no commit (gate/decision only, per protocol); cheerio approved by the coordinator/human
|
||||
2. **Task 2: cheerio install + NetServerAdapter + fixture + spec** - `043ada9` (feat)
|
||||
3. **Task 3: NetServerAdapter registered in TendersModule** - `88572f5` (feat)
|
||||
|
||||
## Files Created/Modified
|
||||
- `apps/api/src/tenders/adapters/netserver.adapter.ts` - config-driven multi-portal `NetServerAdapter` (`fetchTenders`, `parseSearchResults`, `parseNetServerDate`/`parseNetServerDeadline` helpers)
|
||||
- `apps/api/src/tenders/adapters/netserver.adapter.spec.ts` - 14 unit tests against the fixture + mocked fetch
|
||||
- `apps/api/src/tenders/__fixtures__/netserver-search.html` - live-captured NetServer results table (6 rows)
|
||||
- `apps/api/package.json` / `pnpm-lock.yaml` - `cheerio ^1.2.0` dependency added
|
||||
- `apps/api/src/tenders/tenders.module.ts` - `NetServerAdapter` provider + `onModuleInit` registration + `ai-netserver` poll-config seed (`isActive: false`)
|
||||
|
||||
## Decisions Made
|
||||
- **cheerio over node-html-parser** — see `key-decisions` in frontmatter. The automated legitimacy gate's "too-new" flag on `node-html-parser` was investigated and found to be a measurement artifact (latest-version-publish date, not package founding date); both packages are legitimate, but `cheerio` passed the gate cleanly with zero flags, so it was the unambiguous choice for the human approver.
|
||||
- **`sourceUrl` = search-results URL, not a per-notice deep link** — live HTML inspection found no `<a href>` per row (JS-only navigation via `data-oid`); rather than guess/invent an unverified detail-servlet URL pattern, this is documented as a best-effort limitation (Open Question 2).
|
||||
- **`ocdsPayload` used as a generic NetServer field bag** rather than reshaping `RawTenderRecord`'s DÖE-shaped interface — keeps the shared adapter contract stable; full normalizer support for this shape is explicitly out of this plan's scope (see Known Stubs below).
|
||||
- **`isActive: false` on the seeded poll config** — this plan proves the adapter works against a live-captured fixture; it deliberately does not flip the platform-global scheduler into actually polling 3 real portals every tick without an explicit admin/Phase-14 decision.
|
||||
|
||||
## Deviations from Plan
|
||||
|
||||
### Auto-fixed Issues
|
||||
|
||||
None — Rule 1-3 auto-fixes were not needed; implementation followed the plan and 13-RESEARCH.md's Pattern 1 design directly.
|
||||
|
||||
### Checkpoint Handling (not a deviation, documented per protocol)
|
||||
|
||||
**Package selection changed from the plan's primary suggestion.** 13-RESEARCH.md's `[ASSUMED]` recommendation was `node-html-parser` (with `cheerio` as the stated alternative). The blocking-human checkpoint's own automated gate flagged `node-html-parser` as `SUS`/"too-new" (a false positive on version-publish-date vs. package age, verified via `npm view ... time.created`), while `cheerio` passed cleanly. The coordinator/human explicitly approved `cheerio` after reviewing both findings. This is exactly the checkpoint's intended function — no auto-approval occurred, and both plan-sanctioned options were legitimate; the final selection was a human decision, not an executor deviation.
|
||||
|
||||
**Total:** 0 auto-fixed deviations. 1 human-directed package selection (checkpoint outcome, not a deviation).
|
||||
|
||||
## Known Stubs
|
||||
|
||||
- **`ocdsPayload` generic bag is not yet consumed by `TenderNormalizerService`.** The normalizer (`tender-normalizer.service.ts`) is currently hardcoded to the DÖE eForms/OCDS shape (`getOcdsTender`, `getEformsRoot`, etc.) and does not know how to map `NetServerAdapter`'s `{title, buyerName, procedureType, legalFramework, deadlineAt}` bag into `NormalizedTenderFields`. This means that even once `ai-netserver`'s poll config is activated, ingested NetServer records would currently normalize incorrectly (falling through to DÖE-shaped field extraction, which would find nothing and default most fields to null/'Unbenannte Ausschreibung'). **This is explicitly out of this plan's `files_modified` scope** (only `netserver.adapter.ts`/`.spec.ts`, the fixture, and `tenders.module.ts` were planned) — normalizer extension for non-DÖE-shaped payloads is a natural follow-up (likely alongside or after 13-05's cosinex adapter, which will face the identical problem) and should be a deliberate next plan, not silently patched in here. `activePortalCount >= 2` (the dedup-gate signal) is unaffected by this — it counts active poll configs, not correctly-normalized records — but real cross-source dedup value from NetServer will only materialize once the normalizer is extended AND the poll config is activated.
|
||||
|
||||
## Issues Encountered
|
||||
|
||||
None beyond the expected checkpoint pause. Live network access was available in this environment, which let the fixture be captured genuinely live (not synthesized) — matching the plan's "live-gecapturte HTML-Fixture" requirement exactly.
|
||||
|
||||
## User Setup Required
|
||||
|
||||
None. `ai-netserver` remains `isActive: false` — no live polling will start until an admin explicitly activates it (Phase 14 UI) or a manual DB update is made. No test/prod server was touched (local dev API only, host-side Vitest).
|
||||
|
||||
## Next Phase Readiness
|
||||
- `SourceRegistry`, `TenderDedupService`, and `pollDueSources`' fan-out (13-02/13-03) now have a second real, tested adapter to exercise — 13-05 (cosinex/DTVP) can follow the identical pattern (config-driven-per-portal is N/A there since it's one portal, but the fetch/parse/error-isolation/registration shape repeats).
|
||||
- **Follow-up needed before NetServer data is usably normalized end-to-end:** extend `TenderNormalizerService` (or introduce a per-sourceType normalization strategy) to understand the `ocdsPayload` generic bag this adapter produces — flagged above under Known Stubs, not blocking for INGEST-02's own success criteria (adapter parses to `RawTenderRecord[]` correctly) but blocking for a live end-to-end NetServer ingest to produce correct `Tender` rows.
|
||||
- Manual/live verification (13-VALIDATION.md) — an optional live poll against `tender24.de` to confirm real records ingest — remains a manual step, not executed here (no `ai-netserver` activation was performed; the adapter's live-fetch path was exercised for the fixture capture itself, confirming reachability, but not through the full poll/ingest pipeline).
|
||||
- No blockers for 13-05.
|
||||
|
||||
---
|
||||
*Phase: 13-scraping-adapters-cross-source-dedup*
|
||||
*Completed: 2026-07-23*
|
||||
|
||||
## Self-Check: PASSED
|
||||
|
||||
All created files verified present on disk; both task commit hashes (043ada9, 88572f5) verified in git log.
|
||||
Reference in New Issue
Block a user