tessera-ctl/.planning/phases/13-scraping-adapters-cross-source-dedup/13-04-SUMMARY.md

phase, plan, subsystem, tags, requires, provides, affects, tech-stack, key-files, key-decisions, requirements-completed, coverage, duration, completed, status

phase	plan	subsystem	tags	requires	provides	affects	tech-stack	key-files	key-decisions	requirements-completed	coverage	duration	completed	status
13-scraping-adapters-cross-source-dedup	04	backend	cheerio html-scraping adapter-pattern netserver package-legitimacy-gate	phase plan provides 13-scraping-adapters-cross-source-dedup 01 SourceType open union (includes 'ai-netserver'), RawTenderRecord contract phase plan provides 13-scraping-adapters-cross-source-dedup 02 TenderSourceAdapter.portals[], SourceRegistry with DeniedPortalError denylist gate phase plan provides 13-scraping-adapters-cross-source-dedup 03 pollDueSources fan-out + TenderDedupService, TendersModule onModuleInit registration hook	NetServerAdapter — ONE config-driven adapter serving all 3 AI-AG Vergabe@Net portals (tender24, lhs-vpbw, vergabe.landbw) cheerio (^1.2.0) as the project's HTML-parsing dependency (package-legitimacy-gated, human-approved) Live-captured netserver-search.html fixture for stable, network-free adapter tests NetServerAdapter registered as a TendersModule provider + at DI boot via SourceRegistry.register(); ai-netserver TenderSourcePollConfig seeded (isActive: false)	13-05-cosinex-adapter phase-14-admin-source-activation-ui	added patterns cheerio ^1.2.0 (HTML parsing, package-legitimacy-gated: human approved over node-html-parser after gate flagged the latter's most-recent-version-publish-date as a 'too-new' false positive) Config-driven multi-portal adapter: one class + a hardcoded portal-to-baseUrl Record, iterated per fetchTenders() call, each portal wrapped in its own try/catch so one blocked/broken portal never prevents the other two from being polled in the same tick Per-row try/catch inside cheerio .each() — a single malformed table row is skipped (logger.warn) rather than aborting the whole portal's parse; cheerio.load() itself is also wrapped, so a totally unparsable page returns [] instead of throwing RawTenderRecord.ocdsPayload used as a generic field bag (title/buyerName/procedureType/legalFramework/deadlineAt) for a non-OCDS source — deliberately NOT reshaping the raw eForms/OCDS-shaped RawTenderRecord interface; the shared TenderNormalizerService does not yet understand this shape (out of scope for this plan, see Known Stubs)	created modified apps/api/src/tenders/adapters/netserver.adapter.ts apps/api/src/tenders/adapters/netserver.adapter.spec.ts apps/api/src/tenders/__fixtures__/netserver-search.html apps/api/package.json pnpm-lock.yaml apps/api/src/tenders/tenders.module.ts	cheerio chosen over node-html-parser at the blocking-human package-legitimacy checkpoint — the automated gate flagged node-html-parser 'SUS'/'too-new' purely because it measures the latest-*version*-publish date (2026-07-06), not true package age (first published 2017-06-14, 129 versions, v9.0.0); cheerio passed the gate cleanly (OK, 27M weekly downloads, published since 2011) so the human approver picked the unambiguous option sourceUrl falls back to the portal's search-results URL, NOT a per-notice deep link — live inspection of the captured HTML found NO <a href> per result row; navigation to a notice's detail page is client-side-JS-only (data-oid + an external click handler script), with no statically-derivable detail URL anywhere in the search response. This resolves 13-RESEARCH.md Open Question 2 with a documented limitation rather than a guessed/unverified URL pattern (avoiding a fabricated, unverifiable servlet path). sourceNoticeId = the row's data-oid attribute (e.g. '54321-NetTender-19f800ce3b1-...') — stable and unique per notice, confirmed unique across all rows in the live fixture dayCursor parameter is accepted (interface conformance) but not used as a query filter — the NetServer search servlet has no documented incremental/date-range parameter (unlike DÖE's day-batch export); first-page-only fetch is accepted as sufficient for the MVP proof (13-RESEARCH.md Open Question 2 recommendation), pagination is out of scope for this plan NetServer's '24:00' deadline convention (meaning midnight at the END of the stated day) is normalized to 00:00 of the next day, rather than left as an invalid Date TenderSourcePollConfig row for 'ai-netserver' is seeded with isActive: false — this plan builds, registers, and tests the adapter; it does not force-activate live polling of the 3 real portals (D-02: activation is a later admin/seed decision, Phase 14 UI)	INGEST-02	id description requirement verification human_judgment D1 ONE config-driven NetServerAdapter declares portals=['tender24','lhs-vpbw','vergabe.landbw'] and a hardcoded per-portal baseUrl map; fetchTenders() fetches and parses all 3 portals independently, sourcePortal set correctly per parsed row INGEST-02 kind ref status unit apps/api/src/tenders/adapters/netserver.adapter.spec.ts — 'declares sourceType ai-netserver and the 3 AI-AG NetServer portals', 'fetches all 3 portals and aggregates their parsed records' (14/14 tests pass) pass false id description requirement verification human_judgment D2 Adapter parses the live-captured NetServer <table> results into RawTenderRecord[] with correct sourceNoticeId (data-oid), umlaut/&-entity-decoded buyer names, and deadline mapped to a Date or null (never throwing on empty deadlines) INGEST-02 kind ref status unit apps/api/src/tenders/adapters/netserver.adapter.spec.ts — 'decodes an umlaut + &-entity buyer name correctly', 'uses the row data-oid attribute as sourceNoticeId', 'maps a populated deadline row to a Date and an empty-deadline row to null' pass false id description requirement verification human_judgment D3 Parse-Totalausfall (empty HTML, no matching rows, malformed row, portal fetch throwing, portal responding non-2xx) returns [] instead of throwing, per portal and in aggregate (D-01 error tolerance) INGEST-02 kind ref status unit apps/api/src/tenders/adapters/netserver.adapter.spec.ts — 'returns [] for empty HTML', 'returns [] for HTML with no matching result rows', 'skips a row missing data-oid without throwing', 'returns [] for a portal whose fetch throws, without aborting the other portals', 'returns [] for every portal when fetch always throws', 'returns [] without throwing when a portal responds non-2xx' pass false id description requirement verification human_judgment D4 NetServerAdapter is a TendersModule provider and is registered with SourceRegistry at DI boot alongside DoeOpenDataAdapter; the INGEST-07 denylist gate structurally applies (tender24/lhs-vpbw/vergabe.landbw are not denylisted, so registration succeeds); project-wide typecheck and the full tenders test slice remain green INGEST-02 kind ref status unit cd apps/api && npx tsc --noEmit -p tsconfig.json (clean) pass kind ref status unit cd apps/api && npx vitest run src/tenders (20 files, 205/205 pass, includes source-registry.spec.ts 6/6 unchanged and netserver.adapter.spec.ts 14/14 new) pass false id description requirement verification human_judgment D5 Package-legitimacy checkpoint (blocking-human) ran BEFORE any install; cheerio was human-approved and installed only after explicit confirmation, never auto-approved INGEST-02 kind ref status manual_gate gsd-tools query package-legitimacy check --ecosystem npm node-html-parser cheerio (run pre-install) + npm view node-html-parser time.created / npm view cheerio time.created (cross-checked); human approved 'cheerio' via explicit resume instruction pass true	55min	2026-07-23	complete

Phase 13 Plan 04: NetServer-Adapter (tender24/lhs-vpbw/vergabe.landbw) Summary

ONE config-driven NetServerAdapter (cheerio-based, human-approved at a blocking package-legitimacy checkpoint) now serves all three AI-AG "Vergabe@Net" portals from a single class, parsing the live public PublicationSearchControllerServlet results table into RawTenderRecord[] with full row/portal error isolation — the second real ingestion source that makes the Phase-13 dedup core (13-03) live-observable (activePortalCount >= 2 once activated).

Performance

• Duration: ~55 min (including the blocking package-legitimacy checkpoint pause)
• Tasks: 3/3 completed
• Files modified: 6 (3 created, 3 modified)

Accomplishments

• Blocking package-legitimacy checkpoint honored correctly: paused BEFORE any install, ran gsd-tools query package-legitimacy check against both node-html-parser and cheerio, cross-verified the flagged node-html-parser "too-new" verdict against npm view ... time.created (found it to be a false positive — the gate measures latest-version-publish date, not package age; the package is actually 9+ years old). Reported both findings and waited for explicit human approval. Human selected cheerio — installed only after that confirmation.
• NetServerAdapter: ONE class, portals = ['tender24', 'lhs-vpbw', 'vergabe.landbw'], hardcoded per-portal baseUrl constants (SSRF guard, T-13-04-01), fetches each portal's PublicationSearchControllerServlet?function=SearchPublications via native fetch + AbortController 15s timeout + redirect: 'follow', parses the <table> results with cheerio.
• Fault tolerance exactly per D-01/Pitfall 2: try/catch per row (skip + logger.warn), try/catch per portal (one blocked/broken portal never prevents the other two from being polled), cheerio.load() itself wrapped so a totally unparsable page returns [] for that portal instead of throwing.
• Live-captured fixture (__fixtures__/netserver-search.html, 6 real rows from a live tender24.de GET, 2026-07-23) exercises umlaut + &-entity buyer decoding, a populated deadline, and multiple empty-deadline rows.
• 14 spec tests, all green: sourceType/portals declaration, full-fixture parse with correct sourcePortal/sourceNoticeId/sourceUrl, umlaut/entity decoding, data-oid-uniqueness, deadline-to-Date-or-null mapping (including NetServer's 24:00 = next-day-midnight convention), empty/broken-HTML []-fallback, missing-data-oid-row skip, per-portal fetch-fail isolation, total-fetch-failure []-fallback, non-2xx []-fallback, no-axios guard, no-input-interpolated-URL guard.
• NetServerAdapter registered as a TendersModule provider and at DI boot via SourceRegistry.register() (denylist gate structurally applies — none of the 3 portals are denylisted); ai-netserver TenderSourcePollConfig seeded with isActive: false (activation deferred to admin/Phase 14, per D-02).
• npx tsc --noEmit clean; src/tenders slice: 20 files, 205/205 tests pass (191 pre-existing + 14 new).

Task Commits

Each task was committed atomically:

1. Task 1: Package-legitimacy checkpoint (blocking-human) — no commit (gate/decision only, per protocol); cheerio approved by the coordinator/human
2. Task 2: cheerio install + NetServerAdapter + fixture + spec - 043ada9 (feat)
3. Task 3: NetServerAdapter registered in TendersModule - 88572f5 (feat)

Files Created/Modified

• apps/api/src/tenders/adapters/netserver.adapter.ts - config-driven multi-portal NetServerAdapter (fetchTenders, parseSearchResults, parseNetServerDate/parseNetServerDeadline helpers)
• apps/api/src/tenders/adapters/netserver.adapter.spec.ts - 14 unit tests against the fixture + mocked fetch
• apps/api/src/tenders/__fixtures__/netserver-search.html - live-captured NetServer results table (6 rows)
• apps/api/package.json / pnpm-lock.yaml - cheerio ^1.2.0 dependency added
• apps/api/src/tenders/tenders.module.ts - NetServerAdapter provider + onModuleInit registration + ai-netserver poll-config seed (isActive: false)

Decisions Made

• cheerio over node-html-parser — see key-decisions in frontmatter. The automated legitimacy gate's "too-new" flag on node-html-parser was investigated and found to be a measurement artifact (latest-version-publish date, not package founding date); both packages are legitimate, but cheerio passed the gate cleanly with zero flags, so it was the unambiguous choice for the human approver.
• sourceUrl = search-results URL, not a per-notice deep link — live HTML inspection found no <a href> per row (JS-only navigation via data-oid); rather than guess/invent an unverified detail-servlet URL pattern, this is documented as a best-effort limitation (Open Question 2).
• ocdsPayload used as a generic NetServer field bag rather than reshaping RawTenderRecord's DÖE-shaped interface — keeps the shared adapter contract stable; full normalizer support for this shape is explicitly out of this plan's scope (see Known Stubs below).
• isActive: false on the seeded poll config — this plan proves the adapter works against a live-captured fixture; it deliberately does not flip the platform-global scheduler into actually polling 3 real portals every tick without an explicit admin/Phase-14 decision.

Deviations from Plan

Auto-fixed Issues

None — Rule 1-3 auto-fixes were not needed; implementation followed the plan and 13-RESEARCH.md's Pattern 1 design directly.

Checkpoint Handling (not a deviation, documented per protocol)

Package selection changed from the plan's primary suggestion. 13-RESEARCH.md's [ASSUMED] recommendation was node-html-parser (with cheerio as the stated alternative). The blocking-human checkpoint's own automated gate flagged node-html-parser as SUS/"too-new" (a false positive on version-publish-date vs. package age, verified via npm view ... time.created), while cheerio passed cleanly. The coordinator/human explicitly approved cheerio after reviewing both findings. This is exactly the checkpoint's intended function — no auto-approval occurred, and both plan-sanctioned options were legitimate; the final selection was a human decision, not an executor deviation.

Total: 0 auto-fixed deviations. 1 human-directed package selection (checkpoint outcome, not a deviation).

Known Stubs

• ocdsPayload generic bag is not yet consumed by TenderNormalizerService. The normalizer (tender-normalizer.service.ts) is currently hardcoded to the DÖE eForms/OCDS shape (getOcdsTender, getEformsRoot, etc.) and does not know how to map NetServerAdapter's {title, buyerName, procedureType, legalFramework, deadlineAt} bag into NormalizedTenderFields. This means that even once ai-netserver's poll config is activated, ingested NetServer records would currently normalize incorrectly (falling through to DÖE-shaped field extraction, which would find nothing and default most fields to null/'Unbenannte Ausschreibung'). This is explicitly out of this plan's files_modified scope (only netserver.adapter.ts/.spec.ts, the fixture, and tenders.module.ts were planned) — normalizer extension for non-DÖE-shaped payloads is a natural follow-up (likely alongside or after 13-05's cosinex adapter, which will face the identical problem) and should be a deliberate next plan, not silently patched in here. activePortalCount >= 2 (the dedup-gate signal) is unaffected by this — it counts active poll configs, not correctly-normalized records — but real cross-source dedup value from NetServer will only materialize once the normalizer is extended AND the poll config is activated.

Issues Encountered

None beyond the expected checkpoint pause. Live network access was available in this environment, which let the fixture be captured genuinely live (not synthesized) — matching the plan's "live-gecapturte HTML-Fixture" requirement exactly.

User Setup Required

None. ai-netserver remains isActive: false — no live polling will start until an admin explicitly activates it (Phase 14 UI) or a manual DB update is made. No test/prod server was touched (local dev API only, host-side Vitest).

Next Phase Readiness

• SourceRegistry, TenderDedupService, and pollDueSources' fan-out (13-02/13-03) now have a second real, tested adapter to exercise — 13-05 (cosinex/DTVP) can follow the identical pattern (config-driven-per-portal is N/A there since it's one portal, but the fetch/parse/error-isolation/registration shape repeats).
• Follow-up needed before NetServer data is usably normalized end-to-end: extend TenderNormalizerService (or introduce a per-sourceType normalization strategy) to understand the ocdsPayload generic bag this adapter produces — flagged above under Known Stubs, not blocking for INGEST-02's own success criteria (adapter parses to RawTenderRecord[] correctly) but blocking for a live end-to-end NetServer ingest to produce correct Tender rows.
• Manual/live verification (13-VALIDATION.md) — an optional live poll against tender24.de to confirm real records ingest — remains a manual step, not executed here (no ai-netserver activation was performed; the adapter's live-fetch path was exercised for the fixture capture itself, confirming reachability, but not through the full poll/ingest pipeline).
• No blockers for 13-05.

---

Phase: 13-scraping-adapters-cross-source-dedup Completed: 2026-07-23

Self-Check: PASSED

All created files verified present on disk; both task commit hashes (043ada9, 88572f5) verified in git log.