tessera-ctl/.planning/phases/13-scraping-adapters-cross-source-dedup/13-04-SUMMARY.md

---
phase: 13-scraping-adapters-cross-source-dedup
plan: 04
subsystem: backend
tags: [cheerio, html-scraping, adapter-pattern, netserver, package-legitimacy-gate]

requires:
  - phase: 13-scraping-adapters-cross-source-dedup
    plan: 01
    provides: SourceType open union (includes 'ai-netserver'), RawTenderRecord contract
  - phase: 13-scraping-adapters-cross-source-dedup
    plan: 02
    provides: TenderSourceAdapter.portals[], SourceRegistry with DeniedPortalError denylist gate
  - phase: 13-scraping-adapters-cross-source-dedup
    plan: 03
    provides: pollDueSources fan-out + TenderDedupService, TendersModule onModuleInit registration hook
provides:
  - "NetServerAdapter — ONE config-driven adapter serving all 3 AI-AG Vergabe@Net portals (tender24, lhs-vpbw, vergabe.landbw)"
  - "cheerio (^1.2.0) as the project's HTML-parsing dependency (package-legitimacy-gated, human-approved)"
  - "Live-captured netserver-search.html fixture for stable, network-free adapter tests"
  - "NetServerAdapter registered as a TendersModule provider + at DI boot via SourceRegistry.register(); ai-netserver TenderSourcePollConfig seeded (isActive: false)"
affects: [13-05-cosinex-adapter, phase-14-admin-source-activation-ui]

tech-stack:
  added:
    - "cheerio ^1.2.0 (HTML parsing, package-legitimacy-gated: human approved over node-html-parser after gate flagged the latter's most-recent-version-publish-date as a 'too-new' false positive)"
  patterns:
    - "Config-driven multi-portal adapter: one class + a hardcoded portal-to-baseUrl Record, iterated per fetchTenders() call, each portal wrapped in its own try/catch so one blocked/broken portal never prevents the other two from being polled in the same tick"
    - "Per-row try/catch inside cheerio .each() — a single malformed table row is skipped (logger.warn) rather than aborting the whole portal's parse; cheerio.load() itself is also wrapped, so a totally unparsable page returns [] instead of throwing"
    - "RawTenderRecord.ocdsPayload used as a generic field bag (title/buyerName/procedureType/legalFramework/deadlineAt) for a non-OCDS source — deliberately NOT reshaping the raw eForms/OCDS-shaped RawTenderRecord interface; the shared TenderNormalizerService does not yet understand this shape (out of scope for this plan, see Known Stubs)"

key-files:
  created:
    - apps/api/src/tenders/adapters/netserver.adapter.ts
    - apps/api/src/tenders/adapters/netserver.adapter.spec.ts
    - apps/api/src/tenders/__fixtures__/netserver-search.html
  modified:
    - apps/api/package.json
    - pnpm-lock.yaml
    - apps/api/src/tenders/tenders.module.ts

key-decisions:
  - "cheerio chosen over node-html-parser at the blocking-human package-legitimacy checkpoint — the automated gate flagged node-html-parser 'SUS'/'too-new' purely because it measures the latest-*version*-publish date (2026-07-06), not true package age (first published 2017-06-14, 129 versions, v9.0.0); cheerio passed the gate cleanly (OK, 27M weekly downloads, published since 2011) so the human approver picked the unambiguous option"
  - "sourceUrl falls back to the portal's search-results URL, NOT a per-notice deep link — live inspection of the captured HTML found NO <a href> per result row; navigation to a notice's detail page is client-side-JS-only (data-oid + an external click handler script), with no statically-derivable detail URL anywhere in the search response. This resolves 13-RESEARCH.md Open Question 2 with a documented limitation rather than a guessed/unverified URL pattern (avoiding a fabricated, unverifiable servlet path)."
  - "sourceNoticeId = the row's data-oid attribute (e.g. '54321-NetTender-19f800ce3b1-...') — stable and unique per notice, confirmed unique across all rows in the live fixture"
  - "dayCursor parameter is accepted (interface conformance) but not used as a query filter — the NetServer search servlet has no documented incremental/date-range parameter (unlike DÖE's day-batch export); first-page-only fetch is accepted as sufficient for the MVP proof (13-RESEARCH.md Open Question 2 recommendation), pagination is out of scope for this plan"
  - "NetServer's '24:00' deadline convention (meaning midnight at the END of the stated day) is normalized to 00:00 of the next day, rather than left as an invalid Date"
  - "TenderSourcePollConfig row for 'ai-netserver' is seeded with isActive: false — this plan builds, registers, and tests the adapter; it does not force-activate live polling of the 3 real portals (D-02: activation is a later admin/seed decision, Phase 14 UI)"

requirements-completed: [INGEST-02]

coverage:
  - id: D1
    description: "ONE config-driven NetServerAdapter declares portals=['tender24','lhs-vpbw','vergabe.landbw'] and a hardcoded per-portal baseUrl map; fetchTenders() fetches and parses all 3 portals independently, sourcePortal set correctly per parsed row"
    requirement: INGEST-02
    verification:
      - kind: unit
        ref: "apps/api/src/tenders/adapters/netserver.adapter.spec.ts — 'declares sourceType ai-netserver and the 3 AI-AG NetServer portals', 'fetches all 3 portals and aggregates their parsed records' (14/14 tests pass)"
        status: pass
    human_judgment: false
  - id: D2
    description: "Adapter parses the live-captured NetServer <table> results into RawTenderRecord[] with correct sourceNoticeId (data-oid), umlaut/&amp;-entity-decoded buyer names, and deadline mapped to a Date or null (never throwing on empty deadlines)"
    requirement: INGEST-02
    verification:
      - kind: unit
        ref: "apps/api/src/tenders/adapters/netserver.adapter.spec.ts — 'decodes an umlaut + &amp;-entity buyer name correctly', 'uses the row data-oid attribute as sourceNoticeId', 'maps a populated deadline row to a Date and an empty-deadline row to null'"
        status: pass
    human_judgment: false
  - id: D3
    description: "Parse-Totalausfall (empty HTML, no matching rows, malformed row, portal fetch throwing, portal responding non-2xx) returns [] instead of throwing, per portal and in aggregate (D-01 error tolerance)"
    requirement: INGEST-02
    verification:
      - kind: unit
        ref: "apps/api/src/tenders/adapters/netserver.adapter.spec.ts — 'returns [] for empty HTML', 'returns [] for HTML with no matching result rows', 'skips a row missing data-oid without throwing', 'returns [] for a portal whose fetch throws, without aborting the other portals', 'returns [] for every portal when fetch always throws', 'returns [] without throwing when a portal responds non-2xx'"
        status: pass
    human_judgment: false
  - id: D4
    description: "NetServerAdapter is a TendersModule provider and is registered with SourceRegistry at DI boot alongside DoeOpenDataAdapter; the INGEST-07 denylist gate structurally applies (tender24/lhs-vpbw/vergabe.landbw are not denylisted, so registration succeeds); project-wide typecheck and the full tenders test slice remain green"
    requirement: INGEST-02
    verification:
      - kind: unit
        ref: "cd apps/api && npx tsc --noEmit -p tsconfig.json (clean)"
        status: pass
      - kind: unit
        ref: "cd apps/api && npx vitest run src/tenders (20 files, 205/205 pass, includes source-registry.spec.ts 6/6 unchanged and netserver.adapter.spec.ts 14/14 new)"
        status: pass
    human_judgment: false
  - id: D5
    description: "Package-legitimacy checkpoint (blocking-human) ran BEFORE any install; cheerio was human-approved and installed only after explicit confirmation, never auto-approved"
    requirement: INGEST-02
    verification:
      - kind: manual_gate
        ref: "gsd-tools query package-legitimacy check --ecosystem npm node-html-parser cheerio (run pre-install) + npm view node-html-parser time.created / npm view cheerio time.created (cross-checked); human approved 'cheerio' via explicit resume instruction"
        status: pass
    human_judgment: true

duration: 55min
completed: 2026-07-23
status: complete
---

# Phase 13 Plan 04: NetServer-Adapter (tender24/lhs-vpbw/vergabe.landbw) Summary

**ONE config-driven `NetServerAdapter` (cheerio-based, human-approved at a blocking package-legitimacy checkpoint) now serves all three AI-AG "Vergabe@Net" portals from a single class, parsing the live public `PublicationSearchControllerServlet` results table into `RawTenderRecord[]` with full row/portal error isolation — the second real ingestion source that makes the Phase-13 dedup core (13-03) live-observable (`activePortalCount >= 2` once activated).**

## Performance

- **Duration:** ~55 min (including the blocking package-legitimacy checkpoint pause)
- **Tasks:** 3/3 completed
- **Files modified:** 6 (3 created, 3 modified)

## Accomplishments
- **Blocking package-legitimacy checkpoint honored correctly**: paused BEFORE any install, ran `gsd-tools query package-legitimacy check` against both `node-html-parser` and `cheerio`, cross-verified the flagged `node-html-parser` "too-new" verdict against `npm view ... time.created` (found it to be a false positive — the gate measures latest-version-publish date, not package age; the package is actually 9+ years old). Reported both findings and waited for explicit human approval. Human selected `cheerio` — installed only after that confirmation.
- `NetServerAdapter`: ONE class, `portals = ['tender24', 'lhs-vpbw', 'vergabe.landbw']`, hardcoded per-portal `baseUrl` constants (SSRF guard, T-13-04-01), fetches each portal's `PublicationSearchControllerServlet?function=SearchPublications` via native `fetch` + `AbortController` 15s timeout + `redirect: 'follow'`, parses the `<table>` results with cheerio.
- Fault tolerance exactly per D-01/Pitfall 2: try/catch per row (skip + `logger.warn`), try/catch per portal (one blocked/broken portal never prevents the other two from being polled), `cheerio.load()` itself wrapped so a totally unparsable page returns `[]` for that portal instead of throwing.
- Live-captured fixture (`__fixtures__/netserver-search.html`, 6 real rows from a live `tender24.de` GET, 2026-07-23) exercises umlaut + `&amp;`-entity buyer decoding, a populated deadline, and multiple empty-deadline rows.
- 14 spec tests, all green: sourceType/portals declaration, full-fixture parse with correct `sourcePortal`/`sourceNoticeId`/`sourceUrl`, umlaut/entity decoding, `data-oid`-uniqueness, deadline-to-Date-or-null mapping (including NetServer's `24:00` = next-day-midnight convention), empty/broken-HTML `[]`-fallback, missing-`data-oid`-row skip, per-portal fetch-fail isolation, total-fetch-failure `[]`-fallback, non-2xx `[]`-fallback, no-axios guard, no-input-interpolated-URL guard.
- `NetServerAdapter` registered as a `TendersModule` provider and at DI boot via `SourceRegistry.register()` (denylist gate structurally applies — none of the 3 portals are denylisted); `ai-netserver` `TenderSourcePollConfig` seeded with `isActive: false` (activation deferred to admin/Phase 14, per D-02).
- `npx tsc --noEmit` clean; `src/tenders` slice: 20 files, 205/205 tests pass (191 pre-existing + 14 new).

## Task Commits

Each task was committed atomically:

1. **Task 1: Package-legitimacy checkpoint (blocking-human)** — no commit (gate/decision only, per protocol); cheerio approved by the coordinator/human
2. **Task 2: cheerio install + NetServerAdapter + fixture + spec** - `043ada9` (feat)
3. **Task 3: NetServerAdapter registered in TendersModule** - `88572f5` (feat)

## Files Created/Modified
- `apps/api/src/tenders/adapters/netserver.adapter.ts` - config-driven multi-portal `NetServerAdapter` (`fetchTenders`, `parseSearchResults`, `parseNetServerDate`/`parseNetServerDeadline` helpers)
- `apps/api/src/tenders/adapters/netserver.adapter.spec.ts` - 14 unit tests against the fixture + mocked fetch
- `apps/api/src/tenders/__fixtures__/netserver-search.html` - live-captured NetServer results table (6 rows)
- `apps/api/package.json` / `pnpm-lock.yaml` - `cheerio ^1.2.0` dependency added
- `apps/api/src/tenders/tenders.module.ts` - `NetServerAdapter` provider + `onModuleInit` registration + `ai-netserver` poll-config seed (`isActive: false`)

## Decisions Made
- **cheerio over node-html-parser** — see `key-decisions` in frontmatter. The automated legitimacy gate's "too-new" flag on `node-html-parser` was investigated and found to be a measurement artifact (latest-version-publish date, not package founding date); both packages are legitimate, but `cheerio` passed the gate cleanly with zero flags, so it was the unambiguous choice for the human approver.
- **`sourceUrl` = search-results URL, not a per-notice deep link** — live HTML inspection found no `<a href>` per row (JS-only navigation via `data-oid`); rather than guess/invent an unverified detail-servlet URL pattern, this is documented as a best-effort limitation (Open Question 2).
- **`ocdsPayload` used as a generic NetServer field bag** rather than reshaping `RawTenderRecord`'s DÖE-shaped interface — keeps the shared adapter contract stable; full normalizer support for this shape is explicitly out of this plan's scope (see Known Stubs below).
- **`isActive: false` on the seeded poll config** — this plan proves the adapter works against a live-captured fixture; it deliberately does not flip the platform-global scheduler into actually polling 3 real portals every tick without an explicit admin/Phase-14 decision.

## Deviations from Plan

### Auto-fixed Issues

None — Rule 1-3 auto-fixes were not needed; implementation followed the plan and 13-RESEARCH.md's Pattern 1 design directly.

### Checkpoint Handling (not a deviation, documented per protocol)

**Package selection changed from the plan's primary suggestion.** 13-RESEARCH.md's `[ASSUMED]` recommendation was `node-html-parser` (with `cheerio` as the stated alternative). The blocking-human checkpoint's own automated gate flagged `node-html-parser` as `SUS`/"too-new" (a false positive on version-publish-date vs. package age, verified via `npm view ... time.created`), while `cheerio` passed cleanly. The coordinator/human explicitly approved `cheerio` after reviewing both findings. This is exactly the checkpoint's intended function — no auto-approval occurred, and both plan-sanctioned options were legitimate; the final selection was a human decision, not an executor deviation.

**Total:** 0 auto-fixed deviations. 1 human-directed package selection (checkpoint outcome, not a deviation).

## Known Stubs

- **`ocdsPayload` generic bag is not yet consumed by `TenderNormalizerService`.** The normalizer (`tender-normalizer.service.ts`) is currently hardcoded to the DÖE eForms/OCDS shape (`getOcdsTender`, `getEformsRoot`, etc.) and does not know how to map `NetServerAdapter`'s `{title, buyerName, procedureType, legalFramework, deadlineAt}` bag into `NormalizedTenderFields`. This means that even once `ai-netserver`'s poll config is activated, ingested NetServer records would currently normalize incorrectly (falling through to DÖE-shaped field extraction, which would find nothing and default most fields to null/'Unbenannte Ausschreibung'). **This is explicitly out of this plan's `files_modified` scope** (only `netserver.adapter.ts`/`.spec.ts`, the fixture, and `tenders.module.ts` were planned) — normalizer extension for non-DÖE-shaped payloads is a natural follow-up (likely alongside or after 13-05's cosinex adapter, which will face the identical problem) and should be a deliberate next plan, not silently patched in here. `activePortalCount >= 2` (the dedup-gate signal) is unaffected by this — it counts active poll configs, not correctly-normalized records — but real cross-source dedup value from NetServer will only materialize once the normalizer is extended AND the poll config is activated.

## Issues Encountered

None beyond the expected checkpoint pause. Live network access was available in this environment, which let the fixture be captured genuinely live (not synthesized) — matching the plan's "live-gecapturte HTML-Fixture" requirement exactly.

## User Setup Required

None. `ai-netserver` remains `isActive: false` — no live polling will start until an admin explicitly activates it (Phase 14 UI) or a manual DB update is made. No test/prod server was touched (local dev API only, host-side Vitest).

## Next Phase Readiness
- `SourceRegistry`, `TenderDedupService`, and `pollDueSources`' fan-out (13-02/13-03) now have a second real, tested adapter to exercise — 13-05 (cosinex/DTVP) can follow the identical pattern (config-driven-per-portal is N/A there since it's one portal, but the fetch/parse/error-isolation/registration shape repeats).
- **Follow-up needed before NetServer data is usably normalized end-to-end:** extend `TenderNormalizerService` (or introduce a per-sourceType normalization strategy) to understand the `ocdsPayload` generic bag this adapter produces — flagged above under Known Stubs, not blocking for INGEST-02's own success criteria (adapter parses to `RawTenderRecord[]` correctly) but blocking for a live end-to-end NetServer ingest to produce correct `Tender` rows.
- Manual/live verification (13-VALIDATION.md) — an optional live poll against `tender24.de` to confirm real records ingest — remains a manual step, not executed here (no `ai-netserver` activation was performed; the adapter's live-fetch path was exercised for the fixture capture itself, confirming reachability, but not through the full poll/ingest pipeline).
- No blockers for 13-05.

---
*Phase: 13-scraping-adapters-cross-source-dedup*
*Completed: 2026-07-23*

## Self-Check: PASSED

All created files verified present on disk; both task commit hashes (043ada9, 88572f5) verified in git log.