Case study / data infrastructure
An access layer that turns blocks into technical decisions.
Scraping Access Layer (SAL) is a local-first Go service and CLI that assesses public sources, explicitly selects collection methods, and delivers normalized data to systems such as Cat Euphoria Index, Macrobrief, and AI Briefing.
Access hierarchy
Assess first. Collect through the method the target requires.
SAL prioritizes structured surfaces but is not limited to diagnostics: a requester with named methods tests access hypotheses without turning escalation into invisible behavior.
- 01
Structured Public Sources
Starts with APIs, feeds, sitemaps, and public endpoints that already expose verifiable data.
- 02
HTML and Metadata
Extracts signals from HTML, JSON-LD, OpenGraph, and structural links to qualify the source and available data.
- 03
Explicit Methods
The requester selects HTTP, TLS, a sidecar, or an isolated browser through explicit decisions with observable costs and outcomes.
- 04
Source-Specific Adapters
When a source warrants it, normalizes data, validates freshness, and fails closed instead of hiding a brittle collection path.
Policies and governance
Stronger methods must remain explicit and observable.
Automation is treated as reliability engineering: every route carries cost, evidence, limits, and clear criteria to continue or stop.
Assessment Before Collection
Classifies the target, preserves evidence, and formulates an access hypothesis before widening collection methods.
Explicit Escalation
More costly methods never enter the default fetch path silently: each needs selection, diagnostics, and a stop condition.
Fixture-First Testing
Test suite decoupled from external networks, using versioned fixtures to guarantee determinism and reproducibility.
Evidence and Fail-Closed Behavior
Output records method, provenance, validations, and limits; incomplete or changed data stops the route in an auditable way.
This case study presents architecture and verified outcomes without exposing operational artifacts, session secrets, or details that expand control evasion.