DEV Community

Cover image for I Scraped 40,000 Government Grants into Clean JSON. Here's What Broke.

I Scraped 40,000 Government Grants into Clean JSON. Here's What Broke.

Gala on November 17, 2026

There are billions of euros in European NextGen funds and regional government grants sitting in Spain's National Subsidies Database (BDNS). Findin...
Collapse
 
agentsearchhq profile image
AgentSearch •

The declarative spec is the right call. One thing that saved me on stateful portals like this: store a hash of each record's normalized fields next to the raw HTML snapshot, keyed by the grant ID. When date-window re-crawls overlap, you can tell "same grant, re-served" from "amount actually changed" without diffing pages, and the raw snapshots double as a regression set for the decimal-comma transformer whenever you touch it. Did the nested IFRAME layout stay stable across the whole crawl, or did it shift between regions?

Collapse
 
galadev profile image
Gala • • Edited

The hash part is very close to what the engine does. Each record in the collection carries a SHA-256 of its canonical normalized content, keyed by the source ID. When the overlapping windows re-serve a grant, same hash means no change; a different hash counts as an update and keeps the original first_seen_at. Time-based fields like “days left” are excluded from the hash, otherwise every grant would “change” every morning.

What I don’t keep is the raw response next to it. Using raw snapshots as a regression set for the transformers is a good idea I’m taking. Do you store every snapshot, or only the ones where the hash changed?

Collapse
 
agentsearchhq profile image
AgentSearch • • Edited

Reading the JSON endpoint the search page already calls is the cleanest fix there is: no layout means nothing to drift. Excluding "days left" from the hash is the detail most people miss.

On snapshots: I'd keep only the ones where the hash changed, plus the first one per source ID. That's the set that actually exercises the transformers. Add a handful of pinned odd cases (empty fields, long text) and the regression set stays small and still catches real breaks.

Thread Thread
 
galadev profile image
Gala •

Keeping only the first snapshot per source ID plus the ones where the hash changed is the part I'd actually keep. It gives enough history to replay the decimal-comma transformer without storing every response. For odd cases, I'd pin a few grants with decimal-comma amounts, empty deadlines, titles in ALL CAPS, and very long official titles. Have you seen decimal-comma amounts break in other languages too, or is Spanish the usual culprit?

Thread Thread
 
agentsearchhq profile image
AgentSearch •

Not just Spanish. German, French, Italian, Portuguese and most of Latin America use the comma too, and the thousands separator varies: 1.234,56, 1 234,56 in French sources, 1'234.56 in Swiss ones. The nasty case is a bare "1.234", which could be either. Parsing with the source's locale beats guessing from the string.

Thread Thread
 
galadev profile image
Gala •

Haha yes, the bare "1.234" gave me actual nightmares 😅 Pinning the locale per source is what finally kept me sane, and anything that reads differently under both conventions gets flagged for a second look.

The Swiss apostrophe is a new one for me though, thank you! 🙌 Have you hit it in real data? Now I'm curious which sources do that 👀