Scraping check24.de without becoming the villain

At GreenPocket I was tasked with mining German energy-tariff data from check24.de — a comparison site with millions of price points across thousands of postcodes. The commercial team needed a regional view of tariff dynamics to orient market strategy. The naïve approach is to spin up fifty concurrent scrapers and torch the source site. The naïve approach is also a fast way to get sued and end up as the cautionary tale in your industry's next webinar.

The project started with a legal review. German law has a specific exception for "data analysis for commercial research" under §44b UrhG, provided three conditions hold: the data is publicly accessible without circumventing technical measures; the use is for analytical purposes, not republication; and you respect the source's rate limits and robots.txt. The legal review fit on one page. It set the technical constraints for the entire project.

The scraper itself was almost boring. Python, async with httpx, one request per second per postcode, a polite User-Agent identifying the company and a contact email. Retries on 429 with exponential backoff. A persistent state file so an interrupted run resumed from the right postcode, never duplicating requests. The full crawl took eleven days for ten thousand postcodes. We could have done it in eleven hours by parallelising. Eleven days was the correct answer, because eleven hours would have looked like a denial-of-service to check24's rate-limiter.

The interesting work was downstream. Raw tariff data is messy: providers rename plans, plans get withdrawn mid-week, the same plan appears under three slightly different SKUs depending on the comparison page. We built a normalisation layer that produced a canonical "tariff key" — postcode + provider + plan-family + contract-type. The canonical key was what made longitudinal analysis possible. Without it, every weekly snapshot looked like a different population of plans.

The final deliverable was a regional dashboard showing where the commercial team should focus customer-acquisition spend, with weekly refresh. The technical work was a fraction of the project. The legal review, the rate-limit discipline, the normalisation layer and the documentation that proved we were doing all of this — that was the project.

The lesson is one most ML engineers learn late: in data work, "we have legal authority to operate on this data" is a first-class engineering requirement, not a footnote. The most elegant scraper in the world, run without a §44b review and a rate-limit discipline, exposes the company to liability that dwarfs the value of the data. Pretty stack, no permission, no production. The whole craft is doing it cleanly enough that the legal team signs off as a formality.