DEV Community

Devil Scrapes profile picture

Devil Scrapes

View all public scrapers @ https://apify.com/devilscrapes

Joined Joined on 
We doubled the memory, cut runtime 33%, and raised the cost 41%
devil_scrapes profile

We doubled the memory, cut runtime 33%, and raised the cost 41%

#python #devops #apify #webscraping
Comments
3 min read
We swept 288 scrapers for columns that are empty in every row. 35 had one.
devil_scrapes profile

We swept 288 scrapers for columns that are empty in every row. 35 had one.

#webscraping #python #testing #apify
Comments
4 min read
A note said the run scraped 0 rows. The platform's own record said 6.
devil_scrapes profile

A note said the run scraped 0 rows. The platform's own record said 6.

#python #api #fintech #debugging
Comments
4 min read
Our scraper returned 50 rows and 39 distinct IDs — count identities, not rows
devil_scrapes profile

Our scraper returned 50 rows and 39 distinct IDs — count identities, not rows

#python #webscraping #api #apify
Comments
4 min read
Our QA run passed with rows in the dataset and still proved nothing about pagination
devil_scrapes profile

Our QA run passed with rows in the dataset and still proved nothing about pagination

#python #webscraping #testing #apify
Comments
4 min read
Our Actor got 403'd on every page and still reported SUCCEEDED with zero rows
devil_scrapes profile

Our Actor got 403'd on every page and still reported SUCCEEDED with zero rows

#python #webscraping #debugging #apify
Comments
4 min read
A CONNECT 407 looked like a proxy-entitlement problem. It was a country pin with no group name.
devil_scrapes profile

A CONNECT 407 looked like a proxy-entitlement problem. It was a country pin with no group name.

#python #webscraping #apify #proxy
Comments
4 min read
Bing's ad-library search box accepts any query and returns the exact same ads
devil_scrapes profile

Bing's ad-library search box accepts any query and returns the exact same ads

#python #api #webscraping #marketing
Comments
4 min read
44 rows of Apple's 'Icon + Asset Ad' format, and not one had an asset. Bug or null?
devil_scrapes profile

44 rows of Apple's 'Icon + Asset Ad' format, and not one had an asset. Bug or null?

#python #webscraping #api #debugging
Comments
4 min read
We queried Wallapop for zzznonsensequery9999xyz. Zero results is what proved the search actually works.
devil_scrapes profile

We queried Wallapop for zzznonsensequery9999xyz. Zero results is what proved the search actually works.

#python #webscraping #testing #apify
Comments
4 min read
pump.fun's search parameter accepts any string and returns the same feed. We shipped without it.
devil_scrapes profile

pump.fun's search parameter accepts any string and returns the same feed. We shipped without it.

#python #api #crypto #webscraping
Comments
5 min read
OLX Brazil embeds a decoy listings array with the exact shape of the real one. We had to prove which was which.
devil_scrapes profile

OLX Brazil embeds a decoy listings array with the exact shape of the real one. We had to prove which was which.

#webscraping #python #brazil #classifieds
Comments
5 min read
We doubled Mercado Libre's scraper memory and clearance went from 50% to 90% - with zero code changes.
devil_scrapes profile

We doubled Mercado Libre's scraper memory and clearance went from 50% to 90% - with zero code changes.

#webscraping #python #ecommerce #mexico
Comments
6 min read
Kleinanzeigen has two page-2 URLs. One is a redirect trap. We verified the one we shipped actually advances.
devil_scrapes profile

Kleinanzeigen has two page-2 URLs. One is a redirect trap. We verified the one we shipped actually advances.

#webscraping #python #germany #classifieds
Comments
6 min read
We shelved a finished scraper because the site ignored its own search box. Now we test for it first.
devil_scrapes profile

We shelved a finished scraper because the site ignored its own search box. Now we test for it first.

#api #python #webscraping #crypto
Comments
5 min read
FurnishedFinder Scraper: caps every search at 50, and the fix isn't in the docs
devil_scrapes profile

FurnishedFinder Scraper: caps every search at 50, and the fix isn't in the docs

#webscraping #python #graphql #api
Comments
5 min read
Our QA harness certified this Actor at 4096 MB. It ships on 512.
devil_scrapes profile

Our QA harness certified this Actor at 4096 MB. It ships on 512.

#testing #python #webscraping #devops
Comments
3 min read
A run delivered 0 rows and still got billed $0.30 — the bug was ours, not Bayut's
devil_scrapes profile

A run delivered 0 rows and still got billed $0.30 — the bug was ours, not Bayut's

#webscraping #python #playwright #realestate
Comments
3 min read
The TikTok 'block' that was actually Camoufox running out of memory
devil_scrapes profile

The TikTok 'block' that was actually Camoufox running out of memory

#webscraping #python #api #tiktok
Comments
3 min read
A datacenter IP gets 0 listings; a residential one gets 12. Same build.
devil_scrapes profile

A datacenter IP gets 0 listings; a residential one gets 12. Same build.

#webdev #python #scraping #api
Comments 1
3 min read
Our scraper went 0-for-19 against Cloudflare. One boolean took it to 9-for-10.
devil_scrapes profile

Our scraper went 0-for-19 against Cloudflare. One boolean took it to 9-for-10.

#webscraping #python #devops #api
Comments
5 min read
We cleared PeoplePerHour 4/4 locally. The first cloud run got HTTP 202 from every URL.
devil_scrapes profile

We cleared PeoplePerHour 4/4 locally. The first cloud run got HTTP 202 from every URL.

#webscraping #python #devops #api
Comments
4 min read
A Dockerfile typo turned 33 real Depop rows into a silent, SUCCEEDED zero
devil_scrapes profile

A Dockerfile typo turned 33 real Depop rows into a silent, SUCCEEDED zero

#webscraping #python #docker #ecommerce
Comments
4 min read
An isinstance(str) check that was always False capped every Facebook Page at exactly 3 posts
devil_scrapes profile

An isinstance(str) check that was always False capped every Facebook Page at exactly 3 posts

#webscraping #python #api #debugging
Comments
4 min read
Alibaba renamed one JavaScript variable and our price field silently dropped to 0%
devil_scrapes profile

Alibaba renamed one JavaScript variable and our price field silently dropped to 0%

#webscraping #python #javascript #ecommerce
Comments
5 min read
A 586 KB HTTP 200 from YouTube that contains zero videos
devil_scrapes profile

A 586 KB HTTP 200 from YouTube that contains zero videos

#webscraping #python #api #youtube
Comments
4 min read
Our test asked for 4 branches, ran 1, and reported PASS
devil_scrapes profile

Our test asked for 4 branches, ran 1, and reported PASS

#testing #python #webscraping #devops
Comments
4 min read
LinkedIn has an /api/ endpoint that needs no login and returns HTML
devil_scrapes profile

LinkedIn has an /api/ endpoint that needs no login and returns HTML

#webscraping #python #api #linkedin
Comments
3 min read
Our release gate read a docstring and blocked the deploy over a word
devil_scrapes profile

Our release gate read a docstring and blocked the deploy over a word

#devops #python #testing #webscraping
Comments
4 min read
No introspection, no allowlist: reconstructing Whatnot's GraphQL queries from a compiled AST
devil_scrapes profile

No introspection, no allowlist: reconstructing Whatnot's GraphQL queries from a compiled AST

#graphql #python #webscraping #api
Comments
4 min read
Zero results, five rows: the Flippa search response that lies about matching nothing
devil_scrapes profile

Zero results, five rows: the Flippa search response that lies about matching nothing

#python #webscraping #api #webdev
Comments
4 min read
A 200 that isn't reachability: SpareRoom's soft-redirect and the empty run nobody notices
devil_scrapes profile

A 200 that isn't reachability: SpareRoom's soft-redirect and the empty run nobody notices

#python #webscraping #api #webdev
Comments
5 min read
A 404 is not a failure: what Discord's invite endpoint taught us about billing
devil_scrapes profile

A 404 is not a failure: what Discord's invite endpoint taught us about billing

#python #api #webscraping #discord
5
Comments
5 min read
A 25%-failing test is not flaky. It's a bug reporting itself honestly.
devil_scrapes profile

A 25%-failing test is not flaky. It's a bug reporting itself honestly.

#python #testing #webscraping #debugging
5
Comments
4 min read
A bare useApifyProxy: true resolves to datacenter. Patreon 403s every one of those.
devil_scrapes profile

A bare useApifyProxy: true resolves to datacenter. Patreon 403s every one of those.

#webscraping #python #api #cloudflare
Comments
4 min read
grep '"aggregate_rating":' finds zero matches on Zomato. The field is there, just double-escaped.
devil_scrapes profile

grep '"aggregate_rating":' finds zero matches on Zomato. The field is there, just double-escaped.

#webscraping #python #javascript #api
Comments
4 min read
?q=chairs on Chairish isn't a search - it's a redirect to a category page wearing a search URL.
devil_scrapes profile

?q=chairs on Chairish isn't a search - it's a redirect to a category page wearing a search URL.

#webscraping #python #api #ecommerce
Comments
5 min read
?q=chairs on Chairish isn't a search - it's a redirect to a category page wearing a search URL.
devil_scrapes profile

?q=chairs on Chairish isn't a search - it's a redirect to a category page wearing a search URL.

#webscraping #python #api #ecommerce
Comments
5 min read
?page=2 on Immowelt returns page 1, byte for byte. It's a button, not a route.
devil_scrapes profile

?page=2 on Immowelt returns page 1, byte for byte. It's a button, not a route.

#webscraping #python #api #realestate
Comments
5 min read
window.__NUXT__ is not JSON. It's a function you have to run first.
devil_scrapes profile

window.__NUXT__ is not JSON. It's a function you have to run first.

#webscraping #python #javascript #api
Comments
4 min read
Our build wave shipped four scrapers that scraped nothing
devil_scrapes profile

Our build wave shipped four scrapers that scraped nothing

#testing #python #webscraping #devops
Comments
4 min read
A one-step probe told us YouTube was blocked. It was asking the wrong endpoint.
devil_scrapes profile

A one-step probe told us YouTube was blocked. It was asking the wrong endpoint.

#webscraping #python #api #youtube
Comments
4 min read
The top-earning product on Indie Hackers reports $10,000,000,000,000,000 a month
devil_scrapes profile

The top-earning product on Indie Hackers reports $10,000,000,000,000,000 a month

#webscraping #python #saas #data
Comments
3 min read
Grailed's __NEXT_DATA__ is empty. The listings come from Algolia, and the key is on the page.
devil_scrapes profile

Grailed's __NEXT_DATA__ is empty. The listings come from Algolia, and the key is on the page.

#webscraping #python #algolia #nextjs
Comments
3 min read
Coursera ships two big JSON blobs on every search page. Neither has the results.
devil_scrapes profile

Coursera ships two big JSON blobs on every search page. Neither has the results.

#webscraping #python #html #api
Comments
4 min read
dict.get(key, [None])[0] is not safe, and our green test suite agreed with us
devil_scrapes profile

dict.get(key, [None])[0] is not safe, and our green test suite agreed with us

#python #testing #webscraping #debugging
Comments
4 min read
Pinterest's ad library endpoint was hiding behind one missing curl flag
devil_scrapes profile

Pinterest's ad library endpoint was hiding behind one missing curl flag

#webscraping #python #api #marketing
Comments
4 min read
Google News RSS has no cursor, so the query IS the cursor
devil_scrapes profile

Google News RSS has no cursor, so the query IS the cursor

#news #webscraping #python #api
Comments
4 min read
Trustpilot needs a browser exactly once — the WAF token replays onto curl
devil_scrapes profile

Trustpilot needs a browser exactly once — the WAF token replays onto curl

#webscraping #python #aws #debugging
Comments
4 min read
Meta's Ad Library 403 is 481 bytes of instructions, not a block
devil_scrapes profile

Meta's Ad Library 403 is 481 bytes of instructions, not a block

#webscraping #python #api #debugging
Comments
4 min read
Indeed 403s Chrome on every proxy tier — and our laptop said it was fine
devil_scrapes profile

Indeed 403s Chrome on every proxy tier — and our laptop said it was fine

#webscraping #python #debugging #devops
Comments
4 min read
LinkedIn Ad Library Caps at 24 Ads — Here's the Real Pagination Endpoint
devil_scrapes profile

LinkedIn Ad Library Caps at 24 Ads — Here's the Real Pagination Endpoint

#webscraping #python #api #apify
Comments
7 min read
Fiverr Gig Scraper: One Listing, Two Rows, Only One Says Sponsored
devil_scrapes profile

Fiverr Gig Scraper: One Listing, Two Rows, Only One Says Sponsored

#webscraping #python #api #apify
Comments
5 min read
Why 43% of Yahoo Finance's 1-Minute Candles Come Back Null
devil_scrapes profile

Why 43% of Yahoo Finance's 1-Minute Candles Come Back Null

#webscraping #python #api #apify
Comments
5 min read
CoinMarketCap's lastUpdated Lies About Your EUR Price
devil_scrapes profile

CoinMarketCap's lastUpdated Lies About Your EUR Price

#webscraping #python #api #apify
Comments
5 min read
Binance's api.binance.com Blocks Every US IP — Its Own Data Mirror Doesn't
devil_scrapes profile

Binance's api.binance.com Blocks Every US IP — Its Own Data Mirror Doesn't

#webscraping #python #api #apify
Comments
5 min read
The Invisible Character Hiding In We Work Remotely's Job Descriptions
devil_scrapes profile

The Invisible Character Hiding In We Work Remotely's Job Descriptions

#webscraping #python #api #apify
Comments
3 min read
Two Thirds Of Stocktwits Messages Have No Sentiment — And The API Says So Explicitly
devil_scrapes profile

Two Thirds Of Stocktwits Messages Have No Sentiment — And The API Says So Explicitly

#webscraping #python #api #apify
Comments
3 min read
Remotive's Salary Field Mixes Two Different Comma Conventions
devil_scrapes profile

Remotive's Salary Field Mixes Two Different Comma Conventions

#webscraping #python #api #apify
Comments
3 min read
Mastodon's acct Field Hides the Domain You Already Know
devil_scrapes profile

Mastodon's acct Field Hides the Domain You Already Know

#webscraping #python #api #apify
Comments
3 min read
loading...