You copy a selector from Chrome DevTools, run it through BeautifulSoup, and get None (or an empty list from select()). The selector looks right. The element is right there in the Elements panel.
Almost every time, the problem is the same: DevTools shows the live DOM after the browser has run JavaScript, fixed broken markup and applied your headers and cookies. requests gives you the raw bytes the server sent to a Python script. Those are two different documents.
I rebuilt the four most common versions of this on a local test server and ran them with Python 3.13.5, requests 2.34.2, beautifulsoup4 4.15.0, lxml 6.1.3 and html5lib 1.1. Every output below is copied from those runs.
Step 0: look at what you actually received
Before you change a single selector, print three things:
import requests
from bs4 import BeautifulSoup
r = requests.get(url, timeout=10)
print(r.status_code, r.url) # did you get a 200, and from the URL you expected?
print(len(r.text))
print(r.text[:500]) # what does the page really start with?
with open("got.html", "w", encoding="utf-8") as f:
f.write(r.text) # open this file and search it, not DevTools
Search got.html for the text you want. Where it shows up (or doesn't) tells you which of the cases below you're in.
Case 1: the content is rendered by JavaScript
My test page has an empty <div id="app"> and a script that fills it in with <span class="price">$24.00</span>. In a browser, DevTools shows the span. In Python:
soup = BeautifulSoup(r.text, "html.parser")
print(soup.find("span", class_="price"))
print("$24.00" in r.text, '<span class="price"' in r.text)
None
True True
That second line is the trap. A plain "$24.00" in r.text check says the text is there, so the selector must be wrong, right? No. The text is there as a string inside a script, not as an element. Ask BeautifulSoup where the matches live:
import re
hits = soup.find_all(string=re.compile(r"\$24\.00"))
print([h.parent.name for h in hits])
print(soup.select_one("#app"))
['script', 'script']
<div id="app"></div>
If every match has script as its parent, no selector will ever work. You have two honest options:
-
Read the data the page ships with. Many sites (Next.js, Nuxt, a lot of e-commerce themes) embed the data as JSON in a
<script type="application/json">orapplication/ld+jsontag. Parse that instead of the HTML:
import json
data = json.loads(soup.find("script", id="__NEXT_DATA__").string)
print(data["props"]["pageProps"]["products"][0])
{'name': 'Desk Lamp', 'price': '$24.00'}
This is usually more stable than scraping markup, because class names change with every redesign and the JSON shape changes far less often. Also check the Network tab (filter by Fetch/XHR): the page might load the same data from a JSON endpoint you can call directly.
- Use a real browser (Playwright or Selenium) when the data only exists after scripts run. It's slower and heavier, so try option 1 first.
Case 2: tbody is in DevTools but not in the HTML
This one catches people who right-click → Copy selector on a table row. My test table has no <tbody> in the source, like a lot of hand-written and server-generated HTML:
<table id="t"><tr><td>Python</td><td>3.13</td></tr><tr><td>Node</td><td>22</td></tr></table>
Browsers insert a <tbody> while parsing, so DevTools gives you table#t > tbody > tr. Here's that selector under each parser BeautifulSoup supports:
for parser in ["html.parser", "lxml", "html5lib"]:
soup = BeautifulSoup(r.text, parser)
print(parser, len(soup.select("table#t > tbody > tr")), len(soup.select("table#t tr")))
html.parser 0 2
lxml 0 2
html5lib 2 2
html.parser and lxml keep the source as written, so the copied selector finds nothing. html5lib follows the same HTML5 parsing rules as a browser and adds tbody, so it matches DevTools. The fix doesn't need a new parser, though: drop the strict > steps that came from DevTools and use table#t tr, which works with all three.
The general lesson: a selector copied from DevTools describes the browser's repaired tree. Trim it down to the parts that are really in got.html.
Case 3: multi-class elements and class_
Given <div class="price sale">$19.99</div>:
print(soup.find("div", class_="price"))
print(soup.find("div", class_="price sale"))
print(soup.find("div", class_="sale price"))
print(soup.select_one("div.sale.price"))
<div class="price sale">$19.99</div>
<div class="price sale">$19.99</div>
None
<div class="price sale">$19.99</div>
A single class name matches any element that has that class. A string with a space in it is compared to the whole class attribute, in that exact order, so "sale price" misses. If you need several classes, use a CSS selector (div.sale.price), where order doesn't matter. And it's class_, with the underscore: class is a reserved word in Python.
Case 4: the server sent you a different page
My /guarded route acts like many real sites: it returns a "Please enable JavaScript" page with a 403 when it sees the default requests User-Agent.
r = requests.get(url, timeout=10)
print(r.status_code, r.request.headers["User-Agent"])
print(BeautifulSoup(r.text, "html.parser").find("td"))
403 python-requests/2.34.2
None
Your selector is fine. You're parsing an error page. Two habits make this obvious instead of mysterious:
r.raise_for_status()
requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: http://127.0.0.1:18923/guarded
Call it right after every request, so a 403, 404 or 500 fails loudly at the request instead of quietly as None three functions later. And check r.url too: a 200 from a login or cookie-consent page you got redirected to is just as wrong.
On my test server, sending a normal browser User-Agent returned the real page (200, and find("td") gave <td>Python</td>). On real sites, treat a block as information: read the site's terms and robots.txt, use an official API if there is one, and keep your request rate low. Don't treat "send more headers" as a way to get around a site that has said no.
The checklist
When find() returns None:
-
raise_for_status()and printr.url. Are you even on the right page? - Save
r.textto a file and search it. Is the text there at all? - If it's only inside script tags, look for embedded JSON or the XHR endpoint before reaching for a headless browser.
- Remove
tbodyand long>chains copied from DevTools. - For several classes, use
select("tag.a.b")instead ofclass_="a b". - Guard the result before you call
.texton it, so a missing element gives you a clear message instead ofAttributeError: 'NoneType' object has no attribute 'text':
el = soup.select_one("div.sale.price")
if el is None:
raise ValueError("price not found. Check got.html: layout changed or the page is JS-rendered")
price = el.get_text(strip=True)
If you're new to scraping, I wrote a longer walkthrough on Blogs World covering setup, CSS selectors, pagination and saving to CSV: How to Scrape a Website with Python and BeautifulSoup. That NoneType error from step 6 is also covered in Common Python Errors and How to Fix Them. When the data turns out to be embedded JSON, the free JSON formatter makes it much easier to find the path you need before you write data["props"][...].
Top comments (0)