Scraping product prices from protected sites

[Helios]

I'm working on a small price comparison project and need to collect publicly listed prices from several online stores. My Python script works fine on some websites, but others use Cloudflare and keep returning 403 errors or verification pages instead of product data. I've tried switching to Selenium, but the results are still inconsistent. What tools or browser automation setups are worth trying for sites with this kind of protection?

 

[FallenAngel]

A price checker gets old fast when half the stores work normally and the other half need babysitting every time the script runs. I wouldn't make Selenium handle everything just because plain requests stopped working. Check first whether those stores expose product feeds, APIs or structured data, and keep the browser only for pages that genuinely need it. Cache the last good price too, so one failed check doesn't spoil the whole run.

 

[Sandman]

This is usually the point where changing one library for another stops being the whole answer. One store may let the same browser through all day, while another starts throwing verification pages after a few visits even though the script hasn't changed. Cloudflare browser rendering is another option to look into, but it's also worth comparing how different scraping tools handle protected sites. This article covers Selenium, Playwright and their behavior with Cloudflare: gologin.com/blog/best-web-scraping-tools-bypassing-cloudflare/ . Pick one of the stores that gives you the most trouble and use it as a test case. Run the same setup against it for a few days, because something that works once isn't much use for a price tracker.