You open a website, read a product price, and move on. Somewhere else, software may be doing something very different. It opens that same page, reads the code behind it, pulls out useful information, and stores the result. That process is called web scraping.
What Happens When a Website Gets Scraped?
A scraper usually starts with a web address. It sends a request to the website, much like a browser does when you type a URL and hit Enter. The server responds with the page content. The scraper then looks through that response to find the pieces it actually wants.
Reading the Page
• A URL goes in first, which sounds almost too simple until you’re dealing with thousands of pages.
• The website sends back page data, and the scraper works through that response.
• Specific information gets pulled from the page. The exact method depends on how the site is built.
• The collected data is stored somewhere else, often in a format that’s easier to search or analyse later.
How Do Scrapers Move From Page to Page?
One page usually isn’t the whole job. A scraper follows links or works through a set of known URLs. It might start with a category page and then visit individual product pages. From there, it keeps moving.
What About JavaScript Websites?
This is where things get more interesting. Some modern websites don’t put all their content directly into the first HTML response. JavaScript runs in the browser and fetches extra information afterward.
A basic scraper can struggle here. A browser-based scraper handles it differently by opening the page and allowing scripts to run before extracting the content. That’s slower, but for complicated websites, it’s often the approach that actually works.
A Small Example From Real Life
Raj once used a scraper to track prices across a few shopping pages. He had been reopening the same five tabs every morning, so he set up a script to collect the prices for him instead. The script checked each page, picked out the price, and saved the results.
Nothing fancy. Just fewer tabs.
Why Don’t Websites Simply Stop Scraping?
They try. Websites can watch request patterns and block traffic that looks unusual. Some require users to complete challenges before continuing. Others limit how quickly pages can be requested.
But stopping scraping completely is difficult because normal users and automated visitors can look surprisingly similar. A browser loads a page. A scraper can load a page too. The difference often appears in the pattern of behaviour rather than the request itself.
And that’s really the interesting part. Scraping isn’t some secret trick hidden behind the internet. It’s software reading information that a website sends to a visitor, then doing that job repeatedly at a scale a person never would.