There are multiple ways to deal with pagination, but how about dealing with an infinite scroll?
Today we’ll show you how you can scrape a website with infinite scroll (it’s pretty easy).
So let’s dive right into it!
Web scraping websites with infinite scroll
For this example, you will need a powerful web scraping tool. We recommend ParseHub! It’s free to use and has a suite of features we think you’ll enjoy!
We’ll be scrapping our blog for this web scraping project!
So let’s get started!
- Install and open ParseHub. Click on “new project” and enter the URL for the page you will be scraping. Once submitted the URL will render inside the app.
- A select command will automatically be created, start by clicking on the first blog title on the page. It will be highlighted in green to indicate that it’s been selected. The rest of the blog posts on the page will be highlighted in yellow. In the left sidebar, rename your selection to “blog_title”.
- Now click on the second blog title on the page to select them all. They should all be highlighted in green. If not, continue to click on the blog titles to teach ParseHub what to extract.
- Now we can extract more data from this page. Let’s start with the description. Use the PLUS (+) sign next to your “blog_title” selection and choose the Relative Select command.
- Using the Relative Select command, click on the first blog title on the list and then on the date. An arrow will appear to show the association you’re creating. Rename your new selection to “description"
- You can repeat steps 4-5 to extract additional information like blog author and length. You can choose to scrape the blog image if you choose to.
Dealing with Infinite Scroll
Now we’ll show you how to deal with a website that is infinite scroll
- To do this, click on the PLUS + sign beside the page selection and click select. You will need to select the main element to this, in this case, it will look like this.
- Once you have the main Div clicked you can add the scroll function. To do this, on the left sidebar, click the PLUS (+) sign next to the main selection, click on advanced, then select the scroll function.
- You will need to tell how long the software to scroll, depending on how big the blog is you may need a bigger number. But for now, let’s put it 5 times and make sure it's aligned to the bottom.
Simply drag your new scroll command and bring it to the top of the page, it should look like this now:
Running Your Scrape
It’s now time to run our scrape job and extract all the data we have selected.
Start by clicking on the green Get Data button on the left sidebar. Here you will be able to Test, Run or Schedule your project. In this case, we will run it right away.
ParseHub will now go and scrape the data you’ve selected. You will be notified when it’s done.
Once the scrape is completed, you will be able to download your data as a CSV or JSON file.
Closing thoughts
Now you know how to scrape websites with an infinite scroll! Pretty easy huh?
While we understand that not all websites are infinite scroll, you can read our other guide on dealing with pagination or multiple pages
