Web scraping Forums - Using a forum web scraper

Chris Miranda -

There are many forums for many different topics and industries. You can extract valuable data and find ideas of what people are asking, and create content that your competitors aren’t targeting!

This is a great way to find low competitive and zero searched volume keywords!

Well show you how you can use a forum web scraper like ParesHub to extract questions, topics and backlink opportunities in just a few minutes!

Scraping forum data in just a few minutes

For this example, we’ll scrape warrior forum for growth hacking tips and questions

Now it’s time to start setting up our web scraping project.

  1. Install and open ParseHub. Click on “new project” and enter the URL for the page you will be scraping. In this case, we will be scraping growth hacking questions and tips from warrior forum. Once submitted the URL will render inside the app.
  1. A select command will be created by default, start by clicking on the first question on the page to select it. It will be highlighted in green to indicate that it’s been selected. The rest of the questions on the page will be highlighted in yellow. In the left sidebar, rename your selection to “question”.
  1. Now click on the second question on the page to select them all. They will all now be highlighted in green.
  2. We can now extract more data from this page. Let’s start with the number of replies for each post. Use the PLUS (+) sing next to your “question” selection and choose the Relative Select command.
  3. Using the Relative Select command, click on the first question on the list and then on the number of replies under it. An arrow will appear to show the association you’re creating. Rename your new selection to “replies”.
  1. You can repeat steps 4-5 to extract other data like date of the last reply

Your project should now look like this:

We will now extract even more pages from this Forum

Adding pagination

ParseHub is currently only extracting the details we’ve selected from the first page of search results. We will now set it up to extract data from further pages of search results.

  1. Click on the PLUS(+) sign next to your page selection and click on the “next page: link at the bottom of the page. Rename your selection to next.
  1. Click on the Expand icon next to your next command.
  2. Delete both extractions below the next command.
  3. Click on the PLUS(+) sign next to the next command and choose the “click” command.
  4. A pop-up will appear asking you if this is a “next page” link. Click on Yes and enter the number of times you’d like to repeat your scrape. To scrape 3 pages, we will need 2 repetitions.

Running your scrape

You can now run your web scraping project and download the data you have selected.

To do this, click on the green “Get Data” button in the left sidebar. Here you can choose to Test, Run or Schedule your scrape project.

For larger projects, we recommend running a test scrape. In this case, we will run it right away.

Closing thoughts

Now you know how to use a forum web scraper to scrape any forum.

If you want to learn how to scrape bigger forums like Quora and Reddit, you can continue reading our guides here: