How to collect and download big data sets using a web scraper
For many data scientists to complete their tasks and research, they will need to collect the data first. There are many ways to collect data online. One way data scientists can collect big data is from websites that display public data to use.
You can use websites like:
These data set libraries have valuable information you can use for your research and development!
We are ParseHub, and we’re going to show you how you can collect and download big data sets using a free web scraping tool.
Do note that you should only scrape data that is publicly available and can be accessed by anyone.
Scraping a data library website like Data Description
For this big data project, we are going to extract data sets from the website datadescription.com. They have a library of data sets you can use for your research and development. It’s a useful website data scientists can use to collect data they need for their project. You will also be able to extract the download link for the TXT file of the data.
To get started, you will need to download a free web scraper. We think you’ll enjoy ParseHub! It’s easy to use, cloud-based scraping, powerful and includes other features we think you’ll find useful.
So let’s get started
If you want to follow along you can use the following link.
Scraping big data sets
- Install and Open ParseHub, click on “New Project” and enter the URL you will be scraping. In this case, we are scraping the data sets that have a statistical method of correlation. The page will now render inside of the app.
- A select command will automatically be created, (if not, just click on the PLUS (+) next to the page to create one). Make your first selection by clicking on the first headline on the list. Once selected, it will turn green. ParseHub will now suggest the other elements you want to extract in yellow, in this case, the other headlines.
- Now click on the data headline that is in yellow. ParseHub is now extracting all data headlines on the list.
- ParseHub is now extracting the data headline and the big data information page link for each data set on the page. Let’s extract more data. Start by clicking on the PLUS(+) sign next to your data heading selection and click on the “Relative Select” command.
- Now click on the first data headline that is highlighted in orange on the page and then on the Methods. An arrow will appear to show the association you’re creating. On the left sidebar, rename your selection to “methods”.
- Repeat steps 4-5 to select and extract more data from this page. We will repeat these steps and extract the source, the number of cases, except, and the download link.
Your project should look like this:
Adding pagination to grab more data sets
Right now ParseHub is only extracting data sets on the first page, but let's grab from multiple pages! If you want to extract multiple pages of bid data, we will need to add pagination.
1. Now click on the PLUS(+) sign next to your “page” selection and choose the select command.
2. Scroll down to the bottom of the page and click on the “Next >” button. Rename your selection to “next_page”
3. Expand your next_page command and delete both commands that are being extracted
4. Now select the PLUS(+) command next to your “next_page” selection and choose the “click” command.
5. A pop-up will appear asking you if this a “next page” link. Click on “Yes” and enter the number of additional pages you’d like to scrape. In this case, we will scrape 4 more pages.
Running your scrape Project
To do this, click on the green “Get Data” button in the left sidebar. Here, you can test, run or schedule your scrape.
In this case, we will run it right away. ParseHub is now off to scrape the data you have selected from your big data website.
Once ParseHub is done extracting the data, you can download the file in CSV/ Excel file. You'll have the download link for each data on the exported file.
Closing thoughts
Scraping these big data libraries can give you valuable information. Whether you're using the data for product development, industry insights, or market research, a powerful web scraper will make collecting data a lot more efficient and effective.
You can then use the downloaded data to help you make any research articles, presentation or valuable decisions on investments!
What will you use the data for?
Happy Scraping!