How to web scrape RateMDs: Create a list of Doctors in any city

Chris Miranda -

RateMDs is a great way to find local doctors or specialists in your city.  It allows patients to leave reviews about doctors that are in different fields of practice.

It also allows practitioners to manage their listing where they can list their social media profiles, website, location and phone number.

RateMDs allows you to find a doctor or facility in several countries including Canada, United States, India, South Africa and many more.

This is a great way for patients to find the best doctors in the city and share their experiences with other patients.

But why would you want to scrape RateMDs to create a list of doctors and facilities?

We can give you three reasons to web scrape RateMDs

Lead generation

This is a big reason to use web scraping to create a list of doctors. You can use this list to help grow your company.

Say you’re a SaaS company and your target market is the medical industry. You can create a list of your target market in minutes.

You can create multiple lists for multiple cities in just minutes of web scraping. You can even expand your services to another country!

But this doesn’t just help SaaS companies, this can help medical tech/ products and b2b companies like digital marketing agencies.

Competitor analysis

Say you’re a doctor or a specialist yourself. You can easily scrape your competitors in your city. You can see their brand reputation, social media plan, website and targeted keywords.

Since RateMDs rates their practitioners, you can scrape the top 20 specialists and implement what they do into your practice.

Brand Reputation

Since patients can leave reviews, you can scrape yours in minutes. You can scrape both the good and the bad and see the places in your business plan where you can improve.

But let’s show how you can scrape RateMDs and create a list of doctors in minutes.

Scraping RateMDs Doctor’s Info

For this example, we will scrape “verified doctors” in Toronto. We will extract their name, RateMDs profile link, job title, number of reviews, Image URL and business website.

Getting started

  1. First, make sure to download and install ParseHub. We will use this free web scraper for this project.
  2. Open ParseHub, click on “New Project” and use the URL from RateMDs’s result page. The page will now be rendered inside the app.

Scraping RateMDs Results Page

  1. Once the site is rendered, a selection function will automatically be created. If not, you can click on the plus sign next to the page selection.
  2. Click on the first doctor’s name on the page. The name you’ve clicked will become green to indicate that it’s been selected.
  3. ParseHub will now suggest other elements you want to extract. The rest of the doctor’s names will be highlighted in yellow. Click on the second name on the list. Now all of the items will be highlighted in green.

4. On the left sidebar, rename your selection to Doctor_name. You will notice that ParseHub is now extracting the Doctor’s name and RateMDs profile URL for each doctor.

5. On the left sidebar, click the PLUS(+) sign next to the product selection and choose the Relative Select command.

6. Using the Relative Select command, click on the first doctor’s name on the page and then on their job title. You will see an arrow connect the two selections.

7. Expand the new command you’ve created and then delete the URL that is also being extracted by default.

8. Repeat steps 4 through 6 to also extract the number of reviews and images. Make sure to rename your new selections accordingly.

We have now selected all the data we wanted to scrape from the results page. Your project should now look like this:

Scraping the Doctor’s RateMDs Listing

Now, we will tell ParseHub to click on each of the doctors we’ve selected and extract additional data from each page. In this case, we will extract their company website.

First, on the left sidebar, click on the 3 dots next to the main_template text.

  1. Rename your template to doctor_results_page. Templates help ParseHub keep different page layouts separate.

2. Now use the PLUS(+) button next to the product selection and choose the “Click” command. A pop-up will appear asking you if this link is a “next page” button. Click “No” and next to Create New Template input a new template name, in this case, we will use Doctors_profile.

3. ParseHub will now automatically create this new template and render the doctor profile page for the first doctor on the list.

4. Click on the “Visit website” link. ParseHub will extract the doctor’s website URL.

Adding Pagination

If you were to run your project right now, you would only scrape the 1st results page. RateMDs only has 10 doctors per page, but we want to create a list that will give us multiple doctors maybe 50 – 100 depending on how many pages you want to scrape.

We have to tell ParseHub to click on the next button to scrape the following pages.

Let’s setup ParseHub to navigate to the next 10 results pages.

  1. On the left sidebar, return to the doctor_results_page template. You might also need to change the browser tab to the search results page as well.
  2. Click on the PLUS(+) sign next to the page selection and choose the Select command.

3. Then select the Next page link at the bottom of the RateMDs page. Rename the selection to next_button.

4. By default, ParseHub will extract the text and URL from this link, so expand your new next_button selection and remove these 2 commands.

5. Now, click on the PLUS (+) sign of your next_button selection and use the Click command.

6. A pop-up will appear asking if this is a “Next” link. Click Yes and enter the number of pages you’d like to navigate to. In this case, we will scrape 9 additional pages.

Your final project should look something like this

Running and Exporting your Project

Now our project is ready to scrape RateMDs. To do this, simply click on the left sidebar and click on the green “Get Data” button.

You’ll be brought to this page:

This is where you can test, run or schedule your project. For longer and bigger projects, we recommend doing a Test Run just to make sure your data will be extracted and formatted correctly.

But for this project, click on the “Run” button to begin your scrape.

Once ParseHub is done scraping the website, you will be notified by email and you’ll be able to download your extracted data as an Excel/CSV or as a JSON file.

Closing thoughts

Now you know how to scrape RateMDs to create a list of doctors and specialists in any city!

But ParseHub can scrape other websites to create a list for lead generation and competitor analysis. Read more on our other blogs:

How to Scrape Yellow Pages Data: Business Names, Addresses, Phone Numbers, Emails and more.

How to Scrape Yelp Reviews: a step-by-step guide

What will you scrape?

Happy Scraping!

Download ParseHub For free