I want to learn to webscrape for data analysis. how should i go about it?

Hi all,

I’m looking for advice about the best way to learn to web scrape using scrapingdog or some other service.

The project I’m working on is to do an analysis of a particular housing market. I need to scrape websites such as Realtor and Zillow that seem to be pretty good at stopping bots.

I am competent in SQL and PowerBI. I know a teeny weeny bit of javascript and html/css.

1- what is the best way to go about learning how to scrape? My thought is to learn some python / beautiful soup, as that seems the most preferred way to scrape websites.

2- is it unethical to scrape these sites if I am only going to use this information for my own purposes?

Appreciate any guidance on this.

1 Like

There is actually a python class in FCC that goes into this idea with some detail. It is the Scientific Computing with Python class. The specific lesson(s) that discuss web scraping are

Edit: the teacher mentions some of the concerns with scraping such as various protections sites have against it. There is also a quantity of documents on this subject here

Great. I’ll check it out. Thank you.

1 Like

Make a search engine
Try scraping on Wikipedia

This topic was automatically closed 182 days after the last reply. New replies are no longer allowed.