I want to learn to webscrape for data analysis. how should i go about it?

Python
1

Hi all,

I’m looking for advice about the best way to learn to web scrape using scrapingdog or some other service.

The project I’m working on is to do an analysis of a particular housing market. I need to scrape websites such as Realtor and Zillow that seem to be pretty good at stopping bots.

I am competent in SQL and PowerBI. I know a teeny weeny bit of javascript and html/css.

1- what is the best way to go about learning how to scrape? My thought is to learn some python / beautiful soup, as that seems the most preferred way to scrape websites.

2- is it unethical to scrape these sites if I am only going to use this information for my own purposes?

Appreciate any guidance on this.

1 Like
2

There is actually a python class in FCC that goes into this idea with some detail. It is the Scientific Computing with Python class. The specific lesson(s) that discuss web scraping are

Edit: the teacher mentions some of the concerns with scraping such as various protections sites have against it. There is also a quantity of documents on this subject here

3

Great. I’ll check it out. Thank you.

1 Like
4

Make a search engine
Try scraping on Wikipedia

5

This topic was automatically closed 182 days after the last reply. New replies are no longer allowed.