Hi using Python 3.8 on windows 10. Would like to learn to scrape both html and dynamic javascript sites what would be the recommended package(s) to install? BeautifulSoup, Scrapy or Selenium. Please could you also point me to a good intro for the one recommended.
Thanks
Hello, david.
In the past, I have used BeautifulSoup, and Selenium. I would say that those two are good for different purposes. BeautifulSoup is useful for webscraping and parsing the data. Selenium is useful for automated websurfing.
So, for webscraping, I would recommend BeautifulSoup. I found the bs4 Documentation poorly formatted, but the module is easy enough to get going from reading it.
Otherwise, you can watch some tutorials: freeCodeCamp - BeautifulSoup tutorial
One thing to note, a “dynamic javascript site” is just an HTML site; the content rendered is always HTML.
Finally, be careful to abide to website’s content use and privacy terms, when webscraping, as many sites do not allow webscraping.
Hope this helps