Puppeteer Instagram Scrapping

Code Feedback
1

Hey guys , recently i start building a scrapping project with pupeteer. I didnt know where to upload it, since codepen cant run puppeteer, so i put the source code on github.
i would appreciate it If you could spare some time to check my project :slight_smile:

2

Hey there,

great work and cool idea, I like it!

My ideas:

  • you hardcoded all of your CSS selectors to find elements, e.g.
const followersCount = await page.evaluate(() =>
  document.querySelector('.k9GMp li a span').textContent 
);

Instagram probably uses React and its toolchain, so the CSS classes probably are created randomly, meaning on the next Instagram update your tool probably won’t work anymore, because the classes will have been changed.

I think you will have to find a better method to find the correct HTML elements.
If you don’t want to do this, I think a good next step would be to extract the CSS selectors into some variables at the top of your file so it will be easier for you to update the classes/selectors manually.

Keep up the good work!

3

thanks for the feedback!!
do you have any idea on how i can extract the css selectors in some different way and solve this problem?

4

I think the simplest solution is to search for a related text,
e.g. for the followersCount this would be followers.
This could break if they would translate followers based on your computer’s language settings or something like that.

An alternative solution would be to search for an href with /[username]/followers, because they probably don’t change url routes depending on your language.

Screenshot_2020-08-07_12-00-03

5

Hey after a while :stuck_out_tongue:

i have a question on this project. When i opened instagram now a window pops up with a button for accepting cookies. And if i dont click accept the rest of the code wont work. So i added that to my code.
My question now is. How can i tell my puppeteer code that IF this window pops up then click the accept button, otherwise move to the next step ?

6