Web Scraper Outputs Just One Item Instead of Many

JavaScript
21

I could loop through the array of course with a for statement and an index but I don’t see how I would access the results in the return statement.

22

I did this but of course it didn’t work:

for (i = 0; i < 7; i++) {
	img = images[i];
}  
    return {
	  img,
    }
23

This didn’t work either:

var i = 0;
while(img = images[i++]){
    return {
	  img,
    }
}
24

Trying the .each method now.

25

Tried this

const images = cheerio('td.productListing-data > a > img', html)
.each( (i, image) => {cheerio(image).attr('src')})
.get()

but got error:

UnhandledPromiseRejectionWarning: TypeError: Converting circular structure to JSON

26

I don’t understand the connection you have between the link/category and image. There also isn’t a one-to-one relationship when it comes to the numbers of elements. There are 100 images per page (per category) and 39 categories in total.

If the second map (link/category) ran as many times as the first map (images) then you would just get the image from the images array inside the second map using its index.

const image = images[i];

return {
  link,
  cat,
  image
};

But this will only give you 39 images.

Can you show what you want the CSV to look like?

27

@lasjorg How are you seeing 100 images per page and 39 categories in total? What url are you using to see this?

28

I did a search for the image names in another thread and found the site. Not sure about linking it here? It is a online store.

29
30

@makamo66 We merged two of your threads to give some more context. Please continue your current questions in this thread. Sorry for any confusion.

1 Like
31

You merged together two different code bases. The first one console logs all the output and the second one saves to a CSV file. It seems more confusing now than it was before.

32

It is all apart of the same project, so we wanted to keep it together.

33
34
35
36
37

@makamo66 Well, in any case, if you have questions post them here. You also never answer the question I posted.

38

I don’t know what I want the CSV to look like. That’s why I didn’t answer.

39

Well, then I can’t really help you transform the data. But you see why I asked that question, right? You have many more images then links/categories.

40 
const images = cheerio('td.productListing-data > a > img', html)
  .map((i, image) => cheerio(image).attr('src'))

const img = images.each( (i, image) =>
  {image}
).get()

    return {
	  img,
    }

The code above resulted in the entire array being outputted to each cell of the spreadsheet.