Web Scraper Outputs Just One Item Instead of Many

JavaScript
26

I don’t understand the connection you have between the link/category and image. There also isn’t a one-to-one relationship when it comes to the numbers of elements. There are 100 images per page (per category) and 39 categories in total.

If the second map (link/category) ran as many times as the first map (images) then you would just get the image from the images array inside the second map using its index.

const image = images[i];

return {
  link,
  cat,
  image
};

But this will only give you 39 images.

Can you show what you want the CSV to look like?

28

I did a search for the image names in another thread and found the site. Not sure about linking it here? It is a online store.

30

@makamo66 We merged two of your threads to give some more context. Please continue your current questions in this thread. Sorry for any confusion.

31

You merged together two different code bases. The first one console logs all the output and the second one saves to a CSV file. It seems more confusing now than it was before.

37

@makamo66 Well, in any case, if you have questions post them here. You also never answer the question I posted.

38

I don’t know what I want the CSV to look like. That’s why I didn’t answer.

39

Well, then I can’t really help you transform the data. But you see why I asked that question, right? You have many more images then links/categories.

40 
const images = cheerio('td.productListing-data > a > img', html)
  .map((i, image) => cheerio(image).attr('src'))

const img = images.each( (i, image) =>
  {image}
).get()

    return {
	  img,
    }

The code above resulted in the entire array being outputted to each cell of the spreadsheet.

41

I would expect the spreadsheet to have many more cells filled with images than with links/categories.

42

Even when I changed the names of the parameters, it still didn’t work.

43

So I tried to learn how to do this on my own by taking two video courses about node and cheerio on udemy.com but I haven’t gotten any better at it. The code produces an array of image URLs when I do this:

const request = require("request-promise");
const cheerio = require("cheerio");

const url = "https://www.example.com";

const scrapeResults = [];
async function scrapeJobHeader() {
  try {
    const htmlResult = await request.get(url);
    const $ = await cheerio.load(htmlResult);
	$("td.productListing-data > a ").each((index, element) => {
      const resultTitle = $(element).children("img");
	  
	  const img_url = resultTitle.attr("src");
	  
      const scrapeResult = { img_url };
    
      scrapeResults.push(scrapeResult);
    });
    return scrapeResults;
  } catch (err) {
    console.error(err);
  }
}

async function scrapeWebsite() {
  const jobsWithHeaders = await scrapeJobHeader();
 console.log(jobsWithHeaders);

}

scrapeWebsite();

but I get the error “TypeError: Cannot read property ‘replace’ of undefined” when I do this:

const img_url = resultTitle.attr("src").replace("images\\/more_color.png","");

The images that are returned include “images/more_color.png” but I just want to return the actual product images and not the pngs.

The results look like this:

{ img_url: 'images/more_color.png' },
  { img_url: undefined },
  { img_url: undefined },
  {
    img_url: 'images/20191206/thumb/AK1501-@RH-CRY-LOVE@22X06-825_3L@467400@200@01@200.jpg'
  },
  { img_url: 'images/more_color.png' },
  { img_url: undefined },
  { img_url: undefined },
  {
    img_url: 'images/20191206/thumb/AK1501-@GD-CRY-LOVE@22X06-825_3L@467399@200@01@200.jpg'
  },
  { img_url: 'images/more_color.png' },
  { img_url: undefined },
  { img_url: undefined },
  {
    img_url: 'images/20191206/thumb/AK1500-@GD-CRY-QUEEN@3X06-825_3L@467397@200@01@200.jpg'
  },
  { img_url: 'images/more_color.png' },
  { img_url: undefined },
  { img_url: undefined },

The next step would be to get rid of the undefined image URLs but I haven’t gotten that far yet.

44

I got rid of the undefined ones by using:

  if (img_url != undefined){
      scrapeResults.push(scrapeResult);
	 }
45

You can check the Truthiness of the img string and see if it does not end with .png before pushing to the array.

async function scrapeJobHeader() {
  try {
    const htmlResult = await request.get(baseURL);
    const $ = await cheerio.load(htmlResult);
    $('td.productListing-data > a ').each((index, element) => {
      const resultTitle = $(element).children('img');
      if (
        resultTitle.attr('src') &&
        !resultTitle.attr('src').endsWith('.png')
      ) {
        let img_url = resultTitle.attr('src');
        scrapeResults.push({ img_url });
      }
    });
    return scrapeResults;
  } catch (err) {
    console.error(err);
  }
}
46

Thank you lasjorg! That worked!

47

No problem, glad to help. I should have linked to the endsWith method docs.

I suggest you keep these three links handy, on the left-hand side is the list of all the methods on the different objects (Object, Array, String).



Happy coding!