SetTimeout / setInterval on row by row data output

Hello guys

Below is a code snippet of a larger script. The snippet reads data from a csv file and captures the start and end post code for input into a function that calculates the distance between the two postcodes via two external api’s.

The problem i have is that the data is captured and sent to the function too quickly and returns false data.

I have tried using a setTimeout and setInterval as the data is sent to the function but all i get is a delay before all the data is sent to the function.

I would appreciate a little input into how to delay each row and therefore a delay in each input to the function.

Thanks for any input.

Simon

/ Get start and end postcodes from the data.csv
const csv = require('csv-parser')
const fs = require('fs')
const filepath = './data.csv';

fs.createReadStream(filepath)
  .on('error', () => {

  })
  .pipe(csv())
  .on('data', (row) => {
    let id = `${row['id']}`;
    let start = `${row['start_postcode']}`;
    let end = `${row['end_postcode']}`;

    console.log(id + ': ' + start + ' , ' + end);
    
    calculateDistance(start, end);
  })
  
  .on('end', () => {
    // handle end of csv
  });
// End of get start and end postcodes from the data.csv

Hi. I’m not sure of your coding level, but there are multiple ways to approach this.

  1. Use recursion and closures. Basically, you create a function within a function and repeatedly call that function until some condition is met, then exit the internal function.
    Closures - JavaScript | MDN

  2. ( Easiest and Recommended )Iterate the rows, construct an unfulfilled promise for each dataset, and push the promise into an array. Exit the iteration and call Promise.all(). That will execute the calls in parallel, but after the iteration is complete.

https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise/all

Hope that helps.

Hello Matt

Thanks for the input. I’m at a pretty basic level. Still working through udemy courses and books. I’ve read about closures recently so I think I will have a crack at that. Thanks again for the input.

Simon

Hello again guys

I came up with the following solution to my problem, but the strange thing is the console log will display row by row at a set interval but the caluclateDistance function just all gets updated at once it ignores the interval.

Do you have any idea where i have gone wrong with this.

Please see my snippet below.

// Sleep function
function sleep(milliseconds) {
    const date = Date.now();
    let currentDate = null;
    do {
        currentDate = Date.now();
    } while (currentDate - date < milliseconds);
}
// End of Sleep function

// Get start and end postcodes from the data.csv
const csv = require('csv-parser')
const fs = require('fs');
const filepath = './data.csv';

fs.createReadStream(filepath)
  .on('error', () => {

  })
  .pipe(csv())
  .on('data', (row) => {
    function gatherData() {
        let id = `${row['id']}`;
        let start = `${row['start_postcode']}`;
        let end = `${row['end_postcode']}`;
        function sendData() {  
            sleep(500);
            console.log(id + ': ' + start + ',' + end);
            calculateDistance(start, end);
        }
        sendData();
    }
    gatherData();
  })
  
  .on('end', () => {
    // handle end of csv
  });
// End of get start and end postcodes from the data.csv

Thanks for any help

Simon

1 Like