Trying to clean up data in a not-too-deep JSON

JavaScript
1

Hello all,

I am trying to clean up some data before using it. The JSON structure is not too complicated but my head is spinning from the object, object of arrays, object of arrays of objects…aaaagh.

My first step in cleaning up the data is to find all the objects that have null values and remove them from the dataset completely. So for example, if there is an object that looks like this:

{
            "24h_vol": null,
            "date": "12/5/2013",
            "market_cap": null,
            "price_usd": null
        },

it would have to be removed from the cleaned up dataset.

I keep getting lost in the data structure of the JSON so any helpful tips would be appreciated.

Dataset >>> https://github.com/SabahatPK/d3-line-graph/blob/master/data/coins.json
Entire project >>> https://github.com/SabahatPK/d3-line-graph

2

It’s fairly easy tbh, it seems nested and it is (techically 3 levels) but you only have to worry about 2 since the outer object only has 1 property called “bitcoin”.

So you just need to re-assign the “bitcoin” property to the filtering of objects that have any null value.

If you don’t remember the filter function you can research on google “array filter javascript”

To detect whether an object has a value equal to something, you do:

function hasNullValue(obj) {
  for (let key in obj) {
    if (obj[key] === null) return true
  }

  return false
}

Or you can use Object.values (ES7 only) in combination with Array.prototype.includes (also ES7 I think?) or indexOf !== -1.

3

Thanks @luishendrix92.

I think I already tried what you are suggesting.

But there are actually 5 keys on the outer object: bitcoin, bitcoin-cash, ethereum, litecoin, ripple. And I have to get to the properties of each of them. But each of their properties is an array. Then THAT array holds a bunch of objects and it’s these final objects that I need to check null values for.

In trying to understand the data structure and how to access it, I did a bunch of console.logging:

d3.json("data/coins.json").then(function(data) {
  console.log("---data (an object)---");
  console.log(data);

  console.log("---data's values (an array of arrays)---");
  console.log(Object.values(data));

  console.log("---each of those array's holds objects ---");
  Object.values(data).map(eachObj => console.log(eachObj)); //I need to filter at this level

  let cleanData = Object.values(data).filter(
    eachObj => eachObj["24h_vol"] !== null
  );
  console.log("---cleanData----");

  console.log(cleanData);

//a bunch more data...

}

But that did not filter anything out. I ended up with all the objects still in the array.

Code located here: https://github.com/SabahatPK/d3-line-graph/blob/master/js/main.js

4

Seems easier, using the same for-in loop technique but instead with the outer keys. And I didn’t understand you at first when it came to what you defined as “null” values so instead of any null value for any prop it was just '24h_vol'

Here’s the code for it, and I made sure to clean it up and sent it to another JSON file in case you want the json file instead.

5

Thanks @luishendrix92. That is really helpful. A couple more questions:

  1. Is this an example of destructing, i.e. the argument for filter function:
    .filter(({ '24h_vol': prop }) => prop !== null)

  2. My understanding of for...in is that post-ES6, we could replace that with Object.values. Do you know why I could not filter out using Object.values the same way you did in your code?
    I really want to understand why didn’t this work:

let cleanData = Object.values(data).filter(
    eachObj => eachObj["24h_vol"] !== null
  );
6

1- Yeah it’s called “destructuring with an alias”, basically since you’re not allowed to have variables starting with a number; and leveraging the fact that keys in JSON are basically strings, you can alias it to a proper variable name like “prop” or something.

2- Well yes, you can do that but what I understood is that you needed to preserve the same object structure as the source object that’s why I used object mutation (re-assigning properties) instead of converting everything to a tuple-array or something else.

And that code won’t work I think because Object.values(data) will yield N amount of arrays with objects, not individual objects. So eachObj would be an array instead of an object.

JS doesn’t provide a mapping function for objects by default but a library like lodash does:

https://underscorejs.org/#mapObject

This’d allow you to write it as

const clean = _.mapObject(data, filterOutNulls)
// where
// const filterOutNulls = eachObj => eachObj["24h_vol"] !== null
1 Like