How to write reusable filter functions in JS

Hi all,

I am practicing filters and sorting data in JS by building a simple Vue app. So far things are going well, but I have some questions as to how to optimize my filter functions and make them more reusable / generic.

LINK TO CURRENT LIVE CODE: https://awesome-lichterman-e60bed.netlify.com/#/city-search
As you can see below, it works fine, but it feels like I’m not being very DRY with my code.

I just have 2 questions:

  1. Does anyone have any tips/suggestions/links to help me refactor this code.
  2. And what is the best practice / method for chaining filters ontop of each other
    (eg) Filter by letter ‘A’ and then whatever cities start with ‘A’ can then be filtered by Rank, Population etc but without overriding the current filter.

In my Vuex store, I have the following mutation functions, all which filter data but just a little differently for each function.

// state.js
export const state = {
  cities: [],
  filteredCities: [],
  filters: {
    byLetter: '',
    byPopulationSize: '',
    sortByAtoZ: false,
    sortByRank: []
  },
  loadingComplete: false
}

// a typical city item
{
  city:"New York"
  growth_from_2000_to_2013:"4.8%"
  latitude:40.7127837
  longitude:-74.0059413
  population:"8405837"
  rank:"1"
  state:"New York" 
}

// mutations.js
// TODO: Create Filter Util Function to just call ONE main filter func.
export const mutations = {
  SET_FILTER_BY_LETTER (state, payload) {
    state.filters.byLetter = payload
    const filter = state.cities.filter((city) => {
      return city.city[0].toLowerCase() === payload.toLowerCase()
    })
    state.filteredCities = filter
  },

  SET_FILTER_BY_POPULATION (state, payload) {
    state.filters.byPopulationSize = payload
    const filter = state.cities.filter((city) => {
      return city.population >= payload
    })
    state.filteredCities = filter
  },

  SORT_CITIES_A_TO_Z (state, payload) {
    state.filters.sortByAtoZ = payload
    return (!payload)
      ? state.filteredCities.sort((a, b) => b.city.toLowerCase() < a.city.toLowerCase())
      : state.filteredCities.sort((a, b) => b.city.toLowerCase() > a.city.toLowerCase())
  },

  SORT_CITIES_BY_RANK (state, payload) {
    state.filters.sortByRank = payload
    let [ low, high ] = payload
    const filter = state.cities.filter((city) => {
      return city.rank >= low && city.rank <= high
    })
    state.filteredCities = filter
  }
}

Thanks in advance :slight_smile:

You always have to think carefully about whether it’s worth trying to generalise things (just copy pasting stuff is fine in a large % of cases). But here, where you want to dynamically apply filters it kinda makes sense, though what you have works well.

So this is super rough, but one way to do this is to collect the values from the filter UIs into an object, then pass that to a function that runs them all at once in a single filter operation. Note that for this to work you’d want to have a form where you input the filters first (which updates the current state of some filters object), then hit submit so they all get applied at once (as opposed to just updating constantly as the UI changes as currently happens). This may/may not be acceptable.

Just realised the stringStartsWith bit is wrong as well, as always needs to be regex (can use new Regex(some string) though)

function cityFilter(cities, filtersPayload = {}) {
  // Set up the defaults: these should cover all possible cases,
  // so if no extra filters are passed, you get everything. This may
  // not be necessary: you can just pass in the current state of the
  // all the filter controls in the UI and use that, getting rid
  // of these defaults -  was more so I could test locally
  const filtersDefault {
    rankLow: 0,
    rankHigh: 1000,
    populationGreaterThan: 0,
    nameStartsWith: /[A-Z]/
  }
  
  // Assign any extra filters applied in the UI over the
  // top of the default filters
  const filters = Object.assign(filtersDefault, filtersPayload);
  
  // Return the filter function that will be used in the
  // actual `filter`
  return cities.filter(city => {  
    // Destructure the `city` object to grab the values you want
    const { city, rank, population: pop } = city;
    
    return
      rank >= filters.rankLow &&
      rank <= filters.rankHigh &&
      pop >= filters.populationGreaterThan &&
      city[0].match(filters.nameStartsWith) &&
  });
}

                       
// SO, as an example:
cityFilter(cities, {rankLow: 300, rankHigh: 700, popGreaterThan: 250000, nameStartsWith: 'N'})
1 Like

Hey Dan! Thanks that is awesome.

So what you propose is to add the filters all at once on submit/save. That sounds like a smart idea. I was also thinking of providing two options / modes, if you will. One for ‘strict’ filtering (like your suggestion) and one for general filtering (which overrides the data on each click. Although, I haven’t really seen this on many websites.

What I do see a lot on eCommerce and real estate sites are UI’s that update on each filter change. They do seem to use a ‘strict’ filtering approach, so I assume you could use the functionality that you have, but just recall it again and again on each filter update.

I’ll test out your solution and let you know how it goes :slight_smile:
Thanks

I also found this example codepen, which does a good job of filtering. I need to figure out the code a bit more, but it looks like a good approach:

There are 2 ways I’d approach this:

  1. multiple iterations
  2. lazy sequences

I’m a firm believer in not prematurely optimizing, and instead coding for clarity and ease of troubleshooting. So I’d start with multiple iterations, and switch to lazy sequences if my performance tests dictate it.

Because of this, I’d start with what my actual data set will be.

In this case, I’d first need to test some assumptions:

Because of this, and the fact that each filter returns a smaller subset of the results, I’d just chain filters together.

But first I’d run tests with chained filters, each returning the complete set as a stress test. Here I used jsperf https://jsperf.com/chained-filters.

  1. 20k cities – actual set
    • 80k total iterations
    • 317 ops/sec
    • 3ms per result
  2. 100k cities – stress test
    • 400k total iterations
    • 52 ops/sec
    • 19ms per result
  3. 200k cities – ridiculously extreme stress test
    • 800k total iterations
    • 22 ops/sec
    • 45ms per result

So my assumptions are correct, and chained iterations will not be the bottleneck. At least for the foreseeable future.

I will also stay below the 16ms jank threshold up to 100k cities. Although jank is not an issue for searching, as people are easily ok waiting up to 1 second for results.

This means I can now write my code like this

const filterByLetter = () => {}
const filterByPopulation = () => {}
const filterByRankLow = () => {}
const filterByRankHigh = () => {}

// then

cities
  .filter(filterByLetter)
  .filter(filterByPopulation)
  .filter(filterByRankLow)
  .filter(filterByRankHigh)

Easy to read and troubleshoot. And easy to implement as a first trial.

I also prefer this approach because it gives the flexibility to possibly have the user decide which filters to apply using something like a checkbox.

// push desired filter variables into an array onClick
filters.push(filterByLetter)
filters.push(filterByPopulation)

// this is now
const filters = [ filterByLetter, filterByPopulation ]

filters.reduce((cities, filter) => {
  return cities.filter(filter)
}, cities)

Later on, if I see performance drags a bit, I’d introduce a lazy sequence package. https://github.com/dtao/lazy.js/

If you don’t know what lazy sequences are, all it means is that when you chain methods it will do as little work as possible.

It does this by applying each method to a single item before moving on. So if you wanted to chain multiple filters but only return the first 5 matches, lazy sequences will stop once it finds the 5th match.

But I doubt you’d need this since each subsequent filter is a smaller subset of your results.

1 Like

Wow thanks!

That’s an amazing answer! I really appreciate your help. Makes a lot of sense and I love the functional approach to chaining filters.

So just to clarify, do you propose on the click of each checkbox that the filter / chained filter function run, or would you click a checkbox and the a ‘save’ button, and then run the entire filter sequence?

Thanks again!

This is actually more of a UX question.

If you’d like to show results on each click, my first instinct would be to add some check so that a specific filter returns true if it isn’t selected. This allows you to just chain them all together and not worry about helping the computer be performant.

And like the tests showed, I highly doubt someone will click so fast as to have this be the bottleneck. So try each and see which you prefer.

Your welcome! Although I like to think of myself as a functional programmer, I don’t know if that’s because I’m clever, or just lazy lol :smile:

Oh, forgot to mention it would also depend on the speed of the api you’re using. This will influence whether you can filter on each click or not.

Oh yeah for now it will be fine. The dataset is just a github gist of 1000 cities and it’s purely for practice so it’ll be fine. Thanks for the lazy.js recommendation as well, looks cool.

1 Like

I would like if anyone can check my solution at excerise: Functional Programming: Use the filter Method to Extract Data from an Array.
I saw that this topic is locked for discussing but if anyone is willing to help me. My code so far:

var filteredList = watchList.map(function(obj){

return { title : obj[“Title”], rating: obj[“imdbRating”]}

}).filter(function(obj)

{return Number(obj[“imdbRating”]) >= 8.0});