Find difference between rounding errors and real decimals

I am currently working on a program that divides decimals (generating the coordinates for graphing 3D vectors). This produces many rounding errors on almost every single operation.

[ 89.00000000000007, 8.900000000000007, 4.450000000000004 ],
  [ 90.00000000000007, 9.000000000000007, 4.5000000000000036 ],
  [ 91.00000000000007, 9.100000000000007, 4.550000000000003 ],
  [ 92.00000000000007, 9.200000000000006, 4.600000000000003 ],
  [ 93.00000000000007, 9.300000000000008, 4.650000000000004 ],
  [ 94.00000000000007, 9.400000000000007, 4.700000000000004 ],
  [ 95.00000000000007, 9.500000000000007, 4.7500000000000036 ],
  [ 96.00000000000007, 9.600000000000007, 4.800000000000003 ],
  [ 97.00000000000007, 9.700000000000008, 4.850000000000004 ],
  [ 98.00000000000007, 9.800000000000008, 4.900000000000004 ],
  [ 99.00000000000007, 9.900000000000007, 4.950000000000004 ]

I am currently using the following regex to check if a number is a floating-point error:


It checks if it either starts with a number then a decimal sign or just starts with a decimal sign.


After that, it checks for zero or more numbers


From here it checks if there are 3 or more 0 or 9’s in a row and ends with an optional amount number(s).


I am just curious if there is a better way to check. For example, the number 0.000452998 is not a floating-point error, but this regex would return so. Is there a defined amount of 0’s or 9’s in a floating-point error?

In general, computers have precision issues with math. There are limits. There is limited memory and in terms of floating point numbers, any number that is not an exponent of 2 cannot be accurately stored in a binary system.

Those rounding errors are usually very, very small and are not a problem for the vast majority of applications. I had an engineering teacher that told me that the early NASA missions only used 8 digits of pi, for example. They landed a man on the moon with that. How many do you need?

Is there a defined amount of 0’s or 9’s in a floating-point error?

No. The system just does the math and gives you back the answer to the best of it’s ability. JS provides a constant, Number.EPSILON, that tells you what the max margin of error for the system is. On my system it is 2.220446049250313e-16. For scale, the size of a hydrogen atom is 1.2e-10 meters.

If you want more precision than that, then I’d look into math libraries.

1 Like

A example of a issue I am having with precision is when generating 3d vectors.

The code below is supposed to generate coordinate points on a line between two 3d coordinates.

function curveType(vertices, type) {
    let p1 = vertices[0]
    let p2 = vertices[1]
    let xDiff = matrixTools.differenceBetween(p1[0], p2[0]);
    let yDiff = matrixTools.differenceBetween(p1[1], p2[1]);
    let zDiff = matrixTools.differenceBetween(p1[2], p2[2]);
    let curveCoords = [] 
    let distance = matrixTools.distanceCalc(vertices[0], vertices[1])
    console.log(`Generating coordinates between: ${JSON.stringify(p1)} and ${JSON.stringify(p2)}`)
    let density = 1/distance;
    for (let t = 0; t <= 1; t += density) {
      let x = vertices[0][0] + xDiff * t
      let y = vertices[0][1] + yDiff * t
      let z = vertices[0][2] + zDiff * t
      console.log(`t: ${t} \n Den: ${density}`)
      if (t+density > 1) {
        console.log(`t: ${t} \n Den: ${density} \n t+den: ${t+density} \n distance: ${distance}`)
      console.log(`Generated: ${[x, y, z]}`)
      curveCoords.push([x, y, z])
    // if (!programTools.compare2d(curveCoords[curveCoords.length - 1], p2)) {
    //   curveCoords[curveCoords.length - 1] = [...p2]
    // }
    console.log(`returning: ${JSON.stringify(curveCoords)}`)
    return curveCoords

Here are some of the helper functions that go along with it:

These two functions find the difference (rate of change), positive or negative between the points to use in the 3d vectors. The distance functions finds the direct distance between two points

let matrixTools = {
  differenceBetween: function(n1, n2) {
    if (n1 >= 0 && n2 >= 0) {
      return n2 - n1
    } else if (n1 < 0 && n2 >= 0) {
      return (n2 + Math.abs(n1))
    } else if (n1 >= 0 && n2 < 0) {
      return -(n1 + Math.abs(n2))
    } else {
      return n1 < n2 ? Math.abs(n1) + n2 : (Math.abs(n1) - Math.abs(n2))
  distanceCalc: function(p1, p2) {
    return Math.sqrt(Math.pow((p2[0] - p1[0]), 2) + Math.pow((p2[1] - p1[1]), 2) + Math.pow((p2[2] - p1[2]), 2))

There is a infinite amount of coordinates on a line between any two points. I want to generate enough coordinates that, when each coordinate’s values are rounded to a whole number, all whole numbers coordinates between the two points were found. This is so I could then use these coordinates as index’s on a 3d matrix to capture my 3d object (can’t have matrix3d[1.5][2.5][3]).

To generate the coordinates, I find the distance between the two points. I then divide one by that distance. I then count up with that decimal number from 0 to 1. This should make it so no matter the size of the line, I always generate enough coordinates to span that line.

If you use the program, you will notice that rounding errors seem to always cause a snowball affect. A rounding error on the second coordinate calculation will cause the next one to be off, and the next one’s rounding error adds to that.

Here is a example of this happening (modeled in Three.js using 0.25 sized cubes at the coordinate positions) :

The program generates coordinates along the edge of a triangle, but due to the snowball affect the corner cubes always seem have rouge cubes due to coordinates generating at 0.05025 instead of just zero. This create technically a new coordinate next to my target end coordinate of (0,0,0), causing a overlap

Any ideas on how to fix these rounding errors? Should I just cut off the calculations at two decimal points and just accept a margin of error in the calculations?

Roundoff error is just inherent in floating point calculations. The more calculations you do, the more error accumulates.

The only fix is to choose algorithms that are robust against floating point errors.

1 Like

There is another route to consider, depending on your precision: BigInt math! A great article on the subject (mostly theoretical, though yours is a very valid case for it): Sick of the stupid jokes? Write your own arbitrary-precision JavaScript math library

1 Like

Yeah, you can usually go the arbitrary precision route, though usually that isn’t needed. Typically double precision is adequate.

One simple change you can make is to only compute the location of your endpoints once.

Though, I’m not sure why you need to put your lines on an integer lattice, this is certainly a solved problem and you might be able to find better algorithms for doing so.