Demystifying Floating Point Precision
Do you know that 0.2 + 0.4 = 0.6000000000000001. That’s floating point numbers precision limitation. This post aims to answer questions: 1) What level of precision do I have at a number? 2) When will I hit precision issues?
Appears in lists (1)
More like this (3)
Instead of the fixed-sized exponent and fixed-sized fraction used in IEEE floating point numbers, posits encode...