#Float precision

1 messages · Page 1 of 1 (latest)

late bane
#

Why does 0.1f + 0.2f give 0.3 but 0.1 + 0.2 (double) doesn't return the same.
IK it's partially due to the way it stores it in the IEEE 754 Format but can someone elaborate on this?

jovial sapphireBOT
#

<@&987246964494204979> please have a look, thanks.

civic prism
#

u have to look at how the values are represented internally

#

0.1f is stored as

#

u also notice that its not a clean 0.1 and 0.2

#

are considered the same value

#

so 0.10000000000000000000000123f == 0.10000000000000000000000004f is for example true

#

now, what happens when u add these two numbers? u get 0.30000000447

#

and 0.30000000447f == 0.3f is still considered true because its within that delta

#

but for a double that delta is much smaller, since it has a higher resolution

#

so when u do the same

#

then u get 0.30000000000000004

#

and the delta doesnt include 0.3 bc the distance is greater

#

like, the number 0.3 as double is represented by the range 0.3 plus minus 5.5 * 10^-17

#

so roughly 2.9999999999999999945 to 0.30000000000000000055

#

while 0.1+0.2 for double lands beyond that number

#

and for float within

cerulean meadow
#

float.exposed links to the article the author wrote, which has much more in depth information

formal fox