☆ Yσɠƚԋσʂ ☆@lemmy.ml to Programmer Humor@lemmy.mlEnglish · 1 month agoFirst exposure to floating-point arithmeticpeertube.mesnumeriques.frexternal-linkmessage-square5fedilinkarrow-up118arrow-down11cross-posted to: programmer_humor@programming.dev
arrow-up117arrow-down1external-linkFirst exposure to floating-point arithmeticpeertube.mesnumeriques.fr☆ Yσɠƚԋσʂ ☆@lemmy.ml to Programmer Humor@lemmy.mlEnglish · 1 month agomessage-square5fedilinkcross-posted to: programmer_humor@programming.dev
minus-squarevext01@lemmy.sdf.orglinkfedilinkarrow-up1·1 month agoI looked into this at some point in the past. From memory, despite 0.3 being unrepresenatable, the routine that formats the output rounds it (stringily) to 0.3. Someone can correct me, because it’s only a vague memory at this point.
I looked into this at some point in the past.
From memory, despite 0.3 being unrepresenatable, the routine that formats the output rounds it (stringily) to 0.3.
Someone can correct me, because it’s only a vague memory at this point.