Losing precision with dc

Expressed as decimal (using dc to convert), this corresponds to 999999.98 (rounded down) × 256, i.e. 255999994.88, which is F423FFA.E1 in hexadecimal.

So the difference comes from dc’s rounding behaviour: instead of calculating 256 × (999999 + 253 ÷ 256), which would give 255999997, it rounds 253 ÷ 256 down and multiplies the result.

dc is an arbitrary precision calculator, which means it can calculate to any precision you want, but you have to tell it what that is. By default, its precision is 0, meaning division produces integer values only, and multiplication uses the number of digits in the input. To set the precision, use k (and bear in mind that precision is always expressed in decimal digits, regardless of the input or output radix):

10 k
16 d i o
F423FFD 100 / p
F423F.FD0000000
100 * p
F423FFD.000000000

(8 digits’ precision would be sufficient since that’s what you need to represent 1 ÷ 256 in decimal.)


Note that just printing the original number shows it is rounded:

$ dc <<<'16 d i o F423F.FD p'
F423F.FA

You can get around it by adding lots of trailing zeroes for more precision:

$ dc <<<'16 d i o F423F.FD000000 100 * p'
F423FFD.0000000

Tags:

Dc