Precision

From Fractal Wiki
Revision as of 09:16, 29 June 2019 by WAUthethird (talk | contribs) (Created page with "Typically, calculations on a computer are performed with a limited number of significant digits. The number of digits (usually expressed as a count of binary bits) is called...")
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search

Typically, calculations on a computer are performed with a limited number of significant digits. The number of digits (usually expressed as a count of binary bits) is called the precision. Some data types are commonly implemented in hardware:

datatype precision (bits)
half (GPU) 11
float 24
double 53
long double (x87) 64

When higher precision is needed, software implementations are needed. Common arbitrary precision libraries include:


Original Page by Claude