Precision

From Fractal Wiki
Jump to navigation Jump to search

Typically, calculations on a computer are performed with a limited number of significant digits. The number of digits (usually expressed as a count of binary bits) is called the precision. Some data types are commonly implemented in hardware:

datatype precision (bits)
half (GPU) 11
float 24
double 53
long double (x87) 64

When higher precision is needed, software implementations are needed. Common arbitrary precision libraries include:


Original Page by Claude