# Precision

Revision as of 09:16, 29 June 2019 by WAUthethird (talk | contribs) (Created page with "Typically, calculations on a computer are performed with a limited number of significant digits. The number of digits (usually expressed as a count of binary bits) is called...")

Typically, calculations on a computer are performed with a limited number of significant digits. The number of digits (usually expressed as a count of binary bits) is called the precision. Some data types are commonly implemented in hardware:

datatype | precision (bits) |
---|---|

half (GPU) | 11 |

float | 24 |

double | 53 |

long double (x87) | 64 |

When higher precision is needed, software implementations are needed. Common arbitrary precision libraries include:

Original Page by Claude