Comprehensive Reference: Precision Arithmetic, Significant Figures, and Scientific Notation
In numerical mathematics, experimental physical sciences, financial engineering, and computer science, numbers represent measured physical phenomena or accounting balances rather than abstract ideals. Handling these quantities requires an exact understanding of three interconnected mathematical disciplines:
- Rounding Methodologies: How continuous real numbers are mapped to discrete intervals without introducing cumulative statistical bias.
- Significant Figures (Sig Figs): The systematic rules governing how measurement uncertainty, instrument precision, and tolerance propagate through arithmetic operations.
- Scientific & Engineering Notations: Standardized powers-of-ten representations that make cosmological distances and subatomic dimensions human-readable and computationally tractable.
1. Rounding Mechanics: Understanding the 5 Core Algorithms
When a calculated value contains more decimal digits than an instrument or financial ledger permits, a rounding decision must be executed. While school arithmetic teaches the Round Half-Up heuristic, scientific and banking systems employ specialized algorithms:
| Algorithm Mode | Formal Definition | Tie-Breaking (e.g. 2.5 & 3.5) | Primary Industrial Use |
|---|---|---|---|
| Round Half-Up | If the discarded digit $\ge 5$, increment previous digit | 2.5 → 3 | 3.5 → 4 | Commercial retail, elementary mathematics |
| Banker's (Round Half-Even) | If discarded digit is exactly 5, round to nearest even integer | 2.5 → 2 | 3.5 → 4 | IEEE 754 floating-point, financial accounting, banking |
| Ceiling (Round Up) | Always round toward positive infinity ($\lceil x \rceil$) | 2.1 → 3 | -2.9 → -2 | Inventory procurement, shipping pallet sizing |
| Floor (Round Down) | Always round toward negative infinity ($\lfloor x \rfloor$) | 2.9 → 2 | -2.1 → -3 | Computer graphics grid snaps, tax bracket floors |
| Truncation | Discard trailing digits without sign modification toward 0 | 2.9 → 2 | -2.9 → -2 | Typecasting float-to-int in C/C++, high-speed DSP |
Why Does IEEE 754 & Modern Banking Require Banker's Rounding?
If millions of transactions are rounded using Round Half-Up, every exact midpoint (0.50) rounds upward. Over time, this creates an upward statistical bias that artificially inflates balance sheets and yields systemic drift. Banker's rounding eliminates this drift because 50% of midpoints round down to even digits (e.g. 2.5 → 2, 4.5 → 4) and 50% round up (e.g. 1.5 → 2, 3.5 → 4), achieving a mean drift of exactly zero.
2. The Definitive Rules of Significant Figures
In empirical measurements, digits indicate precision. Stating that a steel beam measures $12.30 cm$ conveys far greater precision than stating $12.3 cm$. Follow these universally accepted ASTM and NIST criteria:
- Rule 1 (Non-Zero Digits): All non-zero digits ($1, 2, 3, \dots, 9$) are always significant. Example: $487$ has 3 significant figures.
- Rule 2 (Captive Zeros): Zeros located between non-zero digits are always significant. Example: $5004$ has 4 significant figures; $1.0008$ has 5 significant figures.
- Rule 3 (Leading Zeros): Zeros preceding the first non-zero digit serve purely as placekeepers and are never significant. Example: $0.00052$ has only 2 significant figures ($5$ and $2$).
- Rule 4 (Trailing Zeros with Decimal): Zeros appearing at the end of a number that contains an explicit decimal point are significant. Example: $45.00$ has 4 significant figures; $0.0200$ has 3 significant figures.
- Rule 5 (Trailing Zeros without Decimal): Trailing zeros in an integer without a decimal point are ambiguous. Example: $1200$ is conventionally assumed to have 2 significant figures. To state that it has 4 significant figures, write $1200.$ with a terminating decimal point or express it as $1.200 \times 10^3$.
Propagation of Significant Figures in Arithmetic
The result cannot have more decimal places than the measurement with the least number of decimal places.
125.17 + 1.2 = 126.37 → 126.4 (1 decimal place)
The result cannot have more total significant figures than the measurement with the fewest significant figures.
4.56 × 1.4 = 6.384 → 6.4 (2 significant figures)
3. Scientific vs Engineering Notation & Metric Prefixes
Scientific notation expresses numbers as $m \times 10^n$, where $1 \le |m| < 10$. In contrast, engineering notation requires the exponent $n$ to be an integer multiple of 3 ($n = 3k$). This ensures that every engineering quantity maps directly to a standard SI unit prefix:
| Standard Decimal | Scientific Notation | Engineering Notation | SI Metric Symbol & Name |
|---|---|---|---|
| 1,000,000,000,000 | 1.0 × 10¹² | 1 × 10¹² | T (Tera) |
| 4,500,000,000 | 4.5 × 10&sup9; | 4.5 × 10&sup9; | G (Giga) |
| 350,000 | 3.5 × 10&sup5; | 350 × 10³ | k (Kilo) |
| 0.000045 | 4.5 × 10−&sup5; | 45 × 10−&sup6; | μ (Micro) |
| 0.0000000028 | 2.8 × 10−&sup9; | 2.8 × 10−&sup9; | n (Nano) |