1. The Anatomy of LLM GPU Memory: Weights, KV Cache & Activation Tensors

Serving and training Large Language Models (LLMs) requires rigorous GPU High Bandwidth Memory (HBM) budgeting. Out-Of-Memory (OOM) errors occur when total allocation exceeds physical VRAM. Total memory consumption during inference is governed by four distinct components:

1. Model Parameter Weights: The parameter footprint is determined strictly by parameter count P and quantization precision:

• FP32 (32-bit float): 4.0 Bytes / parameter (A 70B model requires 280 GB).
• FP16 / BF16 (16-bit float): 2.0 Bytes / parameter (A 70B model requires 140 GB).
• INT8 (8-bit integer): 1.0 Byte / parameter (A 70B model requires 70 GB).
• INT4 / AWQ / GPTQ (4-bit integer): 0.5 Bytes / parameter (A 70B model requires 35 GB).

2. Attention Key-Value (KV) Cache: During autoregressive generation, the model must retain the key and value projection tensors for all prior tokens across all attention layers to avoid redundant recalculation:

In traditional Multi-Head Attention (MHA), n_kv_heads = n_heads. In a 70B model running 8k context at batch size 4 in FP16, MHA requires over 160 GB just for the KV cache. Modern architectures employ Grouped Query Attention (GQA) (e.g. Llama 3 with 64 query heads and 8 KV heads), which slashes KV cache memory by 8x down to ~20 GB.

2. Chinchilla Scaling Laws & Training Compute FLOPs Derivations

The foundational economics of deep learning training are defined by compute scaling laws. Prior to 2022, models were trained according to Kaplan et al., which prioritized scaling parameter count over token volume.

In 2022, DeepMind published the Chinchilla Scaling Laws (Hoffmann et al.), demonstrating that for compute-optimal pre-training, parameter count and dataset token count must be scaled in equal proportion:

The total theoretical floating-point operations (FLOPs) required to train a decoder-only transformer model are calculated using Kaplan's constant:

Where 2 FLOPs per parameter occur during the forward pass (matrix multiplication), and 4 FLOPs per parameter occur during the backward pass (2 for activation gradients, 2 for weight parameter updates).

For example, training a 70 Billion parameter model on 15 Trillion tokens (such as Meta Llama 3) requires:

On a cluster of 1,024 NVIDIA H100 GPUs (each achieving ~400 effective TFLOPS at 40% Model FLOPs Utilization), this requires approximately 45 days of uninterrupted distributed execution.

3. Inference Latency Mechanics: Prefill vs Autoregressive Generation

Serving transformer language models involves two fundamentally distinct computational regimes:

Phase 1: Prompt Prefill (Time to First Token – TTFT): During prefill, the user's prompt tokens are ingested simultaneously. The attention matrix is computed in parallel across all input tokens using General Matrix Multiply (GEMM). Because arithmetic intensity is high, prefill is strictly Compute-Bound by GPU Tensor Core FLOPS.

Phase 2: Autoregressive Decode (Generation Throughput): To generate token N + 1, the model takes the single newly generated token, queries the cached KV activations, and generates one output token. In this phase, the GPU must stream the entire model parameter weights from High Bandwidth Memory (HBM) into SRAM cache to compute a single matrix-vector multiplication (GEMV).

Because arithmetic intensity is less than 2 FLOPs per byte, the theoretical maximum generation speed is capped by the GPU Memory Bandwidth Ceiling:

For a 70B model quantized to INT4 (35 GB) running on a single NVIDIA H100 (3,350 GB/s bandwidth):

This formula proves why quantizing from FP16 (140 GB) to INT4 (35 GB) yields a nearly 4x speedup in token generation throughput—not because the tensor cores compute faster, but because the memory bus transfers 4x fewer bytes per token.

4. Vector Database Embeddings: High-Dimensional Geometry & HNSW Graphs

Retrieval-Augmented Generation (RAG) and semantic search depend on vector embeddings generated by neural encoders (such as OpenAI text-embedding-3-small or BERT). Embeddings map unstructured text into continuous D-dimensional vector spaces where semantic similarity corresponds to cosine distance:

Memory budgeting for vector databases (such as Pinecone, Milvus, Qdrant, and pgvector) requires accounting for both raw tensor storage and approximate nearest neighbor (ANN) graph indexes:

Hierarchical Navigable Small World (HNSW): HNSW builds a multi-layer geometric graph where upper layers have long-range skip connections and lower layers provide fine-grained local clustering. HNSW achieves logarithmic search time O(log N), reducing query latency from 150ms down to 2ms on millions of vectors. However, storing the graph adjacency lists (typically M = 16 to 32 edges per vector) introduces an additional 50% to 65% RAM overhead above raw vector storage.

5. Model Evaluation: Precision-Recall Tradeoffs & Matthews Correlation (MCC)

Evaluating machine learning models requires choosing metrics aligned with dataset topology. In real-world enterprise applications (fraud detection, medical diagnosis, malware detection), datasets are severely imbalanced (e.g. 99.5% negative samples and 0.5% positive samples).

In imbalanced regimes, standard Accuracy is actively misleading: a dummy classifier predicting negative 100% of the time achieves 99.5% accuracy while detecting zero true positives.

The Harmonic F1-Score: The F1-score balances Precision (how many predicted positives are correct) and Recall (how many actual positives were found):

The Superiority of Matthews Correlation Coefficient (MCC): While F1 completely ignores True Negatives (TN), the Matthews Correlation Coefficient (MCC) incorporates all four cells of the confusion matrix:

MCC produces a score between −1.0 (total inverse prediction), 0.0 (random coin flip), and +1.0 (flawless prediction). Statistically, an algorithm can only achieve a high MCC if it predicts well across both the majority and minority classes, establishing MCC as the gold standard in competitive AI benchmarking.

1. The Anatomy of LLM GPU Memory: Weights, KV Cache & Activation Tensors

Serving and training Large Language Models (LLMs) requires rigorous GPU High Bandwidth Memory (HBM) budgeting. Out-Of-Memory (OOM) errors occur when total allocation exceeds physical VRAM. Total memory consumption during inference is governed by four distinct components:

1. Model Parameter Weights: The parameter footprint is determined strictly by parameter count P and quantization precision:

• FP32 (32-bit float): 4.0 Bytes / parameter (A 70B model requires 280 GB).
• FP16 / BF16 (16-bit float): 2.0 Bytes / parameter (A 70B model requires 140 GB).
• INT8 (8-bit integer): 1.0 Byte / parameter (A 70B model requires 70 GB).
• INT4 / AWQ / GPTQ (4-bit integer): 0.5 Bytes / parameter (A 70B model requires 35 GB).

2. Attention Key-Value (KV) Cache: During autoregressive generation, the model must retain the key and value projection tensors for all prior tokens across all attention layers to avoid redundant recalculation:

In traditional Multi-Head Attention (MHA), n_kv_heads = n_heads. In a 70B model running 8k context at batch size 4 in FP16, MHA requires over 160 GB just for the KV cache. Modern architectures employ Grouped Query Attention (GQA) (e.g. Llama 3 with 64 query heads and 8 KV heads), which slashes KV cache memory by 8x down to ~20 GB.

2. Chinchilla Scaling Laws & Training Compute FLOPs Derivations

The foundational economics of deep learning training are defined by compute scaling laws. Prior to 2022, models were trained according to Kaplan et al., which prioritized scaling parameter count over token volume.

In 2022, DeepMind published the Chinchilla Scaling Laws (Hoffmann et al.), demonstrating that for compute-optimal pre-training, parameter count and dataset token count must be scaled in equal proportion:

The total theoretical floating-point operations (FLOPs) required to train a decoder-only transformer model are calculated using Kaplan's constant:

Where 2 FLOPs per parameter occur during the forward pass (matrix multiplication), and 4 FLOPs per parameter occur during the backward pass (2 for activation gradients, 2 for weight parameter updates).

For example, training a 70 Billion parameter model on 15 Trillion tokens (such as Meta Llama 3) requires:

On a cluster of 1,024 NVIDIA H100 GPUs (each achieving ~400 effective TFLOPS at 40% Model FLOPs Utilization), this requires approximately 45 days of uninterrupted distributed execution.

3. Inference Latency Mechanics: Prefill vs Autoregressive Generation

Serving transformer language models involves two fundamentally distinct computational regimes:

Phase 1: Prompt Prefill (Time to First Token – TTFT): During prefill, the user's prompt tokens are ingested simultaneously. The attention matrix is computed in parallel across all input tokens using General Matrix Multiply (GEMM). Because arithmetic intensity is high, prefill is strictly Compute-Bound by GPU Tensor Core FLOPS.

Phase 2: Autoregressive Decode (Generation Throughput): To generate token N + 1, the model takes the single newly generated token, queries the cached KV activations, and generates one output token. In this phase, the GPU must stream the entire model parameter weights from High Bandwidth Memory (HBM) into SRAM cache to compute a single matrix-vector multiplication (GEMV).

Because arithmetic intensity is less than 2 FLOPs per byte, the theoretical maximum generation speed is capped by the GPU Memory Bandwidth Ceiling:

For a 70B model quantized to INT4 (35 GB) running on a single NVIDIA H100 (3,350 GB/s bandwidth):

This formula proves why quantizing from FP16 (140 GB) to INT4 (35 GB) yields a nearly 4x speedup in token generation throughput—not because the tensor cores compute faster, but because the memory bus transfers 4x fewer bytes per token.

4. Vector Database Embeddings: High-Dimensional Geometry & HNSW Graphs

Retrieval-Augmented Generation (RAG) and semantic search depend on vector embeddings generated by neural encoders (such as OpenAI text-embedding-3-small or BERT). Embeddings map unstructured text into continuous D-dimensional vector spaces where semantic similarity corresponds to cosine distance:

Memory budgeting for vector databases (such as Pinecone, Milvus, Qdrant, and pgvector) requires accounting for both raw tensor storage and approximate nearest neighbor (ANN) graph indexes:

Hierarchical Navigable Small World (HNSW): HNSW builds a multi-layer geometric graph where upper layers have long-range skip connections and lower layers provide fine-grained local clustering. HNSW achieves logarithmic search time O(log N), reducing query latency from 150ms down to 2ms on millions of vectors. However, storing the graph adjacency lists (typically M = 16 to 32 edges per vector) introduces an additional 50% to 65% RAM overhead above raw vector storage.

5. Model Evaluation: Precision-Recall Tradeoffs & Matthews Correlation (MCC)

Evaluating machine learning models requires choosing metrics aligned with dataset topology. In real-world enterprise applications (fraud detection, medical diagnosis, malware detection), datasets are severely imbalanced (e.g. 99.5% negative samples and 0.5% positive samples).

In imbalanced regimes, standard Accuracy is actively misleading: a dummy classifier predicting negative 100% of the time achieves 99.5% accuracy while detecting zero true positives.

The Harmonic F1-Score: The F1-score balances Precision (how many predicted positives are correct) and Recall (how many actual positives were found):

The Superiority of Matthews Correlation Coefficient (MCC): While F1 completely ignores True Negatives (TN), the Matthews Correlation Coefficient (MCC) incorporates all four cells of the confusion matrix:

MCC produces a score between −1.0 (total inverse prediction), 0.0 (random coin flip), and +1.0 (flawless prediction). Statistically, an algorithm can only achieve a high MCC if it predicts well across both the majority and minority classes, establishing MCC as the gold standard in competitive AI benchmarking.