In the field of artificial intelligence, OpenAI's GPT‑3 (Generative Pre‑trained Transformer 3), released in 2020, is undeniably a milestone. With an astonishing 175 billion parameters, it demonstrated the power of large language models, able to perform translation, writing, reasoning and other tasks with virtually no additional training. The paper "Language Models are Few‑Shot Learners" shocked the world: a model could exhibit few‑shot learning and perform well from only a few prompts.
But what exactly is this "175 billion parameters"? It is not a mysterious black‑box number but the total of all trainable weights in the model. These parameters mainly come from neural network weight matrices and biases, optimized through training on massive datasets. In short: 175 billion parameters = Transformer body (multi‑layer attention and feed‑forward networks) + token embedding matrix + output projection matrix + layer norms and small bias terms. The vast majority (about 98%) resides in the 96 Transformer layers.
GPT‑3's core architecture: Decoder‑only Transformer
GPT‑3 is a "decoder‑only" model based on the Transformer architecture, inheriting from GPT‑2 but scaled up by orders of magnitude. Its largest (175B) version uses the following key hyperparameters:
- Layers (Transformer blocks): 96 layers
- Hidden dimension (d_model): 12,288 (the vector size per token)
- Attention heads: 96 heads (each head dimension 128, 96 × 128 = 12,288)
- Feed‑forward intermediate dimension (d_ff): 49,152 (about 4 × 12,288)
- Context length: 2,048 tokens
- Vocabulary size: about 50,257 (using Byte‑Pair Encoding)
The model structure simplifies to: input token embeddings → 96 Transformer blocks → output projection (to produce next‑token probabilities).
Parameter origins explained: step‑by‑step calculation to 175 billion
Parameters are mainly distributed across three parts; we use precise figures based on the official paper and community verification:
- Token embeddings
- Each vocabulary token is mapped to a 12,288‑dimensional vector.
- Calculation: vocabulary × hidden dim = 50,257 × 12,288 ≈ 617,558,016 (about 617.6 million)
- Additionally, positional embeddings: context length × hidden dim = 2,048 × 12,288 ≈ 25,165,824 (about 25.2 million, typically negligible)
- Share: about 0.35% (a very small portion)
- Transformer body (96 layers)
- This is the parameter "bulk"; each layer includes two submodules:
- Multi‑head self‑attention includes Query, Key, Value and Output projection matrices. Standard calculation: 4 × d_model² (because number of heads × head dimension = d_model, equivalent to 4 full‑size matrices). Per layer: 4 × 12,288 × 12,288 ≈ 603,979,776 (about 604 million), plus biases and small terms.
- Feed‑Forward Network (FFN) consists of two linear transforms: d_model → d_ff → d_model. Calculation: (12,288 × 49,152) + (49,152 × 12,288) ≈ 1,207,959,552 (about 1.208 billion), plus biases.
- Per‑layer minor terms: LayerNorm and biases, about 0.1%, negligible.
- Per‑layer total: attention ≈ 604 million + FFN ≈ 1.208 billion = about 1.8 billion.
- 96 layers total: 1.8 billion × 96 ≈ 172.8 billion (precise calculation about 173.95 billion).
- Share: about 98–99%, with the FFN accounting for roughly two‑thirds and attention one‑third.
- Output layer (language model head)
- Projects hidden states back to the vocabulary to produce next‑token probabilities.
- In GPT‑2 weights were tied with the embedding (parameter cost = 0); GPT‑3 does not tie them.
- Calculation: 12,288 × 50,257 ≈ 617,558,016 (about 617.6 million).
- Share: about 0.35%.
Total: embeddings ≈ 617.6M + Transformer layers ≈ 17.395B + output ≈ 617.6M + minor terms ≈ 17.456B ≈ 175 billion parameters (official rounding).
Parameter proportions roughly: Transformer layers ~98%, embeddings + output ~0.7%, remaining biases/LayerNorm negligible.
Why so many parameters? The underlying "scaling laws"
OpenAI did not pick these numbers at random but followed "scaling laws": as model size, data and compute increase, performance improves following power‑law behavior. The jump from GPT‑2's 1.5 billion parameters to 175 billion aimed to capture linguistic complexity and enable emergent abilities—such as few‑shot learning without fine‑tuning.
Choices like 12,288 dimensions, 96 layers and 96 heads reflect an engineering balance: at the time training used thousands of NVIDIA V100 GPUs, took weeks, and cost millions of dollars. Scaling further faces exponentially rising costs.
What the parameters mean: like synapses in a brain
By analogy to the human brain (about 100 trillion synapses), these parameters are the model's "connection strengths." More parameters mean greater capacity to store complex language patterns, grammar, knowledge and reasoning paths. This helps explain why GPT‑3, after a scale leap, acquired abilities such as translation, poetry and simple arithmetic—scale enabled qualitative changes.
Because parameters are central to LLMs, most leading closed‑source models no longer publish full parameter breakdowns; open‑source large models generally range from hundreds of billions to a few trillion parameters, reflecting trade‑offs between efficiency and performance.
2025 Global Top 10 LLMs: Comprehensive Comparison and Outlook of the US, China and Europe's Mistral
Conclusion: scale rules, but it's not the only way
GPT‑3's 175 billion parameters demonstrate that sheer scale can unlock surprising capabilities and inspired subsequent models such as GPT‑4. Today we also see the importance of efficiency optimizations (sparse models, quantization). Although most parameters come from repeated Transformer layers, understanding these numbers both demystifies GPT‑3 and illuminates the scaling path for AI's future.
▶️ Enjoyed this article? Creating great content takes effort!
Subscribe to our YouTube channel:
AI Rosetta
Your likes, subscriptions and shares mean the world to us! 🙏
For more AI news, visit:
https://www.airosetta.com