1. What Is an LLM's Vocabulary? (Vocabulary)

At the core of large language models (LLMs) such as GPT, LLaMA and Claude, the vocabulary (Vocabulary, also called the token set) is the foundation for handling natural language. It is the set of all possible tokens the model uses during training and inference and directly determines how text is converted into numerical sequences.

It defines:

➡ the smallest language units the model can recognize and process

➡ a unique integer ID assigned to each token

➡ these IDs are looked up and trained in the embedding table

In NLP systems, these tokens are not "words" in natural language but encoding units at the subword level.

For example:

"playing" → "play", "ing"

📌 Mapping text to these subword units is the process known as tokenization.

2. How is a vocabulary constructed?

A vocabulary is not a handcrafted "dictionary" but is learned automatically from large corpora:

1) Character‑level (Char)

The most basic approach, treating each character as a token.

Advantages: covers all languages

Disadvantages: sequences are very long and inefficient.

2) Subword‑level (Subword): mainstream methods

⭐ BPE (Byte Pair Encoding)

• Start from characters (e.g., all Unicode characters)
• Count the most frequent substring merges in the corpus
• Merge high‑frequency combinations into new tokens
• Iterate until the target vocabulary size is reached
• The result is a set of "most common subwords"
• Used by the GPT family and many autoregressive models.

SentencePiece (Unigram / BPE)

• Similar to BPE but uses a probabilistic or unigram model to select tokens
• Can naturally cover multilingual cases
• Adopted by many open models (e.g., LLaMA, Mistral).

WordPiece

• Mainly used in the BERT family
• Also builds subword tokens based on statistics
• Different from BPE merging rules: WordPiece uses a probabilistic search for maximum mutual information.

3) Special tokens

In addition to subwords, there are special tokens such as:

• <PAD> padding
• <UNK> unknown token
• <CLS> / <SEP> structural separators
• These are also counted toward the vocabulary size.

3. What are common vocabulary sizes for popular LLMs?

Vocabulary sizes vary widely across models and tokenizers depending on the target language, multilingual coverage, and tokenization strategy.

The following are reliable ranges and known information:

Open‑source model vocabulary sizes (public)

Closed‑source commercial vocabularies (typically not publicly disclosed)

⚠ These closed models usually do not publish vocabulary contents or tokenizer details. In engineering practice sizes can only be inferred or community‑reconstructed.

4. Why do different models have different vocabulary sizes?

Vocabulary size affects:

📌 input sequence length (smaller vocab → longer sequences)

📌 model parameter count (larger vocab → more embedding parameters)

📌 multilingual coverage (larger vocab better covers diverse scripts)

So different trade‑offs apply depending on goals:

• Monolingual models (English) can get by with smaller vocabularies
• Multilingual models need larger vocabularies
• Tasks rich in code/symbols also require larger vocabularies

This is why LLaMA 3 expanded to ~128k vocab⁠—⁠to better cover Chinese and special symbols. Rohan's Bytes

5. Which vocabularies are publicly available and open‑source?

✅ Common sources for open vocabularies:

📌 HuggingFace Model Hub

Most open models' tokenizers and vocabularies can be downloaded and used directly.

👉 For example:

• LLaMA vocabulary
• Bloom vocabulary
• T5 / mT5 vocabulary
• Phi series vocabularies

These are usually provided as files such as vocab.json, merges.txt or spm.model and can be loaded with tokenizer tools. Rahat Ibn Rafiq

6. What are common vocabulary file formats?

Format explanations: vocab.json + merges.txt for BPE (e.g., GPT‑2 series); spm.model / spm.vocab for SentencePiece model; plain txt lists; word‑level or legacy vocab; custom bin/jar proprietary formats

These files can be found in model release packages or downloaded from HuggingFace/Model Zoo. Rahat Ibn Rafiq

7. How to train your own vocabulary?

If you want to build your own LLM or a domain‑specific vocabulary (e.g., legal/medical):

1️⃣ Collect large‑scale corpora

2️⃣ Use tools to train a tokenizer

• SentencePiece (supports Unigram/BPE)
• tokenizers libraries (Rust/C/Python)
• 3️⃣ specify vocabulary size (e.g., 32k, 64k)
• 4️⃣ generate vocab files
• 5️⃣ use it to train or fine‑tune a model

This customization is common in industry and can significantly improve recognition of specialized terminology.

8. Conclusion

📌 A vocabulary is the model's "basic dictionary" for understanding language

📌 Learned from large corpora using algorithms like BPE / SentencePiece

📌 Popular open vocabularies typically range from 32k to 250k

📌 Vocabularies for closed‑source models such as GPT‑3, GPT‑4, Claude are undisclosed

📌 HuggingFace and the open‑source community are the richest sources for vocabularies

▶️ Enjoyed this article? Creating great content takes effort!

Subscribe to our YouTube channel:
AI Rosetta

Your likes, subscriptions and shares mean the world to us! 🙏

For more AI news, visit: https://www.airosetta.com