Introduction
Conversational text‑to‑speech (TTS) is undergoing a transformative shift in AI voice synthesis. ChatTTS is an open‑source generative speech model designed for everyday dialogue scenarios, aiming to produce natural, expressive audio suitable for chatbots, virtual assistants and multi‑speaker interactions. It focuses on expressive prosody (pauses, laughter, interjections), multi‑speaker support and fine‑grained controls to help developers build realistic human–machine conversational experiences. Led by the 2Noise digital content team in the Netherlands, ChatTTS is open‑sourced on GitHub and Hugging Face and has quickly attracted global researchers and enthusiasts, currently accumulating nearly 38,000 stars. According to the project documentation, the model was trained on over 100,000 hours of Chinese and English conversational data; the openly released version is based on 40,000 hours of pretraining and is limited to academic use. The project fills a gap in open‑source TTS for conversational naturalness and expands application boundaries through a community ecosystem, advancing the democratization of speech AI. This report analyzes ChatTTS background, techniques, installation and usage, and future directions to provide a comprehensive reference for developers.
Background and Origins
ChatTTS originated from 2Noise's audio research efforts; the company focuses on digital content generation and voice technology innovation. The project began around 2023 as part of an open‑source initiative to lower the barrier to building conversational TTS. Unlike traditional TTS models (e.g., Tacotron or WaveNet) that prioritize single‑sentence reading, ChatTTS is optimized for LLM assistant scenarios and simulates the dynamics of real conversations. Project code, inference examples and pretrained checkpoints are hosted on the GitHub repository (https://github.com/Michael‑Zhang‑sz/ChatTTS) and distributed through the Hugging Face model hub. Training data come from public audio sources, covering thousands to 100k+ hours of conversational corpora with emphasis on Chinese–English mixed input and multi‑speaker scenarios.
Open‑source licensing is a core attraction of ChatTTS. The code uses AGPLv3+, while model weights follow CC BY‑NC 4.0 (non‑commercial, research/educational use), prohibiting commercial or illegal applications. To mitigate misuse, the released model embeds high‑frequency noise and MP3 compression in the 40k‑hour version, and the team plans to open‑source detection models. The community response has been rapid: after release, an Awesome‑ChatTTS index repository emerged (https://github.com/libukai/Awesome‑ChatTTS), aggregating UI tools, acceleration forks and mobile deployment extensions. 2Noise interacts with users via Discord and QQ groups (for example QQ group 808364215), encouraging PRs and issue reports. This reflects an active open‑source TTS ecosystem: ChatTTS is bridging research prototypes and end‑user products.
Technical Approach and Capabilities
The core of ChatTTS lies in prosody modeling and a generative architecture, influenced by projects such as Bark, XTTSv2, Valle and Fish‑Speech. It uses an autoregressive TTS framework that first predicts semantic tokens and then synthesizes audio through a neural vocoder (e.g., pretrained Vocos). Key innovations include:
- Expressive prosody control: The model predicts fine‑grained events such as micro‑pauses, breaths, laughter and interjections, with label‑based controls. For example, [oral_0‑9] denotes colloquial level, [laugh_0‑2] triggers laughter, and [break_0‑7] adjusts pauses. Sentence‑level or word‑level manual interventions (e.g., inserting [uv_break] for natural breaks) enhance conversational realism.
- Multi‑speaker and mixed‑language support: It supports switching speaker identities and mixed Chinese–English input for interactive agents. Speaker embeddings (spk_emb) use Gaussian sampling for timbre recovery.
- Fine‑grained parameter tuning: Inference supports custom temperature (e.g., temperature=0.3), top‑p/top‑k decoding to increase output diversity. English support is experimental and more languages are planned.
- Performance metrics: On an NVIDIA 4090 GPU, semantic token generation runs at about 7 tokens/s, and a 30‑second audio has an RTF (real‑time factor) of approximately 0.3. VRAM requirement is at least 4GB, but artifacts at sentence ends or multi‑speaker confusion may occur (inherent to autoregressive models).
These capabilities make ChatTTS stand out among open‑source TTS: its prosody quality surpasses many competitors, though it has higher computational cost. The documentation emphasizes that stability requires multiple sampling iterations; the roadmap includes streaming generation, zero‑shot inference, multi‑emotion support and a ChatTTS.cpp accelerated build.
Ecosystem and Tools
The ChatTTS ecosystem has grown, with rich community contributions. The Hugging Face model page (https://huggingface.co/2Noise/ChatTTS) provides pretrained weights and a Colab demo (https://colab.research.google.com/github/2noise/ChatTTS/blob/main/examples/ipynb/colab.ipynb). Community projects include WebUI, ComfyUI nodes and TensorRT optimization branches, supporting mobile deployment and voice‑cloning integration. A PyPI package (pip install ChatTTS) simplifies installation, and video demos (e.g., Bilibili BV1zn4y1o7iV) show real‑world effects.
Multilingual documentation covers English, Simplified Chinese, Japanese, Russian, Spanish, French and Korean to help global developers. Contact channels include GitHub Issues, Discord (https://discord.gg/Ud5Jxgx5yD) and email (open‑[email protected]). These tools lower the threshold from research to product, for example in virtual assistants or games.
Installation Guide
Installing ChatTTS is relatively straightforward, but a Python 3.11 environment and GPU support are recommended. Steps:
- Clone the repository:
git clone https://github.com/2noise/ChatTTS\ncd ChatTTS\n
- Prepare the environment:
- Use Conda to create a virtual environment:
conda create -n chattts python=3.11\nconda activate chattts\n
- Or upgrade requirements directly with pip:
pip install --upgrade -r requirements.txt\n
- Core dependencies: torch, torchaudio, safetensors are required. GPU users may optionally use vLLM (Linux only, pip install vllm==0.2.7 torchaudio) to accelerate inference. TransformerEngine (NVIDIA GPU) or FlashAttention‑2 are not recommended (may slow down): pip install git+https://github.com/NVIDIA/TransformerEngine.git@stable or pip install flash‑attn ‑‑no‑build‑isolation.
- Package installation options:
- Stable release: pip install ChatTTS
- Latest: pip install git+https://github.com/2noise/ChatTTS
- Developer mode: pip install ‑e .
After installation, verify the environment: run python ‑c "import ChatTTS; print('Success')". Note: installing offline requires preinstalled libraries such as torch.
Usage Guide
ChatTTS can be used via command line, Python scripts and WebUI.
- Start WebUI:
python examples/web/webui.py
Open a browser to localhost, enter text and generate audio with real‑time preview.
- Command‑line inference (save as output_audio_n.mp3):
python examples/cmd/run.py "Your text 1." "Your text 2."
- Python basic inference:
import ChatTTS
import torch
import torchaudio
chat = ChatTTS.Chat()
chat.load(compile=False) # compile=True can improve performance
texts = ["your first text", "your second text"]
wavs = chat.infer(texts)\nfor i in range(len(wavs)):
try:
torchaudio.save(f"basic_output{i}.wav", torch.from_numpy(wavs[i]).unsqueeze(0), 24000)
except:
torchaudio.save(f"basic_output{i}.wav", torch.from_numpy(wavs[i]), 24000)
- Advanced: speaker sampling and control:
# Randomly sample a speaker
rand_spk = chat.sample_random_speaker()
print(rand_spk) # save for timbre recovery
params_infer_code = ChatTTS.Chat.InferCodeParams(
spk_emb=rand_spk,
temperature=0.3,
top_P=0.7,
top_K=20,
)
params_refine_text = ChatTTS.Chat.RefineTextParams(
prompt='[oral_2][laugh_0][break_6]',
)
wavs = chat.infer(
texts,
params_refine_text=params_refine_text,
params_infer_code=params_infer_code,
)
- Word‑level control example:
text = 'What is [uv_break]your favorite english food?[laugh][lbreak]'
wavs = chat.infer(text, skip_refine_text=True, params_refine_text=params_refine_text, params_infer_code=params_infer_code)\ntorchaudio.save("word_level_output.wav", torch.from_numpy(wavs[0]), 24000)
- English self‑introduction example (experimental):
inputs_en = """
chat T T S is a text to speech model designed for dialogue applications.
[uv_break]it supports mixed language input [uv_break]and offers multi speaker
capabilities with precise control over prosodic elements like
[uv_break]laughter[uv_break][laugh], [uv_break]pauses, [uv_break]and intonation.
[uv_break]it delivers natural and expressive speech,[uv_break]so please
[uv_break] use the project responsibly at your own risk.[uv_break]
""".replace('\\n', '')
params_refine_text = ChatTTS.Chat.RefineTextParams(prompt='[oral_2][laugh_0][break_4]')
audio_array_en = chat.infer(inputs_en, params_refine_text=params_refine_text)
torchaudio.save("self_introduction_output.wav", torch.from_numpy(audio_array_en[0]), 24000)
Outputs are WAV/MP3 at 24,000 Hz. English support is experimental; multiple sampling iterations are recommended to improve stability.
Current Status and Adoption
Since release, ChatTTS has seen active adoption: Hacker News and Reddit communities praised its conversational naturalness while noting latency and end‑of‑sentence artifacts that need improvement. Download volumes are high, and tutorials appear on Bilibili and YouTube. Community feedback focuses on English experimentation and multi‑emotion expansion; 2Noise has pledged to open‑source spk_stats and detection models. Limitations include autoregressive variability, high compute requirements (GPU recommended) and noise embedding to prevent abuse.
Advantages, Limitations and Ethical Considerations
Advantages: open‑source and free, leading prosody, multilingual support—suitable for education, entertainment and assistant development. Limitations: immature English support, slower inference and potential stability issues. Ethically, the non‑commercial license restricts misuse and the project emphasizes responsible use, e.g., avoiding deepfakes or illegal content. Future work will advance streaming, multi‑emotion and cross‑platform support.
Conclusion
As an open‑source TTS pioneer, ChatTTS is pushing speech AI toward conversational realism. With easy installation and flexible use, it empowers developers to build vivid agents. The growing community ecosystem indicates vast potential, though technical and ethical balances remain essential. Developers can start from the GitHub repository to explore this speech revolution.