OpenOmni: A Fully Open-Source Omni Large Language Model with Real-time Self-Aware Emotional Speech Synthesis

[📖 arXiv Paper] [📊 Dataset (Coming Soon)] [🏆 Models(Coming Soon)]

OpenOmni is the end-to-end fully open-source pioneering method that successfully incorporates image,speech and text into the omni large language model. OpenOmni's design for speech generation through language bridging and text-guided speech can be quickly trained in situations where omni-modal data and VRAM resources are scarce. OpenOmni not only supports omni-modal nderstanding, but also supports two real-time emotional speech generation modes, CTC mode and AR mode, so that users can flexibly choose according to their needs to achieve a balance between generation speed and quality. The flexible framework design allows OpenOmni to be easily and quickly applied to a variety of downstream tasks, such as speech embodied navigation, multi-role-playing speech dialogue, etc. Everyone is welcome to come and experience it now!

🔥 Update

• [2025/01/13]🔥OpenOmni is coming! We release the code
• [2025/01/09]🔥After two months of company audit! We release the paper
• [2024/11/14]🔥We submit the paper for peer review

👀 Contents

• Setup
• Model
• Preparation
• Train
• Evaluation
• Example
• Citation

📷 Setup

Please follow the instructions below to install the required packages.

1. Clone this repository

git clone https://github.com/RainBowLuoCS/OpenOmni.git
cd OpenOmni

1. Install Package

conda create -n openomni python=3.10 -y
conda activate openomni
pip install --upgrade pip  # enable PEP 660 support
pip install -e .
pip install openai-whisper
pip install transformers==4.43.4
pip install -r requirements.txt

1. Install additional packages for training

pip install -e ".[train]"
pip install flash-attn --no-build-isolation

🔥 Fast Usage

After downloading the weights and configuring the paths properly. Two open-sourced speech tokenizer are needed for speech discretization and reconstruction with different vocabulary size! CosVoice for 6K CTC Mode and GLM4Voice for 16K AR Mode

Fast inference for omnimodal input (speech,text,image and video)

python inference.py

Fast interation for omnimodal input (speech,text,image and video)

python demo.py

Model

Here are the pretrained weights and instruction tuning weights

Stage	Model	Speech Projector	Image Projector	IT Data	Download
1-1	OpenOMNI-Qwen2-7B-Stage1-1	ckpt	ckpt	openomni_stage1-1.json	ckpt
2-1	OpenOMNI-Qwen2-7B-Stage2-1	ckpt	ckpt	openomni_stage2-1.json	ckpt
2-2	OpenOMNI-Qwen2-7B-Stage2-2	ckpt	ckpt	openomni_stage2-2.json	ckpt
3-1	OpenOMNI-Qwen2-7B-Stage3-1	ckpt	ckpt	openomni_stage3-1.json	ckpt
3-2	OpenOMNI-Qwen2-7B-Stage3-2	ckpt	ckpt	openomni_stage3-2.json	ckpt

Preparation

Dataset

Please follow MMEvol to prepare the corresponding images-text datasets. Here we only provide the details of speech-text datasets.

The following is the data directory tree of OpenOmni

data structure

datasets
├── json # data receipe
│   ├── openomni_stage1-1.json # speech2text pretraining
│   ├── openomni_stage2-1.json # image2text pretraining
│   ├── openomni_stage2-2.json # image2text instruction tuning
│   ├── openomni_stage3-1.json # text2speech pretraining
│   ├── openomni_stage3-2.json # text2speech emotional injection
├── asr # classic bilingual speech corpus
│   ├── AISHELL-4
│   ├── LibriSPeech
│   ├── WeNetSpeech
├── audio_en # synthetic english speech corpus for question
├── audio_llava # synthetic bilingual speech corpus for answer
├── audio_zh # synthetic chinese speech corpus for question
├── audio_unit # synthetic bilingual speech corpus for answer
├── audio_prefer # synthetic emotional bilingual speech corpus for answer
├── audio_reject # synthetic emotional bilingual speech corpus for answer
├── audio_ultrachat # synthetic bilingual speech corpus for answer
├── ai2d
│   ├── abc_images
│   ├── annotations
│   ├── images
│   ├── questions
│   └── categories.json
......

• All file/path starting with "audio" are self-synthesized.
• DPO contains approximately 9k entries for "prefer" and "reject," covering 9 types of emotions.

More details about data curation can be found in our paper.

Train

Speech2Text Pretrain

Please download the MMEvol, AIShell-4, LibriSPeech, WeNetSpeech, OpenOmni Data and organize the data following Preparation before training . Make sure set up the corresponding train script with correct setting (data path, weight path, and hyper-paramaters)

bash scripts/train/llama3/speech2text_pretrain.sh
bash scripts/train/qwen2/speech2text_pretrain.sh

Image2Text Pretrain

Please make sure you download and organize the data following Preparation before training. Make sure set up the corresponding train script with correct setting (data path, weight path, and hyper-paramaters)

bash scripts/train/llama3/image2text_pretrain.sh
bash scripts/train/qwen2/image2text_pretrain.sh

Image2Text Instruction Tuning

Please make sure you download and organize the data following Preparation before training. Make sure set up the corresponding train script with correct setting (data path, weight path, and hyper-paramaters)

bash scripts/train/llama3/image2text_finetune.sh
bash scripts/train/qwen2/image2text_finetue.sh

Text2Speech Pretrain

Please make sure you download and organize the data following Preparation before training. Make sure set up the corresponding train script with correct setting (data path, weight path, and hyper-paramaters)

bash scripts/train/llama3/text2speech_ pretrain.sh
bash scripts/train/qwen2/text2speech_ pretrain.sh

Text2Speech Emotional DPO Tuning

Please make sure you download and organize the data following Preparation before training. Make sure set up the corresponding train script with correct setting (data path, weight path, and hyper-paramaters)

bash scripts/train/llama3/text2speech_ dpo.sh
bash scripts/train/qwen2/text2speech_ dpo.sh

Evaluation

Dataset

Ensure that your api_base, key and dataset are correctly configured before evaluation.

data structure

datasets
├── json # data receipe
│   ├── aishell2_eval.jsonl # aishell evaluation
│   ├── librispeech_eval.jsonl # image2text pretraining
│   ├── wenetspeech_eval.json # image2text instruction tuning
│   ├── openomni_emotion_val.json 
├── OmniBench # OmniBench
│   ├── mmdata
│   ├── dataset
│   		├── eval.json
├── Ov-Odyssey # Ov-Odyssey Bench
│   ├── av_odyssey_part1.parquet
│   ├── av_odyssey_part2.parquet
│   ├── av_odyssey_part3.parquet
│   ├── av_odyssey_part4.parquet
│   ├── av_odyssey_part5.parquet

Speech-Text Evaluation

Make sure set up the corresponding train script with correct setting (data path, weight path, and hyper-paramaters)

python openomni/eval/llama3/asr_eavl.py
python openomni/eval/qwen2/asr_eavl.py

Model	LibriSpeech-test-clean	LibriSpeech-test-other	AIShell2-dev	AIShell2-test	WeNetSpeech-testnet	WeNetSpeech-testmeeting
VITA	8.1	18.4			12.2	16.5
EMOVA	4.0	8.6	10.6	10.3
MINI-OMNI	4.5	9.7
Freeze-Omni	3.29	7.4			8.57	10.09
ours	2.57	5.6	6.81	6.87	7.63

Image-Text Evaluation

Refer to MMEvol for detailed OpenCampass Vision Language Evaluation

# run on all 9 datasets
./script/run_inference.sh OpenOmni-Qwen "MME MMMU_DEV_VAL MathVista_MINI LLaVABench RealWorldQA MMStar MMVet AI2D_TEST OCRBench HallusionBench POPE BLINK" all

# The following are instructions for running on a single dataset
# MME
./script/run_inference.sh OpenOmni-Qwen MME all
# MMMU_DEV_VAL
./script/run_inference.sh OpenOmni-Qwen MMMU_DEV_VAL all
# MathVista_MINI
./script/run_inference.sh OpenOmni-Qwen MathVista_MINI all
.....

Speech-Text-Image Evaluation

Please download OmniBench and run the following command

python openomni/eval/llama3/omni_eavl.py
python openomni/eval/qwen2/omni_eavl.py

Speech-Text-Image-Video Evaluation

Please download Ov-Odyssey and run the following command

python openomni/eval/llama3/ov_odyssey_eavl.py
python openomni/eval/qwen2/ov_odyssey_eavl.py

Text-Speech Evaluation

python openomni/eval/llama3/t2s_eavl.py
python openomni/eval/qwen2/t2s_eavl.py

Emotional Text-Speech Evaluation

python openomni/eval/llama3/et2s_eavl.py
python openomni/eval/qwen2/et2s_eavl.py

📌 Cases

四是四，十是十，十四是十四，四十是四十。	黑化肥发灰，灰化肥发黑，黑化肥发灰会挥发，灰化肥挥发会发黑。	吃葡萄不吐葡萄皮，不吃葡萄倒吐葡萄皮。
zh_0.webm	zh_1.webm	zh_2.webm
八百标兵奔北坡，炮兵并排北边跑，炮兵怕把标兵碰，标兵怕碰炮兵炮。	红凤凰，黄凤凰，粉红凤凰，花凤凰。	牛郎年年恋刘娘，刘娘念念恋牛郎。
zh_3.webm	zh_4.webm	zh_5.webm

She sells seashells by the seashore.	Peter Piper picked a peck of pickled peppers.	Six slippery snails slid slowly seaward.
en_0.webm	en_1.webm	en_2.webm
Six sleek swans swam swiftly southwards.	I saw Susie sitting in a shoeshine shop.	Can you can a can as a canner can can a can?
en_3.webm	en_4.webm	en_5.webm

📚Citation

If you find this repo useful for your research, please consider citing the paper

@article{luo2025openomni,
  title={OpenOmni: Large Language Models Pivot Zero-shot Omnimodal Alignment across Language with Real-time Self-Aware Emotional Speech Synthesis},
  author={Run Luo, Ting-En Lin, Haonan Zhang, Yuchuan Wu, Xiong Liu, Min Yang, Yongbin Li, Longze Chen, Jiaming Li, Lei Zhang, Yangyi Chen, Hamid Alinejad-Rokny, Fei Huang},
  journal={arXiv preprint arXiv:2501.04561},
  year={2025}
}

@article{luo2024mmevol,
  title={Mmevol: Empowering multimodal large language models with evol-instruct},
  author={Luo, Run and Zhang, Haonan and Chen, Longze and Lin, Ting-En and Liu, Xiong and Wu, Yuchuan and Yang, Min and Wang, Minzheng and Zeng, Pengpeng and Gao, Lianli and others},
  journal={arXiv preprint arXiv:2409.05840},
  year={2024}
}

📧 Contact

if you have any question, please consider following concat for help

• Run Luo — [email protected]

• Haonan Zhang — [email protected]

Acknowledgement

- LLaVA and LLaVA-Omni: the codebase we built upon. Thanks for their brilliant contributions to the community! We just can't wait to use OpenOmni.

- VLMEvalKit: the amazing open-sourced suit for evaluating various LMMs!

- CosVoice: the amazing open-sourced speech tokenizer for speech discretization and reconstruction with 6k vocabulary size!

- GLM4Voice: he amazing open-sourced speech tokenizer for speech discretization and reconstruction with 16k vocabulary size!