Uni-Sign: Toward Unified Sign Language Understanding at Scale

If our project helps you, please give us a star🌟 on GitHub, that would motivate us a lot!

💥 News

[2025/1/25] This paper is accepted by ICLR 2025 🎉🎉!

[2025/2/24] Release CSL-News dataset and code implementation.

[2025/6/5] Release lighter pose extraction and online inference implementation. Check it out here 🎊🎊.

🛠️ Installation

We suggest to create a new conda environment.

# create environment
conda create --name Uni-Sign python=3.9
conda activate Uni-Sign
# install other relevant dependencies
pip install -r requirements.txt

📖 Preparation

Please follow the instructions provided in DATASET.md for data preparation.

🔨 Training & Evaluation

All scripts must be executed within the Uni-Sign directory.

For the pre-training code, please refer to Issue #15 for detailed instructions. Thanks to @williams-bert for generously sharing the code implementation.

Training

Stage 1: pose-only pre-training.

bash ./script/train_stage1.sh

Stage 2: RGB-pose pre-training.

bash ./script/train_stage2.sh

Stage 3: downstream fine-tuning.

bash ./script/train_stage3.sh

Evaluation

After completing stage 3 fine-tuning, performance evaluation on a single GPU can be performed using the following command:

bash ./script/eval_stage3.sh

👨‍💻 Todo

• Release CSL-News dataset
• Release Uni-Sign implementation
• Release pose extraction and online inference implementation

📮 Contact

If you have any questions, please feel free to contact Zecheng Li (lizecheng19@gmail.com). Thank you.

👍 Acknowledgement

The codebase of Uni-Sign is adapted from GFSLT-VLP, while the implementations of the pose/temporal encoders are derived from CoSign. We sincerely appreciate the authors of CoSign for personally sharing their code 🙏.
We are also grateful for the following projects our Uni-Sign arise from:

• 🤟SSVP-SLT: a excellent sign language translation framework!
• 🏃️MMPose: an open-source toolbox for pose estimation.
• 🤠FUNASR: a high-performance speech-to-text toolkit.

📑 Citation

If you find Uni-Sign useful for your research and applications, please cite using this BibTeX:

@article{li2025uni,
  title={Uni-Sign: Toward Unified Sign Language Understanding at Scale},
  author={Li, Zecheng and Zhou, Wengang and Zhao, Weichao and Wu, Kepeng and Hu, Hezhen and Li, Houqiang},
  journal={arXiv preprint arXiv:2501.15187},
  year={2025}
}