OneIG-Bench: Omni-dimensional Nuanced Evaluation for Image Generation

Project lead, Correspondence Authors
1Shanghai Jiao Tong University, 2StepFun
  • [2025-07-23] 🌟 We updated the evaluation results of Step-3o Vision.
  • [2025-07-03] 🌟 We updated the evaluation results of Ovis-U1.
  • [2025-06-25] 🌟 We updated the evaluation results of Show-o2 and OmniGen2.
  • [2025-06-23] 🌟 We released the T2I generation script here.
  • [2025-06-10] 🌟 We released the OneIG-Bench benchmark on 🤗 Hugging Face.
  • [2025-06-10] 🌟 We released the tech report and the project page.
  • [2025-06-10] 🌟 We released the evaluation scripts.
OneIG-Bench Image
OneIG-Bench is a comprehensive benchmark framework for fine-grained evaluation of T2I models across multiple dimensions.

Note: If you would like to submit your results, please contact us at oneigbench@outlook.com.

Introduction

Text-to-image (T2I) models have garnered significant attention for generating high-quality images aligned with text prompts. However, rapid T2I model advancements reveal limitations in early benchmarks, lacking comprehensive evaluations, for example, the evaluation on reasoning, text rendering and style. Notably, recent state-of-the-art models, with their rich knowledge modeling capabilities, show promising results on the image generation problems requiring strong reasoning ability, yet existing evaluation systems have not adequately addressed this frontier. To systematically address these gaps, we introduce OneIG-Bench, a meticulously designed comprehensive benchmark framework for fine-grained evaluation of T2I models across multiple dimensions, including prompt-image alignment, text rendering precision, reasoning-generated content, stylization, and diversity. By structuring the evaluation, this benchmark enables in-depth analysis of model performance, helping researchers and practitioners pinpoint strengths and bottlenecks in the full pipeline of image generation. Specifically, OneIG-Bench enables flexible evaluation by allowing users to focus on a particular evaluation subset. Instead of generating images for the entire set of prompts, users can generate images only for the prompts associated with the selected dimension and complete the corresponding evaluation accordingly.

Benchmark Comparison

OneIG-Bench Image
 Unless otherwise specified, OneIG-Bench refers to OneIG-Bench-EN in the subsequent results sections.

Method Comparison Radar Charts on OneIG-Bench

Leaderboard (Methods & Scores for Each Metrics)

BibTeX

@article{chang2025oneig,
  title={OneIG-Bench: Omni-dimensional Nuanced Evaluation for Image Generation}, 
  author={Jingjing Chang and Yixiao Fang and Peng Xing and Shuhan Wu and Wei Cheng and Rui Wang and Xianfang Zeng and Gang Yu and Hai-Bao Chen},
  journal={arXiv preprint arxiv:2506.07977},
  year={2025}
}