UI-Genie-Agent-3B

Model Description

UI-Genie-Agent-3B is a state-of-the-art Multimodal Large Language Model specifically trained for mobile GUI automation tasks. It is part of the UI-Genie framework, which introduces a novel self-improving approach for enhancing MLLM-based mobile GUI agents through iterative agent-reward model co-evolution.

This model achieves state-of-the-art performance on mobile GUI benchmarks by eliminating the need for manual annotation through synthetic trajectory generation guided by our specialized reward model UI-Genie-RM.

Model Architecture

Base Model: Qwen2.5-VL-3B-Instruct
Training Method: Supervised fine-tuning with exisiting trajetory datasets and our synthetic trajectory data
Action Space Coverage: Supports comprehensive mobile interactions (click, swipe, type, etc.) and Set-of-Mark mode.

Performance

AndroidControl Benchmark

Model Size	Low-Level Tasks	High-Level Tasks
UI-Genie-Agent-3B	93.8% SR	72.9% SR
UI-TARS-2B	89.3% SR	68.9% SR
Qwen2.5-VL-3B	90.8% SR	63.7% SR

AndroidLab Benchmark

Model	Success Rate	Sub-Goal Success Rate
UI-Genie-Agent-3B	28.8%	35.4%
AutoGLM	36.2%	-
Qwen2.5-VL-7B	14.9%	18.7%

Training Data

Our model is trained on a combination of:

AndroidControl: 15.3K trajectories (high & low level tasks)
AMEX: 2.9K trajectories (high-level tasks)
AndroidLab: 726 trajectories (high-level tasks)
UI-Genie-Agent-16k: 2.2K synthetic trajectories (our generated data)

Action Space

The model supports a comprehensive action space for mobile interactions:

Action Type	Parameters	Description
`open`	app_name, action_desc	Launch applications
`click`	coordinate/som, action_desc	Tap UI elements
`swipe`	coordinate/som, direction, distance, action_desc	Scroll the screen
`long_press`	coordinate/som, action_desc	Long press interactions
`type`	text, action_desc	Text input
`system_button`	button, action_desc	System button presses
`wait`	time, action_desc	Wait operations
`terminate`	status, action_desc	Task completion

Citation

@misc{xiao2025uigenieselfimprovingapproachiteratively,
      title={UI-Genie: A Self-Improving Approach for Iteratively Boosting MLLM-based Mobile GUI Agents}, 
      author={Han Xiao and Guozhi Wang and Yuxiang Chai and Zimu Lu and Weifeng Lin and Hao He and Lue Fan and Liuyang Bian and Rui Hu and Liang Liu and Shuai Ren and Yafei Wen and Xiaoxin Chen and Aojun Zhou and Hongsheng Li},
      year={2025},
      eprint={2505.21496},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2505.21496}, 
}