📅 July 2026
PhoneCLI combines the reliability of CLI macros with the flexibility of GUI agents. Instead of calling a VLM for every tap — slow, expensive, and error-prone — we pre-build a navigation graph (app map) for each app. Routine operations become deterministic macro replays; the VLM only steps in when genuinely needed. Same intuition as why CLI tools beat GUI for repeatable tasks — now applied to your phone.
- 🗺️ App Maps — BFS crawls each app, recording every screen, element, and navigation path into a structured YAML graph
- ⚡ Macro Replay — High-frequency operations execute deterministically in sub-second time, with zero VLM cost
- 🧠 Smart Fallback — When a task doesn't match any macro, the agent gracefully degrades to VLM reasoning
- 📦 8 Pre-Built Maps — 微博, foodpanda, Calendar, 京东, Dianping, 小红书, Music, Settings — ready to use
- 🔗 Cross-App Planner — Multi-app tasks automatically decomposed into single-app subtasks
- 🎯 What is OpenPhone?
- 🤔 Why 3B Parameters?
- 💡 Research Highlights
- 🚀 Model Release & Resources
- 🚀 Quick Start
- 🖥️ PhoneCLI: GUI × CLI Hybrid Agent
- 🌟 Key Features of OpenPhone
- 🌟 Technical Innovation & Implementation
- 🧪 Testing & Evaluation
- 📊 Result Generation
- 🎯 📊 Key Evaluation Findings for OpenPhone
- 📈 Device-Cloud Distribution Analysis for Phone Agents
- ⚡ Inference Speed Comparison
- 🌟 Citation
- 🔗 Related Projects
- 📜 License
The Problem: Most AI agents rely on expensive cloud APIs and large models that are impractical for real-world on-device deployment. Users face Privacy Concerns, Latency Issues, and High Costs when their phone needs to call external services for every interaction.
Our Solution: OpenPhone introduces the first Open-Source, 3B-parameter Agentic Foundation Model designed specifically for on-device smartphone interaction. This compact vision-language model runs entirely locally — meaning No Privacy Concerns, No Cloud Dependence, and Zero API Costs.
We believe the future of mobile AI lies not only in making models larger, but in making them smarter and more efficient for real-world constraints. Our 3B model is:
- ⚡ Edge-Optimized: Efficient enough for commodity GPUs and next-generation mobile NPUs.
- 🔒 Privacy-First: All computation stays on your device.
- 💰 Cost-Free: No cloud inference and no ongoing API fees.
- 🎯 High-Performance: Achieves performance comparable to 7B–9B models through advanced training.
Considering the compute limitations of today’s edge devices, models with ≤3B parameters strike a practical balance between capability and deployability. Based on this insight, we introduce OpenPhone‑3B, a lightweight yet powerful on‑device agent model.
- Model Size & Architecture: Vision-language model engineered for efficient on-device reasoning under tight mobile compute constraints.
- Edge-Native Design: Primary local agent compatible with consumer GPUs and mobile NPUs, eliminating continuous cloud dependency.
- GUI‑Aware Action Capabilities: Trained for visual interpretation, instruction following, and structured action generation across real mobile tasks.
- Open‑Source Release: Full model weights, configurations, and inference stack enabling community deployment and development.
- Practical Sweet Spot: 3B scale delivers optimal balance—significantly stronger than tiny models while remaining deployable where larger models fail.
- Hardware Fit: 3B parameters align perfectly with consumer GPU memory (8-12GB) and emerging mobile NPU computational budgets.
- Speed Advantage: 3B models deliver 3-5x faster inference than 7B alternatives while maintaining competitive accuracy for sub-second GUI responses.
- Power Efficiency: Smaller footprint extends battery life - essential for mobile deployment where power consumption affects user experience.
- Privacy-First: Enables phone tasks to run entirely on-device, preserving user privacy while eliminating network dependencies.
- Cost Savings: Local processing eliminates expensive cloud APIs and per-request charges for sustainable operation.
- Model Weights: OpenPhone-3B is available on Hugging Face with full licensing for research and commercial use.
- Production-Ready Serving: Pre-configured vLLM inference scripts enable efficient deployment with optimized throughput and memory usage.
- Reproducible Recipe: Full training implementation including our novel two-stage approach (SFT + GRPO-style RL with synthetic GUI data).
- Customization Support: Detailed documentation in model_training/ allows researchers to adapt the model for domain-specific phone tasks or extend to new mobile platforms.
- Data Generation Paradigm: Scripts and methodologies for creating high-quality training data at scale.
This project comprises core components designed for comprehensive mobile agent development and evaluation:
- 🖥️ For the phone agent CLI, check out phonecli — build app maps and run autonomous tasks on iOS devices.
- ⚡ For model training, please refer to the training guide README for comprehensive setup and execution instructions.
- 🔧 For the data generation pipeline, please refer to the data preparation guide README for detailed implementation steps.
Below, we focus on evaluation using the AndroidLab benchmark framework.
Installation: Follow the official AndroidLab documentation AndroidLab for complete setup instructions.
Environment Configuration:
- Recommended Mode: AVD on Mac (arm64) - validated in our experiments.
- App Setup: Manual installation and task-specific configuration required.
- Compatibility Note: Original Docker images are not compatible with AVD environments.
vLLM Integration:
- Inference scripts available in ./vllm_script/ directory
- Optimized for efficient small model serving
Model Access:
- OpenPhone Weights: 3B parameter model hosted on HuggingFace
- Deployment Process: Download weights → Deploy via vLLM → Configure inference service
- Service Ready: Seamless integration with evaluation pipeline
- API Setup Required: Configure cloud model credentials in ./evaluation/evaluation.py: Line 63, Line 75, Line 81
- Coming Soon: Streamlined configuration interface in development
Think about how you use your phone. Turning on Wi-Fi, checking today's calendar, searching for a contact — these are repetitive fixed sequences you execute dozens of times. Doing them through VLM-driven GUI interaction is slow (2–5s per step), expensive (every screenshot costs an API call), and fragile (VLMs hallucinate coordinates).
PhoneCLI takes a different approach. Instead of treating every task as a novel GUI exploration, it pre-builds a navigation graph (app map) for each app, then replays routine operations as deterministic macros. The VLM only gets called when genuinely needed — for novel tasks, verification, or cross-app reasoning.
1. Build an app map — A BFS crawler systematically explores every screen of an app via WebDriverAgent, recording all tappable elements, their coordinates, and the navigation paths between screens. Each element is classified (stable vs. dynamic) and enriched with semantic metadata by an LLM.
# The app map encodes every screen as a node, every element as an edge
screens:
- id: screen_0
description: "Settings main page"
elements:
- text: Wi-Fi
center: [0.50, 0.15] # normalized coordinates
fixed: true # stable across app updates
leads_to: screen_1 # where this tap navigates
screen_macros:
screen_1: [force_stop, launch, tap(195, 126)] # exact replay sequence2. Run a task — The agent maps a natural language task to a specific macro operation, replays it deterministically, then optionally verifies with a single VLM screenshot check. Most routine tasks complete with 0–1 VLM calls.
3. Handle the unexpected — When a task doesn't match any macro (e.g. "Find restaurants near me that are open late"), the agent falls back to pure VLM reasoning. The system gracefully degrades to GUI mode only when needed.
The intuition is simple: don't ask a VLM to re-discover what you already know. Pre-compute the navigation graph once, then replay it reliably forever. This is the same principle that makes CLI tools faster than GUI — applied to phone automation.
# 1. Build a map for any iOS app (~10 minutes, one-time)
python phonecli/cli.py macro auto-build -b com.apple.Preferences -a Settings
# 2. Run a task
python phonecli/run.py --task "Turn on airplane mode"
# 3. Interactive daemon (continuous multi-task session)
python phonecli/run.py --interactiveThe package ships with pre-built maps for 8 apps (微博, foodpanda, Calendar, 京东, Dianping, 小红书, Music, Settings), each covering 20–50 screens and hundreds of elements.
➜ Full phonecli documentation — setup, app map building, CLI reference, troubleshooting.
• Compact Architecture: Specialized 3B-scale Vision-Language Models optimized for mobile GUI tasks with minimal computational footprint.
• On-Device Deployment: True smartphone-compatible models that maintain competitive performance while running locally without cloud dependency.
• Dynamic Orchestration: Real-time task complexity assessment that intelligently switches between device and cloud models based on execution requirements.
• Cost-Performance Optimization: Strategic resource allocation that leverages cost-efficient on-device models while compensating limitations through selective cloud model usage.
• Extended Benchmark Suite: Beyond AndroidLab, incorporating 25+ additional tasks across popular mobile applications for real-world validation.
• Multi-Dimensional Assessment: Comprehensive evaluation covering performance metrics, computational efficiency, and practical deployment scenarios.
• Synthetic Data Generation: Leverages advanced MLLMs to create high-quality reasoning chain training data, addressing the scarcity of manual annotations.
• Two-Stage Training: SFT injects GUI foundational knowledge, while GRPO reinforcement learning optimizes task completion accuracy.
• Small Model Enhancement: Enables 3B models to achieve performance comparable to 7B-9B models on GUI tasks through structured training.
• Dynamic Task Assessment: Real-time complexity evaluation determines when and how frequently to monitor device model performance.
• Intelligent Orchestration: Seamlessly switches between device and cloud models based on execution progress and failure patterns.
• Cost-Performance Optimization: Reduces cloud invocations by ~10% while maintaining high task success rates through strategic resource allocation.
• Long-Horizon Reasoning: Multi-step chain-of-thought reasoning with reflective error correction to enhance decision-making capabilities.
• Text-Based Summarization: Compresses high-resolution screenshots into compact textual representations for efficient memory management.
• Structured Context Retention: Maintains 10-20 steps of historical context in resource-constrained environments through optimized token usage.
Test individual tasks using the following command structure:
python eval.py -n test_name -c your path to config.yaml --task_id task_idExample Usage:
python eval.py -n all_cloud_v1_hyper -c ./configs/example_xml_cloud_hyper.yaml --task_id zoom_1Convenient batch testing scripts are available in ./test_script:
• all_test_cloud_v1_hyper.sh: Evaluates all 138 AndroidLab benchmark tasks
• all_test_cloud_v1_hyper_add.sh: Evaluates tasks for four additional mobile apps
For comprehensive details about the four additional app tasks, refer to the documentation: Additional Apps Documentation
Required Configuration: Set up LLM service credentials in ./evaluation/tasks/llm_evaluator.py:
• Line 10: API configuration
• Line 12: Service URL
💡 Enhancement: Our implementation replaces AndroidLab's rule-based evaluation with LLM-powered assessment, providing more nuanced and accurate task completion evaluation.
Execute result generation with the following command:
python generate_result.py --input_folder ./logs/evaluation/ --output_folder ./logs/evaluation/ --output_excel ./logs/evaluation/test_name.xlsx
• Manual Transfer Required: Move generated evaluation files from script directory to ./logs/
• Then Execute: Run the result generation command above
• Error Prevention: This step prevents file path conflicts and ensures proper result compilation
- Size vs Performance: OpenPhone-3B achieves performance comparable to 9B models while maintaining the deployment advantages of a compact architecture.
- Efficiency Champion: Establishes itself as a genuine "small powerhouse" that challenges the bigger-is-better assumption in mobile AI.
- Against Proprietary Models: OpenPhone-3B shows respectable performance compared to lightweight versions of proprietary models when evaluated on standard benchmarks.
- Potential of Small Models: Demonstrates promising results that validate the viability of compact open-source approaches in mobile agent development.
- Performance with Efficiency: OpenPhone's hybrid architecture delivers near-optimal performance while dramatically reducing cloud model usage.
- Intelligent Routing: Proves that smart task routing creates practical efficiency gains without sacrificing capability.
- Context Matters: Extended prompting strategies only improve performance when paired with sufficiently capable cloud models.
- Smart Matching: Highlights the importance of matching reasoning complexity to model capability rather than assuming longer prompts always help.
To evaluate the practical efficiency of our hybrid approach, we measured key metrics across different MLLMs: average total steps per task, the proportion of steps handled by on-device versus cloud models, and cloud call reduction compared to cloud-only baselines.
Cloud models still handle approximately 65% of execution steps, reflecting the computational limitations of smaller on-device models for complex reasoning tasks.
Introducing on-device processing achieves roughly 10% reduction in cloud API calls, translating to direct cost savings and reduced latency.
Advanced cloud models like GLM-4.5V show smaller reductions in cloud dependency, as their superior capabilities enable more independent task completion without requiring on-device assistance.
We evaluated average inference time per step using vLLM across different GPU configurations to assess real-world deployment feasibility. Note that GLM-4.1V-9B-Thinking could not operate on a single 3090 GPU due to context length constraints.
| Model | GPUs | Size | SR | Time Cost / Step |
|---|---|---|---|---|
| Qwen2.5-VL-7B-Instruct | Single 3090 | 7B | 10.1 | 6289.15 ms |
| OpenPhone | Single 3090 | 3B | 15.2 | 4170.63 ms |
| GLM-4.1V-9B-Thinking | Two 3090s | 9B | 24.6 | 14584.89 ms |
| Qwen2.5-VL-7B-Instruct | Two 3090s | 7B | 10.1 | 4587.79 ms |
| OpenPhone | Two 3090s | 3B | 15.2 | 3524.25 ms |
- Clear Winner: OpenPhone demonstrates significant inference speed advantages thanks to its lightweight 3B architecture
- Real-World Ready: Speed benefits become increasingly pronounced under constrained computational resources, matching typical edge deployment scenarios
- 3.5x Faster: OpenPhone on single 3090 vs GLM-4.1V-9B-Thinking on dual 3090s.
- 4x Faster: OpenPhone on dual 3090s vs GLM-4.1V-9B-Thinking on dual 3090s.
- OpenPhone's Lightweight: GLM-4.1V-9B-Thinking's inability to run on single 3090 severely limits edge deployment options.
The trade-off is clear: while larger models like GLM-4.1V-9B-Thinking achieve higher task performance, OpenPhone's speed advantages make it far more suitable for real-world on-device scenarios where response time and hardware constraints matter.
If you find this work helpful to your research, please kindly consider citing our paper.
@inproceedings{jiang2026openphone,
title={OpenPhone: Mobile Agentic Foundation Models},
author={Jiang, Yangqin and Huang, Chao},
booktitle={Findings of the Association for Computational Linguistics: ACL 2026},
pages={30362--30380},
year={2026}
}
OpenPhone builds upon excellent open-source projects. We sincerely thank their authors and contributors:
- AndroidLab - The benchmark framework.
- R1-V - Implementation details for the GRPO training methodology.
- LLaMA Factory - The unified training framework enabling efficient model fine-tuning.
This project is released under the MIT License.
If this project helps you, please give us a Star🌟
🤖 Empower AI Phone with Agents!
❤️ Thanks for visiting ✨ OpenPhone!