As AI agents rapidly advance, the smartphone—already the primary gateway for daily interactions—is poised for a revolutionary change.In recent days, many Chinese netizens were repeatedly exposed to an AI smartphone assistant called Doubao Smart Phone Assistant; the controversy drew responses from platforms such as Tencent and Alibaba, leading to restrictions and forced downgrades. Just now, however, the Open‑AutoGLM project maintained on GitHub by zai‑org (https://github.com/zai‑org/Open‑AutoGLM) has been open‑sourced. It aims to "unlock AI phones for everyone." The project was officially released in early December 2025 and quickly attracted developer community attention. As of December 11, 2025, the repository has gathered about 5,700+ stars and shows active updates, with the most recent commit on December 9. As part of Z.ai's open ecosystem, Open‑AutoGLM builds on AutoGLM core technology and is designed specifically for Android devices, enabling screen‑aware, multimodal understanding and automated operations via natural language commands. It is more than a toolkit: it is a comprehensive solution for building "autonomous phone agents" to free users from tedious manual operations. This article analyzes the project background, core architecture, features, installation and deployment, practical examples, performance benchmarks, community ecosystem and future outlook of this open‑source "phone brain."
Project background: An open‑source leap from the GLM family to mobile agents
Z.ai (formerly Zhipu AI), a leading Chinese AI company rooted in Tsinghua University technology, is known for the GLM (General Language Model) series. From ChatGLM‑6B in 2022 to higher‑capability inference models such as GLM‑4.5 and GLM‑130B in 2025, Z.ai has emphasized "safe and beneficial general artificial intelligence" (AGI). Open‑AutoGLM extends its agent technology to mobile devices and is inspired by AutoGLM—an autonomous GUI agent framework. AutoGLM was originally designed for desktop environments and can use vision‑language models (such as GLM‑4V) to understand screenshots, plan action sequences, and achieve high autonomy in complex tasks. Open‑AutoGLM ports this ability to smartphones, focusing on the Android ecosystem to address the fragmentation of mobile operations.
The project launch coincides with an AI agent boom: by 2025, models such as OpenAI's GPT‑4o and Anthropic's Claude already supported tool use, but were mostly cloud‑based and lacked mobile optimizations. Z.ai chose to open‑source the AutoGLM‑Phone‑9B model (9B parameters) and provide a full framework emphasizing "local deployment and full control." This lowers the barrier to entry and mitigates privacy risks—user data need not be uploaded to the cloud. According to the project README, Open‑AutoGLM supports mainstream Chinese and English apps and covers over 50 high‑frequency scenarios such as social networking, e‑commerce and maps/navigation. Its goal is to "make AI the invisible assistant on your phone": for example, a user saying "open Xiaohongshu and search for food" would have the agent parse the intention, navigate the interface, input the query and return results automatically.
The repository is released under the MIT license to encourage global contributions. The maintainer zai‑org is Z.ai's official organization, which also hosts popular repositories like ChatGLM3 and CogVLM, totaling over 50k stars. Recent updates include uploading model weights to Hugging Face and ModelScope and optimizing the framework's ADB integration, reflecting rapid iteration.
Core architecture: A closed‑loop system of multimodal perception + intelligent planning
Open‑AutoGLM's technical architecture is concise and powerful, divided into perception, planning and execution layers that form a "understand‑decide‑act" loop.
First, the perception layer relies on vision‑language models (VLMs) for screen understanding. The core model, AutoGLM‑Phone‑9B, is based on the GLM‑4.1V‑9B‑Thinking architecture, combining a 9B‑parameter transformer structure and supporting image inputs (such as screenshots) and text prompts. It can recognize interface elements (buttons, text fields), understand context (current app state) and generate descriptive outputs. For example, given a WeChat chat screenshot, the model can output "currently in WeChat Moments, with a search bar and a post button at the top." This layer draws on CogVLM's multimodal fusion techniques to ensure high‑accuracy OCR and object detection.
Second, the planning layer introduces AutoGLM's "reflection" mechanism and uses a large language model (LLM) to generate action sequences. The framework decomposes user intent into subtasks such as "open app → switch tab → enter keyword → tap search." The planning algorithm combines the ReAct (Reasoning + Acting) paradigm and supports tool calls (e.g., generating ADB commands). To improve robustness, the project integrates an error‑recovery module: if an operation fails, the agent re‑senses the screen and adjusts the plan.
Finally, the execution layer bridges to hardware via Android Debug Bridge (ADB). ADB allows the framework to remotely control a phone from a PC, including tap coordinates, swipes and text input. The architecture supports wireless ADB for real‑device testing. Overall, Open‑AutoGLM is compatible with Python 3.8+ and depends on libraries such as torch, transformers and opencv‑python for model inference and image processing.
Compared with closed‑source competitors like Google's Project Astra, Open‑AutoGLM's openness is its biggest advantage: developers can fine‑tune models and adapt them to custom apps or iOS extensions (although current focus is Android).
Features: From simple commands to complex workflows
Open‑AutoGLM stands out for practicality and extensibility. Key features include:
- Natural language interaction: Users can describe requests in Chinese or English and the agent parses them automatically. Nested commands are supported, e.g., "help me book an Uber to the nearest supermarket and buy groceries."
- Multimodal support: Integrates VLMs to handle screen images, audio (planned) and text, ensuring cross‑language compatibility. It has been validated on 50+ apps, including WeChat (send messages, transfers), Taobao (search and order), Douyin (like videos), Meituan (order food), Gmail (check mail) and Google Maps (navigation).
- Automated workflows: Generates executable scripts with loops (e.g., "refresh 10 news items") and conditional branches (e.g., "if balance is insufficient, switch payment method").
- Local‑first deployment: No cloud dependency, suitable for privacy‑sensitive scenarios. Model weights are open‑source and inference can run on consumer GPUs (e.g., RTX 3060).
- Extension interfaces: Provides API hooks for integrating third‑party tools such as voice input (Whisper) or external database queries.
In addition, the framework includes safety mechanisms: high‑risk actions (such as payments) require user confirmation before execution, and operations are logged to prevent misuse.
Installation and deployment: A zero‑barrier getting‑started guide
Deploying Open‑AutoGLM is straightforward; the README provides step‑by‑step instructions. First, clone the repository:
git clone https://github.com/zai-org/Open-AutoGLM.git cd Open-AutoGLM
Install dependencies (using conda or venv):
pip install -r requirements.txt # Core dependencies: torch>=2.0, transformers, opencv-python, adbutils
Configure ADB: enable USB debugging on the phone, connect to the PC and run adb devices to verify. For wireless mode, use adb connect <IP:PORT>.
Download the model from Hugging Face:
from transformers import AutoModelForCausalLM, AutoTokenizer model = AutoModelForCausalLM.from_pretrained("zai-org/AutoGLM-Phone-9B") tokenizer = AutoTokenizer.from_pretrained("zai-org/AutoGLM-Phone-9B")
Start the agent server: run python main.py ‑‑device_id <ADB_DEVICE_ID> to enter interactive mode. The whole process takes under 10 minutes and is suitable for beginners.
Practical examples: From beginner tasks to advanced workflows
The project provides rich examples that demonstrate the framework's capabilities. Beginner example: open Xiaohongshu and search for food.
User input: "Open Xiaohongshu and search for food recommendations."
Agent flow:
- Perception: capture the home screen and recognize the Xiaohongshu icon.
- Planning: tap the icon → wait for loading → tap the search bar → input "food recommendations."
- Execution: ADB sends tap(coordinates) and input_text commands.
Code snippet (simplified):
from autoglm_phone import PhoneAgent agent = PhoneAgent(model_path="zai-org/AutoGLM-Phone-9B") result = agent.run("Open Xiaohongshu and search for food") print(result) # Output: Task completed, search results page loaded
Advanced example: an e‑commerce ordering workflow—"Buy an iPhone case on Taobao, red, under 50 RMB."
The agent will: open Taobao → search "iPhone case red" → filter by price → add to cart → proceed to checkout (payment requires manual confirmation). Success rates exceed 85% depending on network and interface changes.
Another highlight is multi‑step tasks such as "transfer 100 RMB to friend Xiaoming via WeChat, then send a confirmation message." The framework's reflection mechanism helps it adapt even if the interface hangs.
Performance benchmarks: Empirical efficiency and reliability
Benchmarking shows AutoGLM‑Phone‑9B achieves end‑to‑end latency of 2–5 seconds per step on Android devices, outperforming similar frameworks like AppAgent (7s+ latency). On a dataset of 50 high‑frequency tasks, the success rate is 78% (higher for Chinese apps at 82%), with coverage for social (90%), e‑commerce (75%) and navigation (85%). Compared with closed‑source models, its local GPU energy consumption is about 30% of cloud‑based agents. The project plans to release more benchmark datasets to drive community optimization.
Community ecosystem and contributions: The vitality of open collaboration
The Open‑AutoGLM community is active on GitHub Issues and Reddit (e.g., r/LocalLLaMA). The contribution guide encourages PRs for adding new app support, model fine‑tuning scripts or bug fixes. Currently there is one open issue focused on iOS compatibility. Z.ai provides SDKs (such as a Python interface) to facilitate integration into existing projects. In the future, the community may expand into voice agents or cross‑device collaboration.
Impact and future outlook: An open‑source pioneer for AI phones
Open‑AutoGLM's open sourcing not only democratizes agent technology but also offers practical benefits for people with disabilities and knowledge workers. Imagine an elderly person using voice to automatically book a doctor appointment, or developers automating app testing. Challenges remain—dynamic UI changes and privacy compliance—but Z.ai's AGI vision will drive further iterations.
Looking toward 2026, the project may integrate lighter models such as GLM‑4.5‑Air to support edge computing. Whether you are a developer or an AI enthusiast, Open‑AutoGLM is a repository worth starring. It demonstrates that open source is a key to an intelligent future.
▶️ Creating content is not easy—please subscribe to our YouTube channel: https://www.youtube.com/@airosetta and like and save! 🙏