Google is changing how it measures the software development capabilities of large language models. However, the updated results are making the tech giant look a bit awkward. The Android development team just announced a major overhaul to Android Bench, its specialized leaderboard designed to rank how well different artificial intelligence agents handle real-world Android programming challenges.
The biggest technical shift is that Google dropped its old methodology to adopt the standardized Harbor framework. This update migrates the benchmark into secure sandbox environments, making it significantly easier for developers to run independent evaluations, test individual development setups, and share clear data. Because the rules of the playground changed, Google had to re-run and rescore every single AI model on the board to establish a fresh baseline.
Anthropic takes the crown
Alongside the architectural upgrade, Google added eight new frontier models to the leaderboard, including heavy hitters like Claude Fable 5, Sonnet 5, GLM 5.2, and Qwen 3.7 Max.
The clear standout is Anthropic’s highly restricted flagship, Claude Fable 5, which immediately claimed the top spot. Fable 5 achieved an impressive 84.5% accuracy score, landing a comfortable four points ahead of OpenAI’s GPT-5.5, which scored 80.2%.
Unfortunately for Google, its own native models are struggling on their home turf. Gemini 3.1 Pro currently sits in a disappointing fifth place, trailing behind both Anthropic and OpenAI. Even worse, Gemini 3.5 Flash absolutely cratered in terms of efficiency. It racked up a massive $165 operational cost per run and a brutal 28-hour runtime because it took so long to parse the 100-problem evaluation dataset.
The true cost of clean code
Fable 5 and GPT-5.5 are undeniably the smartest Android developers on paper. However, their accuracy comes at a steep premium. Running the 100-problem, 10-run benchmark costs more than $130 in token consumption for those top-tier models. By comparison, Google’s lower-scoring Gemini 3.1 Pro only cost $87 to finish the exact same workload.
To keep the platform relevant as coding models evolve, Google is opening up Android Bench to the global programming community. Through the project’s official GitHub repository, developers can now submit their own custom Android development tasks and share their independent benchmark evaluations. The Android team will review these community submissions before officially adding them to the live dataset.
The Android Headlines Take
It’s definitely nice to see that Google maintains an objective, transparent leaderboard even when its own internal products lose. Stumbling on Android Bench is an undeniable setback for Google, especially as the company aggressively transitions its core engineering projects toward autonomous, agentic development workflows.
This objectivity, combined with opening the platform up via the Harbor framework and GitHub, helps Google turn Android Bench into the definitive, unbiased destination for AI code evaluation. If you are a developer and want to know the capabilities and objective costs of an AI model outside of all the marketing fluff, you know where to go. Android Bench will show you exactly how many tokens an agent will chew through before delivering working code.
The post Google Changes the Rules for Ranking Android Coding AI—and Gemini Is Still Lagging Behind appeared first on Android Headlines.
