Google’s Next Gemini Move: An AI Agent That Works Your Apps for You

Google’s vision for the future of AI assistants will become a reality via its conversational chatbot interface Gemini Live in the next few months.

That was a revelation at the tail end of its Made by Google event in Mountain View, California, on Tuesday, during which the company also showed off its new Pixel 9 phones (including the Pixel 9 Pro Fold), the Pixel Watch 3 and the Pixel Buds Pro 2.

Rick Osterloh, senior vice president of platforms and devices at Google, said its next AI assistant, an AI agent known as Project Astra, will bring contextual understanding about where we are and what we’re doing to Gemini Live via our phone cameras.

While Project Astra sounds like a top-secret mission from NASA, it’s actually a prototype from Google’s AI research laboratory DeepMind. It extends the concept of an AI assistant from merely a question answerer to what’s known as an agent, which can take action on our behalf, like checking dates on a calendar or messaging a friend. All with our permission, of course.

The idea is that once we have AI agents, we won’t have to open other apps — we can simply talk to Project Astra (or a similar agent) while it pulls in necessary information from elsewhere on our devices. It’s a big opportunity for Google and its competitors as AI and search converge and the way we access information changes. And while Google may win the prize for most futuristic sci-fi moniker, consumer loyalty to an AI agent is still very much up for grabs. 

Project Astra + Gemini Live

There’s one small catch with the upcoming integration: Gemini Live, and therefore Project Astra, are available only to Gemini Advanced subscribers, who pay $20 per month for access to Google’s latest AI model, Gemini 1.5 Pro. 

If you fall into that camp, you’ll soon be able to share your camera during a conversation with Gemini to ask questions about what’s in front of you, whether it’s a calculus problem you don’t know how to solve or furniture you’re struggling to assemble.

Gemini Live will also be able to pull in information from apps like Google Calendar and Gmail to help answer your questions and share information without leaving the Gemini Live interface, Osterloh said.

We’ve seen similar functionality from AI startup OpenAI. At its Spring Update in May, OpenAI introduced conversational interactions with its ChatGPT chatbot, as well as the ability to share photos, videos and documents to help inform those conversations.

The voice functionality, known as Advanced Voice Mode, went live earlier this month for a small group of testers.

Both Project Astra and Gemini Live were introduced at the Google I/O developer event, which was also in May.

“We’re evolving Gemini to be even more agentive, to tackle complex problems with advanced reasoning, planning and memory, so you’ll be able to think multiple steps ahead, and Gemini will get things done on your behalf, under your supervision,” Osterloh said as Made By Google wrapped up. “That’s the promise of a true AI assistant.”