Gemini is getting better at understanding what’s on your phone screen

Google is updating Gemini on Android to let its AI better tap into what’s on your screen. The update should allow Gemini to lean into one of its best use cases, helping you make sense of a limited set of data as you go about your day.

If you set Gemini as the default assist on your Android phone, it can already summarize or answer questions about a webpage or a screenshot. Soon, it’ll also be able to tell if there’s a video on your screen and prompt you to ask questions about it. Gemini uses the video’s automatic captions to find answers — something you could already get it to do in a more roundabout way.

Gemini will also take a similar cue if you’re looking at a PDF, but there’s a catch: you’ll need access to Google’s paid version, Gemini Advanced, to use it. That’s because the feature ingests the entire PDF, so it requires the long context window available to Gemini Advanced subscribers. But once it has taken the PDF on board, you’ve basically turned it into an expert on whatever that topic is — maybe that’s your dishwasher owner’s manual or your local curbside recycling guidelines. Gemini Advanced is part of the $20 per month Google One AI Premium plan.

There’s one more minor update, too — you’ll soon be able to drag and drop images generated by Gemini into whatever it is you’re working on without having to jump between apps. You’ll just long-press an image in the Gemini overlay and drag it into a chat or an email. Altogether, it has the net effect of making Gemini feel less like a Thing You Have To Go Get and something that’s just integrated seamlessly with the rest of the system.

It’s also a reminder that Google has been pursuing this dream of context-aware search for well over a decade — remember Google Now? I think it’s a notable step forward; the best use of Gemini Assistant I’ve come across is asking it to remember a dinner recipe so I can ask it questions as I’m moving around the kitchen cooking. Sounds simple, but it feels a lot more practically useful to me than questioning AI about all the knowledge on the entirety of the internet.

Google’s Gemini on Android updates will be rolling out to “hundreds of millions of devices over the next few months,” and more contextual features are in the works.