Google has begun rolling out new AI capabilities to Gemini Live that enable the assistant to “see” through your smartphone camera or screen and answer real-time questions about what it detects.
Confirmed by Google spokesperson Alex Joseph in an email to The Verge, these features follow the demonstration of Google’s “Project Astra” nearly a year ago. According to reports, a Reddit user first noticed the feature on their Xiaomi phone, as highlighted by 9to5Google.
One of the two key features being rolled out is screen-reading, which allows Gemini to interpret what’s on the user’s screen and respond accordingly. The second feature enables real-time video interpretation via the smartphone camera, where users can ask questions about the feed. A recent video by Google shows someone using the live video feature to ask Gemini for advice on selecting a paint color for their pottery.
These AI tools are part of Google’s push to expand its assistant capabilities, available to Gemini Advanced Subscribers under the Google One AI Premium plan. The features are launching in tandem with similar developments from Amazon and Apple, who are preparing to release upgraded versions of their virtual assistants, Alexa and Siri, respectively. While Samsung’s Bixby remains a competitor, Gemini remains the default assistant on Samsung devices. This rollout marks a significant leap in AI integration into everyday devices.
