According to the blog post, Google has been ramping up its visual search app, Lens, with the ability to answer questions about your surroundings in near-real-time. The users of the Google app on both English-speaking Android and iOS devices can now start a video capture with Lens and then ask about specific things within that video.
According to Lou Wang, director of product management for Lens, the feature taps into a “customised” Gemini model-the company’s suite of AI models that underpin everything from Search to Translate to Autosuggest-to understand the video and related questions.
Imagine that I want to learn more about cool fishes,” Wang said at a news conference. More resources with helpful text will allow the overview to describe why they are taking this kind of circle path. 
To try the new video analysis feature of Lens, you need to first enroll in the Google Search Labs program and enable the experimental Labs feature called “AI Overviews and more”. The video-capturing mode of Lens is activated inside the Google app by long-pressing on the camera shutter button of your smartphone.
You can also ask questions while recording a video, and with Lens, you will be linked to an answer using AI Overviews, a feature of Google Search that leverages AI to compile data from across the internet in one place.
Wang says Lens uses AI to pick out the frames in a video that are most “interesting” and salient – crucially, which are relevant to the posed question – and uses those to “ground” the response from AI Overviews.
“The insight is from observing how users are interacting today with tools like Lens,” Wang said. “By lowering those barriers to these questions and helping users scratch that curiosity itch, they likely will adopt this quite organically.
In fact, video’s introduction to Lens comes not too long after a similar feature that Meta showed last month on its own AR glasses, Ray-Ban Meta. On the latter, Meta will integrate real-time AI-powered video features into the glasses; one day they will have the capability to enable users to query what they are seeing, including questions like, “What flower is this?
OpenAI has hinted at this in the future, where the Advanced Voice Mode tool will have the capability to understand videos. Later, Advanced Voice Mode—a ChatGPT exclusive—will include the ability to analyse videos in real time, keeping context in mind while responding to users.
That means it has managed to outcompete both firms, but we also have to keep in mind that the app works in an asynchronous manner-a proper real-time chatting isn’t possible-and this is in a case when the video feature works as it’s said to. In an attempt to cut them some slack, the feature was not demoed live during the press briefing, and there is rich history when it comes to Google overpromising how good its AI technologies work.
Besides video analysis, Lens now supports image-text searches in one go. Users in English and not part of Labs can open the Google app, hold the shutter button for a photo, and after asking their question, it is fast-tracked to results.
From now on, if Lens finds a product on both Android and iOS, it will highlight useful information about the price, available deals, brand info, reviews, and where the item is in stock. Product ID is available on uploaded photos, as well as new photos taken within the app-except videos. It’s also only available in certain countries and shopping categories, including but not limited to electronics, toys, and beauty.
Imagine you’re walking by something that’s a backpack and catches your eye,” Wang said. “Use Lens to identify that product, and you’ll get access right away to information that you might want to know.”
Part of this is an ad play, too. Per Google, the products identified by Lens will also show “relevant” shopping ads, with options and prices included.
What’s the point in putting ads in Lens? As Google says, about 4 billion Lens searches every month are of a shopping nature. The potential revenue from advertising is too good of an opportunity to pass up for such a significant player in the tech industry.
