Just as promised, Gemini AI is taking its first tentative steps in the Google Home app, with Nest camera feeds and home automations among the first beneficiaries.
Starting this week, Gemini will be able to write descriptions of what’s happening in captured Nest video events. And soon, the LLM will start helping you create Google Home automations based on natural-language queries.
Google is taking a cautious approach to adding AI functionality to the smart home, adding just a couple convenience features for its Nest cams and automation recipes before wading into more far-reaching improvements, such as imbuing Google Assistant with a more “fluid” conversational style.
This news story is part of TechHive’s in-depth coverage of the best security cameras.
The new Gemini features come on the heels of an extension for the Gemini app that lets Google’s AI model control your smart lights, thermostats, and other smart home gadgets.
For now, you’ll need to be a member of Google’s Public Preview program to try out the new Gemini-powered features, and you’ll also need to enable the “Experimental AI features” setting.
AI-powered Nest video descriptions
Until now, video events in your Nest camera feeds have been limited to such generic text descriptions as “animal or activity detected.” With help from Gemini, the Google Home app can now generate more specific labels based on what’s actually happening in the clip.
For example, the Gemini-generated description might read “the dog is digging in the garden,” or “package with ballons.” Tapping the information icon will give you a more detailed description, according to Google.
Gemini can now write descriptions of what’s happening in a captured Nest video event.
Besides just browsing your Gemini-labeled video feeds, you’ll also be able to search on those descriptions.
For example, if you enter “Did I get any packages today?” into the search box, you might get a Gemini-generated result that reads, “Someone left a package and balloons this morning, and 2 people with a small dog dropped off 3 packages a little later.”
Even better, the description could include the name of a detected person in your “familiar faces” library if they’re the only person in the video event. If your “familiar” friend is in a group, their name won’t appear in the description, but you’ll still be able to search on their name, Google says.
The new Gemini-powered video descriptions still have some limitations, starting with the fact that they’ll only work on Nest cameras managed in the Google Home app. In other words, don’t expect AI descriptions with any legacy cams that are still stuck in the old Nest app.
Also, the new descriptions are only available in English, and just in the U.S. for now.
Finally, it’ll be interesting to see if the Gemini-generated descriptions fall victim to the hallucinations that plague most AI models.
“Help me create” automations
Coming soon for those in Google’s Public Preview program is another interesting Gemini-powered feature: the ability to create home automations based on natural-language queries.
For example, you’ll be able to say, “Lock the doors and turn off all the lights at bedtime,” and Gemini will instantly create a Google Home automation for you, which you can then review or tweak as needed.
Gemini will soon be able to create automations based on your natural language queries.
If you’re not quite sure what kind of automation you want, you can just give Google Home the gist (like “Help the kids remember to put their bikes away when they get home from school”) and Gemini will serve up some suggested triggers and actions.
Such “help me create” automations will start rolling out in public preview “later this year,” Google says.
Coming later: A better Google Assistant
Perhaps the most dramatic new Gemini feature slated for Google Home is a souped-up Google Assistant for Nest speakers and displays, complete with a more “fluid” conversational style.
With help from Gemini, the revamped Google Assistant on Nest speakers will sound “more natural” and will “better understand you,” allowing for more “back and forth” as well as follow-up questions.
Sounds enticing, but it’s worth noting that Google didn’t list the retooled Google Assistant in its list of Fall 2024 Google Home improvements, which suggests we’ll need to wait longer for that functionality to arrive. (AI video descriptions and “help me create” automations did make the cut.)
Author: Ben Patterson, Senior Writer, TechHive
Ben has been writing about technology and consumer electronics for more than 20 years. A PCWorld contributor since 2014, Ben joined TechHive in 2019, where he has covered everything from smart speakers and soundbars to smart lights and security cameras. Ben's articles have also appeared in PC Magazine, TIME, Wired, CNET, Men's Fitness, Mobile Magazine, and more. Ben holds a master's degree in English literature.