Hey, here I am again with a Home Assistant tutorial. I’ll explain how you can use AI in Home Assistant this time. I’ll show you two examples: the first one welcomes you home in a funny way, and the second sends a message to your phone describing the person at your door when the doorbell rings. Let’s get started!

---

⭐⭐⭐ NOTE: ⭐⭐⭐

This article accompanies a YouTube video. I wrote it for people who would rather read than watch a video. To keep doing this, I want to ask you to check out the video, leave a comment under the video, give the video a thumbs up, and subscribe to my YouTube channel. This means that the video is offered more often to new visitors so that they also stay informed of the latest Home Assistant tutorials.

Thank you for your support!

Ed

Click here to watch the video

AI has been a hot topic lately, and I’m concerned about its development. Computers are taking over more and more tasks from us, which might seem like a great advancement at first. Life has become much more efficient, but what worries me most is the decline in human interaction. Nowadays, we can scan our groceries, pay with cards, date, and shop online. As a result, we’re experiencing less and less human contact. And that’s not even mentioning all the data that’s being collected about us.

As some of you may know, besides being a YouTuber, I’m also a music therapist, helping people with mental health issues in a mental health care institution and my private practice. I often see clients who have become isolated due to social media and modern communication tools. This has led to depression and, in some cases, even suicide. Social media and AI can contribute to loneliness, and I genuinely hope everyone realizes that technological advancements don’t just bring efficiency—they also pose a risk to people’s mental health. Let me know your thoughts about this in the comments.

So yes, I hesitated about making a video on using AI in Home Assistant. However, you can also use it to make your smart home more fun. In this video, I’ll show you how to use AI to make your smart home less static. So, here’s a way to use AI in a fun manner without causing too much harm to your social interactions.

Use Case 1: Welcome someone when they come home.

In the first use case, we’ll ensure a message is played on your smart speaker when someone comes home. We’ll use the TTS: Speak option in Home Assistant to do this. We’ll first install the Google Translate Text-to-Speech integration to make this work properly in your language.

• Go to Settings.
• Go to Devices and Services.
• Click “Add Integration”.
• Search for “Google”.
• Select “Google”.
• Select “Google Translate Text-To-Speech”
• If you are English, keep the settings to En and Com. If you want, change it to the language of your choice.
• Click Submit.
• Click Finish.

Now that Google Translate Text-to-Speech is installed, we can move on to setting up the AI integration. You can choose from various providers, such as OpenAI or Google Gemini. I’m choosing Google Gemini for this tutorial mainly because it’s still free to use. However, you will need to request an API key from Google. Let’s do that first.

• Go to Google AI Studio. You’ll find the link in the description of this video.
• Click Get API key.
• Click the Create API key button.
• A new API key will be generated for you.
• Click “Copy” to copy this API key to your clipboard.

Now we’re ready to install the AI integration in Home Assistant. We’ll start by installing the Google Generative AI integration.

• Go to Settings.
• Go to Devices and Services.
• Click “Add integration”.
• Search for “Google”.
• Select “Google”.
• Select “Google Generative AI”.
• Paste the API key that we’ve just created.
• Click “Submit”.
• Click “Finish”.

Google Generative AI is now installed, and we can generate messages via AI from now on. I’ll show you an example in the developer tools.

• Go to Developer Tools.
• Click the Actions Tab.
• Select “Google Generative AI: Generate Content” in the Action field.
• Enter a prompt, such as: “Welcome Ed to his home and ask if you can do anything for him. Use maximum two funny lines”.
• Click “Perform Action”
• You will now see a randomly created Welcome message.

In a few minutes, I’ll show you how to generate a description of an image, but first, we’ll work with these messages. Of course, you can experiment with various other prompts, but for now, we’ll use this example to create a new automation.

Creating the Automation to welcome someone home.

To make the automation work, you must ensure that you have assigned some tracking devices to your person entity. You can view your person entity by going to Settings > People and selecting the person you want to welcome. You can select tracking devices like your phone here. You must have installed the Home Assistant Companion App on your phone for this to work.

Go to Settings > Automations and Scenes to create a new automation. I made the following automation to welcome someone home.

• I started with creating an entity state trigger that triggers if the state of the person entity Ed changes to Home.
• In the “Then Do” section I start with a delay of three minutes. This gives me the time to enter my living room before the message will be played.
• Then I proceed with the Google Generative AI “generate Content” action.
• I’ve added the prompt: “Welcome Ed to his home and ask if you can do anything for him. Use maximum two funny lines”.
• In the field “Response Variable” I entered the name of the response variable. I named it “response”. This variable will contain the welcome sentence that Google generative AI generates based on the prompt.
• Next, I set the media player’s volume using the action media_player.volume_set. It’s optional, but I want my media player to play at a certain loudness. There is a way to restore the loudness to its initial state after the message has been played through your speaker, but that is out of the scope of this tutorial. If you want to know how that works, please check this video where I explain how to create scenes on the fly.
• In the last action, I will send the response to the media player using the action “tts.speak”. You need to use a bit of YAML for this. The important part of this action is the “{{ response.text }}” code that contains the response of Google Generative AI.

Once you’ve saved the automation, a message will be played over your speaker when you come home. The speed at which the message is played depends on how quickly your phone is detected as being home. This can sometimes take a little longer if your device tracker isn’t very responsive. But anyway, it’s about the concept. You now know how to generate an AI message and apply it to other things as well. For example, I get such a message when there’s new mail or when the trash needs to be taken out. Let’s go to the next Use Case!

Use Case 2: Send a message describing what is visible in an image.

In the next example, we’ll have AI send a description of an image to our phone.

• Go back to the developer tools and click the Actions tab.
• Select the Google Generative AI: generate Content action if it has not been selected yet.
• We use a different prompt: “Describe in one sentence what you see on the image. If you see a person, describe this person and describe the colors of his clothes. If the person is carrying something, describe what he carries. Don’t describe walls. Make it witty.”
• Next to a prompt, you can also refer to an image.
• I stored a doorbell image on my home assistant server and entered its location in this field. (/config/www/tmp/snapshot-doorbell.jpg)
• Click “Perform Action”.
• As you can see, the description of the image is displayed.

I will show you the automation I created to display this message on your phone.

• First, I check if someone has rung the doorbell at my front door. I’m using the Reolink Video Doorbell, which you can watch my review of in this video.
• In the “Then Do” section, I first take a snapshot of the video feed from the front door. I use the action “camera: take snapshot” and select my front door camera for this. I save the file on my Home Assistant server in the folder /config/www/tmp/ and name it snapshot-doorbell.jpg.
• Then, I add a 1-second delay to give the server time to save the file. This isn’t strictly necessary, but I do it just to be sure.
• Next, I call the action “Google Generative AI: Generate Content” and enter the prompt we used earlier in the Developer Tools. I also reference the snapshot.
• Here, I again use the variable name “response,” just as I did in the first example.
• Finally, I sent a message to my phone using the action “notify.mobile_app_iphone_x_ed.” In your case, you should use the appropriate notify action for your phone.
• We need to configure this action in YAML because we use the response variable. The message contains {{ response.text }}.
• The message’s title, in my case, is “Front door Doorbell rings”.
• And then, I also want to show the screenshot.
• I refer to the local image here. Note that you should use /local/tmp/ instead of /www/tmp/, as I used when saving the image.
• The entity is my camera ID.
• I’ve also added a few actions so I can immediately open the Reolink App, but you can also insert your own URL here, which could link to your dashboard, for example.

Once you’ve saved the automation, an AI-generated message will be sent to your phone whenever the doorbell rings.

As always, you can type the code for free from the screen. Additionally, I’ve created two blueprints: one for playing a message on your smart speaker when you arrive home and one for sending a message when someone is at your door. With the help of these blueprints, you can easily implement AI automations in your own system. You can download the blueprints for the price of a cup of coffee via the link in the description of this video. This helps me continue creating these tutorials! I really appreciate any help you can provide.

I’ve shown you the standard functionality of Home Assistant. You can install the llmvision integration from HACS if you want even more functionality. It’s more extensive and can be linked to multiple LLMs like OpenAI, Anthropic Claude, Google Gemini, Groq, LocalAI, Ollama, and custom OpenAI-compatible APIs. You can also analyze data, streams, and videos with it. However, the standard Home Assistant functionality is more than enough for the use cases I most commonly use. Let me know in the comments if you’d like to see a tutorial for LLMVision from me.

Now, you might be thinking, “Hey, but I’m using a cloud service to use AI.” Yes, that’s true, but you can also use a local service, for example, by installing Ollama on a separate server. Go to the Ollama website to download and install it. To keep this video from being too long, I didn’t include the installation of Ollama in this tutorial. Perhaps I’ll make a separate tutorial for that. Let me know if you’d like me to do that in the comments.

I’m very curious about the AI applications you’ve created. Please let me know in the comments.

Thanks again for watching! Your support is crucial to keeping this channel alive. You can sponsor me monthly with the cost of a cup of coffee via Patreon or Ko-Fi or by becoming a paid member of this channel, just like these people already do. Thanks to you, I can keep doing this work!

Remember to subscribe to this channel and give this video a thumbs up. That way, my videos will be seen by more people on YouTube! I’ll see you soon in my next video. Bye-bye!