Multimodal AI explained simply: when AI can see, hear and speak

What “multimodal” means

A modality is a type of information: written text, a photo, spoken language, a sound or a video. IBM describes multimodal AI as models that can process and integrate data from several such modalities.

The first chatbots could only read and write. A multimodal AI, by contrast, can look at a photo, hear your spoken question about it and answer out loud. It connects what it sees with what you say. That is the real step forward: not just recognising an image, but having a conversation about it.

According to IBM, the main technical difficulty lies precisely in linking such different kinds of data in a meaningful way. That is why it does not work equally well in every situation.

Everyday examples: camera, screen, photo

Gemini Live: in the Gemini app for Android, you can switch on the camera during a voice conversation and show Gemini what you see, or share your screen. You can switch between the front and rear cameras. According to Google, a new interactive guidance feature is gradually rolling out: it can, for example, read out the small print on food labels and help with colour differences, menus in dim light or finding items around the house.

ChatGPT: according to OpenAI, you can share video and your screen in the voice mode of the ChatGPT app for iPhone and Android, but only in “Advanced” mode and only with an eligible subscription. The newer Live mode does not currently support video or screen sharing.

Google Search: according to Google, AI Mode in Google Search is also available in Germany. There you can not only type or speak questions but also upload images and files.

Be My Eyes: the free app for blind and partially sighted people includes the Be My AI feature. You send a photo and get a description or answers to questions about it, in 36 languages according to the App Store listing. According to the provider, more than half a million blind people worldwide use the app.

How to talk to an AI and which assistants can do it is covered in our overview of voice mode.

Limits: seeing is not understanding

An AI can misinterpret an image, misread a number or miss something. Google explicitly warns that its guidance feature can make mistakes, is not a medical device or mobility aid, and should not be used for navigation or detecting obstacles. For medicines, traffic and safety, the AI is therefore something to ask, not a reliable authority.

Images can also be manipulated. The OWASP security project describes how attackers can hide instructions in images that a multimodal AI then reads and follows. How such hidden commands work is explained in our glossary entry on prompt injection.

Privacy: what the camera sees, the provider sees too

When you share your camera or screen, you send those images to the provider. Google points out that screen sharing in Gemini Live shares the entire screen. So close banking apps, private messages or other sensitive content first.

Mind other people in the picture as well. Google's help pages explicitly ask you to respect others' privacy and to ask permission before recording them or bringing them into a live conversation.

One practical detail: according to Google, the camera in Gemini Live switches off automatically when you leave the app or lock the screen, and does not come back on by itself afterwards.