Skip to content
ThreadOut
Google DeepMind@GoogleDeepMind · May 12, 2026

We’re reimagining a 50-year-old interface - the mouse pointer - with AI. 🖱️ These experimental demos show how people…

6 tweets1 min read♥ 8.5Koriginal

Summary

Google is experimenting with AI-enhanced mouse pointer technology that combines gestures and speech to provide contextual assistance. The system understands what's under your cursor, allowing natural interactions like pointing at a PDF to request bullet points or highlighting text to modify it.

Summarized by ThreadOut AI from the full thread. May miss nuance — read the thread below.

  1. #1

    We’re reimagining a 50-year-old interface - the mouse pointer - with AI. 🖱️ These experimental demos show how people can intuitively direct Gemini on their screens using motion, speech, and natural shorthand to get things done 🧵

  2. #2

    With an AI-enabled pointer, help is always available where you’re working - without having to detour to additional apps. 📲 Point at a PDF and request bullet points for an email, hover over a table to ask for a pie chart, or highlight a recipe and simply say: "double these ingredients."

  3. #3

    Current models require precise instructions, but our AI-enabled pointer removes that burden. 💡 By "seeing" what’s under your cursor, it instantly understands the specific word, image, or code block you need help with.

  4. #4

    In the real world, we don't tend to speak in long paragraphs; we point and say: "fix this" or "move that". 💬 By combining gestures with speech, it lets you use natural shorthand to complete tasks.

  5. #5

    For decades, your mouse only tracked where you were pointing. AI helps it understand what you're pointing at. 💭 This means a photo of a scribbled note could turn into an interactive to-do list, or a paused video frame can become a restaurant booking link.

  6. #6

    These capabilities are guiding how we think about the next generation of interfaces. As we continue exploring what an AI-enabled mouse pointer would unlock, try our experiments in @GoogleAIStudio → goo.gle/49HqFeu