6 Google DeepMind Robotics Projects Built at RoboHacks
Robots do not struggle to collect data. Cameras, microphones, and LiDAR sensors can feed them a constant stream of information. The harder part is deciding what that information means and what to do next.
That was the problem builders tackled during RoboHacks, held at Y Combinator’s San Francisco office on April 11 and 12, 2026. The event gave 120 participants access to 25 MARS robots and a weekend to turn multimodal AI into working physical systems.
Google DeepMind’s challenge attracted 15 submissions. Most connected Gemini or Gemma to a live camera feed, then used the model’s interpretation to guide movement, navigation, memory, or manipulation. The results went well beyond conversational robots. Teams built machines capable of mapping an unfamiliar pharmacy, inspecting a rental property, locating lost objects, preparing food, assessing explosive threats, and drawing a human portrait.
The following six projects show how multimodal models can move from interpreting a scene to helping a robot act inside it.
RoboHacks x Google DeepMind: At a glance
- RoboHacks brought together 120 builders and 25 MARS robots at Y Combinator’s San Francisco office on April 11–12, 2026.
- Google DeepMind’s challenge focused on the best use of a multimodal model.
- Fifteen projects entered the challenge.
- Teams used Gemini and Gemma to turn live visual data into navigation, memory, and physical actions.
- Projects ranged from pharmacy assistance and bomb disposal to cooking, hospitality, and portrait drawing.
- Team DimiMax’s spatial-memory system placed first overall at RoboHacks.
1. Zero-Shot Cross-Domain Spatial Memory for MARS
Team DimiMax built a pharmacy robot capable of learning an unfamiliar environment from a single video walkthrough. The project placed first overall at RoboHacks.
The system reconstructs a three-dimensional scene from monocular video using Pi3, then aligns it with LiDAR occupancy grids through RANSAC and ICP. Local vision-language models label the reconstructed environment, turning a geometric map into a semantic one that the robot can understand.
Gemma 3 handles semantic labelling on the device. A GPT-4o Realtime orchestrator interprets spoken requests and dispatches the required navigation or manipulation task. An in-house ACT policy controls grasping.
The result is a robot that can receive a command such as “fetch my prescription,” locate the relevant area, navigate to it, identify the item, and retrieve it without environment-specific training.
Its stack combines Pi3, DUSt3R, VGGT, Gemma 3, ROS 2, LiDAR SLAM, an ACT policy, and the MARS platform.
View Zero-Shot Cross-Domain Spatial Memory for MARS on Iterate
2. Gordy, the Physical AI Sous Chef
Gordy gives the MARS robot the voice and personality of a famously demanding British chef. Behind the joke sits a complete physical workflow.
A user places an order through a voice interface. The robot listens, interprets the request, and prepares the food using the Innate OS platform. Gemini supports natural-language understanding and examines the robot’s progress through its camera. It can visually check whether each physical step worked before the system moves on.
ElevenLabs handles transcription and expressive speech, allowing Gordy to respond throughout the process without dropping character.
The project demonstrates why multimodal feedback matters in robotics. Sending an instruction to a robotic arm is not enough. A reliable system also needs to observe the result and confirm that the intended action actually happened.
The team explicitly submitted Gordy to the Google DeepMind Best Multimodal Use of VLM track. It received seven public votes.
3. RECON
RECON turns the MARS platform into an autonomous explosive ordnance reconnaissance unit.
Gemini Flash 2.5 analyzes the scene at two frames per second to identify possible threats and nearby civilians. The robot combines this visual interpretation with SLAM mapping and LiDAR obstacle avoidance as it moves through the environment.
When it detects danger, RECON can issue spoken evacuation warnings through ElevenLabs. It approaches suspicious devices with a three-step controller that uses the object’s bounding box position to adjust its direction. An arm-mounted camera then provides a closer view for wire-level defusal analysis.
The team also built an operator dashboard that brings together the live video stream, radar motion readings, and SLAM map. A human operator can override autonomous behavior with natural-language commands when necessary.
Safety constraints remain active throughout the process. These include LiDAR stops, capped movement speeds, and limits on how long each command can run.
RECON received seven public votes at RoboHacks.
4. Cloudy
Cloudy is a companion robot designed for older adults living alone. It watches its surroundings, remembers where objects appeared, and answers questions about them later.
A user could ask, “Where did I leave my keys?” Cloudy might respond that they were last seen on the kitchen counter beside a wallet.
Gemini 3 Flash Preview converts camera footage into structured descriptions. These records cover object type, color, location, nearby items, and the surface supporting the object. Gemini 2.5 Flash interprets the user’s question and calls the appropriate memory tools.
The memory system uses PostgreSQL and pgvector. It stores 768-dimensional embeddings alongside three-dimensional positions, room information, and observation history. An association engine compares new observations with existing records based on appearance, proximity, and meaning. This prevents the same item from being saved repeatedly as several different objects.
When Cloudy has not observed an item for five minutes, it marks the object as missing. It therefore retains both an object’s current location and its previous position.
The system runs as a set of ROS 2 Humble nodes on an NVIDIA Jetson Orin Nano.
5. BnBot
BnBot applies physical AI to short-term rental management.
The project combines two agents. The first inspects a property after a guest checks out. It follows a patrol route, compares before-and-after images with Gemini, and identifies new rubbish or damage that may need the host’s attention. It then prepares an email report.
The second agent welcomes arriving guests and leads them through a personalized tour of the property.
This divides a broad hospitality task into two narrow workflows with distinct goals. One agent records and compares the state of the property. The other interacts directly with the guest.
BnBot brings together SLAM navigation, visual scene comparison, speech, servo controls, an LCD, and automated reporting. It shows how a mobile robot could handle work that normally requires a host or property manager to be physically present.
6. Leo
Leo is a robotic portrait artist.
The MARS robot begins by locating a person and holding a short conversation with them. It takes a photograph through its onboard camera, then sends the image to Gemini 3.1 to produce line art. A computer-vision pipeline extracts centerline paths that the robotic arm can follow.
Leo then draws the portrait physically. A custom hardware module allows the robot to change tools, supported by NVIDIA GraspGen and Meta’s SAM3.
The team organized the project into three layers. Agents coordinate the interaction, skills connect those agents to the physical painting process, and the Leo pipeline converts the original image into drawable movements.
Unlike a standard image generator, Leo must account for the physical limits of the arm and the drawing tool. The final output depends on both visual interpretation and accurate manipulation.
What These RoboHacks Projects Reveal About Multimodal Robotics
The projects solve very different problems, but they rely on a similar loop:
- Sensors capture the environment.
- A multimodal model interprets what the robot sees or hears.
- An orchestrator chooses the next task.
- Navigation or manipulation systems execute it.
- New sensor data confirms whether the action worked.
This final verification step separates a physical agent from a conventional AI assistant. An assistant can produce a response and stop. A robot must check whether it reached the correct location, found the intended object, or completed the movement safely.
The strongest submissions also avoid asking one model to control everything. Cloudy separates perception, memory, and question answering. Leo splits coordination, skills, and drawing. RECON combines AI-based scene analysis with conventional safety controls. DimiMax uses several mapping systems before its orchestrator dispatches an action.
Multimodal models provide understanding, but specialist components still handle mapping, control, storage, and physical execution.
Turn Real Technical Problems Into Working Prototypes With Iterate!
A focused hackathon gives technical teams a reason to move beyond slide decks and test whether an idea works under real constraints.
Iterate helps companies and AI teams organize hackathons built around their technologies, datasets, and engineering challenges. Participants receive a defined problem, a limited timeframe, and the support needed to build demonstrable products.
Sponsors gain more than a collection of generic AI demos. A well-designed challenge can reveal unexpected use cases, generate technical feedback, attract skilled builders, and show how developers use a product when they are free to experiment.
RoboHacks did exactly that for multimodal robotics. In one weekend, teams turned models, sensors, and robotic hardware into systems that could navigate, remember, inspect, communicate, and manipulate objects in the physical world.
Conclusion
RoboHacks showed that multimodal AI becomes more useful when it connects perception to action.
Gemini and Gemma helped these robots interpret requests and understand their surroundings. The teams then combined that intelligence with spatial maps, memory systems, control policies, and safety rules to complete physical tasks.
The projects remain prototypes, but their applications are concrete: healthcare assistance, emergency response, independent living, property management, food preparation, and creative robotics. Together, they offer a practical view of what embodied AI teams can already build within the limits of a single hackathon weekend.
Frequently Asked Questions
What was RoboHacks?
RoboHacks was a robotics hackathon held at Y Combinator’s San Francisco office on April 11–12, 2026. It brought together 120 builders and provided 25 MARS robots for teams to develop physical AI prototypes.
What was the Google DeepMind challenge at RoboHacks?
The challenge rewarded strong uses of multimodal models in robotics. Fifteen submissions entered the track, with teams using models such as Gemini and Gemma to interpret visual information and support physical actions.
Which project won RoboHacks?
Team DimiMax placed first overall with Zero-Shot Cross-Domain Spatial Memory for MARS. Its pharmacy robot builds a semantic map from a single video walkthrough and uses it to navigate and retrieve objects.
How were multimodal models used in the RoboHacks projects?
Teams connected multimodal models to cameras, microphones, and other sensors. The models helped robots interpret scenes and instructions before navigation or manipulation systems carried out the required actions.