Giving AI a Body: What I Learned Building Everyday Robots at Google X
It was the first week of January 2016 when I walked through the doors of Google X, Alphabet’s closely guarded secret innovation lab. My new assignment: sort through the teams and technology left behind by the nine robotics firms Google had acquired, and figure out what came next. The entire division was in disarray: Andy Rubin, the “father of Android” who’d led the robotics effort, had left abruptly. Google cofounders Larry Page and Sergey Brin popped in occasionally to offer direction in their spare time, but their input was sporadic. Months earlier, Google X head Astro Teller had agreed to absorb all the robotics staff into the lab, known affectionately as the moonshot factory, so I stepped in to lead the work.
I joined because Astro convinced me Google X (which we soon called just “X”) operated nothing like other corporate innovation labs. The company’s founders were committed to thinking on an unprecedented scale, and they had what’s called “patient capital” — long-term funding that doesn’t demand quick returns — ready to turn big ideas into reality. After decades of founding and selling multiple tech startups, this aligned perfectly with my goals. X felt like exactly the kind of ambitious work Google should be doing. I knew from first-hand experience how hard it is to build something that, as Steve Jobs put it, puts a dent in the universe, and I believed Google was the only place that could pull off the kind of bold, risky bet I believed in: AI-powered robots that could one day live and work alongside ordinary people.
Eight and a half years on, 18 months after Google discontinued its largest robotics and AI bet, a new robotics startup seems to launch every single week. I’m more convinced than ever that these helpful, everyday robots can’t arrive soon enough. But I’m also deeply concerned that Silicon Valley’s obsession with minimum viable products and venture capital’s widespread aversion to hardware investment mean the region lacks the patience needed to win the global race to give artificial intelligence a functional, physical robot body. Worse, much of the capital that is flowing into robotics right now is chasing the wrong priorities. Here’s why.
What a “Moonshot” Actually Means
Google X, the home of our robotics moonshot which we named Everyday Robots, launched in 2010 built on one bold idea: that Google could take on some of the world’s most intractable problems. On purpose, X was housed in its own standalone building a few miles from Google’s main Mountain View campus, to nurture a unique culture and give teams space to think far outside conventional boundaries. A huge amount of work went into encouraging X employees to take big risks, run fast experiments, and even celebrate failure — we saw failure as proof we’d set our goals high enough. By the time I arrived, X had already spun out Waymo, launched Google Glass, and was working on other seemingly science-fiction projects: airborne wind turbines, and stratospheric balloons to deliver internet access to underserved communities around the globe.
What made X projects different from the average Silicon Valley startup was the scale and timeline we were encouraged to think in. To even qualify as a moonshot, X had a clear three-part formula: First, the project had to solve a problem that impacted hundreds of millions, even billions, of people. Second, it needed a breakthrough technology that opened up a completely new path to solving that problem. Third, it required a radical product or business model that sounded just barely on the sane side of crazy.
The Core Problem: Giving AI a Physical Body
No one was better suited to lead X than Astro Teller, whose official title was literally Captain of Moonshots. You’d never find him wandering X’s home — a sprawling three-story converted department store — without his signature pair of rollerblades. Add a ponytail, a constant friendly grin, and that one-of-a-kind name Astro, and you could easily mistake the scene for an episode of HBO’s Silicon Valley.
When Astro and I first sat down to map out what we could do with Google’s acquired robotics companies, we both agreed we needed to build something big. But what exactly? Up to that point, nearly all useful industrial robots were big, unintelligent, and dangerous, locked away in factory floors and warehouses behind cages or under constant supervision to keep human workers safe. How could we build robots that were safe and useful in ordinary, unstructured human environments? We knew we’d need a completely new approach.
The problem we were targeting was a massive global demographic shift: aging populations, shrinking workforces, and growing labor shortages across nearly every sector. Even back in 2016, we knew our breakthrough technology would be artificial intelligence. Our radical solution? Fully autonomous robots that could help with a growing list of everyday tasks in our homes and workplaces.
Put simply, our goal was to give AI a body in the physical world. If any organization could pull off a project of that scale, I was convinced it was X. This work would take decades, endless patience, and a willingness to test dozens of wild ideas and fail at most of them. It would require major technical breakthroughs in both AI and robotics engineering, and almost certainly cost billions of dollars. Yes, billions. Our team shared a deep conviction that if you look just beyond the current horizon, the convergence of AI and robotics is inevitable. Much of what had only existed in science fiction up to that point was about to become reality.
“When Are the Robots Coming?”
Every week I’d talk to my mother on the phone, and her opening line never changed. No hello, no small talk — just: “When are the robots coming?” I’d always answer, “It’s going to be a while yet, Mom,” and she’d shoot back, “They better get here soon.”
My mom lived in Oslo, Norway, where she had access to strong public healthcare. Caregivers visited her apartment three times a day to help with tasks made impossible by her advanced Parkinson’s disease. While that support let her stay living independently in her own home, she longed for a robot to help with the dozens of small, embarrassing barriers that had become insurmountable for her — and sometimes just to have something to lean on when she walked.
Building Everyday Robots Is Much Harder Than It Looks
“You do know robotics is fundamentally a systems problem, right?” Jeff Bingham asked me once, searching my face for an answer. Every team has a Jeff, and ours was Jeff: a lean, earnest bioengineering PhD who’d grown up on a farm, and was our go-to knowledge hub for pretty much everything. To this day, if anyone asks me about robotics, the first thing I tell them is this: it’s a systems problem.
What Jeff was trying to drill into me is that a robot is an incredibly complex system, and it only works as well as its weakest link. If your vision system can’t process what’s in front of it in direct sunlight, your robot will suddenly go blind and shut down the second a sunbeam comes through a window. If your navigation system can’t recognize stairs, your robot could tumble down them, injuring itself or anyone nearby. Building a robot that can safely live and work alongside humans is hard. Really, really hard.
For decades, researchers have tried to program robots to pull off even simple tasks, like grabbing a cup off a table or opening a door. And those programs almost always end up extremely brittle: they fail the second there’s any tiny change to the environment or conditions. Why? Because the real world is completely unpredictable — just ask that stray sunbeam. And that’s before we even get to the really hard stuff, like navigating the messy, cluttered homes and offices we live and work in.
Once you break it all down, you realize that unless you rigidly control every variable — lock all objects into fixed, pre-planned positions, make sure the lighting never changes — even something as simple as picking up a green apple and putting it into a glass bowl on a kitchen table becomes nearly impossible to pull off reliably. That’s why factory robots stay locked in cages: every variable, from lighting to the position of the parts they work on, is controlled, and they never have to worry about accidentally hitting a human.
Two Paths to AI-Powered Learning Robots
“All you need is 17 machine learning researchers,” Larry Page told me, one of his classic, cryptic insights. I pushed back immediately: there was no way we could build the full hardware and software infrastructure for collaborative everyday robots with just a handful of ML researchers. He waved off my objection. “All you need is 17.” I left the conversation confused — why 17? Why not 11, or 23? I was missing his bigger point.
To put it simply, there are two core approaches to integrating AI into robotics. The first is the hybrid approach: different parts of the robot system are powered by AI, then stitched together with traditional coding. In this model, the vision system might use AI to recognize and categorize what it sees. Once it compiles a list of detected objects, the traditional robot code uses pre-written rules to act on that list. If the robot is supposed to pick an apple off the table, the AI vision system spots the apple, the code pulls the “apple” entry from the object list, then the traditional control software guides the arm to grab it.
The second approach is end-to-end (e2e) learning, where the robot learns entire tasks from scratch — from “pick up an object” all the way to “tidy this whole table.” The robot learns by practicing the task over and over with huge volumes of training data, much the way a human child learns physical skills. If you ask a toddler to pick up a cup, they first have to learn what a cup is, that it holds liquid, and they’ll probably spill it or knock it over dozens of times while practicing. But with demonstrations, imitation, and lots of playful practice, they eventually learn to do it without even thinking about the individual steps.
What I eventually realized Larry meant was that nothing mattered unless we could prove robots could learn full end-to-end tasks. Only that approach would give us a real shot at building robots that work reliably in the messy, unpredictable real world, which was the whole point of our moonshot. The number 17 wasn’t literal — his point was that big breakthroughs need small, focused teams, not huge armies of engineers. Of course, there’s a lot more to a robot than its AI brain, so we never paused our other engineering work: we still had to design and build the physical robot. But it quickly became clear that proving end-to-end learning worked would be our ticket past the “gravity well” of incremental innovation, in moonshot terms. In Larry’s view, everything else was just “implementation details.”
The “Arm Farm” That Taught Robots To Grasp
Peter Pastor was a German roboticist who earned his PhD from the University of Southern California. When he wasn’t in the lab, he was usually out kiteboarding trying to keep up with his girlfriend. At work, he spent most of his time wrangling 14 custom robot arms, which we later replaced with seven industrial Kuka arms in a setup we nicknamed “the arm farm.”
These arms ran 24 hours a day, seven days a week, repeatedly trying to grab everyday objects — sponges, Lego bricks, rubber duckies, plastic bananas — out of a bin. When they started, we programmed them to lower their claw gripper into the bin from a random starting position above, close the gripper, lift it up, and check if they’d caught anything. A camera mounted above the bin recorded the contents, the arm’s movement, and whether the attempt succeeded. This went on for months.
At first, the robots only succeeded 7% of the time. But every successful attempt got positive reinforcement: for a robot, that means adjusting the “weights” in its neural network to reward the behaviors that worked and penalize the ones that didn’t. Eventually, the arms learned to successfully pick up objects more than 70% of the time. One day Peter showed me a video of an arm not just grabbing a yellow Lego brick, but nudging other objects out of the way to get a clear shot at it. I knew we’d hit a real turning point: the robot hadn’t been explicitly programmed to make that move with traditional rules. It had learned it on its own.
Even so: seven robots working for months just to learn how to pick up a rubber duck? That wasn’t going to scale. Even hundreds of robots practicing for years wouldn’t be enough to teach robots to do their first useful real-world tasks. So we built a cloud-based simulator, and in 2021, we created more than 240 million simulated robot instances to practice in.
Think of the simulator like a giant open-world video game, with physics accurate enough to replicate the weight of an object or the friction of a surface. Thousands of simulated robots used simulated camera inputs and simulated bodies modeled after our real physical robots to practice tasks like grabbing a cup off a table. All running at once, they tried and failed millions of times, collecting data to train our AI algorithms. Once the robots got reasonably good at the task in simulation, we transferred the algorithms to real physical robots for final training in the real world. I always thought of this as robots dreaming all night, then waking up having learned something new.
Robots Face a Giant Unresolved Data Problem
The day the world woke up to ChatGPT, it felt like magic. Suddenly an AI could write full paragraphs, answer complex questions, and hold a natural conversation. At the same time, we all quickly learned its core limitation: it took an enormous amount of training data to pull that off.
Today’s robots already leverage large language models to understand spoken speech and vision models to recognize what they see, and those capabilities make for great viral YouTube demo videos. But teaching robots to live and work autonomously alongside humans is an even bigger data problem by comparison. Even with simulations and other tricks to generate training data, it’s extremely unlikely that robots will one day “wake up” fully capable, controlled by a single foundation model that runs the whole system.
We still don’t know how complex tasks can get when taught to robots using AI alone. I’ve come to believe we’ll need thousands, even millions of robots operating in the real world to collect enough data to train end-to-end models that can do anything beyond narrow, well-defined tasks. For the foreseeable future, building robots that can deliver useful real-world services — like wiping all the tables in a restaurant or making hotel beds — will require both AI and traditional programming. In other words, don’t expect robots to start acting outside their programming, for better or worse, any time soon.
Do Robots Really Need To Look Like Us?
Horses are incredibly efficient at walking and running on four legs, but we built cars with wheels instead. Human brains are incredibly powerful biological computers, but the silicon chips we built don’t work anything like our brains. Why don’t cars have legs, and why aren’t computers modeled after human biology? The point is, building robots doesn’t have to be about copying humans.
I learned this lesson during a meeting with Everyday Robots’ engineering leaders, where we were arguing animatedly about whether our robots should have legs or wheels. The debate quickly turned into something more like a religious argument than a fact-based discussion. A lot of people were deeply attached to the idea that robots should look like humans. Their logic made sense: we’ve built our homes and workplaces to fit human bodies, and humans have legs, so robots should too.
After 30 minutes of back and forth, the most senior engineering manager in the room, Vincent Dureau, spoke up. He simply said, “If I can get there in my wheelchair, a robot should be able to get there too.” The room went silent. The debate was over.
The truth is, human-like robot legs are mechanically and electronically extremely complex. They don’t move very fast, they make robots far more prone to falling, and they’re much less energy-efficient than wheels. These days, when I see companies racing to build humanoid robots that closely mimic human form and function, I wonder if it’s a failure of imagination. There are so many robot designs that complement human capabilities, why are we so fixated on mimicry? At Everyday Robots, we tried to keep our robot’s design as simple as possible, because the sooner robots can start doing real tasks, the faster we can collect valuable data to improve them. Vincent’s comment reminded us we needed to tackle the hardest, most impactful problems first, not get distracted by replicating human form.
The Day A Robot Tidyed My Desk
I was sitting at my desk one day when one of our single-armed robots — with a rounded rectangular head — rolled up, called me by name, and asked if it could tidy my workspace. I said yes and stepped back. A few minutes later, it had picked up two empty paper cups, my clear Starbucks iced tea cup, and a Kind bar wrapper. It dropped all the trash into a bin attached to its base, nodded at me, and rolled off to the next desk.
This desk-tidying capability was a major milestone: it proved we were making real progress on a previously unsolved piece of the robotics puzzle. The robots were using AI to reliably recognize both people and objects. Benjie Holson, the software engineer (and former puppeteer) who led the team that built this capability, was a vocal advocate for the hybrid approach. He wasn’t opposed to end-to-end learning, he just had a practical attitude: let’s get robots doing something useful right now. If ML researchers solved an end-to-end task better than his team could code it, they’d just swap in the new algorithm.
I got used to seeing our robots rolling around the building doing small chores like tidying desks. Every so often I’d see a new visitor or a new engineer watching them, and their face would light up with wonder and joy. Through their eyes, I remembered how extraordinary this actually was. As our head of design Rhys Newman once said in his Welsh accent as a robot rolled past: “It’s become normal. That’s weird, isn’t it?”
Dancing Robots That Changed How We Saw Our Work
Our Everyday Robots advisory board included a philosopher, an anthropologist, a former labor leader, a historian, and an economist. We debated big, important questions vigorously: If robots lived alongside us, what would the economic impact be? What are the short- and long-term effects on labor? What does it mean to be human in an age of intelligent machines? How do we build these machines to feel welcoming and safe for everyone?
In 2019, after I told my team we wanted to hire an artist in residence to experiment with creative, unexpected uses for our robots, I met Catie Cuan. Catie was working on her PhD in robotics and AI at Stanford, but what caught my eye was her background as a professional dancer, who’d performed with the Metropolitan Opera Ballet in New York.
You’ve probably seen YouTube videos of dancing robots: they perform a pre-programmed sequence of timed moves synced to music. It’s fun to watch, but those dances aren’t any different from the pre-planned animatronics you see on a Disneyland ride. I asked Catie what it would look like if robots