This article was originally posted on the First Round Review.
When Caitlin Kalinowski joined Oculus as the head of product design engineering in 2014, her team was sweating on a challenge: design the controllers that would be paired with the hotly anticipated Oculus Rift virtual reality headset. At the time, VR was largely uncharted territory, with little precedence for what the hardware should look or feel like. Working with designers led by Oculus’s industrial design head Peter Bristol, Kalinowski’s team began to weigh various factors that would determine how the product would ultimately feel. For example, they wanted a broad swath of hand shapes and sizes, from the fifth percentile of females to the 95th percentile of males, to be able to hold the Touch controllers comfortably. Hands had to grip it without interfering with the connection between infrared LEDs and an external sensor. Each new parameter complicated another: the overall weight, the number of buttons, the strap on the controller, the shape and so on.
This is where the magic happened. The team iterated over and over again. Some prototypes were so ugly they were literally a tube of plastic with putty stuck on it. But by the end, they’d created a ring-shaped controller that a user could open and close her hand on in game play, with a sensor on the thumb stick that allowed users to give a thumbs-up. Even more miraculously, they’d iterated enough times that when it came time for the engineering validation test, or EVT, the physical proportions of the controller were nearly perfect. Only a tweak on the diameter of the outer ring was needed.
Kalinowski is a master of the prototyping process, with a deep understanding of where, when and how changes should be slotted in, from the first iteration to the last. It’s made her a highly sought-after engineer in Silicon Valley. Before her work at Oculus, she was the technical lead for Apple’s MacBook Air and Mac Pro. She also led and shipped Facebook’s Bluetooth Beacons, which power location-based prompts for users.
In this exclusive interview, Kalinowski discusses how and why you must define your non-negotiables before starting to build prototypes. She dives deep on specific approaches for squeezing the most iterations and improvements out of the process — the more iterations, the better the product, after all. Kalinowski also lists the warning signs for two of the three bears of this process: prototyping processes that take too long, and also ones that end too quickly. Kalinowski spells out what the just-right scenario looks like and lists the potholes to look out for before you hit the ‘ship’ button. These insights apply to every product development situation, whether you’re in hardware or otherwise. Let’s get started.
Declare Your Non-Negotiables
There is a moment before you start building your product when the deadlines aren’t yet set, the stakeholders are patient and the customers aren’t clamoring. Use this moment to get very clear about which goals you won’t bend or budge on. “Before starting design work, decide what your product absolutely must do before you’ll ship it. New companies are often under a lot of pressure. They don’t always define their non-negotiables clearly at the beginning of the product process. I see this mistake often. Unfortunately, they can get backed into a corner and be forced to ship early without having met those critical benchmarks first,” Kalinowski says. “Choose the product must-haves first. If you do get backed into a corner and you haven’t quite met them, it can be really tempting to ship anyway. Instead, start out with: ‘The product has to do these three things. Nice to have are these next two things. A bonus are these other things.’ As a team, you must agree that if you don’t hit those have-to-haves, you’re not going to ship the product.”
Finding your have-to-haves often means boiling the most critical elements of a product down to one or two key features. In the case of the iPad, the product had to be lightweight enough that customers could hold it up easily. “Apple worked on the iPad for years. Steve Jobs always said it was too heavy. They turned it into the iPhone, worked on that for years, shipped the iPhone, went back to the iPad, worked on that for years and finally shipped the iPad. There probably wouldn’t have even been an iPhone if they hadn’t had that guiding principle for the iPad. They didn’t ship the earlier iPad prototypes because they were too heavy. It was tiring for your hands. It takes a lot a lot of discipline to keep developing until you nail your key features,” Kalinowski says. “Focus on the product and the consumer experience. On the Oculus Rift, if you didn’t experience “presence,” or the feeling that you’re actually in your virtual environment, we weren’t going to ship it. If the Oculus Rift didn’t fit comfortably on a wide set of adult faces, we weren’t going to ship it. Decide at the beginning, what are the goals you must hit in order to deliver those key features? As long as you achieve those product goals, you can have discussions about other matters like secondary features and whether you’ll ship or delay your schedule. Share those must-haves with your entire organization. Get everyone on board with the non-negotiables so you’re moving together towards the same goal.”
No matter what, always solve for consumer experience. “As a designer and an engineer, I’m constantly trying to think about the customer and how much better or how much worse the product will be depending on the decisions we are making on the product level. That is how you determine your North Star. When I’m thinking about product trade-offs and feature trade-offs, I’m really thinking about the customer. I ask myself, is this change materially going to affect their experience of using this product?” Kalinowski says. “This is why we care so much at Oculus about ergonomics and weight. They affect the software experience and the customer’s sense of presence. If you can wear the device longer, you can stay immersed, play games longer and experience more fluid social VR. The calculus of consumer experience is present in every decision from the fabric we use to how heavy the device is.”
Slide into a Prototyping Plan
Map out your approach to prototyping by determining your point on a speed-caution slider. On one end is caution, and on the other end is speed. Which way you tilt depends on your company, what you’re trying to accomplish, and where you are in your development cycle. “If you’re schedule-driven and trying to beat a competitor to market, slide away from caution. If you’re making a product with a lower volume, you can slide away from caution. For example, making 100,000 of a product versus two million changes your risk tolerance. You cannot recover from problems in the design of a two million unit product the same way you can with hundreds of thousands,” Kalinowski says. “If I’m at Apple developing the MacBook Air, I’m going to be far more cautious about risk because it’s a huge, high volume product. It’s expensive to recover from mistakes. If I’m designing the first VR product, I’m going to use less caution and be more ambitious because my volumes will be manageable enough that I’ll be able to fix problems in flight. I’ll try to move faster. There’s a speed-caution slider and you have to understand where you are in order to master your trade-offs. If I’m making a medical product, my caution slider’s generally all the way to the left, because they typically take much longer to develop and achieve compliance. It boils down to the risks associated with what you’re making and how much development time you’re willing to trade to be faster to market.”
Your management of your team should also shift accordingly. “If you’re super cautious, encourage all assumptions to be checked and double checked. Say to the team, ‘Confront each other.’ Ask, ‘Is this true? Is there enough swell space for the battery? Is this flame rating correct for this part? Are you sure? Have we tested it? Have we sent the part out to a third party and verified it? Three months into the product run, will we re-verify that the manufacturer hasn’t changed the material on us to lower their costs?’” Kalinowski says. “When speed is the top concern, only worry about the most critical issues. In those cases, I give a piece of paper to my people and say, ‘Make a list of all the design issues you are worried about in ranked order.’ Then I rip the page under their first five and hand the top part of the page back to them and say, ‘just focus on those.’’”
The Six-Step Approach to Prototyping
Once you’ve figured out your place on the speed-caution slider, you’re ready to start building prototypes. Your singular goal? To iterate as much as possible before the ship date. Kalinowski spells out her six-step approach to building strong prototypes and how to design a process that fits in as many iteration cycles as possible.
First, frontload. Pile on the effort and focus at the beginning of your journey. “My former colleague Doug Field, who was a VP at Apple and now runs engineering at Tesla, had a very useful graph of how to think about design effort. Most people increase their effort and focus as the product develops and they find issues they need to fix or address, peaking at the moment right before you ship. Although this is sometimes unavoidable, this is not what you want to do. Making changes at the end of development is far more difficult, dangerous and costly than in the beginning,” Kalinowski says. “By the EVT [engineering validation test], your goal is to basically be done. Often it’s very difficult to achieve this, but it’s a good goal. If you have to fix major design issues after EVT, you’ll have diminishing returns on your efforts. Ideally, the phases after EVT are a scaling operation in hardware or bug fixes in software. You won’t have much ability to make big changes after this point without causing risk. Focus your effort towards the beginning, where you can squeeze in as many iterations and changes as possible. Go through crazy amounts of work right here. Put the best people on it. Do all of your grinding here.”
Start with what’s hardest. Take one thing at a time with your prototypes, and begin with the gnarliest challenge. “Iterate on the most important, hardest thing first. This seems really simple, but people often don’t actually do it. The best way to look at product development is to say, ‘I want to dedicate the most time to the hardest thing and start to layer in the easier things as I go.’ It’s usually easy to identify the hardest thing. What are you doing that nobody else has done before? This is the thing that you’re not even sure you can do at all.That’s where you want to focus your initial prototyping energy,” Kalinowski. “And momentum isn’t an issue. If you take a typical engineer, show them a problem and say, ‘I don’t know if this is possible,’ they’re usually all in. This principle serves to make sure you’re frontloading the work by putting the hardest, most time-consuming problem first — while also revving up your engineers by giving them a meaty problem to solve.”
Braid in other hard things as you progress. “When you’re prototyping, you don’t need to do everything at once. You just need to prototype the hardest parts of the hardest things. The rest of it can be ugly and unuseful. Then as you iterate, draw in the next hardest thing and the next hardest thing, and integrate those aspects into your prototype — or put them in a separate prototype, then fork them together,” Kalinowski says. “Work forward from the hardest stuff in your prototyping. With Oculus Touch, the team started with a general shape, and once that was close, moved onto details around input, with the electrical innards and sensors split off into a separate set of prototypes. Once we got to far enough along with understanding space constraints, we braided the two prototypes together in our first integration build, proving that everything could fit.”
Make seriously ugly prototypes. No really, your early iterations should be hideous. “You’re focusing on the toughest engineering problems first. You don’t need to worry about how it looks yet. But there’s also a hidden benefit. When something is ugly, people don’t fall in love with it. Especially with influential people inside a company, if you show them something that’s physical and they see it, they will lock on it,” Kalinowski says. “This actually also happens a lot with price point. We have an inside joke at the company, because often, you put a very early estimate for product price on a slide and say, ‘It could be this.’ Then for the next year, everyone’s like, ‘Oh, it’s going to cost this.’ No! Don’t give people anything to anchor to until you’re absolutely ready for it. People do that and it’s detrimental to the process. Add polish when you’re selling something. A strategy for guarding a product’s potential is to keep your prototypes and early ideas ugly longer.”
Let each team own their best-case scenario. When developing Apple’s cylindrical computer, the Mac Pro, Kalinowski had to juggle a multitude of factors. “The industrial design team wanted the device to have a really small diameter. But that meant a small diameter on the heat sink — which was a problem, because we wanted as much heat transfer as we possibly could. When the heat sink was small, more air had to be pulled through to cool the central processing unit (CPU), making it louder. Yet still we needed the computer to be quiet,” Kalinowski says. “The way to solve this problem was to let each team completely own its best-case scenario. There were separate teams inside Apple focused on optimizing for small diameter, focusing on keeping the noise down, worried about the heat transfer — we just let each one own that and drive as hard for their goal as they possibly could. For the thermal team, since this is a high performance machine, we needed the CPU performance as unlocked as possible, and so we let them fight for that. Then the audio team made sure the fan noise didn’t cross a certain threshold. Let everyone work on the best possible outcome for the feature for which they’re responsible, and you’re likely to end up making the best product trade-offs — and that means a better product.”
Smooth the negotiations while merging best-case scenarios. Once each team has come up with their ideal solution, you’ll have to start making tradeoffs — maybe the heat sink has to be tweaked in order to make the computer quieter, for example. “I never go into a meeting where a major decision is going to be made without having pre-baked the outcome. I first do a stakeholder analysis, go to each team, and get my proposal approved by everyone. What I learned at Apple is you really have to go around and explain what you think is right to each team. If you don’t have, for example, buy-in from across the product teams for a product decision, you really aren’t going to get anything out of that meeting. Do a lot of the early upfront negotiation so that meetings are just about finalizing decisions and locking them in,” Kalinowski says. “You also have to understand technical tradeoffs. You can’t just go in to an antenna team and say, ‘We can’t put antennas there. You have to put them here,’ and not understand if that’s not going to work. Do a listening tour and learn from a technical perspective about product tradeoffs. Understand why a team wants something. Form relationships with experts in your team, listen to them, believe them, but also push on them all in different ways at the same time as you’re trying to negotiate for that feature.”
The Three Bears of Prototyping
While you’re executing the above approach, get familiar with three bears of the prototyping process: you can go for too long, not long enough — but also just right. Below, Kalinowski details the warning signs for the first two and illuminates when you’re on the right track.
Prototyping for too long. You can sense when you’re on the wrong track when iterations start to yield tiny, incremental improvements and aren’t converging towards your goal. “If the results of your first, second, third and fourth tries at a problem land pretty close to each other, and you’re still far from your goal, that means you’ve made the wrong assumptions. You keep making changes but you can’t get there. You’ll no longer be closing on your goal in significant ways every time you iterate. At this point, you’ve got it all wrong. Either change your goal or change your assumptions. Scrap the plan. Back up to the beginning and find a different way to solve that problem,” Kalinowski says.
“Engineering teams will often iterate on one path forever and never hit their goal. It’s hard to go backwards when you’re in a product cycle and you want to ship. If you wait too long, that’s the worst. It’s so expensive that sometimes you can’t even get back to the beginning,” says Kalinowski “For the Mac Pro, we struggled to get the extrusion right. Two components, the heat sink and inner enclosure, were supposed to be made from one piece of aluminum. But it was too hard. We wound up cutting it into two pieces, even though the industrial design team really wanted it to be one. We went so far down the path trying to get it to be one piece, but in the end, it wasn’t working. That’s when Matt Casebolt made the call to stop. It’s because he had that feeling. We weren’t going to get there in time. We all wanted to make the industrial designers happy at Apple, but at a certain point, if you’re not converging, you need to make the call. They made this call. It was the right one.”
How to tell if you’re converging on the right solution in your prototyping process
Prototyping too short a time. Similarly, you can also sense when you haven’t spent long enough iterating. “Your engineers will be too worried. You’ll ask them ‘What do you think? How’s this going to work out?’ They’ll say, ‘I feel really uncomfortable. I don’t know if this is going to work.’ Sometimes, they won’t want to tell you. You have to really ask the right questions, like ‘What do you think we could be doing better?’ They’ll get really uneasy. That’s usually a big warning sign they feel you’re moving too fast. Ask, ‘What are you concerned about? What’s concerning to you right now?’ The answer to that is gold. It’s so valuable,” Kalinowski says. “Just because you’re not hearing about it doesn’t mean everything’s okay. You can encourage people to tell you bad news by improving your response. If you get upset or dismiss them, they won’t tell you anymore. Why would they? Even worse is to have an engineer express a concern directly and your response is, ‘cool, thanks.’ If you don’t do anything, they won’t trust you. You have to be like, ‘Oh. Shit. That’s a problem. What do you think we should do?’ Tell them, ‘Thank you so much, that’s really helpful. Tell me more. I want to fix it.’ And be sure to actually do something about it. It often means connecting people or calling a meeting and making it known that you’ll do everything you can to support your team in fixing the problem. Importantly, this goes double for your vendors.”
Prototyping just the right amount of time. Products are never perfect, but at some point you have to call it. “There are always more iterations we can do and there is always something that engineers are worried about, because it’s their job. Your engineers will never feel like the product’s done. Usually team leaders are okay with shipping with a few minor issues, but engineers will struggle with it. My strategy is to say we’ll get it right in the follow-on product. Let the engineer know that you care. You want to get it right. But this product needs to go out,” Kalinowski says. “They’ll still get to do all this cool engineering. They’ll still get to solve this problem that they’re unhappy about. Every engineer has that feeling of, “Man, I wish I did this differently. This isn’t quite where I want it to be.’ Great. Fix it, but you don’t have to fix it right now in this product. You can fix it in the next product and the next product, and make sure that it’s good forever forward. That helps a lot. Start a list of what you’re not going to fix. That doesn’t mean you’ll never deal with it. Fix it next time. Worst-case scenario, you can roll a fix after ramp, when you’re building up a bunch of product right before the sales date, or iterate off-build.”
Getting to Blastoff
Under the “just right” scenario, you have to make a call to halt engineering at the right time. That time is most likely sooner rather than later. “Ending your iterations close to the ship date is really dangerous because of all the interdependencies. Be ready to fight for not fixing problems unless they’re non-negotiables. For example, you can decide to change the material a few weeks before the ship date. Then you’ll find out that the material change means the product is no longer waterfast, or it fails drop testing,” Kalinowski says. “You can screw up something else that you don’t even know about. What I see a lot with immature companies is they try to make changes up until the very end and end up causing themselves more problems than they’re solving. Those small changes could bork the whole thing.”
Iterate Your Way To Success
Start your prototyping process by pinpointing your North Star. Write down your must-haves and don’t ship until you’ve achieved them. Determine your point on the speed-caution slider, and optimize for the most iterations possible by starting with the hardest, thorniest problems. Frontload your work. Make ugly prototypes and fork in additional features in order of their difficulty to engineer. Watch out for the warning signs of prototyping too long — or prototyping not long enough. Determine your stopping point by evaluating the remaining flaws you still want to address. The later you wait to make changes, the more dangerous your decisions will be. Be willing to fight for not making fixes.
“Look on the desks of your engineers. Are they littered with ugly prototypes? If so, you’re doing something right. Engineers often say they have it in their head how something is going to work. Ask for the parts. Ask to be shown. It often doesn’t work like they think it’s going to work. Push people to get to the prototyping faster. Then there’s convergence. All of your prototypes should allow you to converge, folding in more and more iterations and versions. If they’re not doing that, you might not be asking the right questions about your prototypes,” Kalinowski says. “Finally — and this is critical — prototyping is supposed to be really fun. If it’s not fun, you’re doing it wrong. If you’re over-constraining your engineers and they’re not able to come up with really cool solutions, or they do and you squash them, that’s bad. Free your engineers to come up with things — there should be an grand excess of cool ideas. After you’ve popped the champagne to celebrate your successful ship, you can pick up the next great product from your cutting room floor.”
Photography by Christophe Wu.
