Data-Informed Decision Making
Use data as a tool for learning, not as a replacement for vision
Data-Informed Decision Making draws a crucial distinction between being data-driven and being data-informed. A purely data-driven approach, while rigorous, can lead to local maxima: the best possible outcome within a constrained problem space, but not the best possible outcome overall. Like evolution, machine optimization can create the eye but cannot redesign it from scratch. Humans are good at finding new systems; machines are good at optimizing known ones.
The framework argues that quantitative data is excellent for testing hypotheses but poor at generating new ones. If you let algorithms optimize your website without human judgment, you may end up with the highest click-through rates but destroy your brand. Orbitz discovered Mac users would pay more for hotels, but blindly optimizing on that insight without considering the PR implications would have been dangerous.
The solution is to use data as one input into decisions, not as the sole decision-maker. Instincts are experiments. Data is proof. You need a big, hairy, audacious vision that Lean Startup helps you pursue methodically, not a data-driven wandering that leads nowhere. As the book puts it: be Lean, but do not be small.
- Math is good at optimizing a known system; humans are good at finding a new one
- Quantitative data is excellent for testing hypotheses but poor at generating them without human introspection
- Instincts are experiments and data is proof; you need both gut feelings and rigorous measurement
- Lean Startup is the process you use to move toward and achieve your vision, not a replacement for having one
- Be data-informed not data-driven: use data as a tool to guide decisions, not as the sole decision-maker
- Start with a big visionBefore any data collection, articulate your bold, changing-the-world vision. This is your compass. Without it, you are susceptible to every piece of customer feedback and competitive move. Lean Startup helps you get there; it does not replace the destination.Pro tipYou are not building a product. You are building a tool to learn what product to build. Keep this distinction clear.WarningDo not use Lean Startup as an excuse to start without a vision. Landing in Normandy did not mean the Allies lacked a big vision.
- Use gut instincts to generate hypothesesYour intuition, pattern recognition, and domain expertise generate the hypotheses worth testing. Do not expect data to tell you what to build. Instead, let your instincts propose directions and then use data to validate or invalidate them.
- Use data to test and validateRun controlled experiments, A/B tests, and cohort analyses to rigorously test your hypotheses. Let the data tell you whether your instinct was right. If the data contradicts your gut, take it seriously. But if the data only optimizes within a local maximum, step back and consider whether you need a fundamentally different approach.Pro tipWatch for local maxima. If you are given three wheels and asked to evolve the best vehicle, data will give you a tricycle. Only human insight says four wheels would be way better.
- Avoid the ten pitfalls of data analysisMaintain analytical rigor by avoiding common traps: assuming data is clean, not normalizing, mishandling outliers, ignoring seasonality, ignoring context when reporting growth, creating dashboard clutter, setting overly sensitive alerts, ignoring external data sources, and seeing patterns in noise.
- Moderate machine optimization with human judgmentWhen using automated optimization tools, always have human curators who understand the bigger picture. Algorithms optimize the metrics they are given; humans understand the metrics that matter but are not measured. Brand damage, PR risk, and long-term strategic position cannot be quantified in an A/B test.Pro tipHumans do inspiration; machines do validation. Structure your organization so that creative vision and data rigor work together rather than competing.
Airbnb founders had a gut instinct that professional photography would increase bookings. Rather than just implementing it, they built a Concierge MVP to test the hypothesis. Data confirmed that professionally photographed listings got two to three times more bookings. They then iteratively expanded the program, measuring each step.
Orbitz found through data analysis that Mac users were 40 percent more likely to book four or five star hotels. A purely data-driven approach would optimize pricing based on browser type, but the human judgment layer recognized the PR risk of charging customers differently based on their device.
This framework emerged from a tension the authors observed in the Lean Startup community. Some entrepreneurs used Lean as an excuse to start companies without a vision, reasoning that data would tell them what to build. Others rejected data entirely, preferring to trust their gut. The authors saw both extremes fail repeatedly.
The Omniture story was a crystallizing moment: the company's content optimization software quickly learned that scantily clad women generated the highest click-through rates, but this obviously damaged the brands of companies using it. The machine found a global optimum for one metric while destroying value on unmeasured dimensions. This illustrated that human judgment and machine optimization must work together, with humans providing inspiration and machines providing validation.