Stefan H. Thomke is a leading authority on the management of innovation, as well as the William Barclay Harding Professor of Business Administration at Harvard Business School. The author of Experimentation Works, Stefan has written over one hundred articles, case studies, and notes in books and journals, including the Harvard Business Review, Management Science, and MIT Sloan Management Review. He has also chaired numerous executive education programs in companies around the world and is a frequent conference speaker and advisor to global business leaders.
In this episode, Stefan explains the benefits of conducting experiments within your company, as well as the importance of reflecting on all experiment outcomes—even if they are negative—in order to learn from them. Listen in to discover how to successfully incorporate experimentation into your business, the common thread in positive customer experiences, and how to evoke the desired emotion in your customers.
Learn the benefits of conducting experiments within your company or business with Stefan Thomke, author of EXPERIMENTATION WORKS. Share on X
If you enjoy the show, please drop by iTunes and leave a review while you are feeling the love! Reviews help others discover this podcast and I greatly appreciate them!
Listen in:
Podcast: Play in new window | Download
On Today’s Episode We’ll Learn:
- The common thread in positive customer experiences.
- Why companies have a hard time successfully redesigning or re-branding.
- The key ingredient to customer experience.
- How to cultivate curiosity in your employees.
- Why it’s important to reflect on every experimental outcome.
Key Resources for Stefan Thomke:
- Connect with Stefan Thomke: Website | LinkedIn
- Amazon: Experimentation Works: The Surprising Power of Business Experiments
- Kindle: Experimentation Works
Share the Love:
- If you like The Brainfluence Podcast…
- Never miss an episode by subscribing via iTunes, Stitcher or by RSS
- Help improve the show by leaving a Rating & Review in iTunes (Here’s How)
- Join the discussion for this episode in the comments section below
Full Episode Transcript:
- Welcome to Brainfluence, where author and international keynote speaker Roger Dooley has weekly conversations with thought leaders and world class experts. Every episode shows you how to improve your business with advice based on science or data.Roger’s new book, Friction, is published by McGraw Hill and is now available at Amazon, Barnes & Noble, and bookstores everywhere. Dr Robert Cialdini described the book as, “Blinding insight,” and Nobel winner Dr. Richard Claimer said, “Reading Friction will arm any manager with a mental can of WD40.”To learn more, go to RogerDooley.com/Friction, or just visit the book seller of your choice.Now, here’s Roger.Roger Dooley: Welcome to Brainfluence, I’m Roger Dooley.Today’s guest is bringing an incredibly important message to us, and it’s one that we don’t hear often enough. Stefan Thomke is a leading authority on the management of innovation and is a professor at Harvard Business School. Way back in 2003, he wrote Experimentation Matters: Unlocking the Potential of New Technologies for Innovation. In an earlier life, Stefan was an electrical engineer and a McKinsey consultant. And just a few weeks ago, his latest book came out. It’s titled, Experimentation Works: The Surprising Power of Business Experiments. Welcome to the show, Stefan.Stefan Thomke: Thanks for having me, Roger.Roger Dooley: Well, Stefan, I’m kind of partial to business people with an engineering background because I probably am biased too having one myself, but I think it combines science and real-world performance. And you’ve been writing about business experimentation for a long time, longer than most people been thinking about it. Does a bias for experimentation come from your engineering days, do you think?Stefan Thomke: My engineering days, Roger, got me started in experimentation. I had a summer job and it was in 1990, I was working in a chip factory. And they gave me a really tough assignment. They wanted me to optimize a process, when you’re actually pouring basically something on a wafer, it’s photo resistant. And there were many variables and I had no idea how to even get going on this. And then someone told me and said, “There’s this approach called experimental design, and there’s a systematic way in terms of how you could do this.” And so, I started reading on that and I got fascinated by this and then, you know how life sometimes goes. I used it all summer, then I got interested in it and then took a lot of classes, and then got really into sort of engineering experimentation, which then led me to sort of think about the role of experiments in product development, and the role of modeling and simulation, and all these kinds of things because the tools now allowed engineers to run much, much cheaper experiments.Stefan Thomke: And that then, eventually, became my first book, Experimentation Matters, which is mostly about the role of technologies in the context of R&D and product development, and what companies are doing in this space, large manufacturing companies. And I was really fascinated by that, which then was essentially the launching pad for the next round. And that is thinking about experiments in a much more holistic way, as almost like an enterprise activity that affects decision-making by pretty much everyone in an organization, which then later on led to a new set of research, which then led to this latest book.Roger Dooley: I do want to get to the book, but before we go too far down the science oriented rabbit hole and make you sound like a complete left-brain thinker, Stefan, I noticed that one of your most recent articles was at MIT Sloan Management Review, and it was about customer experience, and surprisingly enough you found that emotion was the key ingredient. Why don’t you explain briefly what you were thinking in that article?
Stefan Thomke: Well, that was a big insight that I have and the insight came from participants. So several years ago I got really fascinated by this question, how do you create a sticky experience? Something that you’ll never forget. Sometimes we have these moments where you go into a hotel, or you fly, or something happens and it’s so magical that it stays with you for 5, 10, even 20 years or so. And I was curious, could you actually design something like that? Could you design an experience that is so sticky? And I figured that the way to learn this is perhaps to teach a class on it. This is maybe a good way to find things out. If you don’t know what the answer is, teach a class in it, maybe the students will tell you.
Stefan Thomke: So that’s what I did. I created the class actually using Legos. There’s a method called Lego Serious Play. I became trained as a facilitator, created this class, iterated. And as part of the class participants get up and they tell their own stories. They tell stories about transformational experiences as customers that they’ve had. And I just paid attention to what they were saying. And I found this thread, this common thread in all these stories. And that is that all the stories that they told when they got up and were really, I mean people telling stories, things that happened 30 and 40 years ago. And the common thread was that all these experiences had a strong emotional element and it wasn’t about value. So, it wasn’t about, I got a really good deal, or this worked really well and had great functionality. No, it was all about how they felt as part of that experience. And, by the way, it was true both in B to C and B to B.
Stefan Thomke: So what I essentially did is I took all of these lessons, what I learned and developed a set of principles, which then became that article, which by the way, Roger, and I didn’t expect that, that article went viral. And MIT Sloan Management Review told me that it was the most or requested, the most downloaded article last year. So, I was quite happy about that. I did not expect that outcome, but I guess I touched a nerve here.
Roger Dooley: I guess so. Well, congratulations on that. And do you have just like one example of an emotional experience that would sort of typify what you found?
Stefan Thomke: Well, a lot of the stories that I heard were about, what I call, recovery. And that is, many of the stories started out with a problem. A problem would be, I go to the airport and I have no money. Or I come to a hotel and there’s no reservation. Or I have a medical emergency, I land in a country and I suddenly have a medical emergency. So, many of the stories start out like that. And suddenly, the expectations of what’s going to happen really drop, so you expect the worst. Suddenly you get to a new country and you’re ill, suddenly you have some severe illness or something like that. Or imagine being at the airport and you have no money.
Stefan Thomke: Or there was a story, someone going to Walt Disney, for example, I still remember that story very vividly. And with the whole family planned the vacation the whole year, and on the first day she lost her purse. And expectations, of course, were terrible. A whole vacation is ruined. All the tickets, money, everything is gone. And then, something like a miracle happens. Somebody steps in, someone who has the capability and the license to act and they do this heroic intervention. Like in the Disney example, they come in and they tell her saying, “Don’t worry about anything. We’ll try to find something. And, by the way, here’s some coupons so you can eat and we’ll figure out something, and enjoy the day in the park.” And that intervention kind of drives up your emotional state, and you’re suddenly elated because you expected the worst. And this stickiness, Roger, comes from the change in emotional state, I call it an emotional transition. And that’s kind of where customers are almost invulnerable. I mean, that’s the part where you can really kind of leave a big impression on them.
Roger Dooley: Right. Well, they’re already in a pretty high state of activation at that moment. They’re stressed out. Maybe they’re fearful. Their amygdala’s lit up because they don’t know what’s going to happen to them. And, undoubtedly, I think that there’s more general neuroscience that shows that people form memories more when they are activated like that. So, Ritz Carlton has a policy that each employee is empowered to solve a customer’s problem by spending up to $2,000, which is a huge amount of money for any random employee without manager’s permission solving a guest problem. But undoubtedly, in some cases, that’s going to be what you’re describing. Somebody going to have really what feels to them like a very serious problem. And this person, might be a desk clerk, might even be a maid, can fix it for them. That’s pretty cool.
Roger Dooley: But I do want to move on, but that’s a really fascinating work. And perhaps that could be the topic for another conversation. But Stefan, I really enjoyed reading Experimentation Works. It’s one of those books where as I’m reading it I just kept going sort of, “Yes, yes, yes,” to myself. But I’m curious, as you were working on the book, did you do any experimentation on the book itself? You have the title, the cover design or anything like that?
Stefan Thomke: Well, I did of course some of it and then I checked around, but quite honestly the way I picked the title, I had a different motivation. I didn’t experiment on that really because my first book was Experimentation Matters. And so, I wanted to continue sort of on the same theme. It’s a little bit like, are you a Beatles fan, Roger?
Roger Dooley: Yes. They were definitely around when I was in my youth. So, that’s when you form your music memories.
Stefan Thomke: So, you remember The Red and The Blue Album? They were like two very famous albums that the Beatles had. The Red Album was basically about all the old stuff, the classics, and The Blue Album was about all the new, sort of the new things that they wrote when they were trying to reinvent themselves. So, I thought about maybe my Experimentation Matters is The Red Album and Experimentation Works as The Blue Album, wanted them to be complementary because, as a Beatles fan you had to have both. You had to have The Red and The Blue Album. And so, that was actually my inspiration here. So, I didn’t experiment on the title.
Roger Dooley: Right, kind of like a Star Wars sequel where you’ve got to keep the title consistent so people know that it’s still a Star Wars movie.
Stefan Thomke: It’s a continuation of sort of the same theme, yeah.
Roger Dooley: Right, probably won’t sell quite as many copies as a Star Wars movie. But I have seen some interesting experimentation books. Tim Ferriss, a fellow Austinite here, conducted a very simple experiment when he was trying to decide on cover designs. He mocked up a few covers, went into a bookstore, put the alternate covers on a table near the front of the store, and then sat and watched, which people picked up first, or most often. And, now, that that isn’t a super scientific experiment from a statistical standpoint. You’d have to repeat it over a long period of time in multiple stores in multiple cities and so on. But in terms of being better than nothing, I think that qualifies, sometimes any data is better than no data.
Stefan Thomke: Correct. And, by the way, running experiments doesn’t rule out intuition and other things like that. I think that’s often a myth that people have. I think, using intuition, creativity and all these kinds of things, I mean, I think that complement experiments because even when you run experiments all the hypotheses that you’re testing have to come from somewhere. They could come either from qualitative research, they can come from maybe even more quantitative analysis. They can come from many places, including intuition. Sometimes our intuition actually is correct even though most of the times it’s wrong, but sometimes it is correct.
Stefan Thomke: So the only difference and so I always tell people is the companies that actually experiment a lot, they do all the kinds of things that companies that don’t really experiment do. They do the same kinds of things, but they have that intermediate step that is a company that doesn’t experiment they use intuition, qualitative research, and all these kinds of things. And then, they take all that and they go to market with that. In a company that experiments a lot they have an intermediate step and that is, they have their experiment as what Francis Bacon used to call put nature to the question. And they put nature to the question really early before they actually launch something, so they can learn before it gets really expensive to reverse a decision that you make.
Roger Dooley: My recent book, Friction, I did not do the in store experiment, but I did do some eye tracking experiments on Amazon thumbnail mock ups to see which people would look at. And it actually informed the final cover decision because one seemed to outperform the others in terms of holding people’s gaze in that tiny Amazon thumbnail size. Because, for most of us these days, these books are not being sold so much in Barnes & Noble as they are in Amazon where people encounter them primarily through a picture the size of a postage stamp.
Roger Dooley: But you bring up a great point because intuition is still important, or experience, people’s judgment is still important as a way of guiding what one uses for experimentation. But you’ve got a really interesting statistic in there, at least a more of an estimate perhaps from Microsoft. That when they run experiments only about a third of the experiments, which are presumably based on reasonable hypotheses, they aren’t just randomly testing stuff. They’re coming up with a hypothesis that makes sense, but only about a third of the time do they produce positive results. And about a third of the time they’re neutral. And a full third of the time they produce negative results. So, a company that is relying primarily on intuition and common sense, and managerial experience is going to be wrong pretty often. How common do you think that split is? I mean that’s sort of a general third, third, third split. But do you think that’s pretty typical across different businesses?
Stefan Thomke: I think in reality it’s actually worse than that. Even if you look at Microsoft, if you just look at Bing, the numbers are even more dramatic. And that is I was told by one of my coauthors when we wrote a paper, he said about 80 to 90% of the things that tested Bing turned out not to work. And not to work, again, means you either get a null result, no effect at all, or you get a negative result going in the opposite direction of what you expect. And that number, 8 to 9 out of 10 getting it wrong, 8 to 9 out of 10 was pretty consistent. When I looked at, which was one of the chapters in the book, same thing again, they come 8 to 9 out of 10 day they learn.
Stefan Thomke: By the way, Roger, these are the experts. These are not people who don’t have any experience. They experiment for living, so they know their business really well. Even in these kinds of settings you got a fairly high in quotes, what I call “failure rate.” But these are good failures in some ways because often you learn something, and that leads to a new hypothesis. It leads to some new insights. So, failing is kind of like the default mode here in these settings. That’s what you normally do. And “winning,” or having a successful result is really more the exception rather than the rule.
Roger Dooley: Yeah, I was just at a conference in Holland on the Island of Texel, where I met the head of experimentation at Booking. And one thing too that these people use to drive their experimentation is behavioral science, which is a topic of great interest to many of the folks in our audience. And you can’t just take some principle from behavioral science and say, “Okay, this is going to work.” What you do and what these smart businesses do is they test it. And then, as we’ve been discussing, it doesn’t always work. Often certain things will work most of the time, but you’ll find exceptions you can’t even always explain that, “Hey, it didn’t work this time, but that’s okay. We tested it and now we know,” and maybe you use that information to test something else.
Roger Dooley: And I think you made a key point there, Stefan, in that every experiment teaches you something, and it’s important not just to say, “Oh, that didn’t work,” and discard the idea. It’s important I think to reflect on that a little bit and say, “Okay, did we get some kind of valuable information from this?” Now, perhaps sometimes you really don’t, but there will be times when you take that negative result and look at maybe another negative result and a positive result and you can start drawing some conclusion about your customer’s behavior, their interest and so on.
Stefan Thomke: Absolutely, Roger. And you have to because, if you think about, if you’re an organization that runs, say, thousands of experiments, or some of the organizations run tens of thousands of experiments a year, these hypotheses have to come from somewhere. It’s not so easy to come up with tens of thousands of hypotheses annually. So that’s where a lot of these failed experiments come in, in addition of course to behavioral science, and all sorts of other things. Sometimes it’s a surprise, a surprising customer insight or something like that that drives it. Could come from many different sources.
Roger Dooley: Yeah. As I was reading the book, I was surprised that we have a a case study in common. In my speeches, I often use JC Penney’s sweeping changes to their stores and pricing as an example of how not to do it. And I am focused on their pricing issues where they went away from the sale pricing that was used by just about all their competitors where you have weekly sale flyers, you get these things in people’s mailboxes, and they show markdowns of 40%, 50%. and JC Penney took what seemed to be a logical approach of going to everyday pricing, and these prices were generally as low as their competitors, but they felt that they took a lot of the stress out of the shopping for their customers that they wouldn’t have to worry about, oh gee, is this price still good. Or no, I missed it. They can just go to the store and always know they can get a good price on it. Very logical. But it, unfortunately, took away that sense of urgency that the sale flyers were giving them. Other people said, “Wow, I’d better get into the store and buy this because it’s going to be gone.” But, obviously, that was just one of the things they did.
Roger Dooley: Why don’t you tell sort of the JC penny story from your standpoint Stefan?
Stefan Thomke: Roger, it’s a great story. It’s a story about an executive named Ron Johnson who worked at Apple. And he, in fact, was what you would consider to be a retail god because he and Steve Jobs together, created the Apple Store, the Apple Store concept. And we all know that the Apple Store is quite, possibly, the most successful physical retail concept, probably, in the last two decades. Hugely profitable, highest revenue per square footage in the industry, in all industries, in fact, by a large degree. So, here’s Ron Johnson, who was the retail god, and JC Penney, of course, looks at him and what he’s done at Apple, and they’re thinking, “Well, we want that too for our stores.” And so they hire him, bring him in as the new CEO in 2011.
Stefan Thomke: And, of course, he gets to work right away and he starts to make all these changes. Some of the changes that you just talked about. He eliminated coupons, clearance racks, branded boutiques, used technology, eliminated cashiers, and so on, and so on because all these things worked so well at Apple. About 17 months later, JC Penney’s fighting for its survival. Sales had plunged, losses are soaring, and Johnson is out of his job. And you have to ask yourself sort of what happened? Of course, what happened is that he thought he knew what works in a retail environment and it worked very well for Apple. He brought sort of that knowledge that he had into a different retail environment. And according to what people said later is, and he didn’t test. There were no tests and presumably if he had run a few of those experiments, perhaps, the experiments would have told him that this is not going to work. But he was in a hurry.
Stefan Thomke: What’s really interesting, Roger, later on Ron Johnson actually gave talk, which I heard, and he talked about this experience. And when you listen to him you wouldn’t consider him to be an arrogant person, and so he’s actually quite humble about sort of what he did then. But he talks in it about what he called situational arrogance. And that is normally he’s not an arrogant person, but in this situation he became arrogant because he so strongly believed in these assumptions which came from his experience at Apple. And he was in such a hurry that he didn’t test. And, now, we know what happened.
Stefan Thomke: And you can go back and you can look at other examples whether you look at Evan Spiegel at Snapchat and all kinds of other examples where they kind of fell into the same trap. They all follow their intuitions, make some assumptions, then they’re in a hurry, they launch something. And then, they later on regret not having run the experiment.
Roger Dooley: One good thing, at least, about the digital world is that experimentation is easy compared to, say, testing new retail concepts that’s going to take a lot of time, and be expensive. Busy websites, so like booking.com, or Expedia, or Amazon can run lots of tests continuously, get results back in very short time, and so on.
Roger Dooley: About five years ago, now, I wrote an article at Forbes titled Never Redesign Your Website Again. And my point was that the approach that most companies take is obsolete. They redesign a website based mostly on the ideas of the people in charge. And, certainly, getting information from people at various levels. They turn the designers loose, they get to the point where it’s almost ready. Then, maybe they do a little bit of user testing and try and tweak stuff that might be confusing. They roll it out, then fix bugs for a few weeks and say, “Wow, okay, we’re done with that,” and move on.
Roger Dooley: But if you look at Amazon you are never aware that their website has been redesigned because it has never undergone, at least as long as I can remember, a redesign process. Instead, it’s an evolution process that’s constantly being shaped by A/B testing and other kinds of testing. So, why do companies not get this, especially when the tools are so simple?
Stefan Thomke: Because it’s a completely different approach to doing things. And the environments that you’re describing, Roger, there’s no beginning and no end it’s agile. I mean, in software it’s dev ops. And often companies are sort of in a project mode. They think everything as a project. And a project has a beginning, a project has an end. And they manage it as a project, they resource it as a project. And it’s hard kind of to shift gears. And I have an example about an international shipping company that went through this as well, where they have a project mentality. And then, this comes along and they have a really difficult time adjusting to this kind of environment where things kind of just go. And, like you said, it’s very evolutionary. You just keep on testing, testing, testing, and it never ends. You just keep on making it better.
Roger Dooley: Yeah. I think you really made a key point there, Stefan. You cannot think of these things like a website redesign as a project. And you’re right, that’s what happens. People say, “Oh wow, the website’s really looking kind of dated. Users want something different, so let’s redesign it,” but it’s a project rather than a process. So, great, great point.
Roger Dooley: But do you think there’s a danger in an experimental approach that sort of focuses on incremental changes instead of indicational radical rethinking? I mean, it’s easy to test small things, whether it’s a design element, or a headline, or the sorts of things that Amazon test constantly. But does it make you think small instead of occasionally thinking big?
Stefan Thomke: So, Roger, I think about experimentation in a different way. And that point, by the way Roger, has been made by number of people, but I think they’re just so driven by what they see in A/B testing. I think about it in a much more holistic way. That’s why I call it experiments rather than A/B tests. And because there are different kinds of experiments that you can run there are, of course, these optimization experiments that you just spoke about like A/B testing. And they’re amazingly powerful because you can take a website and you can optimize it with thousands, or tens of thousands of these experiments, and make them incrementally better.
Stefan Thomke: But there are other kinds of experiments that you can run as well. There are experiments that are more exploratory where you can, for example, test sort of bigger strategic directions as well. Of course when you go into that mode you’re giving up some things because you may be changing too many variables at the same time, and you don’t get the cause and effect relationships that you would get in a very, very precise A/B test. But these kinds of experiments serve a purpose as well because they give you kind of a sense, like what you sort of said earlier, with just maybe putting up book title down and giving some sort of a qualitative sense of where things are heading, which you can then later on optimize again with some of the more precise optimization type of experiments.
Stefan Thomke: So, there are different kinds of experiments. And even if you go back to the world where I came from in engineering design there are many different kinds of designs that you can come up with in terms of experiments, multivariate tests, and all sorts of things that allow you to explore much bigger directions. I think the issue often is that people get so wedded to these A/B tests and maybe that’s the only tools that they have. And they forget that there are other kinds of experiments out there, there other experimental designs that they can take advantage off.
Roger Dooley: Yeah, let’s talk just a little bit about culture, Stefan. A pretty recent guest on the show was Safi Bahcall, who wrote a book called Loonshots. And he talks about how, particularly, as organizations grow they can become somewhat risk averse in their culture, and people get punished for failure. So, people do safe things. They don’t innovate in a big way. They do little safe product extensions and whatnot because they know that they aren’t going to get punished for failure. And you make the point too that companies often embrace innovation and they’re all about innovation, but they don’t really embrace failure. And if you’re innovating failure’s pretty much inevitable, how do you change that culture?
Stefan Thomke: Well, first of all, I tell people you have to distinguish between, what I call, failures and mistakes even though the two words are semantically very close to each other. Mistakes, to me, are something that really adds no value. Mistakes don’t really answer a question that you have. And so, if Amazon builds yet another distribution center, and they’ve built many, that’s operational execution they’re not trying to answer a question here. They’re just trying to build another center. That would be a mistake, there’s no learning going on.
Stefan Thomke: But failure, to me, is something that’s preceded by a question, there’s a learning objective here and I think it’s important sort of for companies to acknowledge the fact that failures, in that sense, are something positive because you learn something new sort of coming out of that. So, you want to encourage these kinds of failures while, at the same time, you want to minimize mistakes because they don’t add value. And I think starting with that distinction is very useful because just saying to people, “You’ve got to fail more,” that’s not very useful because there are, obviously, some things where you don’t want people to fail on. And that’s why I call them mistakes.
Stefan Thomke: Then, what I also find is that when companies are trying to scale experiments, and we’re talking about many experiments here, it’s kind of not so hard to run 5 or 10 experiments a year, but I’m talking about hundreds, thousands of experiments. When you start doing that, then you start really running into big cultural obstacles. And I found that the cultural obstacles are probably harder than some of the infrastructure challenges that companies have, and that they can solve very easily. Now, if they have enough budget there are lots of good tools around. And there are five things that identify, Roger, that need to be addressed in a company’s culture if they really want to scale it up.
Stefan Thomke: And the first thing is to cultivate curiosity. Really important because you need to have curious people around. Curiosity needs to prevail and people need to see failures not as costly mistakes, but as opportunities for learning. Then, the second I have is insists that data trumps opinions. That is when you have good data, reliable data, you need to make sure that they also do trump the opinions, so the opinions cannot prevail in the face of very persuasive data.
Stefan Thomke: The third is democratizing experimentation. You need to kind of get it down to the lowest level. You need to empower people to run experiments without creating a lot of red tape. They need to run pretty much experiments anytime they want, and so you need to kind of create that environment. Fourth, I think you need to be ethically sensitive because once you start running experiments you have huge ethical responsibilities. You don’t want to run experiments that hurt people. And a fifth one is to embrace a different leadership model. That is the role of leaders in these kinds of environments is very different than what we normally sort of traditionally think of as a leader in a traditional company.
Roger Dooley: Yeah, well, I think when we think about, say, somebody like Steve Jobs we believe that he was the sort of autocratic genius who simply came up with ideas and told people what to do. But actually he did listen to people, he did embrace the results of experimentation too. And none of us, also, are Steve Jobs. And he made a lot of mistakes, but he had some pretty big wins too. But I do believe there are a lot of managers out there who are more ready to go with their gut, even presented with data. They’ll find fault with the experiment and say, “No, no, that can’t be right because I’ve been doing this for 20 years and I know better.”
Stefan Thomke: Absolutely. The American writer, Upton Sinclair once said, “It’s difficult to get a man to understand something when his salary depends upon his not understanding it.” And so, there are also incentives in place that sometimes get in the way of saying, “Hey, I don’t know.” Or, “I don’t understand,” and just being humble about sort of what you do, that’s really important as well. You may have heard sort of the term HiPPO.
Roger Dooley: Oh yeah, that’s pretty common in the conversion optimization industry.
Stefan Thomke: The highest paid person’s opinion, right? So, people are the most dangerous animals on the planet, so we need to be aware of those HiPPOs, right? So, I think everybody knows about them. So, there are lots of reasons and sometimes, quite honestly, it’s also our fault. We train these MBAs, and we train them to be decision-makers. And they kind of climb up the ladder, and they want to show that they’re good decision-makers and so on.
Stefan Thomke: I also want to say, I think it’s important to say, Roger, as well, that not every decision needs to be made based on experimental outcomes. I mean, sometimes there are good reasons why you may want run experiment and the result of the experiment suggests something that you may not want to do either ethically, or for strategic reasons.
Stefan Thomke: There was a great article in the Wall Street Journal in 2018 about Netflix. I don’t know if you saw that.
Roger Dooley: I didn’t.
Stefan Thomke: And there was a decision that they had to make about sort of a new show called Grace and Frankie. So, they had a promotional feature, and what they did is they ran different tests. They ran one test with Lily Tomlin alone, one of the actresses in it. Then, they ran another one with Lily Tomlin and Jane Fonda who was sort of partner. So, they run all these tests and they found that when they measured viewer engagement, they found that the best promotional feature is actually to show only Lily Tomlin and not Jane Fonda. Now, that creates a bit of a problem because they have a contract with Jane Fonda and they, certainly, don’t want to alienate her. And so, they ended up running the promotional feature with Jane Fonda anyway, in spite of the data telling them not to do it for other reasons.
Stefan Thomke: But the power here of the experiment, or the experimental evidence here is to make these decisions transparent. So, they can’t lie to themselves and say, “Well, we’re doing this because we think it’s good for user engagement.” Well, it’s clearly said it’s not good for user engagement, but it’s good for something else. So, this is another aspect, I think, of experimentation is adding transparency, adding honesty to the discussions, with that we also sometimes underestimate,
Roger Dooley: Sure. And I think even cost can be an example where you were an experiment shows that, “Gee, if we change A to B, that will create better customer engagement or better customer experience.” But then, when the cost comes up it’s like, “Well we simply cannot justify that expense for the level of improvement we’d expect.” And I think your point about leadership was really good as well, Stefan, that leaders tend to think they’re paid to know more stuff, and to make the tough decisions. You worked for McKinsey, right? And if you’re a new consultant coming into an organization, billing hundreds of dollars an hour you can’t really maybe afford to be that humble, and be much of a learner because everybody’s looking to you for the answers.
Stefan Thomke: Exactly. You can’t come in and say, “I’m sorry, I don’t know,” and the meter is running. So, I defined a set of new roles for these leaders. I identified at least three of those. The first one I think what a leader should do is, first of all, sort of to acknowledge the fact that they don’t have all the answers. But the first role I think is what’s really important is the leaders have to set, what I call, a grand challenge. Something that can be broken into testable hypotheses and KPIs. And the reason is very simple. And that is you don’t want to have an organization where everybody just experiments willy nilly without any direction. You want them to be part of these experiments, to be part of an overall program. And to determine sort of what the overall program is about requires a grand challenge. That’s the first.
Stefan Thomke: The second one I sort of think what leaders should do is they should put in place systems. I mean, they have access to resources, they can change the organizational designs and so forth. So, their job is to make sure that people who want to run experiments have sort of the right organization, have the right resource to do so. So, they don’t have to every time, sort of scratch their head and say, “How do I run this experiment?”
Stefan Thomke: And then third, of course, which we already addressed, is to be a role model. It means leaders have to live by the same rules as everyone else and subject their own ideas to test. And that’s really hard for a lot of leaders. But in these kinds of environments you have to do it that way.
Roger Dooley: Right. Well, that’s probably a pretty good place to wrap up. Let me remind our listeners that today we are speaking with Stefan Thomke, author of Experimentation Works: The Surprising Power of Business Experiments. Stefan, how can people find you
Stefan Thomke: Many places. Well, first of all, they could find the book in all the usual channels: Amazon, Barnes & Noble and so forth. Yeah, I’m at Harvard Business School. You can find me just typing in www. T-H-O-M-K-E, thomke.com which takes you directly to my HBS website. I will always be here. I’m not going anywhere. Yeah, you can also find me on LinkedIn, if you want just let me know where you heard me, so I kind of make the connection so I know where you’re coming from. You can write to me email as well, the letter it’s just [email protected]. So, lots of ways to find me, so I’m a very public and I’m always looking forward to engaging with people, and learning about their stories, and what they’re doing.
Roger Dooley: Great. Well we will link to all of those places on the show notes page along with any other resources we spoke about. And the show notes page will be found at rogerdooley.com/podcast. And we’ll have a text version of our conversation there too. Stefan, thanks for being on the show. Great book.
Stefan Thomke: Thanks, Roger. Thanks for having me.
Thank you for tuning into this episode of Brainfluence. To find more episodes like this one, and to access all of Roger’s online writing and resources, the best starting point is RogerDooley.com.
And remember, Roger’s new book, Friction, is now available at Amazon, Barnes and Noble, and book sellers everywhere. Bestselling author Dan Pink calls it, “An important read,” and Wharton Professor Dr. Joana Berger said, “You’ll understand Friction’s power and how to harness it.”
For more information or for links to Amazon and other sellers, go to RogerDooley.com/Friction.
