What user researchers can learn from Sherlock Holmes
The parallels between good research and good detective work are striking. In this article we take a close look at what user experience researchers can learn from the investigative methods used by detectives. And, in the spirit of all the best detective stories, we draw an important conclusion: if you want to become a better researcher you should learn to think like a detective.
The similarities between a good researcher and a good detective are quite striking. Maybe this is not a surprise as both disciplines involve investigation, both seek to establish a trail of evidence, and both aim to arrive at a solution. But it goes further. The knowledge required, the skills and experience needed, and the methods and techniques used also have much in common. In fact, it is not stretching things at all to say that detective work actually is research, and research actually is detective work.
So what can we learn about doing UX research from the greatest detective of them all — Sherlock Holmes? Holmes’s method comprised these five steps:
- Understand the problem to be solved
- Collect the facts
- Develop hypotheses to explain the facts
- Eliminate the least likely hypotheses to arrive at the solution
- Act on the solution
We’ll alternate between wearing our deerstalker and our UX hat as we take a look at what each of these steps can teach us about doing good UX research.
Step 1. Understand the problem to be solved
“I never guess. It is a shocking habit — destructive to the logical faculty.”
— The Sign of Four (1890).
Which do you find most interesting, questions or answers?
It’s no contest — the question always wins. Even asking that simple question got you thinking — but as soon as you answered the interest is over. Answers are seldom exciting in the way that questions are and, like most researchers, Holmes was intrigued by the challenge of the problem.
So it’s puzzling that in the world of design so much prominence is given to “solutions” or answers. Solutions are the goal but they should never be the starting point. The cost of focusing too early on design solutions, as many development teams do, is that you easily lose sight of the problem you are trying to solve.
Sherlock Holmes resisted leaping to solutions, arguing:
“It is a capital mistake to theorise before one has data. Insensibly one begins to twist facts to suit theories, instead of theories to suit facts.”
He always started each case by focusing on the problem. The problem would sometimes arrive in the form of a letter, sometimes as an item in the newspaper, but most often it would announce itself by a knock at the door. The client would then present the mystery to Holmes and he would probe the client for salient information. He would also bring to bear his considerable knowledge on the topic, recalling prior cases and finding out all he could about the likely protagonists. Holmes never relied on guesswork or on assumptions. For Holmes, each new case was unique, and what mattered were reliable and verifiable facts about the case. These gave the investigation an initial focus and direction.
Here are some things we can learn from Holmes’s approach to help focus UX research:
- Move away from solutions.
- Create an explicit research question (actually write it down with a question mark at the end).
- Don’t start doing any research until you have this question.
- Don’t assume the question has never been asked before.
- Find out what your company already knows.
- Do an archival search — start by reading prior research reports.
- Interview team members and stakeholders.
- Use a checklist to collect background information in a systematic manner.
- Leave nothing to guesswork.
Step 2. Collect the facts
“Data! Data! Data!” he cried impatiently. “I can’t make bricks without clay.”
— The Adventure of the Copper Beeches (1892).
Although Holmes made skilful use of questioning, he knew that relying on people to accurately report what they may have seen or heard, or what they know and think, is an unreliable approach to investigation. Opinions are not facts, and speculation is not evidence. Instead, his primary method of collecting facts was careful observation:
“You know my method, Watson. It is founded upon the observation of trifles.”
For Holmes, the seemingly unimportant aspects of a crime scene, and the minutiae of the case were vital. From small clues, large inferences can often be drawn.
Observation is essential to innovation, and is an important technique for UX researchers. When used in field research and site visits, it can help us to understand the “messy reality” of how people work and what they really do (rather than what they say they do). It also helps us look at the minutiae of people’s work, and at the details of the workflow, in a way that users often cannot do themselves. This is the key to identifying latent user needs — things people can’t articulate because they don’t know what’s possible.
A good practice during an observation session is not to worry about the relevance of the information you are capturing. Don’t approach data collection with any kind of filter based on prior expectations, assumptions, or pet theories. Don’t judge or weight information at this stage. Don’t try to interpret the things you observe or fit things into a plan or a solution. All of that comes later. Reflecting on a successful case, Holmes reminds Watson:
"We approached the case, you remember, with an absolutely blank mind, which is always an advantage. We had formed no theories. We were simply there to observe.”
Right now you just need to be sure you catch everything. You can always discard items later, but it may be impossible to revisit the site and collect information that you missed.
You may not need to wear a disguise, or crawl on your hands and knees with a magnifying glass, but here are some things we can learn from Holmes to improve our data collection and observation skills:
- Watch people actually doing their work — don’t just get a demonstration.
- Remember that your participants are the experts, you are the “novice”.
- Focus on the most typical tasks; busiest days; typical days; critical incidents.
- Find out what activities precede and follow the task you are observing.
- Look for inconveniences, delays and frustrations.
- Shadow people: follow them wherever they go.
- Point to things and find out what they are for.
- Get copies or photos of artefacts, samples, forms, and documents.
- Make diagrams of the workspace.
- List the tools people are using.
- Note people dynamics and interactions.
- Be alert to things happening simultaneously.
- Record anything unusual about the scene you are looking at.
- Ask yourself if anything is missing.
- Observe behaviour at a low level of detail — watch what people touch and what they look at.
- Pay attention to the sequences and timing of events and actions.
- Remember to pay attention to trifles.
Step 3. Develop hypotheses to explain the facts
“Watson, you can see everything. You fail, however, to reason from what you see. You are too timid in drawing your inferences.”
— The Adventure of the Blue Carbuncle (1892).
Holmes formulated hypotheses by interpreting facts in light of his considerable knowledge and experience:
“As a rule, when I have heard of some slight indications of the course of events I am able to guide myself by the thousands of other similar cases which occur to my memory.”
His knowledge was very deep but it was also very narrow. He had unparalleled understanding of chemistry, footprints, various poisonous flowers (though not gardening), and bloodstains, and he was an accomplished violinist. His narrow focus is evidenced by his monograph on the distinction between 140 different forms of cigar, pipe and cigarette tobacco.
Similarly, we must bring to bear our knowledge of human behaviour, technology advances, market trends, and our company’s business goals, to help us formulate the models and solutions that best fit the facts we collected in our field research.
Our hypotheses now help us to identify the gaps in the way people work — a gap being the opportunity that emerges when we compare the way something is currently being done, and the improved way it might be possible to do it in the future. To help our innovation and design teams spot these gaps, this stage of our work must adequately answer questions about the user, the tasks and the environment of use (Who? Doing what? Under what circumstances?).
Our models, personas, scenarios and stories should include:
- The primary goals that people have.
- The workflow.
- The mental models people build.
- The tools they use.
- The environment they work in.
- The terminology they use to describe what they do.
When the analysis is completed, all of the salient facts should have been explained, and the gaps and opportunities should start to emerge, and we can — finally — begin to propose solutions.
Step 4. Eliminate the least likely hypotheses to arrive at the solution
“It is an old maxim of mine that when you have excluded the impossible, whatever remains, however improbable, must be the truth.”
— The Adventure of the Beryl Coronet (1892).
At this point the detective is faced with a number of possible suspects, and, if we have done our jobs well, we will be faced with a number of possible product ideas and solutions. In this step we begin eliminating the solutions least likely to succeed. The detective asks, “Does the theory fit the facts?” and we ask, “Does the solution fit the data we observed?” We start to drop the weaker solutions — those that don’t quite account for everything we observed; and we apply Occam’s Razor and drop the solutions that only fit the data by dint of being overly complex.
And we carry out tests. Holmes, remember, was a student of science. He carried out experiments.
Eliminating potential concepts is a high stakes game. The evidence put forward in favour of one solution versus another must be compelling. This is nothing new for detective work but strength of evidence seems to be rarely considered in UX research. I don’t mean statistical significance, I mean reliability and validly of data, and the ability for data to predict an outcome. Most of all I mean evidence that resists bias. Not all data are equal in this respect. Here’s a useful hierarchy:
- Strong evidence: Carefully designed and independently conducted usability tests; beta tests employing committed testers; archival research and meta-analyses of such studies.
- Moderate evidence: In-house usability tests; user testing with a company’s own employees; feedback from usability experts using the product.
- Weak evidence: Opinion-based data from focus groups and surveys; feedback from friends and colleagues; opinions of internal management; anecdotal evidence.
As we move into the actual design cycle, testing should continue as an iterative process, with the team prototyping their way towards success. James Dyson famously tested 5,127 prototypes before he achieved success with his Dual Cyclone bag-less vacuum cleaner. You may not need so many iterations, but you also should not expect to get there with a one-shot approach.
So what about the role of intuition? Doesn’t that fit in anywhere? Yes. But, first, let's clear up a misconception.
Intuition is not guesswork. We only have intuition for things we’re familiar with. That’s what intuition is — familiarity. And familiarity means experience. So, yes there is a role for intuition because we bring our experience to bear on our decision-making. When we read of CEOs like Steve Jobs making intuitive judgment calls, he was, like Sherlock Holmes, drawing on his vast experience of previous similar situations. He knew what worked and what didn’t. But we are not talking here about a ‘shoot from the hip’, ‘seat of the pants’, ‘shot in the dark’ kind of intuition. And, frankly, if you are going to just guess, there’s no sensible reason for starting at this stage. You might as well use guesswork from the beginning.
Step 5. Act on the solution
“Nothing clears up a case so much as stating it to another person.”
— Silver Blaze (1892)
Once Holmes had solved a case, he would explain to the client, and to Watson and the police, how he had solved the crime. Then Inspector Lestrade of Scotland Yard would arrest the culprit and the case was closed. Holmes’s job was done. He archived the experience in his great mental storeroom and moved on to the next adventure.
Here are some recommendations that can help ensure the design team takes action on the results of your investigation:
- Conduct a one-day design workshop to transition the UX solutions to the design team.
- Present the team with specific and actionable design recommendations.
- Agree accountability for implementing your UX recommendations.
- Create and present a clear series of next steps — both tactical and strategic.
- Promote iterative design by arranging to test the new version of the design.
- Educate the team in UX and user-centred design methods.
- Don’t just attend design meetings, chair them.
Thinking like a detective
We set out to discover what it means to think like a detective. Have we got there?
Perhaps, but I still want to capture just the essence of the detective’s method, to make things even more striking — some kind of emblematic statement that we might put on a wall poster. Much as we may admire Sherlock Holmes, there’s no escaping the fact that he did have one characteristic that most detectives would consider a hindrance — he didn’t actually exist. So to get a real life perspective I decided to talk to a real life detective. I got in touch with an old school friend, recently of the West Yorkshire Criminal Investigation Department, and asked him, “If you had just one piece of advice to give to a novice researcher, what would it be?” He didn’t hesitate to reply:
“Never, ever, ever, act on assumptions. Search out the facts and act on those.”
Holmes himself could not have put it better. Facts and evidence, not guesswork and assumptions. That’s how to think like a detective.
Philip Hodgson, October 2011