Analyzing User Research Data
To me, the most fun part of user research is analyzing the results and watching patterns emerge. The process is an interesting mix of cataloging data and trusting your intuition, wading around in the minute details while keeping in mind the larger research objectives. The goal of analyzing the results is to find patterns that will lead to new insights. The approaches to discovering those patterns in qualitative and quantitative data are somewhat different.
The user research methods that produce qualitative data are methods like user interviews, ethnographic field studies, focus groups, or case studies. Researchers track and organize this type of data starting with a process called ‘coding’: anytime something interesting in the data appears, it is given a ‘code’ or brief description (Mortensen, 2019). The code often uses the same wording the participants used which helps maintain the unbiased aim of this technique (Cresswell & Cresswell, 2018). The types of codes used will depend on the goal of the research study – whether it aims to explore new themes, or confirm a theory (Mortensen, 2019). Once all the data has been given a code, the codes can then be arranged into themes.
Themes are broad categorizations by which the researchers start to apply an interpretation to the data (Mortensen, 2019). The insights from the theming process tell a larger story of the participants' experiences, and may fit together chronologically or narratively (Cresswell & Cresswell, 2018). Researchers examine how the themes relate to each other, and the implications for overall research question (Mortensen, 2019). This coding and theming process for qualitative data uncovers valuable insights on how users think and feel, while quantitative data analysis uncovers insights on what users actually do.
Quantitative data in user research might include things like surveys, task times, error rates, or attitudinal measures like satisfaction rating. One element in analyzing this type of data is examining causation between variables. Causation is when a change in one variable can be shown to cause a change in another variable. There are three main criteria for examining causation, and two additional principles that when studied, can increase confidence in the causation hypothesis. First, there must be a correlation between two variables – when one increases or decreases, the other does the same. Second, the changes in variables must happen in succession – one must occur after the other. Thirdly, the interference of an additional variable must be ruled out. It is also helpful to identify a possible process, or mechanism, for one variable to act on the other. Lastly, understanding the larger context around the changes in variables is important, as all causation is due to multiple interrelated peripheral factors (Chambliss & Schutt, 2006). Another way to find meaning from quantitative data is to look for common patterns of user behavior.
For instance, a concentration of data points (known as a ‘cluster’) or multiple clusters could “represent something as simple as the task completion times on two different versions of a design” (Baty, 2009, p.16). Conversely, a gap where no data points appear might represent an audience that hasn’t been reached. A pathway is a sequential pattern of data, and represents how users travel from point to point. This sort of pattern leads to insights around user journeys or flows (Baty, 2009).
Making sense of user research data is a process of analysis and synthesis: breaking down broad goals into micro-level observations which when reinterpreted, have new meaning. Both qualitative and quantitative data types tell a story, and the different approaches uncover the truth of a situation. But, reaching insights is not enough; sharing those insights with a client or project team is the final step. Giving a voice to the users’ experience, good or bad, is the best way to advocate for greater investment in human-centered design.
Baty, S. (2009, February 23). Patterns in UX Research. Retrieved from: https://www.uxmatters.com/mt/archives/2009/02/patterns-in-ux-research.php
Chambliss, D.F., & Schutt, R.K. (2006). Making Sense of the Social World: Methods of investigation (2nd ed.). Thousand Oaks, CA: Pine Forge Press, an imprint of SAGE Publications, Inc.
Cresswell, J.W., & Cresswell, J.D. (2018). Research Design: Qualitative, quantitative, and mixed methods approaches (5th ed.). Thousand Oaks, CA: SAGE Publications, Inc.
Goodwin, K. (2009). Designing for the Digital Age. Indianapolis, IN: Wiley Publishing, Inc.
Mortensen, D. (2019, June 13). How to Do a Thematic Analysis of User Interviews. Retrieved from: https://www.interaction-design.org/literature/article/how-to-do-a-thematic-analysis-of-user-interviews