Recently, I reviewed a number of research proposals in which some applied qualitative or mixed (i.e. quantitative and qualitative) methods to answer health questions. I had my “reviewer hat” on as I assessed the proposals for research quality. After 3-4 proposals, it occurred to me that while assessing quantitative research proposals for research quality and reproducible practices was straightforward, doing the same for qualitative research proposals was trickier. There did not seem to be a clear way to assess research quality of qualitative research.

By definition, qualitative researchers study phenomena that can’t be counted. Some examples include the credibility (believability) of a physical intervention, the usability or acceptability of an information flyer, or patients’ impressions of a therapy. The personal and subjective nature of qualitative outcomes seems to imply they can’t be reproduced (e.g. my subjective impressions can’t be reproduced by you because we are different people). In a sense, how could qualitative research ever “not be good quality”? It made me wonder whether qualitative research seemed to have escaped the “reproducibility crisis” and scrutiny which quantitative research has been subjected to in the last few years.

At the same time, my colleague Dr Darren Reed was marking a thesis that applied mixed methods and wondered the same thing out loud: how is research quality assessed when outcomes are subjective? That was the second trigger. I had to look into this.

First, the Equator Network recommends using the Consolidated criteria for reporting qualitative research (COREQ) checklist when reporting on interviews and focus groups studies. Equator Network reporting guidelines are endorsed by many key medical journals, so it is the first place to check if a reporting guideline for a study design exists at all. The COREQ is a 32-item checklist that ensures authors report on characteristics of the qualitative study such as investigator relationship with participants, theoretical framework, participant selection, data collection and analysis, and reporting. This is good. But, remember that reporting guidelines are designed to ensure consistent and transparent reporting of key design features. They are not designed to instruct investigators on conducting good research. This leads to the second point.

Economics researchers Aguinis and Solarino performed a literature search to develop transparency criteria for qualitative research. They developed 12 transparency criteria covering research design, measurement, data analysis and data disclosure. The criteria are presented in Table 1 of the paper, and summarised below:

Transparency criteria	Definition
1. Kind of qualitative method	The particular qualitative methodology used in the study. (e.g. action research, case study, grounded theory)
2. Research setting	The physical, social and cultural milieu of the study. (e.g. firm conditions, industry, participants’ social status)
3. Position of researcher along the insider-outsider continuum	The researcher’s relationship with the organization and study participants; the closer the relationship, the more the researcher is an insider rather than an outsider
4. Sampling procedures	The procedures used to select participants or cases for the study (e.g. convenience, purposive, theoretical)
5. Relative importance of the participants/cases	The study’s sample and the relative importance of each participant or case
6. Documenting interactions with participants	The documentation and transcription of the interviews and all other forms of observations
7. Saturation point	It occurs when there are no new insights of themes in the process of collecting data and drawing conclusions
8. Unexpected opportunities, challenges and other events	Unexpected opportunities (e.g. access to additional sources of data), challenges (e.g. a firm’s unit declines to participate in the last data collection stage and is replaced by a different one), and events (e.g. internal and external changes such as a new CEO or changes in market conditions during the study) that occur during all stages of the research process
9. Management of power imbalance	The differential exercise of control, authority, or influence during the research process
10. Data coding and first-order codes	The process through which data are categorized to facilitate subsequent analysis (e.g. structural, descriptive or narrative coding)
11. Data analysis and second- and higher-order codes	The classification and interpretation of linguistic or visual material to make statements about implicit and explicit dimensions and structures, and is generally done by identifying key relationships that tie the first order codes together into a narrative or sequence (e.g. pattern, focused or axial coding)
12. Data disclosure	Raw material includes any information collected by the researcher before any manipulation (i.e. analysis) (e.g. transcripts, video recordings)

Even a quick read shows that some of these criteria are not all that different from reproduciblity criteria in quantitative research (e.g. data disclosure). And some could even be used to make the progress of a quantitative research study transparent (e.g. unexpected opportunities, challenges and other events). Marty once commented how difficult/impossible it is to determine from the published report how a study develops and is conducted over time. For example, how did investigators pilot test an experimental setup and protocol, discover some hypotheses are testable but others can’t be tested with the current methods, decide on which hypotheses to keep or pass, and decide on key comparisons. That is, it is not always easy to work out how a study evolves in practice and what goes through the minds of investigators, just from the published paper alone.

Aguinis and Solarino used these criteria to assess published papers in strategic management, and made recommendations on how they could be implemented. We’ll take a look at these recommendations in the next post.

Summary

It is important to ensure that qualitative research is itself of good quality. The criteria to ensure high quality research deserve further consideration to ensure robust research practices in these fields.

Reference

Aguinis H and Solarino AM (2017) Transparency and replicability in qualitative research: The case of interviews with elite informants. Strat Mgmt J 40:1291-1315.