The first level of analysis focuses on the research problem. To determine the soundness of a study, look for:
- Clear and correct explanations of the research problem
- Insights into the significance or relevance of the problem, or the purpose and significance of the study. A problem may be genuine, but still may be either insignificant or too obvious to warrant a study
- Theoretical and philosophical base of the study. Consider whether the research problem flows naturally from such a base
- Assumptions underlying the study
- Description of context and location of the study
- Clarity in conceptual definitions
- Justification for hypothesis. Many qualitative studies do not have hypothesis. In such cases, look for justifications in this regard. For studies with a hypothesis, compare the researcher’s effort with the standard convention associated with formulating hypothesis, such as developing a null and alternative hypothesis
- Comprehensive literature review. A sound research project will assess all major studies connected with the topic from primary sources, to undertake a comprehensive review that covers all aspects of the study. The researcher updating knowledge base to incorporate latest trends during the course of the research lends further credibility to the study.
Methodology and Process
The second level of analysis concerns methodology and process. To determine the credibility of a study, consider:
- Research Design: A good design allows for a thorough and in-depth examination of the research problem.
- Participants: A good study explains the basis for selecting targets, the adequacy of the sampling plan, and provides a description of selected targets. Look for a detailed description of the study participants, and a rationale for their selection
- Data Collection Tools: evaluate the adequacy or relevance of the tools applied to collect data, to meet research objectives. Good studies provide justification for selecting the tools, and provide proof for the validity of tools such as questionnaire
- Data Collection Process: a good study explains in detail the methodology used to collect data, the profile of people engaged in data collection, and a detailed overview of the actual process. Derive information on the researcher spending time with the participants, gaining their trust before collecting data, and crosschecking data. Check for consistency in the data collection process
- Data Coding: Good studies provide an overview of the coding process, the methodology and technologies used, and the profile of people undertaking the task
- Data Analysis: Good studies provide a description of the tools used to analyze data, and justify the usage of such tools.
One optional but important dimension when critiquing qualitative analysis is the ethical dimension of the study. Studies grounded in ethical considerations do not put participants into harm, risk, or inconveniences, or violate their rights or privacy. Participants remain aware that they are a part of the study and under observation.
Instances such as researchers subjecting animals to sleep deprivation torture, or prisoners subject to experimentation on their body without their consent rank as examples of unethical dimensions, as does coercing people into participating, not obtaining their consent when making observations or administering drugs, and so on.
Another important set of guidelines for critiquing qualitative analysis concerns the results or the research report. A sound research:
- Makes the findings within the context of the research framework. The findings directly offer a solution to the problem statement and becomes a natural addition to the theoretical base of the study
- Presents findings with clarity, to yield a meaningful picture. Some common methods used to lend clarity and communicate the results effectively include conceptual maps, models, diagrams and other methods
- Provide sufficient detail. The research report displaying data and decision making trial to verify findings, provide evidence to back claims and conclusions, use participant quotes to support findings, and making explicit the rules and conventions adopted when making conclusions. The findings remain free of bias or subjective judgments
- Adopts a simple and lucid writing style. The main elements of such a style are avoidance of excessive jargon, incorporating well structured sentences, paragraphs, and sections with logical flow of thought and order of sequence, application of style features consistently through the report, and including a complete list of reference in correct format.
- Includes strengths and weakness of the study, with specific examples, in an objective manner.
A sound research provides sufficient details that allow a second researcher with a similar background and philosophical approach to use the same set of data, follow the same methodology, and arrive at the same conclusions. Checking for validity of the research corroborate its soundness.
Be balanced in the critique. Do not be overly critical to pick up condonable weaknesses or overly flattering to identify strengths that do not exist. Undertaking a critique of qualitative study is harder comparing to undertaking a critique of quantitative study, as the latter primary requires checking figures for consistency.
- Sonoma State University. “Guidelines for Critiquing Research Articles.” https://www.sonoma.edu/users/n/nolan/n400/critique.htm#qualanal. Retrieved 08 June 2011.
- Rodgers, Beth, L. “Guidelines for Critique of Research Reports.” University of Wisconsin-Milwaukee, School of Nursing. https://pantherfile.uwm.edu/brodg/www/Handout/critique.htm Retrieved June 08, 2011.
Image Credit: flickr.com/aprilzosia