How to interpret surveys in medical research: A practical approach
ABSTRACTSurveys are being used increasingly in health-care research to answer questions that may be difficult to answer using other methods. While surveys depend on data that may be influenced by self-report bias, they can be powerful tools as physicians seek to enhance the quality of care delivered or the health care systems they work in. The purpose of this article is to provide readers with a basic framework for understanding survey research, with a goal of creating well-informed consumers. The importance of validation, including pretesting surveys before launch, will be discussed. Highlights from published surveys are offered as supplementary material.
KEY POINTS
- Most survey reports do not adequately describe their methods.
- Surveys that rely on participants’ self-reports of behaviors, attitudes, beliefs, or actions are indirect measures and are susceptible to self-report and social-desirability biases.
- Informed readers need to consider a survey’s authorship, objective, validation, items, response choices, sampling representativeness, response rate, generalizability, and scope of the conclusions.
Surveys are common in medical research. Although survey research may be subject to inherent self-report bias, surveys have a great impact on policies and practices in medicine, often forming the basis for recommendations or new guidelines.1,2 To interpret and use survey research results, clinicians should be familiar with key elements involved in the creation and validation of surveys.
The purpose of this article is to provide readers with a basic framework for evaluating surveys to allow them to be more informed as consumers of survey research.
IMPORTANT TOOLS IN MEDICAL RESEARCH
Surveys are important tools for answering questions on topics that are difficult to assess using other methods.3 They allow us to gather data systematically from subjects by asking questions, in order to make inferences about a larger population.3,4 Clinicians use surveys to explore the opinions, beliefs, and perceptions of a group, or to investigate physician practice patterns and adherence to clinical guidelines. They may also use surveys to better understand why patients are not engaging in recommended behavioral or lifestyle changes.
Survey methods include interviews (in person, by phone) and questionnaires (paper-and-pencil, e-mailed, online).4
A well-constructed, validated survey can provide powerful data that may influence clinical practice, guide future research development, or drive the development and provision of needed programs and services. Surveys have the potential to transform the ways in which we think about and practice medicine.
READER BEWARE
While survey research in health care appears to have grown exponentially, the quality of reported survey research has not necessarily increased over time.
For consumers of survey research, the adage “reader beware” is apt. Although a considerable number of studies have examined the effects of survey methodology on the validity, reliability, and generalizability of the results,4 medical journals differ in their requirements for reporting survey methods.
In an analysis of 117 articles, Bennett et al3 found that more than 80% did not fully describe the survey development process or pretesting methods. They also found limited guidance and lack of consensus about the best way to report survey research. Of 95 surveys requiring scoring, 66% did not report scoring practices.
Duffett et al5 noted that of 127 critical care medicine surveys, only 36% had been pretested or pilot-tested, and half of all surveys reviewed did not include participant demographics or included only minimal information.
Because journal reporting practices differ, physicians may be unaware of the steps involved in survey construction and validation. Knowledge of these steps is helpful not only in constructing surveys but also in assessing published articles that used survey research.
LIMITATIONS OF SURVEY RESEARCH
Indirect measures of attitudes and behaviors
Surveys that rely on participants’ self-reports of behaviors, attitudes, beliefs, or actions are indirect measures and are susceptible to self-report and social-desirability biases. Participants may overestimate their own expertise or knowledge in self-report surveys. They may wish to reduce embarrassment6 or answer in ways that would make them “look better,”7 resulting in social-desirability bias. These issues need to be mentioned in the limitations section in papers reporting survey research.
Questions and response choices
The data derived from surveys are only as good as the questions that are asked.8 Stone9 noted that questions should be intelligible, unambiguous, and unbiased. If respondents do not comprehend questions as researchers intended, if questionnaire response choices are inadequate, or if questions trigger unintended emotional responses,10–14 researchers may unwittingly introduce error, which will affect the validity of results. Even seemingly objective questions, such as those related to clinical algorithm use, practice patterns, or equipment available to hospital staff, may be interpreted differently by different respondents.
In their eagerness to launch a survey, clinician researchers may not realize that it must be carefully constructed. A focus on question development and validation is critical, as the questions determine the quality of the data derived from the survey.8 Even the position of the question or answer in the survey can affect how participants respond,15 as they may be guided to a response choice by preceding questions.16
WHAT DO YOU NEED TO KNOW ABOUT ASSESSING SURVEY RESEARCH?
What follows are questions and a basic framework that can be used to evaluate published survey research. Recommendations are based on the work of survey scientists,4,7,10,14,15,17,18 survey researchers in medicine and the social sciences, and national standards for test and questionnaire construction and validation (Table 1).4,19,20
Who created the survey? How did they do it?
How the survey was created should be sufficiently described to allow readers to judge the adequacy of instrument development.3–5 It is generally recommended that feedback from multiple sources be solicited during survey creation. Both questionnaire-design experts and subject-matter experts are considered critical in the process.4
What question was the survey designed to answer?
Is the objective of the study articulated in the paper? 3,20 To judge survey research, readers need to know if the survey appears to adequately address the research question or questions and the objectives of the study in terms of methods used.4
