Cognitive Interview Testing, A More Rigorous Method than Pilot Testing

Traditionally, researchers use pilot testing of instruments with a small sample similar to the study participants to assure content validity. However, this method is not as rigorous as other methods such as cognitive interviews. Cognitive interviews are used in pre-testing questionnaires to detect items not understood by respondents, to assess the conceptual adequacy of the instrument, and to assess cultural fitness of the questionnaire.

In crafting an instrument to assess basic and continuing education learning needs of bedside nurses in Lebanon, cognitive interviews were used to improve questionnaire design and inform revisions by identifying sources of response errors in the study questionnaire.

The purpose of this paper is to illustrate the use of cognitive interviews in ensuring the quality and accuracy of the survey instrument.

Method: prior to conducting the cognitive interviews, the questionnaire was developed in the English language based on literature review and the objectives of the study guided by Morrel-Samuels’ guidelines for designing surveys; ; it was formed of 60 items/questions. Next, a panel of experts scrutinized the questionnaire from theoretical and practice perspectives. Thirdly, the English version questionnaire was translated into Arabic and French by a nursing director and a nursing faculty member respectively who are fluent in English, French and Arabic. Fourthly, back translation was done by two other members who were blind to the original questionnaire and translations. A fifth step included a panel of nursing faculty members fluent in the three languages do simultaneous translation from English into Arabic and French. Then, the research team collected all the versions and revised the instrument accordingly. Institutional review board approval was granted to conduct the cognitive interview testing.

After that, cognitive interviews were used to test the questionnaire on a sample of bedside nurses who will be excluded from the main study. The sample of nurses included two nurses per language version per district. There are five districts in Lebanon, so the test sample was 30 nurses. The cognitive interview entailed four steps that each participant performed when answering the survey questions: demonstrate understanding of each question and interpret it, show ability to retrieve information accordingly, judge the relevance and accuracy of each question, and respond to the questions. The interviews were conducted by four members of the research team trained in conducting cognitive interviews.

Results: some problems with questionnaire items were identified; necessary changes were made in the survey instruments based on the cognitive interviews. The changes included re-writing 5 items in the best way respondents understand them. Others included re-formatting 3 questions in an easier way for the respondents to answer such as organizing response in a table form rather than on simple lines. Yet other changes included removing or adding responses in 6 questions.

Conclusion: cognitive interviews were used to identify errors and problems in the instrument “assessment of basic and continuing education learning needs of bedside nurses” and to make necessary changes to improve the quality and accuracy of the tool.