Research Methods

Since the early 1900s, researchers have relied on verbal data to gain insights about thinking and learning. Over the years, however, the perceived value of verbal data for gaining such insights has waxed and waned. In 1912 Edward Titchener, one of the founders of structural psychology, advocated the use of introspection by highly trained self-observers as the only method for revealing certain cognitive processes. At the same time, this technique of observing and verbalizing one's own cognitive processes drew much criticism. Researchers questioned the objectivity of the technique and the extent to which people have knowledge of and access to their cognitive processes. With behaviorism as the dominant perspective for studying learning in the United States, verbal data were treated as behavioral products, not as information that might reveal something about cognitive processing. From about the 1920s to 1950s, most U.S. researchers abandoned the use of introspective techniques, as well as most other types of verbal data such as question answering.

While U.S. learning theorists and researchers were relying almost solely on nonverbal or very limited verbal (e.g., yes/no response) techniques, the Swiss cognitive theorist Jean Piaget was relying primarily on children's verbal explanations for gaining insights into their cognitive abilities and processes. Piaget believed that children's explanations for their responses to various cognitive tasks provided much more information about their thinking than did the task responses alone. United States theorists, however, were not ready to consider Piaget's work seriously until about 1960, when cognitive psychology was beginning to emerge and there was declining satisfaction with a purely behavioral perspective.

With the rise of cognitive psychology beginning in the 1950s and 1960s, educational and experimental psychologists became interested once again in the usefulness of verbal data for providing information about thinking and learning. Cognitive researchers rarely use Titchener's original introspective technique in the early twenty-first century. Since the 1980s, however, researchers have increasingly used verbal protocol analysis, which has roots in the introspective technique, to study the cognitive processes involved in expert task performance, problem solving, text comprehension, science education, second language acquisition, and hypertext navigation.

What Are Verbal Protocols?

Verbal protocols are rich data sources containing individuals' spoken thoughts that are associated with working on a task. While working on a particular task, subjects usually either think aloud as thoughts occur to them or they do so at intervals specified by the researcher. In some studies, researchers ask subjects to verbalize their thoughts upon completion of the task. The verbalizations are recorded verbatim, usually using a tape recorder, and are then coded according to theory-driven and/or empirically driven categories.

Verbal protocols differ from introspection. Subjects are not instructed to focus on the cognitive processes involved in task completion nor are they trained in the self-observation of cognitive processing. The goal is for subjects to express out loud the thoughts that occur to them naturally. Researchers use these data in conjunction with logical theoretical premises to generate hypotheses and to draw conclusions about cognitive processes and products.

What Can Verbal Protocols Reveal about Thinking and Learning?

In order to verbalize one's thoughts, individuals must be aware of those thoughts and the thoughts must be amenable to language. Thus, verbal protocol analysis can reveal those aspects of thinking and learning that are consciously available, or activated in working memory, and that can be encoded verbally.

One major advantage of verbal protocol data is that they provide the richest information regarding the contents of working memory during task execution. In studies of reading comprehension, for example, verbal protocols have provided a detailed database of the types of text-based and knowledge-based inferences that might occur during the normal reading of narrative texts. Data using other measures such as sentence reading time and reaction time to single-word probes have corroborated some of the verbal protocol findings. For example, corroborating evidence for the generation of causal inferences and goal-based explanations exists. Verbal protocols have also provided information about the particular knowledge domains that are used to make inferences when reading narratives, and about differences in readers' deliberate strategies for understanding both narrative and informational texts.

Verbal protocols have been used extensively in the study of expert versus novice task performance across a variety of domains (e.g., cognitive-perceptual expertise involved in chess, perceptual-motor expertise such as in sports, science and mathematical problem-solving strategies, skilled versus less-skilled reading). While the specific insights about the differences between expert and novice approaches vary from domain to domain, some generalities across domains can be made. Clearly, experts have more knowledge and more highly organized knowledge structures within their domains than do novices. But the processes by which they solve problems and accomplish tasks within their domains of expertise also differ. Verbal protocols have revealed that experts are more likely to evaluate and anticipate the ever-changing situations involved with many problems and to plan ahead and reason accordingly. Knowledge about expert and novice problem-solving processes has implications for developing and assessing pedagogical practices.

Another advantage of verbal protocol analysis is that it provides sequential observations over time. As such, it reveals changes that occur in working memory over the course of task execution. This has been useful in studies of reading comprehension where the information presented and the individual's representation of the text change over time, in studies of problem solving where multiple steps are involved in reaching a solution and/or where multiple solutions are possible, in studies of expert versus novice task performance, and in studies of conceptual change.

Limitations of Verbal Protocol Data

As is the case with most research methods, verbal protocols have both advantages and limitations. Obviously, subjects can verbalize only thoughts and processes about which they are consciously aware. Thus, processes that are automatic and executed outside of conscious awareness are not likely to be included in verbal protocols, and other means of assessing such processes must be used. Also, nonverbal knowledge is not likely to be reported.

Most authors of articles examining the think-aloud procedure seem to disagree with the 1993 contention of K. Anders Ericsson and Herbert A. Simon that thinking aloud does not usually affect normal cognitive processing. It is thought that the think-aloud procedure may lead to overestimates and/or underestimates of the knowledge and processes used under normal task conditions. The need to verbalize for the think-aloud task itself might encourage subjects to strategically use knowledge or processes that they might not otherwise use. Alternately, the demands of the think-aloud task might interfere with subjects' abilities to use knowledge and/or processes they might use under normal conditions. Self-presentation issues (e.g., desire to appear smart, embarrassment, introversion/extroversion) might affect subjects' verbal reports. Finally, the pragmatics and social rules associated with the perception of having to communicate one's thoughts to the researcher might also lead to overestimates or underestimates of knowledge and processes typically used.

Unfortunately, it is not possible to know if a verbal protocol provides a complete picture of the knowledge and processes normally used to perform a task. Typically, however, no single research technique provides a complete picture. Only the use of multiple measures for assessing the same hypotheses and for assessing various aspects of task performance can provide the most complete picture possible.

A final limitation of verbal protocol methodology is that it is very labor intensive. The data collection and data coding are extremely time consuming as compared with other methodologies. The amount of potential information that can be acquired about the contents of working memory during task performance, however, is often well worth the time required.

Optimizing the Advantages and Minimizing the Limitations

Several suggestions have been put forth for increasing the likelihood of obtaining verbal protocol data that provide valid information about the contents of working memory under normal task conditions. The most frequent suggestions are as follows:

• Collect verbal protocol data while subjects are performing the task of interest.
• Ask subjects to verbalize all thoughts that occur. One should not direct their thoughts or processing by asking for specific types of information unless one wishes to study the planned, strategic use of that type of information.
• Make it clear to the subjects that task performance is their primary concern and that thinking aloud is secondary. If, however, a subject is silent for a relatively long period as compared to others during task execution, prompts such as "keep talking" may become necessary.
• To minimize as much as possible the conversational aspects of the think-aloud task, the researcher should try to remain out of the subject's view.