Literacy

The predominant means of instruction has traditionally been through verbal medium, either as spoken lecture or written text. As more instructional resources of many different media types become available to students through the Internet, there is a need for educators to understand when these sources may be used effectively for instruction, as well as a need for students to develop an additional set of literacy skills in order to learn from these sources. Although there is much optimism that multimedia sources will be a great tool for instruction, research in cognitive science has demonstrated that the use of these resources does not always lead to better learning. It is important to recognize the potential cognitive implications of multimedia presentations, including text, graphics, video, audio, and virtual reality simulations. Multimedia has been incorporated into instructional materials in a variety of ways: decoration, illustration, explanatory simulation, and "situating" simulation. The first three uses may be best thought of as adjuncts to a verbal lesson, while in the final use, the entire "lesson" is embedded and conveyed by situating the learner in a virtual context.

Uses

Often multimedia is used to decorate text, with the goal of making the text more interesting for the reader. A second use for multimedia, illustration or description, can be used to help a reader visualize a place or time or object. A third use of multimedia involves the explanation or explication of concepts. Especially in complex domains, understanding often requires that learners develop a dynamic mental model of phenomena or processes. Multimedia animations, narrations, and diagrams have all been used to support the understanding of complex subject matter by illustrating or highlighting important relations, thereby attempting to convey a correct mental model directly to the student.

Improved Learning

One reason why multimedia might be expected to lead to improved learning is consistent with a constructivist approach that posits that conditions that make knowledge acquisition more self-directed and active are beneficial for student understanding. The presentation of loosely connected texts and images in hypermedia environments allow learners to navigate information with more flexibility. At the same time, in order to build coherence, students must construct their own elaborations, inferences, and explanations. Thus, there has been reason for optimism surrounding the benefits of learning from hypermedia.

Another reason one might expect benefits from illustrated text and multimedia presentation in general is that it allows for information to be represented in multiple ways (i.e., both verbal and visual). A great deal of previous research within cognitive psychology, such as that of Allan Paivio in 1986, has suggested that the more codes one has for a given memory, the more likely one is to remember that information. Multiple media may also make the learning experience more vivid or distinctive. And, given the different preferences of different learners, multimedia may allow learners to choose the code best suited to their abilities.

A related reason why multimedia, and graphics in particular, may improve learning is that some particular domains may lend themselves to visual presentation, such as when information is inherently spatial. For instance, learning about different ecologies and climate zones may benefit greatly from the presentation of a map. Further, even when the understanding of the subject matter does not require a visual representation, images can still facilitate understanding if the image provides the basis for an abstract model of the content of the text. Figures, graphs, or flow charts that may allow the reader to think about abstract concepts and relations through images support the creation of more complete mental models and as a consequence may improve comprehension of text. Also, when subject matter is as complex and dynamic as streams of data from weather satellites or space stations, then visuals and animations may be especially important. Similarly, visual or audio representations (sonitizations) of complex data can give human thinkers the ability to consider many more dimensions, and the salient relationships between those dimensions, than they might otherwise.

Finally, it should not be overlooked that instructional materials with visual or audio adjuncts are simply more interesting to readers than plain text. Such motivational issues may contribute to advantages in learning with any multimedia presentation.

Criticisms

With all these potential benefits of images, it is perhaps surprising that since the 1960s, the empirical results on learning from illustrated text have been less than positive. In a 1970 review of studies using illustrated texts, Spencer Jay Samuels found little support for the superiority of illustrated text over plain text. In fact, in some cases illustration leads to poorer learning than simple text presentation. Follow-up investigations suggest that one reason for the lack of a consistent positive effect of images on learning is that any learning effect depends greatly on the kind of image that is used. In a 1987 review of Joel Levin and colleagues that discriminated between decorative illustrations, and conceptually-relevant images, decorative illustrations were found to lead to the smallest improvements and sometimes negative effects in learning. Decorative illustrations are often not relevant for the concepts that are described by the text, yet they are still interesting for the reader, and will attract the reader's attention. For this reason, interesting but irrelevant illustrations can be seen as part of a larger class termed seductive details as coined by Ruth Garner and colleagues, and others. Similarly, color, sound effects, and motion are preattentive cues that necessarily attract a reader's focus. If they are not used to emphasize conceptually important information, they too can seduce the reader.

Even when images are relevant for understanding the target concept, there is a further danger that images or animations can make learners overestimate their level of comprehension. People tend to feel that a short glimpse of an image is generally sufficient for understanding. This can lead to an illusion that they understand a graphic or image, even when they have not really engaged in deeper thought about the information. Further, students are notoriously bad at comprehending complex graphics, especially data-related charts or figures, and will interpret the data in support of their own ideas.

Another danger with images and especially animations, is that they can provide so much information so easily that although the reader is able to grasp a basic idea of "how" a dynamic system works, a good understanding of "why" the system works the way it does is lacking (i.e., they are unable to recreate the system or apply their knowledge to a new instance). This effect has in fact been demonstrated in several studies. The research of Mary Hegarty and colleagues, and other research, indicated that still pictures, or still sequences of pictures, in which the reader needs to infer movement for themselves, led to better understanding of dynamic systems than animations that actually show the motions. Similarly, animations that are reproductions of real-world actions are more effective if they are "doctored" to emphasize important features of the display. And, animations that are stoppable and restartable under the learner's control may lead to better learning than real-time simulations. However, images that provide readers with the basis of a mental model, and animations that show the dynamics of a model, may be especially important for people who lack knowledge and spatial ability. Finally, even conceptually relevant adjuncts run the risk of distracting the learner, and they need to be presented in a way that does not compete with the processing of the text. A number of studies, such as the work of Wolfgang Schnotz and Harriet Grzondziel, have shown that students learn better from diagrams and animated graphics when they were presented separately from text. Alternatively, learning from multimedia has been supported by structured computer learning environments, where different media and sources are presented to students, but students are given instruction both in how to use the environment, and are given a specific learning goal.

Other multimedia adjuncts have been studied, most notably narrations and sound effects. The bottom line from these investigations is that narrations only benefit learning when they are nonredundant with text. However, narrations may be especially helpful for poorer readers, especially when they accompany diagrams, and highlight the conceptually important features. Sound effects and music in general are distracting, and as adjuncts to text, they do not contribute to better conceptual learning. They do however help simulated environments seem more authentic, and may be helpful for situated learning and anchored instruction. Realistic sound effects may be especially important in skill learning environments. Similarly, in terms of conceptual learning, animations may help only when readers cannot generate mental models on their own, although realistic video may help when learning a procedural task and also in "situating" contexts.

The Simulation of Reality

The final use of multimedia considered here is where multimedia is used, not as an adjunct to verbal instruction, but more extensively as the entire means of presentation. In these lines of application, multimedia is used to simulate reality, through video and audio streams, to produce a sense of learning "in context." This may be especially important in skill-training environments, when learning in an actual cockpit or surgical operating room would be unsafe and costly. In more academic domains, simulations can give students the feel of an authentic experience, and both situated learning and anchored learning approaches have attempted to capitalize on this advantage of multimedia presentation. Another application of multimedia simulation is the creation of artificial agents that can act as tutors or peers. The presence of an interactive human-like entity may be an especially important coaching tool, and multimedia simulation may make such tutoring experiences more effective than feedback or prompts that appear in text messages. Simulations may also be used to support distance education and collaboration, again by providing a sense of real "co-presence" to the users.

Virtual reality is the ultimate multimedia tool, combining realistic video and audio streams (i.e., three-dimensional), and sometimes even tactile experience. Here, the potential exists to convey an understanding of new concepts in ways that surpass real experience, and many have heralded virtual reality as a powerful educational tool. Most researchers refer to the multisensory-based sense of "presence" that virtual reality affords the user as the characteristic that separates it from other training approaches. Where procedural knowledge and visuo-spatial skills are concerned there seems to be support for this optimism. However, results on more academic subject matter understanding have been less convincing.

Most studies that have been performed on people's uses of virtual reality have included only self-report data that reflect the user's interpretation of her or his experience in the virtual environment, while fewer investigations include more direct measures of learning. Among the few virtual reality studies with learning measures, Chris Dede and colleagues examined in 1999 students' understanding of electromagnetic field concepts from a virtual environment called Maxwell World. Students in the virtual reality condition were better able to define concepts and demonstrate them in three-dimensional terms.

At the same time, pilot studies, such as those of Andrew Johnson and colleagues in 1999, on using virtual reality to promote understanding of astronomical physics concepts have found that the virtual reality environment can also be prone to seductive distractions. For example, to learn that the earth's shape is round, younger elementary school-age children engaged in a virtual reality game, which included traversing a spherical asteroid to gather objects. The students failed to exhibit a substantial and robust improvement in their understanding, presumably because of the distracting and nonrelevant aspects of the game.

It would seem that virtual reality would be a prime candidate to demonstrate the positive effects of multimedia sources on learning. Yet, virtual reality experiences are not easily translated into learning experiences, and the results of studies on the educational uses of virtual reality underscore the same principles as have been discussed above. Virtual reality may add value to educational contexts when real training is not possible, and where it goes beyond "realistic" experiences in ways that emphasize conceptual understanding.

As an Effective Strategy

As the nature of instructional materials changes to include more images, sounds, animations and simulations, it is important to recognize the conditions under which multimedia can be an effective learning tool, and that new literacy skills are needed to learn from those materials. Instead of being presented with a single message, multimedia learners are presented with many information sources on a topic, and those sources can represent a number of media. The sheer number of resources available through the Internet is enormous. The availability of so much information means that students have the ability to direct their own learning, by performing searches and selecting documents, evaluating sources of information, and allocating attention to images and animations, without being confined by the linear structure of a single text or lecture. Although this flexibility in the learning environment has been seen as an opportunity for more active student learning, it is clear that in order to learn from multimedia and electronic sources, students will likely need additional skills in searching, document evaluation, strategic reading, strategic understanding of graphics, and integrating information across sources, including the integration across text and graphics. A specific set of skills may also be necessary for learning in more immersive multimedia environments.

A recurrent question focuses on how multimedia helps learning. Some have suggested that positive effects due to multimedia may be simply due to the motivating effect of its novelty. Unfortunately the literature of the early twenty-first century contains few controlled studies and few tests of when multimedia helps understanding. Further research is needed to identify what conditions enable the best learning from multimedia, and what new literacy skills students will need to support that learning.