A Framework to Examine the Effectiveness of Computerized Training

Two key elements must be taken into account when considering the effectiveness of technology-mediated training, namely instructional effectiveness; and instructional efficiency. Instructional effectiveness and efficiency are two elusive terms for which no accurate definitions can readily be found in the literature. The difficulty in defining these terms is probably due to the number of factors extraneous to the material itself, which confounds measurement related to the quality of instruction.

In previous studies the efficiency and effectiveness of an instructional product have been used as dependent variables. Nathenson and Henderson (1980) note that research has had a very narrow focus with regards to the effectiveness of instructional materials. In many studies effectiveness has been viewed only in terms of learning gains on post-tests. The authors argued that although improved student performance is an important element, it should not be the only indicator of instructional material effectiveness (Nathenson & Henderson, 1980). Chinien (1990) suggests that instructional material effectiveness should be viewed within a framework, which encapsulates three major elements: achievement, study time, and the students’ attitude toward the material (Chinien, 1990).

Achievement 

Several studies (see Chinien & Boutin, 1994) have demonstrated that the quality of instructional material can help to significantly improve students’ achievement on post-tests. Two indicators of instructional material effectiveness are used with respect to achievement. The first relates to the ability of the material to help a predetermined percentage of students reach a designated level of mastery on post-tests.  The gain in learning is a second indicator of effectiveness related to achievement. Learning gain is usually expressed as the difference between post-test and pretest scores (learning gain equals post-test score minus pre-test score, (Romiszowski, 1986).

Study Time

The amount of time that students spend interacting with an instructional product is another critical element of instructional material effectiveness. Nathenson and Henderson (1980) cite many research studies that have reported achievement at the expense of increased study time. These authors quote (Faw, H. W.; Waller, 1976) to emphasize the critical relationship between study time and the achievement component of instructional material effectiveness: “ [Since] the single most important determinant of how much is learned is probably total study time…it is hardly surprising that the manipulation which tend to extend the period of time spent in study…are in general accompanied by superior levels of learning.” There are also some studies demonstrating improved student performances on post-tests while keeping study time constant. Study time is also commonly referred to as a measure of efficiency (Davis, R. H.; Alexander, L. T.; Yelon, 1974) , (Futrell, H. K.; Geisert, 1984).

Attitude 

A third dimension of instructional material effectiveness is the student’s attitude toward the material. Studies conducted by (Abedor, 1972), (Stolovitch, 1975), and (Wager, 1980) indicate that effective instructional materials generate more positive student attitudes. On the other hand, (Berthelot, 1978) and Chinien (1990) found no significant differences in students’ attitude related to the quality of instructional material. Romiszowski (1986) cautioned that the novelty effects may confound measures of students’ attitude. He argues that the novelty may not only inspire negative attitudes that diminish over time, but may also generate excessive praise and enthusiasm that may also disappear. Although research on time-on-task indicates that a positive correlation between achievement and time engaged in learning tasks, time is not generally used as an independent variable in research on distance education.

The effectiveness of instructional material can be conceptualized within a framework of three major elements: student achievement, study time, and student attitude. All three elements are important and need to be considered collectively when assessing instructional material.  Nesbit and his colleagues have developed a useful instrument for evaluating e-learning objects, which can be adapted and used for Neurogenesis. This instrument comprises nine key elements as described below (Nesbit, J.; Belfer, K.; Leacock, n.d.):  

Table 11. Effectiveness of E-learning Objects

Elements Description of elements
Content quality Veracity, accuracy, balanced presentation of ideas and appropriate level of detail
Learning goal alignment Alignment among learning goals, activities, assessments, and learner characteristics
Feedback adaptation Adaptive content or feedback driven by differential learner input or learner modeling
Motivation  Ability to motivate and interest an identified population of learners
Presentation design Design and auditory information for enhanced learning and efficient mental processing
Interaction usability Ease of navigation, predictability of the user interface, and the quality of the interface help features
Accessibility Design of controls and presentations format to accommodate disabled and mobile learners
Reusability Ability to use in varying learning contexts and with learners from different backgrounds
Standards compliance  Adherence to international standards and specifications

 

Series NavigationElements of Program Quality >>

Submit a Comment

Your email address will not be published. Required fields are marked *