Remember that . . .

Formative Evaluation is an ongoing process designed for use at each phase of the ADDIE Model. Formative evaluation directs the project and allows for ongoing improvement and adjustment. Developers collect data and information at each stage of development to improve the effectiveness of the product. Summative Evaluation evaluates the product in its final form and is conducted to determine whether or not the learning objectives have been met.

Product format: Text with CD.

Target Learners: Middle School Teachers in Idaho

Process: Four Hour hands-on Training for middle school social science teacher in Idaho.

Learner Objective: Given the Oregon Trail CD, Teacher’s Guide and four hours of Training, the learner will be able to navigate the Oregon Trail CD and demonstrate it to middle school students.

A pre-test and a post-test will be given to be used in both formative and summative evaluation. A student attitude questionnaire will also be given.

There are three stages of formative evaluation, however, prior to the implementation of these stages, the Oregon Trail CD and Teacher’s Guide will be reviewed by both a Subject-Matter and Learner Specialist to ensure accuracy in both areas.

Stage 1: One-to-One Evaluation
The evaluator will work individually with three teachers, two with average ability and one with below average ability. The evaluator will gather information to see how the learner reacts to the instruction and interacts with computer technology. The following elements will be evaluated by gathering descriptive data in the following areas:

Clarity: Is the information presented on the Oregon Trail CD and Teacher’s Guide clear to the target audience?

• Message: How clear is the message?
• Links: How clear are the examples, illustrations, and demonstrations?
• Procedures: How clear is the sequencing, segmentations, and transitions?

Impact: How does the instruction on the Oregon Trail CD and Teacher’s Guide impact the learner?

• Is the information personally relevant?
• Can the information, quizes and problems be accomplished?
• Is the instruction interesting?

Feasibility: How is the instruction in the Oregon Trail CD and Teacher’s Guide managed?

• How will the independence and motivation of the learner impact the amount of time needed to complete the instruction?
• Is the equipment necessary for the instruction easily operated?
• Is the learner comfortable with the computing environment?
• Is the cost of delivering instruction via the CD reasonable?

Stage 2: Small Group Evaluation
Stage 2 evaluation expands on Stage 1. Small group evaluation determines if changes made in the first stage were effective and whether or not the instruction is effective without the aid of a monitor. Fifteen teachers, representative of the target population, will be selected. In contrast to One-on-One evaluation, the evaluator will present the Oregon Trail CD and Teacher’s Guide; explain that the materials are in the developmental stage, allow the learners to forge through the instruction as if in its final form, and request feedback. Quantitative data will be added to the descriptive data collected in Stage 1. A student attitude survey will be given. The attitude survey will take into account, ease and familiarity of equipment, instructor competency and helpfulness, interaction with instructional module as well as teacher and other learners, and the learning environment.

Stage 3: Field Trial
The final stage of formative evaluation encourages the close simulation of the actual learning environment and instructional materials. At this juncture, the Oregon Trail CD and Teacher’s Guide have been revised and recommendations from the previous two stages have been incorporated.

Use of well-designed pre and post tests to determine knowledge gain is recommended. The evaluation tool must:

• Accurately reflect the instructional goals to ensure content validity.
• Be based on multiple observers to ensure reliability.
• Be consistent among different classrooms, teachers, and rotations

• Test for criterion-related referenced items
• Provide means for anonymous feedback as well as face-to-face
• The evaluator will discuss, one-on-one, with the learner particular responses and reactions of the questionnaire and pre-test, post test as well as the above mentioned questions to search for mistakes and reasons for mistakes.
• The various evaluation instruments will be evaluated for clarity, reliability, and accuracy.
• Amount of time needed to complete the instruction will be calculated
• A Computer/learner interaction check list for consistency, matching of instructional goals, and coding errors will be used.