Final Redesign - Evaluation


Evaluation, the process of determining a system's effectiveness, is typically split into three categories:
  • formative evaluation: the evaluation of the instruction performed while the instruction is being formed
  • summative evaluation: the evaluation of the instruction performed after the instruction has been implemented
  • learner evaluation: determining the performance change of the learners due to the instruction implemented

Formative Evaluation:
Formative evaluation is used throughout the design process to test a design and make revisions and refinements. The design and evaluation process should be recursive - each revised instruction should be evaluated.

Formative Evaluation consists of several stages:
  • design review - checks whether the instruction that is designed meets the analysis
  • expert review - determines if the content is accurate and consistent
  • subjects testing - tests the instruction with various groups (one-on-one, small groups, and field trials)
  • ongoing evaluation - continually examines the design with respect to possible change in content, audience, or size

The redesign that has taken place for this project is a great example of formative evaluation. The analysis stage determined that there was a gap between the desired outcomes and the learning that was taking place. I served as the content expert and determined that while the content was accurate, it was not consistent and the difficulty level was varied from beginner to advanced. Since this module had been already been taught as a part of an ongoing course, I was able to see it with actual subjects. I was also able to see how well the learning transfered when I saw these same students again in higher level courses.

Before the re-design it used in MUSC 365 next spring semester, I hope to test it on students who have recently completed MUSC 365. They will be able to view the new presentation, listen to the audio examples, and then fill out a survey. Their responses can be used to further refine the instruction.

Summative Evaluation:
Summative Evaluation is carried out after the design has been implemented. It seeks to determine if the instruction is effective. In short, have the goals and objectives been met?

While I or an outside party can analyze the materials and lecture content, the true test of the instruction will be how well it the material is understood by the learners. After utilizing the materials in the actual course, I can ask the students to complete a survey about its effectiveness. I will also have their exam performance to look at.

Several things can be used to complete the summative evaluation:
  • the results of the survey
  • reaction and interaction of the students in class
  • the learners' performance on the final exam
  • the learners' comprehension as demonstrated in later courses

Learner Evaluation:

Evaluation of the learners is the ultimate test of an instructional design and it's presentation. Learner evaluation can take many forms, including written exams, hands-on projects, research papers, etc. The material covered in this module on the basics of digital audio is tested on final exam for the course. Typically two to three questions test the student's recall and understanding.

The students also complete several recording projects, gaining hands-on experience. These projects are designed to setup the students through using the concepts of the course in a real-world situation. They often work in groups, providing support for each other.

I look forward to utilizing this re-designed module next spring!