ELearning/Foundations/Quality requirements



Introduction

Defining quality is a process of describing the qualitative and quantitative criteria that together make up the exemplary whole. While quantitative criteria can be objectively measured, qualitative criteria must be judged, most likely by a human. Informed judgment comes from understanding the requirements and spirit of the criteria, finding and citing examples as evidence, and logically explaining the relationship between the criteria and evidence. More on this later.

There are many factors that affect the quality of a course, some over which the designer can exert little control. The list reminds us that education and training do not take place in isolation, and that learner success is dependent on much more than course design.

• Course Design - Involves the forethought and planning that goes into a course.
• Course Delivery - The way that a course is taught, also known as instructor performance. Delivery may be automated, in which case is subsumed under course deign.
• Course Content - All of the subject matter and its rigor.
• Learning Management System - The platform by which the course is delivered, its functionality, and the technical support that enables its use.
• Supporting Infrastructure – Organizational functions that enable student support beyond course-based instruction. Examples include enrollment, fee collection, financial support, advising, counseling, technical support, and tutoring.
• Instructor Readiness – Instructor training and planning to teach online.
• Learner Readiness - Learner training and planning to learn online.

Our primary focus here is on course design. We will look at a number of approaches to assessing design quality, progressing from general formulations to more specific ones. As you read through each one, contemplate the qualitative and quantitative elements and how easy or difficult it will be to arrive at conclusions.

e³ approach

David Merrill (2013) defines quality instruction as effective, efficient, and engaging (e³). A course is effective if learners, after completing instruction, possess the level of knowledge and skill they previously did not. It is efficient if learners can complete tasks and solve problems in less time than prior to instruction, and/or if learning using a particular strategy requires less time than other strategies. Finally, a course is engaging if learners are motivated to complete the course, are persistent in performing tasks and using problem solving methods they have learned, and seek additional opportunities to pursue similar instruction. Merrill's approach is well-suited to workplace training designs.

His prominent contribution to the field is his First Principles of Instruction, which form the basis of his evaluation rubric.

• Activation principle: Learning is promoted when learners recall relevant prior experience as a structure for organizing new knowledge, and when this structure is the basis for guidance during demonstration and the basis for coaching during application.

• Demonstration principle: Learning is promoted when learners observe a demonstration of the skills to be learned that is consistent with the type of content being taught.
  • Demonstrations are enhanced when learners are guided to relate general information or an organizing structure to specific instances.
  • Demonstrations are enhanced when effective multimedia principles are implemented.

• Application principle: Learning is promoted when learners engage in application of their newly acquired knowledge or skill that is consistent with the type of content being taught.
  • Application is effective only when learners receive intrinsic or corrective feedback.
  • Application is enhanced when learners are coached and when this coaching is gradually withdrawn for each subsequent task.

• Integration principle: Integration is enhanced when learners share their new skill for peer-collaboration and peer-critique.

• Problem- or task- centered principle: Learning is promoted when learners are engaged in a problem- or task-centered instructional strategy involving a progression of whole real-world tasks.

Approach to rating

Following his prescriptive approach to instructional design (see Introduction to instructional design models) , the rubric tests for the prescribed elements of the instruction. The steps, in brief:

1. List component skills taught in the course and classify by: kind-of (classification), how-to (procedure), or what happens (conditions and consequences).
2. For each component skill, evaluate the quality of the demonstration(s) included in the instruction.
3. For each component skill, determine the type(s) of application included in the instruction.
4. Conduct the same analysis for a whole problem, if used.

Merrill suggests a professional review, one-on-one trials, and small group trials during formative assessment using prototypes. In trials, he advises recording performance scores, time required for completing the course and for significant subsections, and attitude data using brief learner questionnaires.

Sample standard

How-to demonstration.

• Does the demonstration tell learners the steps and sequence in the procedure?
• Does the demonstration show a specific instance of the task and demonstrate each of the steps required to complete the task?
• Does the demonstration use effective multimedia principles?
• Does the demonstration provide guidance by calling attention to the execution of each step?
• Is the procedure demonstrated in a progression of at least 3 increasingly difficult situations?
• Does guidance during demonstrations show learners how the steps in the procedure relate to an organizing structure?

e³ Evaluation Rubric

Thalheimer Course Review Template

Summary

Will Thalheimer is a learning and performance consultant for business with thirty years of experience. He is widely recognized as a leader in the training and development field. The course review template is based on his research and experience. Last revision 2014. Licensed under Creative Commons.

5 categories, 25 standards (2-9 per category). Designed specifically to evaluate workplace performance-related training courses, with special emphasis on integrating learning into the workplace.

Approach to rating

Each standard and its rationale is briefly described. Most standards are answered yes/no and points assigned, ranging from 1 to 3. The rater is also asked to specify how each standard is achieved. Total 50 points possible divided into 6 ranks.

Rating categories

• Is the content right?
• Are there clear performance-related goals?
• Is measurement effective in enabling course improvement?
• Is the course well designed?
• Is the learning well-integrated into the workplace?

Sample standard

Is the learning well‐integrated into the workplace?

• Are learners’ supervisors prompted to encourage and monitor the learners’ efforts in applying the learning, both before the course and after? Rationale: Learners are more likely to apply what they’ve learned if they have their supervisors support and encouragement.

Thalheimer Course Review Template

California State University Chico

Summary

6 categories, 25 standards (3-5 per category). Comprehensive and easy to use, but with superficial explanation of the standards. Last revision 2009. Licensed under Creative Commons.

Approach to rating

Each standard includes descriptors for baseline, effective, and exemplary execution. The emphasis is upon self-evaluation by the instructor/designer and no numerical scoring is used. No explicit weight or criticality is assigned to the categories or standards.

Rating categories

• Learner support & resources
• Online organization & design
• Instructional design & delivery
• Assessment & evaluation of student learning
• Innovative teaching with technology
• Faculty use of student feedback

Sample standard

Instructional design & delivery category: Course goals.

• Baseline. Course goals are not clearly defined and do not align to learning objectives.
• Effective. Course goals are adequately defined but may not align to learning objectives.
• Exemplary. Course goals are clearly defined and aligned to learning objectives.

Chico State Rubric

Blackboard Exemplary Course Program

Summary

4 categories, 17 standards (3-7 per category). Broad categories with detailed explanation within the categories. Combines detail with ease of administration. Last revision 2014. Copyrighted by Blackboard Inc.

Approach to rating

Each standard includes descriptors for incomplete, promising, accomplished, and exemplary execution. The rubric is used in the annual Blackboard Exemplary Course Program. Courses are submitted for peer review, then a reviewer council, and finally by Exemplary course directors. While no explicit scoring mechanism is included, raters can easily assign point values to each rating level. Also, no explicit weight or criticality is assigned to the categories or standards.

Rating categories

• Course design
• Interaction and collaboration
• Assessment
• Learner support

Sample standard

Interaction and collaboration: Development of learning community.

• Exemplary. Communication activities are designed to help build a sense of community among learners. Student-to-student interactions are required as part of the course. Students are encouraged to initiate communication with the instructor. Collaboration activities (if included) reinforce course content and learning outcomes, while building workplace-useful skills such as teamwork, cooperation, negotiation, and consensus-building.
• Accomplished. Communication activities may help learners build a sense of community, but do not appear to be designed with this in mind. Some student-to-student interaction is built into the course. Students interact with the instructor, although primarily as a result of instructor-initiated contact. Collaboration activities (if included) support some team-building skills, but may not purposefully integrate these elements.
• Promising. Effort has been devoted to fostering a sense of community in the course,but only minimally. More focus is needed on designing activities and a course climate that foster student-to-student interactions as well as student-to-instructor interactions.
• Incomplete. Little to no attention has been devoted to building a sense of community in this course.

Blackboard Examplary Course Rubric

Quality Matters Program

Summary

8 categories, 41 standards (4-8 per category). Comprehensive, extensive depth of explanation, extensive time commitment for each course review. A peer review model with three reviewers per course. Last revision 2014. Copyright Quality Matters, Use is restricted to member institutions. We look at the higher education rubric; QM also provides rubrics for K-12 education, publishing, and continuing professional development.

Approach to rating

Ratings for each standard are restricted to “met” or “not met”. Standards are assigned point values of 1, 2, or 3 to indicate their relative weight, with a value of 3 indicating a critical standard. Critical standards must be “met” for the course to be considered passing. Reviewers are provided an annotated guide for decision making. Additionally, raters are instructed to use the 85% rule. Individual standards must be met to an 85% confidence level and the overall score must be 85% or better.

Rating categories

• Course overview and introduction
• Learning objectives (competencies)
• Assessment and measurement
• Instructional materials
• Learner interaction and engagement
• Learner supporting
• Accessibility

Sample standard

Assessment and measurement Standard 3.3. Specific and descriptive criteria are provided for the evaluation of students’ work and participation and are tied to the course grading policy.

Reviewer annotations. Students are provided with a clear and meaningful description of the criteria that will be used to evaluate their work and participation in the course. These criteria are stated up-front at the beginning of the course. The description and/or statement of criteria provide students with clear guidance on the instructor’s expectations and on the required components of coursework and participation. The criteria give students the information they need to understand how a grade on an assignment or activity will be calculated. As a reviewer, you will ascertain that the criteria used to evaluate students’ performance align with the course objectives and contribute to students’ future growth and improvement. Note, however, that as a reviewer you are not asked to look for and evaluate the instructor’s specific feedback to students in Standard 3.3. Your focus is the design of the course, not the delivery of the course. Examples and non-examples are also provided.

2011-2013 Quality Matters Rubric

Hybrid Approach

We have created a hybrid approach to course review that is adapted in large measure from the Blackboard Exemplary Course Review Program (with permission), with additional inputs from the Quality Matters and Chico rubrics, research literature, and university faculty. It was designed for use with LMS-based courses. A tier system helps identify and rank the need for upgrading.

Summary

4 categories, 11 subcategories, and 43 standards (3-7 per subcategory). Comprehensive, with a tiered rating system. Last revision 2012.

Approach to rating

Ratings begin at the individual standard level, which must be rated as “met”, “not met”, or in certain instances “not applicable”. Ratings of incomplete, promising, accomplished, or exemplary are then assigned for each subcategory. Finally a summary tier rating of 1-4 is assigned according to the number of met criteria. Note that the final rating comes directly from the standard ratings. Item criticality is not differentiated, but the tier ratings are more stringent than the Quality Matters rubric. The subcategory ratings are used to summarize course strengths and areas for improvement.

Rating categories (and subcategories)

• Course design (goals and objectives, content acquisition, learner engagement, tool use)
• Interaction & Collaboration (communication strategies, development of learning community)
• Evaluation of learning (learning evaluation, evaluation design)
• Learner support (course and LMS, technical issues)

Sample standard

Interaction & collaboration: Communication strategies Standard 5. Criteria for course participation are provided indicating how interaction will be assessed (e.g., what constitutes a “good” versus “poor” discussion posting)

eLearning Wiki course rubric

Check for understanding: Create a table with each course evaluation approach describe above as rows. Add columns for rating categories (and subcategories if appropriate), rating method, type of weighting if any, and descriptiveness of standards). Here’s a sample. Rating Categories (and subcategories) Rating Method Weighting Descriptiveness Ease of Use e3 Course Review Template/Thalheimer Chico State Blackboard Quality Matters Hybrid Descriptiveness refers to how well the standards provide specific guidance for determining whether a standard is more or less useful for raters. Before completing the Descriptiveness and Ease of Use cells, determine the criteria you will use to rate or categorize the standards. Once you complete the table, compare the approaches and decide on the most useful overall. Consider the use of each for objectivity, thoroughness, ease of use, and any other criteria you identify. Explain your conclusion and describe any improvements you would make to your chosen approach.

---

#top

⇑ Up to Foundations

⇑ ⇑ Up to Home