We all understand the value of measurement of learnability or learning effectiveness can bring to an organization’s learning strategy and training delivery. In this article, I share a case study that showcases how you can practically evaluate the learnability of online courses.

How To Evaluate The Learnability Of Online Courses: Case Study

Before outlining the case study that will illustrate how you can evaluate the learnability of online courses, I recap a few essential aspects from my previous article How To Predict And Measure The Learnability Of Online Courses.

Why Is Measuring Learnability Or Learning Effectiveness Of Online Courses Vital For Learning And Development Teams?

Learner experience has a direct bearing on the effectiveness of your training initiative and eventually your ROI. If the learnability/learning effectiveness quotient of your online training is high, application of learning and the desired impact are taken care of automatically. Evaluation/measurement of learnability will help you predict the eventual impact and identify ways to better the learning experience.

What Are The Challenges That Learning And Development Teams Face In Determining Learnability Of Their Online Courses (Legacy As Well As New)?

While there are a few measures in place to evaluate learnability, Learning and Development professionals continue to look for an approach that can address the related challenges and help them:

• Analyze the learner experience in their existing courses.

• Gain insights and develop guidelines to enhance learnability in future courses.

• Predict learning effectiveness.

• Evaluate/measure learnability and offer remediation/suggestions to enhance the learning experience.

• Validate the impact on learning experience from users in real time.

• Analyze and identify the gaps.

• Get insights on how to bridge those gaps.

• Make amendments to the existing approach based on the user feedback.

• Bring about the desired level of learnability/learning effectiveness.

What Are The Highlights Of Our Framework That Can Predict And Measure Learnability Of Online Courses?

As I had mentioned in my previous article, at EI Design, we have been working on the creation of a framework that can predict and evaluate the learnability of online courses since Jan'15. This maps very closely to the wish-list you may have to predict, measure, and validate the learnability of your online courses.

In this article, I use the same framework to illustrate exactly how we have used it to:

1. Evaluate and measure the current learnability.
2. Analyze these results and come up with enhancements that helped us improve the learnability.

In my next article, I will use another case study to showcase how we can validate our predictive learnability from users through real time surveys.

How Can You Use Our Framework To Measure And Predict Learnability Of Your Online Courses?

You can use our framework to:

• Evaluate the learnability of your existing courses and identify measures to enhance it.

• Predict the learnability for your upcoming courses and validate this prediction with your learners real-time.

You can also use this framework to:

1. Address the extended learnability challenge. This involves monitoring a group of users over a period of time and evaluating them based on the metrics. You can then use this data to plan future measures to enhance the learning experience.
2. Build a framework based on the cues from the data gathered from the users and the metrics and take concrete steps to improve your ROI.

Case Study: How To Evaluate The Learnability Of Online Courses

Background: The selected course is part of our Suite of 15 online courses for Instructional Designers.

Challenges: The course was developed in 2014 and during the last two years, many aspects have changed (from more wide ranging learning design possibilities to learner profiles that now feature millennials).

We felt that this is a good case study material to see how our learnability evaluation can enhance an existing course which was already delivering the required value.

Objective of the exercise: To transform a traditional eLearning course from an Instructional Design, design, and technology standpoint. Specifically, the exercise was carried out to:

• Create a modern design experience for the learners.

• Demonstrate significant improvement in learnability (learning experience and effectiveness).

Activities undertaken in the exercise: We followed the following steps to evaluate the current learnability and provide recommendations to enhance it. Specifically, we:

• Analyzed the requirement and identified the need to redesign and revamp.

• Did a “Learnability Evaluation” and came up with recommendations.

• Conducted a rapid design workshop to address the recommendations and remediation suggested in the evaluation.

• Created low and high fidelity prototypes to test the designs.

• Developed the course tested against the same “Learnability” parameters.

• Created a set of guidelines to revamp such courses and have a predictive learnability standard.

Outcomes Of The Learnability Evaluation And Recommendation Exercise

The learnability evaluation was done through several metrics notably:

1. Metrics based on task performance.
2. Metrics based on interface usage.
3. Metrics based on content and cognition.
4. Metrics based on feedback on design elements.
5. Metrics based on interface and navigation help.
6. Metrics based on usability.

Here is a screenshot of one of the metrics used for learnability evaluation.

Analysis and recommendations: The recommendations provided in the learnability evaluation helped the learners in getting a significantly better learnability (learning experience and effectiveness) as it provided them with:

• A better learning experience by making them focus on the key content.

• A better way to navigate in the course and know where they are in the course.

• Better ways to interact in the course.

• More controls to learn at their own pace.

• A greater level of intuitive learning through better hierarchy in information presentation and engaging layouts.

View of the enhanced learnability: Here is a video (featuring before and after views) that reflects how the recommendations enhanced the existing learnability of the course and met the goals of the current exercise.

What more is possible: We have tested this framework and methodology with our courses and we have seen tangible gains accrue as a result.

Organizations looking to achieve a similar mandate have a lot to gain from our framework. Do contact me if you would like to sign up to measure and predict learnability of your online courses.

This article is one of the three articles on how to predict, measure, and validate the learnability of online courses.

Do refer to my previous article How To Predict And Measure The Learnability Of Online Courses for an overview of our learnability framework.