TAADES501A Design and develop learning strategies

Stage 4 Learning Strategy Process: Improve

Figure 10 Improve-Stage 4 of the learning strategy process

Evaluation of learning strategies should verify the total contribution that the strategy make to improvement as determined by the measurement of individual competency development and capacity to undertake future learning.

The evaluation process occurs throughout the learning strategy implementation process. It determines the value-adding effectiveness and efficiency of a strategy and program. The Improvement stage delivers the evaluation, review and valid insights necessary for a community, an organisation or the individuals involved (learners, facilitators/teachers, supervisors, designers, etc.) to determine whether their investment not only has been engaging, but also has generated the desired result. Reporting and recording attainment of capabilities to fill a known 'skill' deficiency is one approach to mapping and reporting that the investment in learning has resulted in tangible improvement. However, the larger challenge for the instructional designer, learning manager or learning provider is to demonstrate that this value-add has occurred within an efficient and effective learning strategy that can continuously improve.

Proficiency at the Improvement stage means locking in enhancements to the overall learning strategy process and generating reliable data that in turn can improve the Analyse, Build and Implement stages.

Major actions in Stage 4 Improve will include:

Collect formative and summative evaluation data;
Apply metrics;
Evaluate tools and materials and the learning process(es);
Report and suggest improvements.

Step 4.1. Collect formative and summative evaluation data

Kirkpatrick's model for Summative Evaluation-and its four levels-has become a classic reference:

Level 1: Reaction
Level 2: Learning
Level 3: Behaviour
Level 4: Results

Level 1: Students' Reaction

In this first level, students are asked to evaluate the training after completing the program. The aim is to assess the learner's satisfaction with the session, or program, of learning. While often reduced to simplistic surveys (so-called happy sheets) completed at the end of a session, this type of evaluation can survey individual responses on matters that can be compared across the student population. This survey or questionnaire could for instance investigate:

The degree to which the learner felt in control of the learning process;
Access and ease of progress through the course (modularised, timeliness, etc.);
The relevance and portability of learning to life and/or work;
The ease of navigation;
Satisfaction with the learning experience; and
Level of personalisation in both content and assessment.

Level 2: Learning Results

This level measures what participants have learnt from a learning program. It seeks to quantify the extent to which participants' knowledge, skills and other attributes have changed. The results emphasised by Kirkpatrick refer to the learning outcomes from structured modules, programs or courses. This form of evaluation may involve comparative test scores before and after a course, or across different classes and demographic groups. This type of evaluation is not as widely conducted as Level 1, but is still very common.

Level 3: Behaviour in the Workplace

Competency retention is an important attribute for the knowledge-based worker. Learners may score well on post-tests, but it is more important to determine whether they can transfer the new knowledge, skills and other attributes to immediate and longer-term job application (Horton, 2000). Level 3 involves follow-up evaluation over longer periods than Level 2 after the learning programs (training) has been completed. Over three to twelve months after completion of learning, evaluation attempts to answer whether students' behaviours actually change as a result of new learning.

This data is harder to collect and analyse but has real value in terms of planning future elearning strategies and investment. Surveys or behavioural analysis assessments may be reviewed to show how learning outcomes change behaviours in ways that can be evidenced through customer and business outcomes. This evaluation evidence may include how service training has impacted customer satisfaction or behaviour, or how training for government compliance has resulted in overall compliance of service with government requirements.

Level 4: Business Results

The fourth level seeks to evaluate the business impact that results from learning. The results may be organisation wide, or across populations of learners (i.e. an occupational group, team or business unit, or functional unit. This level requires evaluation metrics to move beyond the individual learner and examine the contribution in learning to the organisation sponsoring the investment.

Various means of evaluating learning could be applied to elearning. While Kirkpatrick's is one approach, it does have weaknesses. Direct and implicit criticisms include:

It takes too long to get to Level 4 evaluation, which is the most important one for managers making investment decisions;
There is no substantial research base to prove that a linear progression from Level 1 (learner reactions) to Level 4 (business outcomes) exists (i.e. if you complete one you can move on to the next level);
The available electronic environments and requirements for knowledge work in the New Economy have progressed so far and are so dynamic that the original four Kirkpatrick levels bear little relevance to measurement of the transfer of knowledge in its many diverse forms;
The model ignores the situation within which learning transfer occurs and focusses on the learning outcomes or immediate knowledge transfer (Holton, 1996);
As an extension of the above, the evaluation approach needs to accommodate the learning so that it includes learning objects and evaluation of technology design; and
Organisational learning or factors impacting learning capacity and knowledge transfer within organisations and communities are not accommodated or cannot fit within a linear, non-adaptive system model.

While surveys consistently show that more than 80 per cent of training managers use Level 1 forms of evaluations and up to 30 per cent Level 2 approaches, less than 20 per cent use Level 3 and less than 5 per cent deploy Level 4 forms of evaluation. Clearly Level 4 evaluation provides the best indication of the business benefits of an elearning design or intervention. Continuously improving elearning metrics at this level directly translates into improved integration into systems and overall organisational strategic success. Yet it is at this level that least evaluation is conducted.

Step 4.2. Apply metrics

The overall advancement in building capabilities (competencies such as skills and knowledge, and identity attributes such as behaviours, roles, culture) at individual, group or organisation levels should be revisited to ensure that evaluation is using the best metrics or if they need to be updated. This will provide a gauge of overall learning efficiency.

Senior managers in organisations also seem to like metrics that confirm the value the investment in the learning strategy has yielded. Various methods have been used and supported to evaluate organisational effectiveness in terms of return on investment at Kirkpatrick's Level 4. While Kilpatrick suggests a cost-benefit ratio (CBR):

CBR =	Program benefits
Program costs

others, such as Phillips (1997), argue for a return on investment (ROI) approach and its addition as an extra level of evaluation. Phillips adds an ROI step and identifies ten strategies to convert data to monetary values depending on the type of data and the particular approach being used. The ten strategies follow:

Output data is converted to profit contribution or cost savings;
Cost of quality is calculated;
Wages and benefits are used as value for time;
Historical costs are examined;
Internal and external experts are sourced;
External databases are used;
Participants provide estimates;
Participants' supervisors provide estimates;
Senior management provides estimates; and
HRD staff provide estimates.

Kirkpatrick's is, at best, an indicative model for elearning to accommodate many levels within the evaluation metrics. It also emphasises the need to evaluate beyond reaction and individual learning. The capture, analysis and reporting of information must enable a picture to be formed of how well elearning is effected at different organisational levels. These levels include:

The company;
Major functional or occupational areas (e.g. finance department or clerical work);
Job-specific requirements (e.g. clerical or accounting);
The operational unit (e.g. department, team, division); and
The individual.

Step 4.3. Evaluate tools and materials and the learning process(es)

Evaluation needs to focus not just on the outcomes from the learning strategy, but on the instructional design, the learning process and the materials used to support the learning strategy. The core focus for evaluation at this level is on:

Content;
Overall design;
Learner satisfaction.

Activity 9

Use the following links to reflect on different means to evaluate your instructional design and overall process of delivering the learning strategy.

The Evaluation Matrix http://mime1.marc.gatech.edu/MM_Tools/EM.html .

IACET's Criteria and Guidelines for Quality Continuing Education and Training Programs ('IACET Criteria and Guidelines') http://www.iacet.org/standards/intro_CG.htm

Evaluating the Effectiveness of E-Learning Strategies for Small and Medium Enterprises by Eduardo Figueira . Sourced January 2005 at http://www.theknownet.com/ict_smes_seminars/papers/Figueira.html

Dangers

Common causes of problems with learning strategies can be traced to content or a design remedy that is:

Inappropriate or irrelevant to the learning objectives;
Set at too high or too low a level of rigour for the intended audience;
Unclear or does not provide adequate coverage of the topic;
Unable to balance knowledge transfer with applied practice or individual reflection; and
Illogically sequenced.

Step 4.5. Report and suggest improvements

Evaluating the impact of learning on individuals, groups and an organisation provides management with a critical means to assess current investments in learning. Evaluation not only can determine current progress, it also can analyse and confirm where improved results can be achieved.

Beyond the business and investment dimensions, evaluation is a formal part of the overall instructional design cycle. It s also a critical part of the quality assurance cycle. As such, evaluation triggers improvements to the overall system of learning strategy design.

Evaluation results need to be properly documented and formal responses monitored.

Activity 10

Complete an evaluation of an actual learning strategy as follows.

Source some useful resources and templates for evaluating a learning strategy. You may source these from an organisation or visit the ANTA site http://www.anta.gov.au/pubBundle.asp?qsID=28
Use the resources to establish the parameters to be evaluated.
Confirm the level at which the evaluation will occur (reactions, business outcomes, etc.).
Document, report and suggest actions relating to any evidence collected on the evaluation results.
Depict and describe a process for record keeping, monitoring and reporting progress on the actions resulting from the evaluation.
Gain sign-off for the evaluation documentation (supporting all the above actions) from an independent expert or the leader responsible for the overall learning strategy implementation.