Measuring success: evaluating evaluation

In early December last year, a member of the Training Zone website posted a query about measuring success that addressed one of the eternal issues of training and development – impact on business performance.

I am looking to try and find the best way to evaluate the success of my training sessions – by success I mean the affect it is having on the performance of those being trained.

The training I deliver is usually on fairly technical topics based in legal and medical areas – there is a lot of theory to cover but also how to apply that theory to real life, I suppose what I am trying to get at is that its not really procedural type training.”

Already testing learners two weeks and three months after the training, and getting informal managers’ feedback, the TrainingZone community provided tips and advice in a follow-up article, Measuring Success, last week. But it struck us there were some hints and tips missing. So, if we may so bold …

Timescales
We’re encouraged that testing is taking place more than just immediately after the event – all too often, evaluation is undertaken at the easiest but lowest level, that effectively just confirms that the training took place, people attending, and enjoyed it (or didn’t, as the case may be). But where learning ‘how to apply theory to real life’ is concerned, we’d counsel that embedding and sustaining new ways of behaving and interacting in the workplace is an on-going process than can take substantially longer than three months.

This is especially true where the ability to apply new learning in new contexts is required – new behaviours must be well on the way to becoming new, unconscious habits before this can come either naturally or easily (if either are quite that right word).

Training for transfer and application
For new behaviours and learning to stand a real chance of becoming embedded, maintained and sustained over this longer timescale (which is, after all, the real objective, even if the timescale is counter-intuitive to those seeking or hoping for immediate impact), the learner needs not just training interventions that are designed to support the transfer and application process. While measuring success at various points after the intervention has taken place provides a series of snapshots, there must also be a focus on the bigger picture.

Apart from the issue of defining outcomes in advance – which we’ll come onto shortly – this embraces a wide range of aspects: this means, for instance, involving line managers throughout, so learners are supported both before and after the training in not just developing their abilities but understanding why – in the organisational context – it is important that they do. As the follow-up article pointed out:

To do this requires the line manager or team leader to agree not just *what* performance outcomes the training must support but also *how* to facilitate that in terms of pre, during and post event activities. Unless the culture and environment of the organisation allows for this, it can cause a blockage.”

The pressures of daily work and the comfort of old habits and ways of working are powerful factors that weight against transfer and application for any of us: change is uncomfortable as well as difficult, and the overwhelming majority of us only truly achieve it with help and support. This means designing learning interventions that provide:

the momentum to initiate change
the self-knowledge to identify what must be changed
skills that enable them to change
support and encouragement to sustain change
reward and recognition to ensure change becomes permanent.

The right outcomes
The imperative to implement change initiatives that can lead to impatient timescales (see above) can also blindside both organisations and trainers to another critical factor – defining the right outcomes. If training is to lead to better workplace performance, then the outcomes must be not just ‘learning outcomes’ (nothing wrong with them in principle, but training can often fall prey to an ‘education model’ that misses the larger point) but outcomes that are linked to, and defined in terms of, business outcomes. (One of the surest ways to not achieve something is, perhaps obviously, to fail to define it in the first place.)

Learning outcomes that are linked to business outcomes give not just the learner but the line manager and the wider organisation clarity as to what is to be achieved, or at least striven for. (They may also help the organisation to see the role it will need to play in supporting learners pre- during and post-event).

But this point is fundamental in other sense: if the intended outcomes are not defined in terms of business objectives, who can performance or progress against them – even where it is being achieved – be measured in a way that indicates that business performance is improving?

While the original poster was perhaps anxious that their use of line managers’ feedback was ‘informal’, it’s arguable that the formality (or otherwise) is not the real issue: informal involvement is certainly better than none at all. The type of evaluation and measurement is more critical: if qualitative assessment is what is required, the evaluation process needs to make sure it is not based predominantly on quantitative measures.

And is evaluation purely focused on the learner? What measures are taken to evaluate the degree of line manager involvement and support, the opportunities provided to implement and apply new learning, reward and recognition practices that could encourage the learner in their progress, and the impact of the organisational culture in supporting (or discouraging) it? Are some learners getting coaching or mentoring? Are other support tools or technologies available to help to help maintain their motivation and progress?

Evaluation that measures and assesses these elements can identify differences in practice and process across organisations that provide evidence of approaches that most strongly support transfer and application of learning. And that evaluation could be invaluable for the organisation not just in improving the outcome of the original question-raiser’s training, but all its learning and development activity.