Learning metrics and business metrics

“It worked” is four different claims.

Evaluation and impact are not the same word

Sixty years of argument in this field rest on one conflation. Two different activities have been carrying one name, and almost every unanswerable measurement question is a symptom of it.

Learning metrics answer whether the instruction is any good. Can these people now make the decisions the work requires, and perform the tasks in something like real conditions? That is evaluation, it belongs to whoever designed the program, and there is a serviceable instrument for it.

Business metrics answer whether the result moved. Days to deliver, defects per thousand, incidents per shift, first-visit fix rate: the measure the organization already reports and already argues about. That is impact, it belongs to the line manager who owns the result, and the instrument is the measure itself. Nothing has to be invented.

These are different questions with different owners, different instruments and different timescales. Treat them as two rungs of one ladder and you get the last sixty years: a profession trying to climb from a satisfied room into a business result by inference, and a business unable to see why the climb should be believed.

Whether the instruction is any good

Sometimes the binding condition genuinely is knowledge and skill, and number 10 says so without hedging. Then the question is not whether the program moved the business. It is whether the program is any good, and that has an instrument.

Will Thalheimer’s Learning-Transfer Evaluation Model sets out eight tiers of evidence, and its usefulness is in the ordering. Attendance, learner activity and learner perceptions are the bottom three, and he is blunt that they validate nothing. Knowledge is the fourth and is split, because knowing a foundational concept and knowing a piece of trivia are not the same achievement. Decision-making competence is the fifth, task performance the sixth, transfer to work performance the seventh, and the effects of that transfer the eighth (Thalheimer, 2024).

The value for this series is precise. Number 09 establishes whether a knowledge-and-skill deficit is the binding condition; number 10 establishes that instruction is the remedy that can reach it. Neither says how you would know the instruction was well made. LTEM answers that, at tiers four through seven, and it does so without asking anybody to attribute a business result to a course.

And business metrics, which you already have

Through the business door of this series the measurement problem barely exists, and that is not a happy accident. It is a consequence of where the work started.

Somebody named a result the organization already counts (number 05). Two values were established on it and the shortfall was cut three ways (number 06). Somebody already achieving it was found (number 07), and the binding condition was established (numbers 08 and 09). Afterwards you look at the same measure again, computed the same way. There is no evaluation framework to choose, no proxy to defend and nobody to convince that the number matters: the finance function has been reporting it for years.

Through the training door the same question is close to unanswerable, and that too follows from where the work started. A program acts on individuals; the result being claimed is produced by teams and held by the organization. No measurement design repairs that, because it is a fault in what was bought rather than in how it was counted.

The ladder that promised to connect them

The conflation has a name and a diagram. Kirkpatrick’s four levels (reaction, learning, behavior, results) put learning metrics and business metrics on one ladder and implied you could climb from the bottom to the top. That promise is the whole problem, and it is worth being precise about it, because the criticism is usually made carelessly and waved away.

Start with what the model actually is. In the late nineteen-fifties Donald Kirkpatrick named four kinds of outcome you could measure after training. As a list of four outcomes worth measuring it has never been bettered, and this paper does not try. It was not offered as a validated causal chain, and for a long time nobody read it as one.

What it acquired in practice is an arrow. The four became a ladder, and a good score at one level came to be treated as evidence for the next. That arrow is what has been coming apart for thirty-five years, and it has come apart further with each decade rather than less.

And the two objections that end the argument

Two more objections have landed, and either on its own would be enough. Together they make the ladder indefensible as a way of evidencing a business result.

The first is a level mismatch, and it is not a technicality. All four measures are taken on individuals: this person reacted, this person learned, this person behaves differently. The result the organization wants at level four is not an individual outcome. Safer handovers, fewer defects, reliable service, faster adaptation: these are produced by teams and held by the organization. So the ladder asks individual evidence to support a collective claim, and no amount of measurement care fixes that, because it is a design fault rather than a measurement fault. The distance between the two levels is real work (teams, embedding, maintenance) and no evaluation form crosses it. The next pages place the ladder among the models that followed it, and give this mismatch its name and its history.

The second is that the model’s own custodians have moved. The organization that carries Kirkpatrick’s work now teaches planning from level 4 backwards to level 1: begin with the result and the reason the program exists, then design towards it (Kirkpatrick Partners, 2026). That is not a defense of the ladder. It is its abandonment as a ladder, by the people with the most to lose from abandoning it, and it points in the same direction as this whole series: start at the result.

Every attempt starts with the learner

Since Kirkpatrick, the profession has tried again and again to show impact through individual learning. Each model adds something worth having. Almost all of them keep the same starting point.

Eight models, one entry point

Set side by side, the models differ in how high they reach. They do not differ in where the evidence enters.

Every column rests on three rungs measured in the learner: reaction, learning and behavior. Above the dashed line the models differ. Most add the organization as a rung; Hamblin, Kaufman and Keller, and Phillips add a further one beyond it. Brinkerhoff looks above the line only through individual cases, and deliberately not through an average. What none of them does is measure a team or an organization as a unit. That is the gap the next pages name.

The level error underneath the ladder

The mismatch named two pages back has a name in organizational research, and a long history. The level error draws a conclusion at one level from evidence gathered at another.

Robinson (1950) showed that correlations between groups cannot stand in for correlations between individuals. The reverse fails as well: data on individuals do not reveal how groups differ, which epidemiologists call the atomistic fallacy (Diez Roux, 1998). Rousseau (1985) and Klein, Dansereau and Hall (1994) turned this into a rule for organizational research: the level of the theory, of the measurement and of the analysis must match.

Kozlowski and Klein (2000) then described how properties of a team or an organization emerge from individuals. By composition, similar individual properties combine into a shared one. By compilation, different contributions form a pattern that exists only at the higher level. For training, Kozlowski and colleagues called the move from individual outcomes to team and organizational outcomes vertical transfer, and treated it as something to design for rather than to assume (Kozlowski et al., 2000).

Why it undermines the ladder

• The result is often a pattern, not a sum. Safe handovers, reliable service and low defect rates depend on how contributions fit together. Ten people who each learned more do not add up to that pattern by themselves (Kozlowski & Klein, 2000; Ployhart & Moliterno, 2011).
• Context decides what reaches the result. Whether individual gains count depends on the work around them: coordination, tools, incentives. The ladder has no place for that, which is why firm-level research keeps finding that training cannot be separated from its system (Garavan et al., 2021).
• The inference fails both ways. A positive effect in individuals does not show an effect on the organization, and its absence does not rule one out. Where results emerge by compilation, a team can change how it works together while individual scores barely move.

The ladder against how results emerge

On the left, the inference the ladder makes. On the right, the two ways a result at a higher level actually comes about.

The ladder measures three rungs in individuals and infers the fourth for the organization. Composition is the case where that inference comes close: similar contributions, independent tasks, a result that is roughly their sum. Compilation is the case where it fails: different contributions that only produce the result through how people work together, so the result exists only at the higher level and has to be designed and measured there (Kozlowski & Klein, 2000).

What the level error says, and what it does not

The level error is easy to overstate. Four limits keep it honest.

• Where adding up holds. When people work side by side on independent tasks, such as agents who each handle their own calls, the result of the unit comes close to the sum of individual results. That is composition, and there the error is small.
• The error lies in the inference, not in the measurement. Measuring individuals remains worth doing. The four outcomes Kirkpatrick named survive as a list; what fails is the step from one level to the next.
• Organizational effects of training exist. Across 159 studies, training relates to organizational performance, weakly and with wide variation by context (Kim et al., 2025). What fails is the route of proof, not the possibility of an effect.
• Isolation is sometimes possible. Phillips’s methods for isolating effects, and the designs in section 06, can show an effect at the level of a unit when conditions allow. They then measure at that level, which is the point.

Or stop trying to prove impact at all

There is a serious argument, made from inside the profession, that the whole pursuit of business impact is a mistake. It deserves a hearing rather than a footnote, because on its own ground it is largely right.

Will Thalheimer sets out ten reasons not to measure impact. Four of them bite hardest. Measuring impact costs a great deal, and the money spent proving a program worked is money not spent making it work. A function measuring its own success is open to bias and to the suspicion of bias, whatever the design. Learning spend is small enough that it does not register against the scale of organizational funds, so the number nobody asked for is also a number nobody weighs. And impact evidence has never been observed to save a function from a cost review anyway (Thalheimer, 2025).

His conclusion is that learning professionals have far more influence over learning than over organizational outcomes, and should spend their effort where that influence is. That is not defeatism. It is a correct reading of where a training function actually stands.

Two ordinary facts that fake a result

Before any question of design, two statistical facts produce improvement out of nothing at all. Both are entirely mundane, both are well documented, and between them they account for a great many celebrated successes.

Regression to the mean. Unusually extreme measurements tend to be followed by measurements closer to the average, and the effect is larger the noisier the measure and the more the group was selected on its extreme values. Read that second clause slowly, because it describes standard practice: the organization identifies its worst-performing region, team or branch, intervenes there, measures again, and finds improvement. Some of that improvement was going to happen with no intervention whatever, and it is a property of arithmetic rather than of the remedy.

Selection. Who ends up in the measured group is rarely random, and the way people arrive in it can produce an apparent effect that has nothing to do with what was done. In an organizational setting it arrives in three ordinary forms: people who volunteer for a program differ from those who do not; people who leave before the post-measurement are not in it; and the sites that agreed to go first were the sites that were readier.

Four designs you can actually run

The counterfactual is usually unavailable. It is not always unavailable, and the situations that supply one for free are more common than they look: most of them created by operational decisions nobody thought of as research.

Which gives the single highest-yield instruction in this paper, and it costs nothing. Decide the evaluation design before the rollout, not after. A phased deployment whose order is recorded in advance is a usable design; the identical deployment with the order lost is a before-and-after with a story attached. The difference is one email, sent a month earlier.

Contribution rather than attribution

Where no design is available, which is most of the time, there is still a disciplined way to make a defensible claim. It is not a technique so much as a form of words that commits you to work.

The claim is that the intervention contributed: that it was one cause among others, that the mechanism by which it would have worked was stated in advance, and that the change observed is consistent with that mechanism and not better explained by something else. That is the second of the two statements on the business side, and it is precisely what a completed cause analysis is already holding: a stated condition, a stated remedy, and a prediction about which number would move and roughly by how much.

The discipline is in the second half of the sentence. A contribution claim is worth nothing unless the rival explanations were named in advance and examined. Which rivals, in an ordinary organization? The market moved. A second change landed in the same quarter. The measure’s definition changed. The comparison period was unusual. The people who were measured are not the people who started. Each of those is checkable from records the organization already holds, and a contribution claim that does not visibly check them is a story with a diagram attached.

The rollout that proved itself by accident

A retail bank, 214 branches. A change to the mortgage application handover (a fixed form traveling with the file, replacing a phone call) was deployed over nine months because the training and system work could not be done everywhere at once.

The measure was applications reaching the underwriter complete: 68 percent before, required 90. After the full rollout it stood at 81. The report that was drafted said the change had delivered thirteen points. That is the measure moved written as we moved it, and it was wrong in three ways.

None of that made the change worthless. It made the first version of the claim indefensible, which is a different thing and a recoverable one.

What the rollout turned out to be

The nine-month deployment was, without anybody intending it, a phased design. The order had been recorded (in a project plan, for scheduling reasons) and that record was the whole asset.

Branches that had received the change could be compared against branches that had not yet received it, in the same months, with the pre-existing difference between the two groups measured rather than assumed. The system upgrade in month six landed on both groups simultaneously, so it moved both and did not explain a difference between them. And the rising trend was present in both, so it likewise could not account for the gap between them.

The instructive part is what it cost. Nothing was measured that the bank was not already measuring. The only thing that made the stronger claim possible was that somebody had written down the order of the waves nine months earlier, for reasons that had nothing to do with evaluation. That is the whole of the practical advice in this paper.

What this does not claim

This paper raises the evidence standard. That has costs, and pretending otherwise would be the same error in the opposite direction.

• Rigor has a price and it is not always worth paying. A change that cost two days of an analyst’s time does not need a quasi-experimental design. The standard should scale with what is at stake: the measure moved is fine for a small change nobody will cite, and the second statement is required the moment the result is used to justify further spending.
• The four designs come from other fields. Interrupted time series, difference-in-differences, stepped rollouts and threshold comparisons are established in epidemiology and econometrics, and they are described here at the level a practitioner needs rather than at the level an analyst does. Where the stakes justify it, get somebody who knows the arithmetic.
• A contribution claim is not a certificate. It is a disciplined way of reasoning towards a defensible statement, and two competent people applying it to the same case can still disagree about how much of the movement to attribute. What it rules out is the undisciplined version, which is most of what gets published internally.

Six lines, on the claim in front of you

Twenty minutes, on a result somebody is about to report. Line one decides how much of the rest you need, and line six is the one that should have been done first.

Two problems, one method, sixteen papers

Two models carry the method. Gilbert’s six conditions ask what kind of cause; Rummler and Brache’s three levels ask where it sits. Both are sound and this series uses both, and it leads with Gilbert, because his unit is the accomplishment, which is what an organization already counts, and because his six conditions are the categories an organization can actually change.

Where the method comes from

Two further pieces close the set and carry no number, because each points at all the numbered papers. Before the guide sits Arets L&D Business Models, the destination piece for the people who receive the requests: four models a learning function can run, and the route from HR-driven to business-driven. Last sits Solutions that cannot work, the field guide, ordered by the sentence that lands on your desk: seventeen forms a performance problem arrives in, each with an address in one of the numbered papers. Three orders in all: the destination you choose, the question you are on, and the form that landed on your desk.

Sixteen papers, six parts

Number 01 is the map and number 02 is the foundation. From number 05 on, every number is depth on exactly one step of the method. The filled block is the paper you are holding; the two unnumbered pieces close the set.

Where every claim in this paper comes from

Sources, continued

The three books named earlier in this paper are listed in full in number 02, so that they do not recur in every reference list.