Metrics, schmetrics

Has anyone found a small, shiny cylinder with a pointed end? That would be the silver bullet that many are looking for to measure return on investment for intangibles, such as knowledge or learning initiatives. KIN has run a number of workshops on this topic in the past and have concluded that this is the wrong question. Return on impact is more achievable and may provide meaningful metrics.

This is the topic of a new blog post from Steve Dale.
To paraphrase Steve,

Not all changes can be measured in strictly cash value terms, which is what many people consider to be the true meaning of ROI. How do you measure the value of a conversation or some information shared?  The answer is, you don’t... measuring impact can be just as important as measuring value.  The impact might be things like improved customer satisfaction (measured using surveys), or less time to complete a task, or improved staff morale (measured using surveys). Any of these can – and potentially will – have an effect in terms of cash value to the organisation, but I firmly believe that converting impact to cash value is an exercise in futility, since more often than not, there are too many variables. 

I agree that ‘converting impact to cash value is an exercise in futility’. Too many times I have seen organisations attempt to attribute cost savings or efficiency gains to organisational learning initiatives. This is made worse when cash ‘benefit’ is scaled-up across the whole organisation. This invites a coach-and-horses to be driven through the unsupportable multi-variable extrapolation. There is one exception to this; where a repeatable process (think of well-drilling or manufacturing process) is improved through a statistically significant trial. Of course any metric must be made against a benchmark; an oft neglected imperative.

I recommend consideration of 3 categories of measurement:
1. Transactional (documents uploaded, questions posted, time of response, number of online ‘lurkers’, etc). This is the least valuable, however if you don’t have this stuff, someone senior will ask.
2. Impact (decisions made faster, mistakes avoided, connections made, effectiveness of communities of practice etc). Impact measures are often best supported by vignettes or examples.
3. Perception (improved collaboration/cooperation {different things}, goodwill for problem solving)
As Steve suggests, the latter two can be measured using survey results to turn qualitative views into qualitative metrics; a legitimate way, and perhaps the only way, of doing this.

You may notice that I have used the terms metric and measurement. At the risk of being called a pedant, it is worth considering the difference between measurement and metric.

Bill Ravensberg provides a succinct and useful definition in this LinkedIn discussion:

Measurement – A MEASURE (rating, sizing, etc.) of one thing. Examples are cost, function points, effort, time, etc.

Metric – A RESULT of taking two or more measurements to create a value. Using the measurement examples, you can get cost per function point, function points per unit of time, etc.

Basically, the measurement is something you need to create a metric that can then be used for reporting and analysing. A measurement generally will not provide much value or meaning until it is combined with other measurements. The metrics provide the value.