Measuring the impact of intangibles - fairy dust or fair enough?

An organisation I have recently been working with has a problem.

They have used a Capability Maturing Modelling approach to measure the impact of their Knowledge Management program for a number of years, apparently very successfully. The majority of teams/departments that take part in the analysis appear to be achieving very high levels of competency (80%+) leaving little room for cross-department improvement. The organisation asked if there were alternative ‘light-touch’ methods of measuring the impact of their Knowledge Management program.

Whatever method you may use to measure, I believe the imperative requirements are: 

1. A baseline from which to judge improvement. 

2. Credible metrics that will convince both staff and senior managers. Beware of metrics from small samples that are extrapolated to organisation-wide improvement. These are liable to immediate challenge. 

3. Metrics that are empirical, to support other evidence that may be anecdotal

Whilst not imperative requirements, the following are of added benefit: 

4.The ability to leverage results to drive further improvement 

5. Low cost (implying as much self-assessment and automation as sensible and possible)

In addition, it is helpful to differentiate between attributable impact and contributory impact. The latter is the most common scenario, as KM (if embedded in the business process, as it should be) is usually one of many simultaneous improvement activities. The exception to this is where you have the ability/luxury of having control groups for your KM initiatives. For example:

I looked at measurement in intangible organisational assets that are analogous to KM, in particular HR and Corporate Social Responsibility.

Ideas from CSR

A common CSR method is 'The Reputation Index'. Whilst rich in data, this cannot attribute CSR to organisational performance.

Warwick Business School has an excellent article from Prof Kamal Mellahi on measuring CSR impact. In particular note his final ‘cautionary tale’ sentence. 

Some organisations have adapted Delphi Analysis , but this is more suited to forecasting than performance management.

Ideas from HR

Almost all the measurement approaches for HR impact seem to feature some sort of maturity assessment. For example this HR.com measurement table shows, in effect, a maturity model for HR 'stability' 

The leading HR forum is CIPD. They have a sophisticated measurement tool, that relies on users level of agreement with statements. Again, the results look very much like a maturity model with an associated action plan.

This article gives a useful representation of cause and effect from HR. Step 5 emphasises the need to set the level of attribution correctly. This is also shown in the Andrew Mayo diagram below, which we examined at the last KIN Roundtable in May.

Organisational Network Analysis

Other than maturity modelling, the other approach that meets almost all the requirements list above in Organisational Network Analysis, or SNA. The exception may be Point 5, cost.

Andrew Parker at Grenoble University and Rob Cross at the University of Virginia are the ‘go-to people for ONA. Rob has an excellent ONA primer here.

KIN Facilitator Steve Dale is also qualified in SNA modelling.

Other KIN Members’ insights

Below is a mindmap synopsis of the major ‘Take-aways’ from a previous Roundtable I ran on KM measurement.

Conclusion

Assuming that you agree with the criteria I set out at the start…

I have not come across anything that is better than Capability Maturity Modelling that is suited to what the organisation is trying to achieve. My recommendation was therefore that they significantly refresh their maturity modelling process and re-benchmark. The re-benchmarking is needed as I suspect there was ‘grade inflation’ to borrow the phrase from education. If there is a small differentiator between departmental KM performance, this should be re-baselined and exposed to identify gaps and re-invigorate competition.

SNA is a sophisticated alternative that meets most of the criteria. It could be used to measure KM interactions through careful wording of the SNA questions. Strength, direction, density and centrality of bonds can be analysed and teams/departmental KM performance inferred. The resulting network infographics can be particularly useful in presenting results in an empirical way to senior managers. If you have not tried SNA before, this would be worthy of a trial. SNA software is now inexpensive (in some cases free) , although proper set-up and analysis can still take a lot of expert input.

What are your suggestions for a ‘light-touch’ approach to measuring knowledge management program that you can share here?