During World War II, the RAF lost a lot of planes to German anti-aircraft fire. They examined all of the planes that made it back to determine where to put protective armour. The picture below show an aggregation of where bullet holes had damaged the planes. Where would you put the armour? The answer appears at the end of this post (1).
This critical thinking exercise was just one that we used during the Knowledge and Innovation Network workshop on 'Evidence-based Decision Making' that we ran last week. Having reliable and complete evidence on which to understand the problem is the start. We learned from several case studies (VISA, IBM, Healthcare policy, Macmillan Cancer Support, the College of Policing and Norton Rose Fulbright) how they gather, present and analyse data, often in real-time, on which to make decisions. How often are your organisation's decisions made on the basis of the HiPPO (highest paid person's opinion) or symptomatic information?
Another experiential learning exercise I ran required teams to consider a 'scenario' and then rank 8 pieces of evidence in order of trustworthiness (2). The scenario was:
'You are a senior manager at a large Italian IT firm specializing in stock control systems. The productivity of the software engineers is below average for the sector. The Board of Directors wants to introduce a performance-related pay model, which the Finance Director says would ‘give the workers a financial incentive to carry out more work’. Since you are not sure what causes the low productivity of the engineers, you decide to consult a number of sources before you decide to implement a performance-related pay model.'
I shan't give you all 8 pieces of evidence, but here are just 4 for you to have a go at ranking, in order of 'trustworthiness' (a deliberately vague term).
- You contact a senior consultant at a well-known consulting firm. This consultant tells you that based on his experience with a large number of manufacturing engineering firms in the US, the most likely cause for low productivity in firms such as yours, is lack of leadership. He therefore advises you not to introduce the performance-related pay model and provide leadership training for all supervisors and senior managers instead.
- The Chief Operating Officer (COO) shows you the graph below, indicating a wide variation of productivity between teams. It appears that teams with a large proportion of senior engineers (5 years experience and more) are twice as productive as teams with a large number of young and inexperienced engineers. She therefore advises you to invest in the training and support of young and inexperienced engineers and a more balanced distribution between the teams.
- The union representing the majority of the engineers has vigorously resisted previous attempts at introducing performance-related pay. The most recent of which resulted in a threatened strike ballot and even greater reduced output whilst negotiations were underway. The Board decided not to pursue the performance-related pay scheme.
- The HR director tells you that the most likely cause for low productivity is lack of teamwork: it is well known that engineers in general lack social skills, and as a result don’t share task-relevant knowledge necessary to solve practical problems and improve performance. She therefore advises you to not to introduce the performance-related pay model and invest in team-building instead.
As you might imagine, having 8 pieces of evidence and 6 teams ranking them, resulted in a huge and passionate debate. This was of course the purpose - applying the principles of evidence-based decision making that we had learned earlier, to situations that are often ill-defined, with missing data and a host of assumptions that need to be made. This was a brilliant and stimulating exercise. The answers (or at least the consensus view!) are given below (3).
As an example of the importance of having a rich source of evidence on which to make decisions, how about this: "When making important strategic decisions, only 29 percent of organizations considered more than one alternative. Just adding a second option made it six times more likely that the decision would later be rated as ‘very successful.’ " (4)
Here's a final example of why critical thinking needs to be applied to data (5).
-----------------------------------------------------------------------------------------
Answers and credits
(1) Answer to plane exercise (presented by Steve Dale): You only have data for the planes that have returned, not the planes that have been (presumably) shot down. You should improve protection in all of the places that have NO bullet holes.
(2) Based on an idea from Eric Barends of the Center for Evidence Based Management http://www.cebma.org/
(3) Answers to evidence ranking exercise (in order of trustworthiness and rationale):
#2 'The Chief Operating Officer tells you that …'. This is based on your own organization's data, which appears to provide objective and reliable evidence and comparison groups. Be careful about statements such as ‘appears to’; we would like to actually see the data regarding age distribution. Data points per team member may be a good indicator of productivity, but what about quality?
#1 'You contact a senior consultant at a well-known consulting firm...'. Represents professional opinion or experience since it is not clear what measures of productivity and teamwork were used. The opinion may be based on a general impression rather than reliable or valid data. How much influence does the different context (US) and different sector (manufacturing engineering) make?
#'The HR director tells you…'. Since no specific evidence is provided, the HR director’s advice should be considered a personal opinion or belief.
#4 'The union representing…'. Whilst this threat and previous experience may be a very important factor in how and when such a scheme is implemented, it is not relevant as evidence for the effectiveness of performance-related pay.
(4) Source: Professor Paul Nutt, Fisher College of Business at Ohio State University, strategy-business.com
(5) Source: Gizmodo.com
