Please find our summary of wonderful framework on how to make judgments by Michael Mauboussin. Emphasis Ours.
The process of acquiring a skill requires three stages. The first one reflects the cognitive stage where you simply try to understand the activity and therefore appreciate the opportunity to learn and being humble to errors. You might recall the very first time you learnt a new skill. Each part of the experience required you to exert explicit mental effort, including reading understanding and writing.
Second is the associative stage, where the skill becomes fluid and habitual. Only if you have read and understood deeply only now you can form your own arguments. The people who have adequately understood a topic will fall under this bracket. The skill acquisition generally reaches a plateau here but academic experts keep on exploring the topic and digging deeper into the topic.
Tetlock’s path to good judgement loosely follows these three stages and, similar to most drivers, the majority of us don’t get to the third stage of decision making.
In stage 1, our understanding of what we’re trying to forecast is somewhat superficial and casual. In stage 2, we develop a deeper understanding of the topic but lack sufficient nuance-a key element in keen judgement. Forecasters who make it to stage 3 have developed robust mental models to allow them to get a differentiated point of view.
Clayton Christensen describes his theory of theory building in three steps:
The first step is observation, which includes observing the phenomena at hand and carefully measuring and describing results. This allows researchers to agree on standards so they are all talking about the same issue and are using common terms to describe it.
The next step is classification; where researchers place their observations in categories that allow for clarification of differences between phenomena. Early on, these categories are based primarily on attributes.
The last step is definition; a description of the relationship between categories and the outcomes. These relationships are generally described by correlations.
An example of theory building is the history of manned flight. The first step in developing the theory was examining the animals that could fly.
To test the theory, aspiring fliers built wings, attached feathers, climbed a tall spot, jumped, flapped, and crashed. The crash was an anomaly in the theory, prompting researchers to go back to through the observation-classification-definition steps.
Outsourcing is a good example of theory building from the world of business. Outsourcing is the practice of contracting a service that was previously done-in house to an outside company. Outsourcing appears attractive because it may allow a company to reduce its costs and invested capital. A number of companies that have been very successful have relied on outsourcing. For instance, Apple generated 165 billion dollars of revenue in calendar 2012 using about 20 billion dollars in invested capital.
The experience that Boeing, the world’s largest airplane manufacturer, has had with their newest plane shows the limitation of the correlation between outsourcing and economic profit. Boeing has long used suppliers, but its traditional process was to design the plane in-house and then send detailed blueprints and specifications to the suppliers. They called this system “build to print”. Critical design features and vital engineering functions were handled by Boeing, but the company lowered its costs by using suppliers.
When it launched an aircraft by the name 787 Dreamliner, Boeing used a different approach. The company decided to have its suppliers design and build various sections of the plane, leaving only the final assembly to Boeing. Based on the company’s projections, the time to market could be trimmed by a couple of years and time to assemble a plane of that size would drop from a month to just three days.
The program was a mess. Though the plane enjoyed strong pre-orders, the launch was repeatedly delayed as the suppliers were unable to deliver sections that functioned properly and that were ready for assembly. Boeing hoped to create a final product by clicking together, like legos, the 1200 components that it had ordered. But the first plane came to Boeing in 30000 pieces, any of which didn’t fit or work together properly. Boeing had to pull design work back in-house, at a substantial cost.
In refining the theory, researchers have figured out when outsourcing works. For example, outsourcing does not make sense for products that require complex integration of disparate subcomponents. This is because when coordination costs are high, simply getting a product to work is a difficult task. In the stage of the industry, vertical integration works best.
We all know what to do in order to succeed. Many of those who supply advice on success which include academics, consultants, and practitioners, make a very common mistake that prevents them from improving judgement. The mistake is to observe success, identify common attributes associated with that success, and then proclaim that those attributes can lead others to succeed. This approach doesn’t work because it fails to properly sample failure, does not consider circumstances, and often neglects the substantial role of luck.
The relative weighting of the outside and inside view
One way to determine the relative weighting of the outside and inside view is based on where the activity lies on the luck-skill continuum. Imagine a continuum where on one end luck alone determines results-think of roulette wheels and lotteries-and where on the other end skill solely defines the outcomes-such as running or swimming races. A blend of luck and skill reflects the results of most activities, and the relative contributions of luck and skill provide insight into weighting of the outside versus inside view.
For activities where skill dominates, the inside view should receive the greatest weight. Suppose you first listen to a song played by a concert pianist followed by a tune played by a novice. Playing music is predominantly a matter of skill, so you can base the prediction of the quality of the next piece played by each musician on the inside view. The outside view has little or no bearing.
By contrast, when luck dominates the best prediction of the next outcome should stick closely to the base rate. For example, money management has a lot of luck, especially in the short run. So if a fund has a particularly good year, a reasonable forecast for the subsequent year would be a result closer to the average of all funds.
Knowing where you are on the luck-skill continuum tells you a great deal about the reversion to the mean, a concept that is frequently misunderstood. Reversion to the mean says that for an outcome that is far from average, the expected value of the subsequent outcome is closer to the average.
There are two analytical concepts that can help you improve your judgement. The first is an equation that allows you to estimate true skill:
Estimated true skill: grand average + shrinkage factor (observed average – grand average)
The shrinkage factor, represented mathematically by letter C, has a range of zero to 1.0. Zero indicates complete mean reversion to the mean and 1.0 implies no reversion to the mean. In this equation, the shrinkage factor tells us how much we should reverse the results to the mean, and the grand average tells us the mean to which we should revert.
The second concept, intimately related to the first, is how to come up with an estimate for the shrinkage factor. It turns out that the coefficient of correlation, r, a measure of the degree of linear relationship between two variables in a pair of distributions, is a good proxy for the shrinkage factor.
Say we had a population of violinists, from beginners to concert-hall performers, and on a Monday rated the quality of their playing numerically from 1 (the worst) to 10 (the best). We then have them come back on Tuesday and rate them again. The coefficient of correlation would be very close to 1.0-the best violinists would play well both days, and the worst would be consistently bad. There is very little reversion to the mean and hence little need for appeal to the outside view. The inside view correctly receives the preponderance of the weight in forecasting results.
Three researchers, Don lovallo, Carmina Clarke, and Colin Camerer, studied how executives make strategic decisions and found that they frequently rely either on a single analogy or a handful of cases that come to mind. Investors likely do the same. The weakness in using an analogy or a small sample of cases from memory is that they often prevent a decision maker from sufficiently weighing the outside view.
“Single analogy,” found in the top left corner, refers to cases where an executive recalls one example and places 100 percent of his or her decision weight on that analogy.
“Case-based decision theory,” the bottom left corner, reflects instances when an executive considers a handful of case studies-generally through recall-that seem similar to the decision at hand. There is then an assessment of how similar the cases are to the focal decision, and the cases are weighted appropriately.
Lovallo, Clarke, and Camerer argue for what they call “similarity-based forecasting,” which starts with an unbiased reference class but assigns more weight to the cases that are more similar without discarding the less relevant ones.
Updating Information
The last thing that author mentions in his paper for better decision making is Bayes’s theorem.
Here is the classic example, which comes from Daniel Kahneman’s book, Thinking, Fast and Slow:
We need to update our understanding/probability of an existing belief based on new events that unfold in future.
Tetlock discusses other common mistakes in updating beliefs based on new information. One mistake is to overreact to what he calls “pseudo-diagnostic” information. This is information that superficially appears to explain causality but in fact does not. For example, there have been cases when equity analysts have downgraded a stock following the announcement of an acquisition because of earnings dilution, only to see the stock rise because the market deemed the deal to add value.
Risk Management: Control and Reversibility
Peter Bernstein, a well-known economist and financial historian, suggested that there are two basic ways, beyond diversification, that we can manage risk. The first is to find decisions where we have some control over the outcomes. There’s a huge difference between the roll of a roulette wheel and a business investment. In the former, you have no control over the outcome. In the latter, you can take steps to improve the chance of a profit by tweaking an offering price, changing a product design, shifting marketing spending, or replacing the managers running the business. But control often requires commitment.
Exhibit 4 shows the trade-off between reversibility and control. Public investors, who are generally subject to relatively low transaction costs and high liquidity, have little control but can reverse their decisions readily. Activists seek to exert control, but they must signal their seriousness and commitment by taking larger stakes and reducing their ability to reverse their decision. Private equity firms have a great deal of control, but their cost of reversibility. Not surprisingly, investment time horizons shrink as you move from left to right on the chart.
To read more: https://plus.credit-suisse.com/researchplus/ravDocView?docid=UMso6F