12 <strong>Topic XI. Assigning Credence Levels to Propositions…
Topic XI. Probabilistic Reasoning
- Using meta-judgments of the likelihood that your best judgment is right— how confident you are—enables decisions that take uncertainty into account.
- An important element of the culture of science is the use of “tentative” propositions, often quantified. These can be as confident as 99.99999%—you would bet your life on it—but it would still be understood to be held as a proposition which could be wrong. This makes it psychologically easier for a scientist to be open to being wrong—and to look actively for ways they might have gotten it wrong. This cultural understanding of the importance of recognizing and reporting one's credence level leads to insistence on including error bars on graphs: a data point is completely meaningless without an error bar.
- Addressing the Question: How confident should we be?
- Credence/Confidence Levels #mq
- Calibration of Credence/Confidence Levels
TOPIC RESOURCES
EXAMPLES
- Introductory Examples
- Weather forecasts
- Using polls to predict elections
- You might decide your credence level that your crush will say yes if you ask them to the dance, and use that to decide whether to ask or not.
- Credence levels predicting the probabilities of natural disasters within specific time frames (earthquakes, floods, wildfires etc.), and using these to make decisions about disaster preparation.
- Use credence levels about getting into various colleges to decide what to use as a safety school.
- Using credence levels about passing tests to decide whether to study more.
- Saul's story of a physicist who cancelled a lecture five minutes in because the presenter wasn't sure how his error bars were calculated.
- Exemplary Quotes
- "I'm 95% confident that this battery is not going to explode. But more than even a 1% chance of our robot exploding would lead is too risky, so we shouldn't use this battery until we are more confident it won't explode."
- “Uncertainty, in the presence of vivid hopes and fears, is painful, but must be endured if we wish to live without the support of comforting fairy tales. It is not good either to forget the questions that philosophy asks, or to persuade ourselves that we have found indubitable answers to them. To teach how to live without certainty, and yet without being paralyzed by hesitation, is perhaps the chief thing that philosophy, in our age, can still do for those who study it.” -Bertrand Russell, History of Western Philosophy, p. xiv
- Cautionary Quotes: Mistakes, Misconceptions, & Misunderstandings
- Some students mistake "confidence" for the colloquial sense of "high confidence."
- "Dr. Ryan, are you absolutely certain? We can't authorize spending hundreds of millions of dollars sending a fleet to Patagonia unless we're completely certain about the outcome."
LEARNING GOALS
- A. ATTITUDES
- Recognize that every proposition comes with a degree of uncertainty.
- e.g., not be impressed by statements made with 100% confidence, no error bars or confidence intervals on claimed measurements, seeking definitive answers when only probabilistic information is available, not recognizing that probabilistic information is better than no information.
- Value and defend scientific expressions of uncertainty.
- B. CONCEPT ACQUISITION
- Credence: level of confidence that a claim is true, from 0 to 1.
- Confidence: essentially a synonym for credence, as in “level of confidence,” instead of colloquial meaning, “state of having a lot of confidence.”
- Accuracy: How frequently one is correct; proximity to a true value.
- Calibration: How closely confidence and accuracy correspond; that is, how accurate a person or system is at estimating the probability that they are correct.
- Because every proposition comes with a degree of uncertainty:
- a. Partial and probabilistic information still has value.
- b. Back-up plans are important because no information is absolutely certain.
- c. It is important to invest in calibrating where you are more and less likely to be right, as opposed to being overinvested in being “right.”
- d. Scientific culture primarily uses a language of probabilities, not certain facts.
- e. Even correctly-done science will obtain incorrect results some of the time.
- B. CONCEPT APPLICATION
- Appropriately weigh uncertainty in decisions involving risk. Identify a reasonable threshold of confidence for a given decision.
- Recognize situations where confidence levels can high enough for risky action (e.g. sometimes confidence is high enough to bet your life or the lives of others, even without perfect certainty).
- Explain how the treatment of uncertainty in scientific work allows scientists to follow the truth, even when that means changing their minds.
CLASS ELEMENTS
- Introductory Examples
- Saul's story of a physicist who cancelled a lecture five minutes in because the presenter wasn't sure how his error bars were calculated.
- Suggested Readings & Reading Questions
- Clicker Questions
- If a President put forward a new policy proposal on health-care reform, which statement would make you feel more confident?
- A. “The policy that I’m putting forward is the right policy for America. I guarantee that it is what’s best for the country.”
- B. “I think that the policy I’m putting forward is the one that is most likely to be the right policy for America. There is no guarantee that it is will work -- in fact, I give it only a 75% chance, but the alternatives that have been presented are all much less likely to succeed.”
- If you cross Oxford Street after class next Thursday, what is the likelihood that you will get hit by a car?
- A. about 1 in 1,000 (one in a thousand)
- B. about 1 in 100,000 (one in a hundred thousand)
- C. about 1 in 10,000,000 (one in ten million)
- D. about 1 in 1,000,000,000 (one in a billion)
- E. about 1 in 100,000,000,000 (one in a hundred billion)
- How large would the risk of getting hit by a car when crossing Oxford St. need to be in order to affect your plans?
- A. about 1 in 1,000 (one in a thousand)
- B. about 1 in 100,000 (one in a hundred thousand)
- C. about 1 in 10,000,000 (one in ten million)
- D. about 1 in 1,000,000,000 (one in a billion)
- E. about 1 in 100,000,000,000 (one in a hundred billion)
- Discussion Questions
- Do you have any beliefs for which you have less than 1.0 credence but do not know how to do without?
- Discuss any current controversial topic, but for each statement anyone makes, they have to give it a credence level.
- Class Exercises
- Students guess the answers to 10 binary questions, write credence levels for each one. See the answers, calculate your calibration.
- Many examples given from recent science presentations.
- A arbitrary topic is chosen for small group discussion (e.g. “Does testing in the schools help or hurt education?”), but during the discussion the students have to state their credence level (by saying a number between 0 and 100%) after every statement that they make which could have a credence level associated with it.
- Data Science Applications
- If we could get some system to institute credence level toggles for each statement, we could get a lot of data to test for calibration and see if it helps. something like that.