Topic XII. Calibration of Credence Levels

  • OVERVIEW

    • It is important to check the calibration of credence levels; that is, how good one's judgments are about how likely each of one's claims is to be right.
    • Most people’s estimates of their confidence is wrong in characteristic ways: high confidence tends to be over-estimated and low confidence tends to be under-estimated. This can be trained to be closer to an accurate calibration, most effectively with repeated, unambiguous, and immediate feedback. One problem that arises from poor calibration is that juries often use witness’ confidence to gauge the likelihood that they are correct, but this often yields poor results due to the poor calibration of the witnesses. 
    • Addressing the Question: How confident should we be?
      • The Value of Uncertain Information
      • Epistemic Caution

  • TOPIC RESOURCES

  • EXAMPLES

    • Exemplary Quotes
      • "Weather forecasters often seem wrong, but they only give probabilities, and their probabilities are really well calibrated. So we should trust weather forecasters, but remember that a 90% chance of rain also means a 10% chance of no rain."
      • "Being well calibrated does not require always predicting the correct outcome but requires being able to predict how often one will be wrong."
    • Cautionary Quotes: Mistakes, Misconceptions, & Misunderstandings
      • Some confusion between confidence, calibration, and accuracy.
      • The temptation to rely on confidence over calibration is sometimes hard to resist.
      • "He seems super confident, and she said she was only 85% sure, so we should trust him over her."

  • LEARNING GOALS

    • A. ATTITUDES
    • B. CONCEPT ACQUISITION
      • Confidence Interval: A range within which a true value of interest lies with a specified probability.  
        • Most commonly a 95% confidence interval, which means there is a 5% chance the true value lies outside the range specified.
          Related Items (#CL.B.1)
      • Error Bars: Smaller bars on a graph that show the range of likely true values around the observed value, typically a 95% confidence interval, or the observed value +/- the standard error or standard deviation. 
      • Scientific culture at its best reinforces the importance of uncertainty by offering respect and career advancement to people on the basis of calibration as well as accuracy. In attaching the ego to calibration as well as accuracy, this discourages scientists from being overly attached to their ideas being “right,” encouraging them to prioritize truth over having been right. 
      • People (including many experts) tend to over-estimate their accuracy at high confidence levels (and under-estimate it at low-confidence levels).     
      • People often use a source’s confidence as a cue to credibility, but appropriately discount confidence when they have evidence of poor calibration.   
    • C. CONCEPT APPLICATION

  • CLASS ELEMENTS

    • Suggested Readings & Reading Questions
    • Clicker Questions
    • Discussion Questions
      • What are the costs of underestimating one's credence level at a low level of confidence?
        Related Items (#CL.B.4)
      • What are the costs of overestimating one's credence level at a high level of confidence?
        Related Items (#CL.B.2)
      • What are some ordinary-life scenarios in which talk of confidence intervals might be useful? Hint: When deciding whether or not to stay in a job & wait for a raise, when financial planning, when betting, etc.
        Related Items (#CL.B.1)
    • Class Exercises

      • Credence-calibration questionnaires to show students’ calibration, and training exercises.   

    • Homework Questions
      • Make ten predictions about ten events you expect to happen within the next week (total of 10, not 100!). Write your credence level for each one: that is, how confident you are it will happen within the next week, 1-100. After the week has passed, mark the ones that happened and calculate your calibration score. How well did you do? Did you predict some kinds of events better than others?
      • What’s a subject in which you have to make repeated predictions in everyday life? It could be academic, social, or personal. (Some possible examples: how you’ll do on tests, what grade you’ll get on papers, whether you will enjoy a new course, whether you’ll get jobs or grants you apply for, whether someone will respond to your message on a dating site, whether what you’re wearing will be appropriate for an event, etc.)
        • A.  How accurate are your predictions? What makes it difficult to be more accurate?
        • B. Is there any way that you could make your predictions more accurate? Why don’t you do this? (You may have a good reason; sometimes increased accuracy requires more time and effort than it’s worth.)
        • C.  How well calibrated is your confidence in your predictions? Do you tend to be overconfident or underconfident? Why?
        • D. Is there any way that you could make your confidence in your predictions better calibrated?