Talk: Debugging with the Scientific Method, S. Halloway

A really good talk about debugging.

Talk: Debugging with the Scientific Method, S. Halloway

  • Github:

https://github.com/stuarthalloway/presentations/wiki/Debugging-with-the-Scientific-Method

  • Talk:

https://www.youtube.com/watch?v=FihU5JxmnBg

People see debugging as an occult art, when often it is the easiest part of software development. Debugging is a perfect closed world in which to apply the scientific method.

The scientific method integrates:

  • problem statements
  • testable hypotheses
  • observation
  • refinement

Read Andreas Zeller’s excellent Why Programs Fail, particularly chapters 5-7 and 11-14.

Zeller is also the instructor for the Udacity course on debugging.

David Agans’ Debugging book is a good introduction to many of the same ideas.

Blithe Rocher gave a nice Scientific Method of Troubleshooting talk at Distill 2014.

TL;DR

             +------------+  something is wrong
             |  FAILURE   | <-----------------------+
             +------------+                         |
               |                                    |
               | why                                |
               v                                    |
             +------------+                         |
+----------> | HYPOTHESIS | <+                      |
|            +------------+  |                      |
|              |             |                      |
|              | suggests    |                      |
|              v             |                      |
|            +------------+  | needs refinement     |
| falsified? | EXPERIMENT |  |                      |
|            +------------+  |                      |
|              |             |                      |
|              | produces    |                      |
|              v             |                      |
|            +----------------------------------------+
+----------- |              OBSERVATION               |
             +----------------------------------------+
               |
               | ?
               v
             +------------+
             |   THEORY   |
             +------------+

Definitions

FAILURE:

  • lack of success
  • omission of expected action

HYPOTHESIS:

  • a proposed explanation made on the basis of limited evidence as a starting point for further investigation

EXPERIMENT:

  • test, trial of tentative procedure

OBSERVATION:

  • active acquisition of information from a primary source

falsified?:

  • deductive process using modus tollens:

needs refinement:

  • The process of removing impurities or unwanted elements from a substance

THEORY:

  • A hypothesis offering valid predictions that can be observed

CAUSE:

  • an event preceding an effect without which the effect would not have occurred

ACTUAL CAUSE:

  • difference between the actual world and the closest possible world in which the effect does not occur

FIX:

  • an experiment that establishes an actual cause

WHAT TO DO

  • clear problem statement

  • efficient hypotheses

  • good experiments

  • useful observations

  • writing it all down

Problem Statements

  • steps you took
  • what you expected
  • what actually happened

Good Experiments

  • reproducible
  • driven by hypotheses
  • small
  • change only one thing
  • try to keep the experiment environment clean (no gulp, pluggins, ect)

Making Observations

  • understand all the outputs
  • suspect correlations
  • use good tools

Write It Down

  • problem statement
  • hypotheses
  • what experiment should show
  • why experiment even make sense
  • observations

Software Specific

  • work at a high level of abstraction
  • the failure is not the defect
  • RTEFM, a bug start as an “unknown unknown” so that means read the entire manual
  • don’t trust, reproduce
  • automate

Cheers and happy coding :slight_smile: