Notes on Python Error Handling

Thu Feb 23 2023E.W.Ayers

An error is something that goes wrong when parsing or executing a python program. An exception is an error that occurs during execution.

Exceptions are a way to 'escape' the usual control flow of a python program. When an exception is raised, the thread stops what it is doing and moves up the call stack. It performs any cleanup it needs along the way (closing with blocks, decrementing reference counts, running finally blocks.) until encountering an try block with an except clause that matches the type of the raised exception. If the exception bubbles all the way up the stack the interpreter will exit and print the exception's message and traceback.

1. Exceptions

In Python, all exceptions derive from the BaseException class. However you should use Exception in the majority of cases. Exceptions all have the following fields:

To make an exception, just construct it (eg RuntimeError()). You can usually pass whatever you want. Making the first argument a human-readable description of the error is a good idea.

1.1. Exception chaining

Sometimes you catch an exception and you want to raise it, but with a different type. For example you could fetch a document from an API or database, which will raise an HTTPException with a 'not found' status code or whatever random library-specific error the database library spits at you. But you want to raise that as a KeyError, but you want to keep the information from the original error e. You do this with raise KeyError() from e.

1.2. Notes

In 3.11 you can now add notes to exception instances by calling e.add_note("blah") you can use this to just add some extra information that might be useful to the user. Eg if a AuthenticationError comes in you could add the note "are you logged in? sign in at https://logloglogloglogin.com and try again."

1.3. Exception groups

New in 3.11 is exception groups. The ExceptionGroup object wraps a list of exception instances.

You can raise with raise ExceptionGroup('oh no', [OSError(), SystemError()]). There is a new syntax except * OSError as e:. The block will be run for each exception in the group that is an OSError.

2. How to decide what exception to raise

  1. Is the error to do with a function argument being invalid?

    1. Is the type wrong? → Use TypeError

    2. Is the value out of range or invalid? → Use ValueError

  2. Is the error because you haven't got round to implementing it yet? → Use NotImplementedError

  3. Is the error because you tried looking something up in a table or other datastructure and it didn't exist?

    1. Is it something like a dictionary? → Use KeyError

    2. Is it something like an array which can be out of bounds? → Use IndexError.

    3. Something else? → Inherit from LookupError

  4. Is the error to do with a connection to something being broken? → Use ConnectionError or one of its subclasses BrokenPipeError, ConnectionAbortedError, ConnectionRefusedError and ConnectionResetError.

  5. Doesn't fit any of these?

    1. cba to make own exception type? → useRuntimeError

    2. → inherit from Exception.

2.1. Making your own exception types.

You should make your own exception types. They should always end with Error to make it clear you can raise them.

This is usually enough for your exception:

(1)
class MyError(Exception):
    pass


raise MyError('oh no')

For bonus points you should inherit from whatever base exception best captures the spirit of the error type. If you want to pass in extra data, I recommend making it a dataclass and overriding __str__.

3. What exception strings should look like

When writing your own exceptions, please make sure str(e) returns just the human-readable message of the exception (not the type). You can set this behaviour by implementing the __str__(self) method on the exception type.

The default behaviour of str(e) for Exceptions is to return str(e.args) if len(e.args) > 1 else str(e.args[0]) or '' if there are no args.

Going multiline is fine if you want to report a lot of data. The general convention is that the first line should be a human-readable summary. By convention this first line should not capitalize and not add a fullstop.

4. Documenting what exceptions get raised

If your function can raise exceptions, you should add a "Raises:" section to the docstring (see the Google example docstrings).

(2)
def foo(x):
    """Computes the foo of x.


    Args:
        x: The value to compute


    Raises:
        AttributeError: when x doesn't have the right attributes


    """
    ...

5. Handling exceptions

6. How to print exceptions properly

In general, calling str(e) will stringify just the message of the exception (not the type of the exception). Don't rely on things like e.message.

If you just want to print the message, use str(e). If you want to print the exception type + message + traceback + notes, use traceback.print_exception(e). If you don't want the traceback, use traceback.print_exception(e, tb = None).

See the traceback module for more ways of printing exceptions.

I recommend logging exceptions with the logging system. Using the logging module, in an except block you can call logging.exception("oh no") and it will print "oh no" + type + message + traceback + notes.

I also recommend using rich. Use rich.console.print_exception() to print the exception and the traceback in a pretty way.

7. Atomicity

An exception doesn't have to be explicitly raised by the current thread. The exception could be thrown by a KeyboardInterrupt or some other IO interrupt. This raises the question of what operations in Python are atomic and robust to interrupts. For example, if you are manipulating a database you don't want a keyboard interrupt to cause the database to enter an invalid state.

Python threads are atomic at the level of bytecode instructions, the manual provides a list of example atomic ops, including things like modifying lists and dictionaries. An interrupt will break execution at the next bytecode instruction, so it's possible to end up in weird states. For this reason, you should be very wary of doing anything but exiting the process after a KeyboardInterrupt is caught (see Python docs). Note that thread-safe structures like locks don't fix this.

If you want to have code that resumes after a keyboard interrupt, or some other OS signal, a better way is to use a signal handler:

(3)
import signal


interrupted = False
def handler(sig, frame):
    # This will run in the main python thread, interrupting whatever it is currently doing.
    global interrupted
    interrupted = True
    # raising an exception here will raise an exception in the main thread.


signal.signal(signal.SIGINT, handleInterrupt)

8. Assertions

Assertions are special errors that happen when an assert condition fails. The idea of assertions is that they should be used for debugging; when you release your code, you should be able to delete all of the assert statements and your program should behave identically.

To help out with this, Python has the -O flag where it doesn't even run the assert statements, so you can have really computationally expensive assert statements without needing to remove them for performance. You can also set PYTHONOPTIMIZE=TRUE in the enviornment to ignore asserts.

What I usually do is have a liberal number of assert statements in my code, if an AssertionError is triggered, it means that there is a bug (that I then fix), or it might be a valid error, in which case I should convert the assertion into raising an exception. A general rule of thumb is that it should be considered a critical bug to see an AssertionError in production code.

Another thing to remember is that you can write assert C, "error string" and provide a string message to help with debugging.

9. Warnings

Python has a similar system for dealing with warnings. A warning is something where you want the user to know that something is possibly going wrong but not in a way that means we should pull the ripcord.

You should use the warnings system instead of the logger.warning function.

10. Exception types

This is a glossary of all the exceptions that I have seen in the wild. I will keep updating this list until I have collected them all.

10.1. Function is upset

10.2. Object is upset

10.3. Data is upset

10.4. IO is upset

10.5. Python is upset

10.6. Control flow

These are exceptions that are not errors, but instead are used to control the program flow during normal operation.