Notes on Python Error Handling

Thu Feb 23 2023E.W.Ayers

An error is something that goes wrong when parsing or executing a python program. An exception is an error that occurs during execution.

Exceptions are a way to 'escape' the usual control flow of a python program. When an exception is raised, the thread stops what it is doing and moves up the call stack. It performs any cleanup it needs along the way (closing with blocks, decrementing reference counts, running finally blocks.) until encountering an try block with an except clause. If the exception bubbles all the way up the stack the interpreter will exit and print the exception's message and traceback.

1. Exception objects

In python, all exceptions derive from the BaseException class. However you should use Exception in the majority of cases. Exceptions all have the following fields:

To make an exception, just construct it RuntimeError(). You can usually pass whatever you want. Making the first argument a human-readable description of the error is a good idea.

1.1. Exception chaining

Sometimes you catch an exception and you want to raise it, but with a different type. For example you could fetch a document from an API or database, which will raise an HTTPException with a 'not found' status code or whatever random library-specific error the database library spits at you. But you want to raise that as a KeyError, but you want to keep the information from the original error e. You do this with raise KeyError() from e.

1.2. Notes

In 3.11 you can now add notes to exception instances by calling e.add_note("blah") you can use this to just add some extra information that might be useful to the user. Eg if a AuthenticationError comes in you could add the note "are you logged in? sign in at https://logloglogloglogin.com and try again."

1.3. Exception groups

New in 3.11 is exception groups. The ExceptionGroup object wraps a list of exception instances.

You can raise with raise ExceptionGroup('oh no', [OSError(), SystemError()]). There is a new syntax except * OSError as e:. The block will be run for each exception in the group that is an OSError.

1.4. How to decide what exception to raise

  1. Is the error to do with a function argument being invalid?

    1. Is the type wrong? → Use TypeError

    2. Is the value out of range or invalid? → Use ValueError

  2. Is the error because you haven't got round to implementing it yet? → Use NotImplementedError

  3. Is the error because you tried looking something up in a table or other datastructure and it didn't exist?

    1. Is it something like a dictionary? → Use KeyError

    2. Is it something like an array which can be out of bounds? → Use IndexError.

    3. Something else? → Inherit from LookupError

  4. Is the error to do with a connection to something being broken? → Use ConnectionError or one of its subclasses BrokenPipeError, ConnectionAbortedError, ConnectionRefusedError and ConnectionResetError.

  5. Doesn't fit any of these?

    1. cba to make own exception type? → useRuntimeError

    2. → inherit from Exception.

1.5. Making your own exception types.

You should make your own exception types. They should always end with Error to make it clear you can raise them.

This is usually enough for your exception:

(1)
class MyError(Exception):
    pass


raise MyError('oh no')

For bonus points you should inherit from whatever base exception best captures the spirit of the error type. If you want to pass in extra data, I recommend making it a dataclass and overriding __str__.

When writing your own exceptions, str(e) should return just the human-readable message of the exception (not the type). The default behaviour of str(e) is to return str(e.args) if len(e.args) > 1 else str(e.args[0]) or '' if there are no args. Going multiline is fine if you want to report a lot of data. The general convention is that the first line should be a human-readable summary and you should not capitalize and not add a fullstop.

If your function can raise exceptions, you should add a "Raises:" section to the docstring (see the Google example docstrings).

(2)
def foo(x):
    """Computes the foo of x.


    Args:
        x: The value to compute


    Raises:
        AttributeError: when x doesn't have the right attributes


    """
    ...

1.6. Handling exceptions

1.7. How to print exceptions properly

2. Atomicity

An exception doesn't have to be explicitly raised by the current thread. The thread could be thrown by a KeyboardInterrupt or some other IO interrupt. This raises the question of what operations in python are atomic and robust to interrupts. For example, if you are manipulating a database you don't want a keyboard interrupt to cause the database to enter an invalid state.

Python threads are atomic at the level of bytecode instructions, the manual provides a list of example atomic ops, including things like modifying lists and dictionaries. An interrupt will break execution at the next bytecode instruction, so it's possible to end up in weird states. For this reason, you should be very wary of doing anything but exiting after a KeyboardInterrupt is caught (see Python docs). Note that thread-safe structures like locks don't fix this.

If you want to have code that resumes after a keyboard interrupt, a better way is to use a signal handler:

(3)
import signal


interrupted = False
def handler(sig, frame):
    # This will run in the main python thread, interrupting whatever it is currently doing.
    global interrupted
    interrupted = True
    # raising an exception here will raise an exception in the main thread.


signal.signal(signal.SIGINT, handleInterrupt)

3. Assertions

Assertions are special errors that happen when an assert condition fails. The idea of assertions is that they should be used for debugging; when you release your code, you should be able to delete all of the assert statements and your program should behave identically. To help out with this, python offers a special release interpreter mode where it doesn't even run the assert statements, so you can have really computationally expensive assert statements without needing to remove them for performance. What I usually do is have a liberal number of assert statements in my code, if an AssertionError is triggered, it means that there is a bug (that I then fix), or it might be a valid error, in which case I should convert the error into raising an exception.

Another thing to remember is that you can write assert C, "error string" and provide a string message.

4. Warnings

Python has a similar system for dealing with warnings. A warning is something where you want the user to know that something is possibly going wrong but not in a way that means we should pull the ripcord.

You should use the warnings system instead of the logger.warning function.

5. Exception types

I will keep updating this list until I have collected them all.

5.1. Function is upset

5.2. Object is upset

5.3. Data is upset

5.4. IO is upset

5.5. Python is upset

5.6. Controlflow