Notes on Python Error Handling
Thu Feb 23 2023E.W.Ayers
An error is something that goes wrong when parsing or executing a python program. An exception is an error that occurs during execution.
Exceptions are a way to 'escape' the usual control flow of a python program. When an exception is raised, the thread stops what it is doing and moves up the call stack. It performs any cleanup it needs along the way (closing
with blocks, decrementing reference counts, running
finally blocks.) until encountering an
try block with an
except clause that matches the type of the raised exception. If the exception bubbles all the way up the stack the interpreter will exit and print the exception's message and traceback.
In Python, all exceptions derive from the
BaseException class. However you should use
Exception in the majority of cases. Exceptions all have the following fields:
argsis the tuple of arguments passed to the constructor of the exception.
__traceback__gives the stack trace at the point the exception was raised.
__notes__(≥3.11) is a set of string notes to show with the error.
To make an exception, just construct it (eg
RuntimeError()). You can usually pass whatever you want. Making the first argument a human-readable description of the error is a good idea.
1.1. Exception chaining
If you are in an
exceptblock for exception
eand you call
raise MyError(), Python's traceback will include the traceback for
ewith the message "During handling of the above exception, another exception occurred".
If you write
raise MyError() from e, it will do the same but say "The above exception was the direct cause of the following exception". You can also do this outside of an
If you want it to trash the original exception you can do
raise MyError() from None.
Sometimes you catch an exception and you want to raise it, but with a different type. For example you could fetch a document from an API or database, which will raise an
HTTPException with a 'not found' status code or whatever random library-specific error the database library spits at you. But you want to raise that as a
KeyError, but you want to keep the information from the original error
e. You do this with
raise KeyError() from e.
In 3.11 you can now add notes to exception instances by calling
e.add_note("blah") you can use this to just add some extra information that might be useful to the user. Eg if a
AuthenticationError comes in you could add the note "are you logged in? sign in at
https://logloglogloglogin.com and try again."
1.3. Exception groups
New in 3.11 is exception groups. The
ExceptionGroup object wraps a list of exception instances.
You can raise with
raise ExceptionGroup('oh no', [OSError(), SystemError()]).
There is a new syntax
except * OSError as e:.
The block will be run for each exception in the group that is an
2. How to decide what exception to raise
Is the error to do with a function argument being invalid?
Is the type wrong? → Use
Is the value out of range or invalid? → Use
Is the error because you haven't got round to implementing it yet? → Use
Is the error because you tried looking something up in a table or other datastructure and it didn't exist?
Is it something like a dictionary? → Use
Is it something like an array which can be out of bounds? → Use
Something else? → Inherit from
Is the error to do with a connection to something being broken? → Use
ConnectionErroror one of its subclasses
Doesn't fit any of these?
cba to make own exception type? → use
→ inherit from
2.1. Making your own exception types.
You should make your own exception types. They should always end with
Error to make it clear you can raise them.
This is usually enough for your exception:
For bonus points you should inherit from whatever base exception best captures the spirit of the error type.
If you want to pass in extra data, I recommend making it a dataclass and overriding
3. What exception strings should look like
When writing your own exceptions, please make sure
str(e) returns just the human-readable message of the exception (not the type).
You can set this behaviour by implementing the
__str__(self) method on the exception type.
The default behaviour of
Exceptions is to return
str(e.args) if len(e.args) > 1 else str(e.args) or
'' if there are no args.
Going multiline is fine if you want to report a lot of data. The general convention is that the first line should be a human-readable summary. By convention this first line should not capitalize and not add a fullstop.
4. Documenting what exceptions get raised
If your function can raise exceptions, you should add a "Raises:" section to the docstring (see the Google example docstrings).
5. Handling exceptions
try:block will run up to the point where an exception is raised.
except Exception as e:will catch all exceptions that subclass
Exceptionand the variable
ewill be bound to it. You can also get
except (E1, E2) as e:will check if
e <: E1or
e <: E2.
else:will run only if the try block didn't raise an exception.
finally:block will always run last.
6. How to print exceptions properly
In general, calling
str(e) will stringify just the message of the exception (not the type of the exception).
Don't rely on things like
If you just want to print the message, use
If you want to print the exception type + message + traceback + notes, use
If you don't want the traceback, use
traceback.print_exception(e, tb = None).
traceback module for more ways of printing exceptions.
I recommend logging exceptions with the
logging module, in an
except block you can call
logging.exception("oh no") and it will print "oh no" + type + message + traceback + notes.
I also recommend using rich.
rich.console.print_exception() to print the exception and the traceback in a pretty way.
An exception doesn't have to be explicitly
raised by the current thread. The exception could be thrown by a
KeyboardInterrupt or some other IO interrupt. This raises the question of what operations in Python are atomic and robust to interrupts. For example, if you are manipulating a database you don't want a keyboard interrupt to cause the database to enter an invalid state.
Python threads are atomic at the level of bytecode instructions, the manual provides a list of example atomic ops, including things like modifying lists and dictionaries.
An interrupt will break execution at the next bytecode instruction, so it's possible to end up in weird states. For this reason, you should be very wary of doing anything but exiting the process after a
KeyboardInterrupt is caught (see Python docs). Note that thread-safe structures like locks don't fix this.
If you want to have code that resumes after a keyboard interrupt, or some other OS signal, a better way is to use a signal handler:
Assertions are special errors that happen when an
assert condition fails.
The idea of assertions is that they should be used for debugging; when you release your code, you should be able to delete all of the assert statements and your program should behave identically.
To help out with this, Python has the
-O flag where it doesn't even run the assert statements, so you can have really computationally expensive assert statements without needing to remove them for performance.
You can also set
PYTHONOPTIMIZE=TRUE in the enviornment to ignore
What I usually do is have a liberal number of assert statements in my code, if an
AssertionError is triggered, it means that there is a bug (that I then fix), or it might be a valid error, in which case I should convert the assertion into raising an exception.
A general rule of thumb is that it should be considered a critical bug to see an
AssertionError in production code.
Another thing to remember is that you can write
assert C, "error string" and provide a string message to help with debugging.
Python has a similar system for dealing with warnings. A warning is something where you want the user to know that something is possibly going wrong but not in a way that means we should pull the ripcord.
You should use the warnings system instead of the
10. Exception types
This is a glossary of all the exceptions that I have seen in the wild. I will keep updating this list until I have collected them all.
10.1. Function is upset
ValueErrorthe input to a function is the right type but it's invalid.
TypeErrorthe input to a function has the wrong type.
NotImplementedErroryou were too lazy to implement the function (or it's an abstract method that should have been implemented). Note also there is a
NotImplementedsingleton type that is used for dispatching logic in operator overloads.
RuntimeErrormisc error that doesn't fit the other categories.
10.2. Object is upset
AttributeErrorwhen the attribute you wanted doesn't exist.
LookupError <: Exceptiongeneral error for a failed
IndexError <: LookupErrorraised when the index is out of range in an array lookup.
KeyError <: LookupErrorraised when the key is not found on a dictionary.
10.3. Data is upset
ArithmeticError <: Exception
UnicodeError <: ValueError
weakrefthat you are trying to access got garbage collected.
10.4. IO is upset
BufferError <: Exception
EOFError <: Exceptionunexpected end of file.
10.5. Python is upset
IndentationError <: SyntaxError
TabError <: IndentationError
SystemErrorinternal python error.
MemoryErrorPython ran out of RAM.
NameError; an unbound identifier was used
UnboundLocalError <: NameError
RecursionError, infinite descending chain of function calls.
10.6. Control flow
These are exceptions that are not errors, but instead are used to control the program flow during normal operation.
GeneratorExit <: BaseException
StopIteration <: Exception
AssertionError <: Exceptionwhen an
SystemExit <: BaseException, this is uncatchable (ie
except SystemExit:block will never run). This happens when you call
KeyboardInterruptwhen you whack
ctrl+con your program.