|  | exception handling in complex Python programs | | |
| | | eliben |  |
|  Posted: Tue Aug 19, 2008 5:19 pm Post subject: exception handling in complex Python programs |  |
| |  | |
Python provides a quite good and feature-complete exception handling
mechanism for its programmers. This is good. But exceptions, like any
complex construct, are difficult to use correctly, especially as
programs get large.
Most of the issues of exceptions are not specific to Python, but I
sometimes feel that Python makes them more acute because of the free-n-
easy manner in which it employs exceptions for its own uses and allows
users to do the same.
Now, what do I mean more specifically... When a program starts growing
large, I find myself a bit scared of all the exceptions that might be
thrown: Python's exceptions as a result of runtime-detection of errors
(Python's dynamic typing also comes into play here), exceptions from
libraries used by the code, and exceptions from my lower-level
classes.
Python doesn't allow to specify which exceptions are thrown (C++'s
feature adding 'throw' after a function/method declaration specifying
the exceptions that can be thrown), and this leaves me at loss - what
should be caught and where ? Which errors should be left to
propagate ?
I've tried looking around the Python blogosphere, but there doesn't
seem to be much concern with this topic.
Apologies for the not-too-coherent post, but I suspect you feel the
pain too and can understand my meaning.
Eli
P.S. There's a common case where a method is passed a filename, to do
something with a file (say, read data). Should the method catch the
errors possibly thrown by open(), or leave it to the caller ?
P.P.S. There's a great post on conditions (Common Lisp's exceptions)
here:
LINK
Not really CL specific, and can apply to Python's exceptions. |
| |
| | | Chris Mellon | |
| | Posted: Tue Aug 19, 2008 5:19 pm Post subject: Re: exception handling in complex Python programs | |
| | | |
On Tue, Aug 19, 2008 at 12:19 PM, eliben <eliben@gmail.com> wrote:
| Quote: | Python provides a quite good and feature-complete exception handling
mechanism for its programmers. This is good. But exceptions, like any
complex construct, are difficult to use correctly, especially as
programs get large.
Most of the issues of exceptions are not specific to Python, but I
sometimes feel that Python makes them more acute because of the free-n-
easy manner in which it employs exceptions for its own uses and allows
users to do the same.
|
Lots of people seem to have this fear. They treat exceptions like they
would treat error codes, trying to handle any possible case around any
particular call.
This is the wrong thing to do, and it only leads to more fragile code.
There are only 2 reasonable things to do with an exception:
1) handle it, by which I mean catch the exception knowing what error
condition it signifies, and take an appropriate action to correct the
error and
2) pass it up so something else has a chance at it.
Catching an exception when you don't know exactly what to do to fix it
is an error. At best, it will make debugging a program harder (because
you're losing context information about the error) and at worst it
adds bugs to your program. The way Javas checked exceptions encourage
empty or otherwise useless exception handlers is a major problem with
them.
There's some fear about presenting exceptions to the end user. That's
a user interface issues, not a software quality or engineering issue,
and it's resolvable with top-level handlers that log tracebacks
somewhere a user can't see them if desired.
| Quote: | Now, what do I mean more specifically... When a program starts growing
large, I find myself a bit scared of all the exceptions that might be
thrown: Python's exceptions as a result of runtime-detection of errors
(Python's dynamic typing also comes into play here), exceptions from
libraries used by the code, and exceptions from my lower-level
classes.
Python doesn't allow to specify which exceptions are thrown (C++'s
feature adding 'throw' after a function/method declaration specifying
the exceptions that can be thrown), and this leaves me at loss - what
should be caught and where ? Which errors should be left to
propagate ?
|
You should catch anything that you can correct. If you don't have a
specific answer for a specific exception, don't catch it.
| Quote: | I've tried looking around the Python blogosphere, but there doesn't
seem to be much concern with this topic.
Apologies for the not-too-coherent post, but I suspect you feel the
pain too and can understand my meaning.
Eli
P.S. There's a common case where a method is passed a filename, to do
something with a file (say, read data). Should the method catch the
errors possibly thrown by open(), or leave it to the caller ?
|
Same rules apply. The only sort-of exception (no pun intended) is that
sometimes you want to re-raise as a different type of exception. Make
sure that you preserve all of the original information (including the
original traceback) if you do this.
| Quote: | P.P.S. There's a great post on conditions (Common Lisp's exceptions)
here:
LINK
Not really CL specific, and can apply to Python's exceptions.
--
LINK
|
|
| |
| | | Fredrik Lundh | |
| | Posted: Tue Aug 19, 2008 5:19 pm Post subject: Re: exception handling in complex Python programs | |
Rafe wrote:
| Quote: | Again, this is probably too simple to help, but the only way to ignore
certain types of exceptions, as far as I know, is to catch them and
pass.
e.g. this ignores type errors...
try:
somethingBad()
except TypeError, err:
pass
except Exception, err:
raise TypeError(err)
|
so what kind of code are you writing where *type errors* are not
considered programming errors? (catching them and proceeding is one
thing, but catching them and ignoring them?)
I'd be really worried if I found that in a piece of source code I had to
maintain.
</F> |
| |
| | | Rafe | |
| | Posted: Tue Aug 19, 2008 5:31 pm Post subject: Re: exception handling in complex Python programs | |
| | | |
On Aug 20, 12:19 am, eliben <eli...@gmail.com> wrote:
| Quote: | Python provides a quite good and feature-complete exception handling
mechanism for its programmers. This is good. But exceptions, like any
complex construct, are difficult to use correctly, especially as
programs get large.
Most of the issues of exceptions are not specific to Python, but I
sometimes feel that Python makes them more acute because of the free-n-
easy manner in which it employs exceptions for its own uses and allows
users to do the same.
Now, what do I mean more specifically... When a program starts growing
large, I find myself a bit scared of all the exceptions that might be
thrown: Python's exceptions as a result of runtime-detection of errors
(Python's dynamic typing also comes into play here), exceptions from
libraries used by the code, and exceptions from my lower-level
classes.
Python doesn't allow to specify which exceptions are thrown (C++'s
feature adding 'throw' after a function/method declaration specifying
the exceptions that can be thrown), and this leaves me at loss - what
should be caught and where ? Which errors should be left to
propagate ?
I've tried looking around the Python blogosphere, but there doesn't
seem to be much concern with this topic.
Apologies for the not-too-coherent post, but I suspect you feel the
pain too and can understand my meaning.
Eli
P.S. There's a common case where a method is passed a filename, to do
something with a file (say, read data). Should the method catch the
errors possibly thrown by open(), or leave it to the caller ?
P.P.S. There's a great post on conditions (Common Lisp's exceptions)
here:http://dlweinreb.wordpress.com/2008/03/24/what-conditions-exceptions-...
Not really CL specific, and can apply to Python's exceptions.
|
Maybe I am oversimplifying (and I am here to learn), but I catch all
exceptions which otherwise would be hard to understand as a user. In
other words, when a better error message is useful.
Again, this is probably too simple to help, but the only way to ignore
certain types of exceptions, as far as I know, is to catch them and
pass.
e.g. this ignores type errors...
try:
somethingBad()
except TypeError, err:
pass
except Exception, err:
raise TypeError(err)
I suppose you could write a decorator to do this if you want it at the
function level, but that seems a bit to broad. Shouldn't exceptions be
on a case-by-case basis to add protection and return information
exactly where it is needed?
- Rafe |
| |
| | | Steven D'Aprano | |
| | Posted: Tue Aug 19, 2008 9:12 pm Post subject: Re: exception handling in complex Python programs | |
On Tue, 19 Aug 2008 11:07:39 -0700, dbpokorny@gmail.com wrote:
| Quote: | def do_something(filename):
if not os.access(filename,os.R_OK):
return err(...)
f = open(filename)
...
|
You're running on a multitasking modern machine, right? What happens when
some other process deletes filename, or changes its permissions, in the
time after you check for access but before you actually open it?
This isn't just a theoretical risk. There's a whole class of errors and
security holes based on similar race conditions. I find it amusing that
you consider it "sloppy" to deal with errors raised when actually opening
a file, but then recommend a technique that has a well-known failure mode.
That's not to say that I never use such techniques myself. For quick and
dirty scripts, where I can tolerate the risk of some other process moving
a file behind my back, I've been known to do something similar.
--
Steven |
| |
| | | eliben | |
| | Posted: Wed Aug 20, 2008 5:24 am Post subject: Re: exception handling in complex Python programs | |
""" between file()
and open() in Python 2 and 3, a NameError is thrown with open() in
Python 3 and an IOError is thrown in the other three cases <bashes
head against keyboard>.
"""
This is *exactly* my concern with Python exceptions. You just never
know what can be thrown at you.
| Quote: | You want to look up Easier to Ask Forgivness than Permission (EAFP)
which is touted as the "canonical" error-handling paradigm for Python.
|
Any (semi)complete guides on this canonical paradigm online ? I've
only found some references in maillist discussions.
| Quote: | def do_something(filename):
if not os.access(filename,os.R_OK):
return err(...)
f = open(filename)
...
|
But does os.access cover absolutely all the errors that can happen
during open() ? What guarantees it, and how can I know without you
teaching me, just from the docs ? |
| |
| | | Marc 'BlackJack' Rintsch | |
| | Posted: Wed Aug 20, 2008 5:37 am Post subject: Re: exception handling in complex Python programs | |
On Tue, 19 Aug 2008 22:24:45 -0700, eliben wrote:
| Quote: | You want to look up Easier to Ask Forgivness than Permission (EAFP)
which is touted as the "canonical" error-handling paradigm for Python.
Any (semi)complete guides on this canonical paradigm online ? I've only
found some references in maillist discussions.
|
There's the glossary in the documentation:
LINK
Look under 'duck-typing', 'EAFP', and 'LBYL'.
Ciao,
Marc 'BlackJack' Rintsch |
| |
| | | Steven D'Aprano | |
| | Posted: Wed Aug 20, 2008 11:24 am Post subject: Re: exception handling in complex Python programs | |
| | | |
On Tue, 19 Aug 2008 22:24:45 -0700, eliben wrote:
| Quote: | """ between file()
and open() in Python 2 and 3, a NameError is thrown with open() in
Python 3 and an IOError is thrown in the other three cases <bashes head
against keyboard>.
"""
|
I'm curious about the claim that open() will raise NameError in Python3.
I find it hard to credit that claim, but if it is correct, what's the
justification for that?
| Quote: | This is *exactly* my concern with Python exceptions. You just never know
what can be thrown at you.
|
It's true that documentation of exceptions is relatively weak in Python.
And some functions can raise a bewildering array of exceptions. See for
example this thread where somebody notes that urllib2.urlopen() can raise
any of six different exceptions:
LINK
And I've had it raise socket.error, which makes seven. And the
documentation only mentions one of those exceptions.
However, as Gregory Smith describes, some of those seven exceptions are
subclasses of others, so it is possible to reduce it down to three cases
-- and arguably one of those cases (ValueError) is a bug that needs
fixing, not an exception that needs catching.
That's probably as bad as it gets in Python, at least for the standard
library. Most functions don't raise arbitrary exceptions for sensible
data, and if you pass non-sensible data then you should treat the
exception as a bug in your code and fix it.
--
Steven |
| |
| | | Bruno Desthuilliers | |
| | Posted: Wed Aug 20, 2008 2:40 pm Post subject: Re: exception handling in complex Python programs | |
| | | |
eliben a écrit :
| Quote: |
This is *exactly* my concern with Python exceptions. You just never
know what can be thrown at you.
|
This rarely happen to be a problem in real life. At least not in mine.
Exception that can be expected (ie : IOError when dealing with files)
are usually obvious and more or less documented - or easy to figure out
(like TypeError and ValueError when trying to build an int from an
arbitrary object, KeyError when working with dicts, AttributeError when
inspecting an object, etc) from concrete use.
IOW, it's usually easy to know which exceptions you're able to deal with
at the lower level.
Any other exception is either a programming error - which needs to be
fixed, not hidden - or nothing you can deal with at the lower level - in
which case just let it propagate until some other layer above deal with
it (eventually just logging the error, displaying a user-friendly
message, and crashing if nothing else is possible).
| Quote: | def do_something(filename):
if not os.access(filename,os.R_OK):
return err(...)
f = open(filename)
...
But does os.access cover absolutely all the errors that can happen
during open() ? What guarantees it, and how can I know without you
teaching me, just from the docs ?
|
The above code is a perfect antipattern. It's useless (if you can't
access the file, you'll get an IOError when trying to open it anyway),
it's wrong (things may change between the call to os.access and the call
to open), and it defeats the whole point of exception handling (by
returning some kind of error object instead of using exception handling). |
| |
| | | Steven D'Aprano | |
| | Posted: Wed Aug 20, 2008 3:59 pm Post subject: Re: exception handling in complex Python programs | |
| | | |
On Wed, 20 Aug 2008 09:23:22 -0700, dbpokorny@gmail.com wrote:
| Quote: | On Aug 19, 4:12Â pm, Steven D'Aprano <st...@REMOVE-THIS-
cybersource.com.au> wrote:
On Tue, 19 Aug 2008 11:07:39 -0700, dbpoko...@gmail.com wrote:
 def do_something(filename):
  if not os.access(filename,os.R_OK):
   return err(...)
  f = open(filename)
  ...
You're running on a multitasking modern machine, right? What happens
when some other process deletes filename, or changes its permissions,
in the time after you check for access but before you actually open it?
This is a good point - if you want to use the correct way of opening
files, and
you don't want to worry about tracking down exception types, then we can
probably
agree that the following is the simplest, easiest-to-remember way:
def do_something(filename):
try:
f = open(filename)
except:
handle exception
|
No, we don't agree that that is the correct way of opening files. Simple
it might be, but correct it is not.
If you're using Python 2.6 or greater, then you should be using a with
block to handle file opening.
And regardless of which version of Python, you shouldn't use a bare
except. It will mask exceptions you *don't* want to catch, including
programming errors, typos and keyboard interrupts.
| Quote: | Opening files is a special case where EAFP is the only correct solution
(AFAIK). I still liberally sprinkle LBYL-style "assert isinstance(...)"
|
Oh goodie. Another programmer who goes out of his way to make it hard for
other programmers, by destroying duck-typing.
BTW, assertions aren't meant for checking data, because assertions can be
turned off. Outside of test frameworks (e.g. unit tests), assertions are
meant for verifying program logic:
def foo(x):
# This is bad, because it can be turned off at runtime,
# destroying your argument checking.
assert isinstance(x, int)
# And it raises the wrong sort of exception.
# This is better (but not as good as duck-typing).
if not isinstance(x, int):
raise TypeError('x not an int')
# And it raises the right sort of error.
y = some_function(x)
# y should now be between -1 and 1.
assert -1 < y < 1
do_something_with(y)
| Quote: | and other similar assertions in routines. The point is that EAFP
conflicts with the interest of reporting errors as soon as possible
|
Not necessarily. Tell me how this conflicts with reporting errors as soon
as possible:
def do_something(filename):
try:
f = open(filename)
except IOError, e:
report_exception(e) # use a GUI, log to a file, whatever...
How could you report the exception any earlier than immediately?
--
Steven |
| |
| Page 1 of 5 .:. Goto page 1, 2, 3, 4, 5 Next | |
|
|
