Tests silently consume AssertionError in Python #13

Activity

1. kazk

   transferred this issue fromcodewars/runneron Mar 13, 2021
2. kazk commented on Mar 13, 2021

   

   kazk

   on Mar 13, 2021

   Member

   More actions

   The first one works because it doesn't involve this test framework at all. This test framework wasn't made to work with AssertionError (we did try to retrofit it, but it's not fully functional).

   https://github.com/codewars/python-test-framework/blob/v0.1.0/codewars_test/test_framework.py is the deployed version.
3. kazk

   added

   bugSomething isn't workingSomething isn't working

   on Mar 13, 2021
4. namedots commented on Jul 26, 2021

   

   namedots

   on Jul 26, 2021

   Last edited by namedots

   More actions

   Looks to me like it's supposed to catch AssertionException, which is defined at the top of the module and raised in expect but is not caught anywhere. Presumably an auto-complete failure.

   from https://docs.codewars.com/languages/python/authoring/#calling-assertions :

   One somewhat distinctive feature of Python assertions is that by default, a failed assertion does not cause a test case to fail early. It can lead to unexpected crashes when an actual value had already been asserted to be invalid, but the execution of the current test case was not stopped and following assertions continue to refer to it. This behavior can be overridden by passing the allow_raise=True argument to the assertion functions that support it.

   allow_raise does not actually have this behaviour, but it WOULD if AssertionError was changed to AssertionException here:

   python-test-framework/codewars_test/test_framework.py

   Lines 112 to 113 in 072ae38

   	except AssertionError as e:
   	display('FAILED', str(e))

   should likely be:

   except AssertionException:
       pass

   and allow_raise should actually be named abort_on_failure or some such. Its current behaviour does not match its name at all, nor does it match the above docs:

   python-test-framework/codewars_test/test_framework.py

   Lines 23 to 24 in 072ae38

   	if allow_raise:
   	raise AssertException(message)

   (if it says ALLOW raise, then it should not be raising itself, it should ... ALLOW someone else to do it, except, that's probably not the intention of the flag, rather that seems to be aborting early -- that's the only reason why expect would raise - to abort the function - it's "allow", but from the wrong perspective, it's allowing expect to raise, and it does so to abort, and then there needs to be a matching catch in it -- which is this except AssertionError typo)