Runtime Assertions

All of the runtime assertion checks do a lot of work to try to provide really nice feedback. They will try to provide a lot of information about what exactly they did when running the student code, so that the error is clearly in the students’ code, and not the instructor code.

unit_test(function_name, *tests)
unit_test(function_name)
unit_test(function_name, *tests, score, partial_credit, else_message, assert_function, context)

The unit_test function provides a wrapper around assert_equal and other assertions to make it easier to group together a bunch of tests. The output of the tests will be provided in a concise, tabular form.

The format of the tests is a little confusing. You can provide any number of them, as input/output pairs. The input is the arguments to the function, and must be iterable. The output is the expected result, and can be any type. So often you will end up with somewhat messy nested iterables.

# A function that consumes two numbers and returns their sum
unit_test('add', ([1, 2], 6), ([0, 0], 0))
# A function that consumes a LIST of numbers and returns their sum
unit_test('add', ([[1, 2, 3, 4]], 10), ([[0, 0, 0, 0]], 0))

Alternatively, you can use unit_test as a context manager and then use any assert_* statements inside.

with unit_test('add'):
    assert_equal(call('add', 2, 3), 5)
    assert_equal(call('add', 3, 2), 5)
check_dataclass_instance(value, dataclass) → Feedback

Checks that the value is an instance of the given dataclass. This is a little more flexible than assert_is_instance() because it will check dataclass characteristics and use appropriate terminology.

Also convenient because you can create a dataclass in the instructor namespace and provide it as an argument.

from dataclasses import dataclass

@dataclass
class Point:
    x: int
    y: int

# Instructions said to make instance in `my_point` variable
check_dataclass_instance(evaluate('my_point'), Point)
assert_equal(left, right) → Feedback
assert_equal(left, right, exact_strings=False, delta=DELTA)

The basic unit test, checks whether the left and right values are equal (uses ==, not is).

The exact_strings parameter is whether to require that strings be exactly the same, for each character. If False (the default), then strings will be normalized (lowercased, trailing decimals chopped, punctuation removed, lines are flattened, and all characters are sorted).

The delta controls how float values are compared - how close the values must be. If delta is None, then the default delta will be used (.001).

# This passes
assert_equal(5, 5)
# This fails
assert_equal("five", 5)
# This passes
assert_equal([1,2,3], [1,2] + [3])

# Normally strings are matched imprecisely
assert_equal("Hello, World!", "hello world")
# Disable this for more precise matching
assert_equal("Hello, World!", "hello world", exact_strings=True)

# Floats are matched imprecisely
assert_equal(1.0, 1.0001)
# You might need a more precise delta
assert_equal(1.0, 1.0001, delta=.0000001)

Typically, you will use this in conjunction with call() to check students’ functions.

assert_equal(call("add", 2, 3), 5)

You do not have to call the function first, and you can even call it on either side if you want to do more complicated comparisons.

# This will work
assert_equal(5, call("add", 2, 3))
# And so will this
assert_equal(call("add", 2, 3), call("add", 3, 2))

In the case of more complicated calls, the feedback will be a little more complicated.

Student code failed instructor test.
I ran the code:
    add(2, 3)
    add(3, 2)
The value of the first result was:
    -1
The value of the second result was:
    1
But I expected the first result to be equal to the second result

Note that you can override the “I ran the code” part by providing a context parameter. You can also override the explanation and the assertion parts.

Feedback Function: Failed Instructor Test
  • Label: assert_equal
  • Response: Student code failed instructor test. {context_message}{assertion_message}{explanation}
assert_not_equal(left, right) → Feedback
assert_not_equal(left, right, exact_strings=False, delta=DELTA)

Similar to assert_equal(), but checks that the left and right values are not equal (uses !=, not is not). See that function for more details.

Feedback Function: Failed Instructor Test
  • Label: assert_not_equal
  • Justification: Left and right were equal
  • Response: Student code failed instructor test. {context_message}{assertion_message}{explanation}
assert_less(left, right) → Feedback

Checks that the left value is less than the right value (uses <). See assert_equal() for more details.

Feedback Function: Failed Instructor Test
  • Label: assert_less
  • Justification: Left is not less than right
  • Response: Student code failed instructor test. {context_message}{assertion_message}{explanation}
assert_less_equal(left, right) → Feedback

Checks that the left value is less than the right value (uses <). See assert_equal() for more details.

Feedback Function: Failed Instructor Test
  • Label: assert_less_equal
  • Justification: Left is not less than or equal to the right
  • Response: Student code failed instructor test. {context_message}{assertion_message}{explanation}
assert_greater(left, right) → Feedback

Checks that the left value is greater than the right value (uses >). See assert_equal() for more details.

Feedback Function: Failed Instructor Test
  • Label: assert_greater
  • Justification: Left is not greater than right
  • Response: Student code failed instructor test. {context_message}{assertion_message}{explanation}
assert_greater_equal(left, right) → Feedback

Checks that the left value is greater than or equal to the right value (uses >=). See assert_equal() for more details.

Feedback Function: Failed Instructor Test
  • Label: assert_greater_equal
  • Justification: Left is not greater than or equal to the right
  • Response: Student code failed instructor test. {context_message}{assertion_message}{explanation}
assert_in(item, container) → Feedback

Checks that the item is in the container (uses in/not in). See assert_equal() for more details.

Correctly handles asymmetry of function calls to give proper messages.

# "I expected the result to be in the: [1, 2, 3]"
assert_in(call('guess_number'), [1, 2, 3])

# "I expected the result to contain: 7"
assert_in(7, call('provide_options'))
Feedback Function: Failed Instructor Test
  • Label: assert_in
  • Justification: Needle not in haystack
  • Response: Student code failed instructor test. {context_message}{assertion_message}{explanation}
assert_not_in(item, container) → Feedback

Checks that the item is not in the container (uses in/not in).

See assert_not_in() for more details.

assert_contains_subset(needles, haystack) → Feedback

Checks that the needles are all in the haystacks (uses in/not in). This is a little different from assert_in() because it will check that all of the needles are in the haystacks, not just one.

# "I expected the result to contain: [1, 2, 3]"
assert_contains_subset([1, 2, 3], call('provide_options'))

# "I expected the result to be in: [4, 5, 6]"
assert_contains_subset(call('guess_numbers'), [4, 5, 6])
Feedback Function: Failed Instructor Test
  • Label: assert_contains_subset
  • Justification: Needles not in haystack
  • Response: Student code failed instructor test. {context_message}{assertion_message}{explanation}
assert_not_contains_subset(needles, haystack) → Feedback

Checks that the needles are not all in the haystacks (uses in/not in). See assert_contains_subset() for more details.

Feedback Function: Failed Instructor Test
  • Label: assert_not_contains_subset
  • Justification: Needles in haystack
  • Response: Student code failed instructor test. {context_message}{assertion_message}{explanation}
assert_is(left, right) → Feedback

Checks that the left value is the same as the right value (uses is). In other words, the two objects MUST be strictly equal, the actaul identical object and not just the same contents. This is actually unlikely to occur in most situations, given the way that Pedal executes student code in its own context, but is still sometimes necessary. See assert_equal() for more details.

Feedback Function: Failed Instructor Test
  • Label: assert_is
  • Justification: Left is not identical to right
  • Response: Student code failed instructor test. {context_message}{assertion_message}{explanation}
assert_is_not(left, right) → Feedback

Checks that the left value is not the same as the right value (uses is not). See assert_is() for more details.

Feedback Function: Failed Instructor Test
  • Label: assert_is_not
  • Justification: Left is identical to right
  • Response: Student code failed instructor test. {context_message}{assertion_message}{explanation}
assert_is_none(value) → Feedback

Checks that the value is None (uses is None). See assert_equal() for more details.

Feedback Function: Failed Instructor Test
  • Label: assert_is_none
  • Justification: Value is not None
  • Response: Student code failed instructor test. {context_message}{assertion_message}{explanation}
assert_is_not_none(value) → Feedback

Checks that the value is not None (uses is not None). See assert_equal() for more details.

Feedback Function: Failed Instructor Test
  • Label: assert_is_not_none
  • Justification: Value is None
  • Response: Student code failed instructor test. {context_message}{assertion_message}{explanation}
assert_is_dataclass(value) → Feedback

Checks that the value is a dataclass. Note that this is not a type check, but actually checking if the given object has the special __dataclass_fields__ attribute. This is a little bit of a hack, but it is the best way to check if a value is actually a dataclass (or at least, it’s currently how is_dataclass itself checks). See assert_equal() for more details.

Feedback Function: Failed Instructor Test
  • Label: assert_is_dataclass
  • Justification: Value does not evaluate to a dataclass
  • Response: Student code failed instructor test. {context_message}{assertion_message}{explanation}
assert_is_not_dataclass(value) → Feedback

Checks that the value is not a dataclass. Note that this is not a type check, but actually checking if the given object has the special __dataclass_fields__ attribute. This is a little bit of a hack, but it is the best way to check if a value is actually a dataclass (or at least, it’s currently how is_dataclass itself checks). See assert_equal() for more details.

Feedback Function: Failed Instructor Test
  • Label: assert_is_not_dataclass
  • Justification: Value does not evaluate to a dataclass
  • Response: Student code failed instructor test. {context_message}{assertion_message}{explanation}
assert_true(value) → Feedback

Checks that the value is (truthy) True. In other words, the result is converted to a boolean using bool`, and then checked to see if it is ``True. See assert_equal() for more details.

Feedback Function: Failed Instructor Test
  • Label: assert_true
  • Justification: Value does not evaluate to true
  • Response: Student code failed instructor test. {context_message}{assertion_message}{explanation}
assert_false(value) → Feedback

Checks that the value is (falsey) False. In other words, the result is converted to a boolean using bool, and then checked to see if it is False. See assert_equal() for more details.

Feedback Function: Failed Instructor Test
  • Label: assert_false
  • Justification: Value does not evaluate to false
  • Response: Student code failed instructor test. {context_message}{assertion_message}{explanation}
assert_length_equal(sequence, length) → Feedback

Checks that the length of the sequence is equal to the given length. This function exists because the built-in len actually CANNOT be used in conjunction with call. The len function in CPython must return an integer, or it will segfault. So to get around this, we have provided our version of len that will avoid this problem. Most of the time you won’t even notice that you are using our version of len, but if you do, then you can use this assert_length_equal function and call to check the length of its result more directly.

See assert_equal() for more details on assertions, or This Post for more information about the len problem.

Feedback Function: Failed Instructor Test
  • Label: assert_length_equal
  • Justification: Sequence does not have length
  • Response: Student code failed instructor test. {context_message}{assertion_message}{explanation}
assert_length_not_equal(sequence, length) → Feedback

Checks that the length of the sequence is not equal to the given length. Basically equivalent to assert_not_equal(len(sequence), length), but will correctly provide context since len cannot be used with call. See assert_length_equal() for more details.

Feedback Function: Failed Instructor Test
  • Label: assert_length_not_equal
  • Justification: Sequence has length
  • Response: Student code failed instructor test. {context_message}{assertion_message}{explanation}
assert_length_less(sequence, length) → Feedback

Checks that the length of the sequence is less than the given length. Basically equivalent to assert_less(len(sequence), length), but will correctly provide context since len cannot be used with call. See assert_length_equal() for more details.

Feedback Function: Failed Instructor Test
  • Label: assert_length_less
  • Justification: Sequence length is less than
  • Response: Student code failed instructor test. {context_message}{assertion_message}{explanation}
assert_length_less_equal(sequence, length) → Feedback

Checks that the length of the sequence is less than or equal to the given length. Basically equivalent to assert_less_equal(len(sequence), length), but will correctly provide context since len cannot be used with call. See assert_length_equal() for more details.

Feedback Function: Failed Instructor Test
  • Label: assert_length_less_equal
  • Justification: Sequence length is less than or equal to
  • Response: Student code failed instructor test. {context_message}{assertion_message}{explanation}
assert_length_greater(sequence, length) → Feedback

Checks that the length of the sequence is greater than the given length. Basically equivalent to assert_greater(len(sequence), length), but will correctly provide context since len cannot be used with call. See assert_length_equal() for more details.

Feedback Function: Failed Instructor Test
  • Label: assert_length_greater
  • Justification: Sequence length is greater than
  • Response: Student code failed instructor test. {context_message}{assertion_message}{explanation}
assert_length_greater_equal(sequence, length) → Feedback

Checks that the length of the sequence is greater than or equal to the given length. Basically equivalent to assert_greater_equal(len(sequence), length), but will correctly provide context since len cannot be used with call. See assert_length_equal() for more details.

Feedback Function: Failed Instructor Test
  • Label: assert_length_greater_equal
  • Justification: Sequence length is greater than or equal to
  • Response: Student code failed instructor test. {context_message}{assertion_message}{explanation}
assert_is_instance(obj, cls) → Feedback

Checks that the object is an instance of the given class using isinstance. If either int or float is provided, then they will allow the other to be used. To avoid this behavior, simply wrap the type in a tuple or list. Note that this function does not support generics. See assert_equal() for more details.

# This will pass
assert_is_instance(5, int)
# This will pass
assert_is_instance(5, float)
# This will fail
assert_is_instance(5, [float])
Feedback Function: Failed Instructor Test
  • Label: assert_is_instance
  • Justification: Object is not an instance of class
  • Response: Student code failed instructor test. {context_message}{assertion_message}{explanation}
assert_is_not_instance(obj, cls) → Feedback

Checks that the object is not an instance of the given class using isinstance. If either int or float is provided, then they will allow the other to be used. To avoid this behavior, simply wrap the type in a tuple or list. Note that this function does not support generics. See assert_equal() for more details.

# This will fail
assert_is_not_instance(5, int)
# This will fail
assert_is_not_instance(5, float)
# This will pass
assert_is_not_instance(5, [float])
Feedback Function: Failed Instructor Test
  • Label: assert_not_is_instance
  • Justification: Object is an instance of class
  • Response: Student code failed instructor test. {context_message}{assertion_message}{explanation}
assert_type(value, expected_type) → Feedback

Checks that the value is of the given type, more flexibly than isinstance. Basically, this uses Pedal’s type system, which allows for types to be provided as type objects (e.g., int, str), with generics (list[int]), as strings ("int", "str", "list[int]"), and a few other ways. For more information about Pedal’s type system, see Pedal’s Type System. For more about assertions, see assert_equal() for more details.

# This will pass
assert_type(5, int)
# This will pass
assert_type(["Hello", "World"], list[str])
# This will fail
assert_type([1, 2], list[str])
# This will pass
assert_type([1, 2], "list[int]")
Feedback Function: Failed Instructor Test
  • Label: assert_type
  • Justification: Value is not of type
  • Response: Student code failed instructor test. {context_message}{assertion_message}{explanation}
assert_not_type(value, expected_type) → Feedback

Checks that the value is not of the given type, more flexibly than isinstance. See assert_type() for more details.

# This will fail
assert_not_type(5, int)
# This will fail
assert_not_type(["Hello", "World"], list[str])
# This will pass
assert_not_type([1, 2], list[str])
# This will fail
assert_not_type([1, 2], "list[int]")
Feedback Function: Failed Instructor Test
  • Label: assert_not_type
  • Justification: Value is of type
  • Response: Student code failed instructor test. {context_message}{assertion_message}{explanation}
assert_regex(pattern, text) → Feedback

Checks that the regex matches the string, checking that re.search is not None. See assert_equal() for more details.

Feedback Function: Failed Instructor Test
  • Label: assert_regex
  • Justification: Regex does not match text
  • Response: Student code failed instructor test. {context_message}{assertion_message}{explanation}
assert_not_regex(pattern, text) → Feedback

Checks that the regex does not match the string, checking that re.search is None. See assert_equal() for more details.

Feedback Function: Failed Instructor Test
  • Label: assert_not_regex
  • Justification: Regex matches text
  • Response: Student code failed instructor test. {context_message}{assertion_message}{explanation}
assert_output(execution, text) → Feedback
assert_output(execution, text, exact_strings=False) → Feedback

Determine if the execution outputs text. Uses the == operator to do the final comparison. In this case, you can think of the output as a single string with newlines, as opposed to a list of strings (i.e., it is retrieved with get_raw_output()). See assert_equal() for more details.

You can use the student variable from the Sandbox to get all of the output.

# Accepts "Hello world"
assert_output(student, "Hello, World!")
# Only accepts "Hello, World!"
assert_output(student, "Hello, World!", exact_strings=True)

Otherwise, the first argument can be a call() or run() result. This correctly checks that the desired text is in the output as a result of the given function call (and not for some other reason).

assert_output(call('main'), "Hello world!")

If the exact_strings parameter is set to be False, then output is first normalized following this strategy:

  • Make strings lowercase

  • Remove all punctuation characters

  • Split the string by newlines into a list

  • Split each individual line by spaces (aka into words)

  • Remove all empty lines

  • Sorts the lines by default order

So the default check will be fairly generous about checking output; as long as all the lines are there (in whatever order), ignoring punctuation and case, the text will be found.

Feedback Function: Failed Instructor Test
  • Label: assert_output
  • Justification: Did not print the output
  • Response: Student code failed instructor test. {context_message}{assertion_message}{explanation}
assert_not_output(execution, text) → Feedback
assert_not_output(execution, text, exact_strings=False) → Feedback

Determine if the execution does not output text. See assert_output() for more details.

Feedback Function: Failed Instructor Test
  • Label: assert_not_output
  • Justification: Printed the output
  • Response: Student code failed instructor test. {context_message}{assertion_message}{explanation}
assert_output_contains(execution, text) → Feedback
assert_output_contains(execution, text, exact_strings=False) → Feedback

Determine if the execution outputs text. Uses the in operator to do the final comparison. The normalization for exact_strings is more basic than for assert_output(), since it will only do the lowercase conversion (punctuation is not removed).

See assert_output() for more details.

Feedback Function: Failed Instructor Test
  • Label: assert_output_contains
  • Justification: Did not contain the printed output
  • Response: Student code failed instructor test. {context_message}{assertion_message}{explanation}
assert_not_output_contains(execution, text) → Feedback
assert_not_output_contains(execution, text, exact_strings=False) → Feedback

Determine if the execution does not output text. See assert_output_contains() for more details.

Feedback Function: Failed Instructor Test
  • Label: assert_not_output_contains
  • Justification: Contained the printed output
  • Response: Student code failed instructor test. {context_message}{assertion_message}{explanation}
assert_has_attr(obj, attr) → Feedback

Determine if the obj has the attribute attr using the hasattr function. See assert_equal() for more details.

Feedback Function: Failed Instructor Test
  • Label: assert_has_attr
  • Justification: Contained the attribute
  • Response: Student code failed instructor test. {context_message}{assertion_message}{explanation}
assert_has_variable(sandbox, variable_name) → Feedback

Determine if the sandbox has the variable variable_name. This actually does check that the variable is defined at runtime, and not just that it is defined in the code. If you wish to check the top-level module of the student’s code, then you can use the student variable from the Sandbox.

assert_has_variable(student, "x")
Feedback Function: Failed Instructor Test
  • Label: assert_has_variable
  • Response: Student code failed instructor test. {context_message}{assertion_message}{explanation}
assert_has_function(sandbox, function_name) → Feedback

Determine if the sandbox has the function function_name, and that the function is callable. If you wish to check the top-level module of the student’s code, then you can use the student variable from the Sandbox.

assert_has_function(student, "add")
Feedback Function: Failed Instructor Test
  • Label: assert_has_function
  • Response: Student code failed instructor test. {context_message}{assertion_message}{explanation}
ensure_coverage() → Feedback
ensure_coverage(at_least=.5) → Feedback

Verifies that the most recent executed and traced student code has at least the given percentage of coverage (defaulting to half of the code covered). The ratio is calculated by dividing the number of lines executed by the total number of non-blank, executable lines. The coverage checker does not take into account certain line such as docstrings or comments.

# 50% coverage
ensure_coverage()
# 90% coverage
ensure_coverage(.9)
# 100% coverage
ensure_coverage(1)
Feedback Function: You Must Test Your Code
  • Label: ensure_coverage
  • Response: Your code coverage is not adequate. You must cover at least {at_least_message}% of your code to receive feedback. So far, you have only covered {coverage_message}%.
ensure_called_uniquely(function_name) → Feedback
ensure_called_uniquely(function_name, at_least=1) → Feedback
ensure_called_uniquely(function_name, ignore=None, why_ignored='') → Feedback

Verifies that the most recent executed and traced student code has at_least called the given function uniquely that number of times. In other words, it prevents students from calling the same function repeatedly WITHOUT changing the arguments. Students often try to bypass simpler checks (e.g., call assert_equal three times) by calling the same function multiple times with the same arguments. This at least prevents the most obvious cheating, although you should still think about additional ways to check their tests.

The ignore (set[tuple])) and why_ignored (str) parameters are used to ignore certain sets of arguments (the tuple are the arguments). This is useful if you have provided them with some tests, and you do not want those tests to count towards their total. The why_ignored is used to provide feedback to the student about why their test was ignored. For example, to say something like ````

# Student must call the "add" function at least once
ensure_called_uniquely("add")

# Student must call the "add" function at least three times
ensure_called_uniquely("add", 3)

# We provide them with a few example tests
ensure_called_uniquely("add", ignore={(1, 2), (2, 3)},
                       why_ignored=" because it was provided to you as an example.")
Feedback Function: You Must Test Your Code
  • Label: ensure_called_uniquely
  • Response: You have not tested the function {function_name} enough. You should test it at least {at_least} times. Each time you test it, you should be using a new set of arguments. So far, you have called it {total_calls} times in total and {unique_calls} times distinctively{instructor_ignore_message}.
ensure_function_callable(name) → Feedback

Verifies that the most recent executed and traced student code has a function with the given name that is callable. This is often a little more intuitive to use than assert_has_function().

# Student must define a function called "add"
ensure_function_callable("add")
  • Feedback Function: Function Not Available
    • Label: function_not_available
    • Response: You may have defined `{name_message}`, but it was not available to be called in the top-level scope. Perhaps you mistakenly defined it inside another function or scope?
  • Feedback Function: Name Is Not a Function
    • Label: name_is_not_a_function
    • Response: You defined `{name_message}`, but did not define it as a function.

Equivalent Names

Despite the fact that PEP8 is pretty clear you should use snake_case instead of camelCase, some folks are still just more comfortable with writing assertEqual. Therefore, we provide aliases of all the functions:

Original

Alias

assert_equal

assertEqual

assert_not_equal

assertNotEqual

assert_less

assertLess

assert_less_equal

assertLessEqual

assert_greater

assertGreater

assert_greater_equal

assertGreaterEqual

assert_length_equal

assertLengthEqual

assert_length_not_equal

assertLengthNotEqual

assert_length_less

assertLengthLess

assert_length_less_equal

assertLengthLessEqual

assert_length_greater

assertLengthGreater

assert_length_greater_equal

assertLengthGreaterEqual

assert_in

assertIn

assert_not_in

assertNotIn

assert_is

assertIs

assert_is_not

assertIsNot

assert_is_none

assertIsNone

assert_is_not_none

assertIsNotNone

assert_true

assertTrue

assert_false

assertFalse

assert_is_instance

assertIsInstance

assert_not_is_instance

assertIsNotInstance

assert_equal

assertAlmostEqual

assert_not_equal

assertNotAlmostEqual

assert_regex

assertRegex

assert_not_regex

assertNotRegex

assert_prints

assertPrints

assert_output

assertOutput

assert_not_output

assertNotOutput

assert_output_contains

assertOutputContains

assert_not_output_contains

assertNotOutputContains

assert_has_attr

assertHasAttr

assert_has_function

assertHasFunction

assert_has_variable

assertHasVariable

assert_type

assertType

assert_not_type

assertNotType