Assertions
There are two major kinds of assertions: static and runtime.
Static
- prevent_function_call(function_name: str, at_most=0, root=None) Feedback
Confirms that the function is not called, generating Feedback if it is. This is a very simple check, simply looking for any
function_name
calls inside the code. It does not check if the function is actually called at runtime, or if that code is reachable!If
at_most
is provided, allows them to call the function at most that many times. If aroot
is provided, allows you to start the search from a specific node.Feedback Function: May Not Use Function# Student will not be able to call the function "hello_world" prevent_function_call("hello_world") # Students can't call "hello_world" more than twice prevent_function_call("hello_world", at_most=2) # Students can't call the function "add" prevent_function_call("add")
- Label: prevent_function_call
- Response: You used the function {name_message} on line {location.line}. You may not use that function{capacity}.
- ensure_function_call(function_name: str, at_least=0, root=None) Feedback
Confirms that the function IS called, generating feedback if it is not. This is a very simple check, simply looking for any
function_name
calls inside the code. It does not check if the function is actually called at runtime, or if that code is reachable!If
at_least
is provided, requires they call the function at least that many times. If aroot
is provided, allows you to start the search from a specific node.Feedback Function: Must Use Function# Student must call the function "hello_world" ensure_function_call("hello_world") # Students must call "hello_world" at least twice ensure_function_call("hello_world", at_least=2) # Students must call the function "add" ensure_function_call("add")
- Label: ensure_function_call
- Response: You must use the function {name_message}{capacity}.
- prevent_operation(operation: str, at_most=0, root=None) Feedback
- prevent_operator(operation: str, at_most=0, root=None) Feedback
Confirms that the operation is not in the code, generating Feedback if it is. This is a very simple check, simply looking for any
operation
calls inside the code. It does not check if the operation is actually called at runtime, or if that code is reachable!The
operation
should be provided as a string like"+"
or"<<"`
. All comparison, boolean, binary, and unary operations are supported.If
at_most
is provided, allows them to use the operation at most that many times. If aroot
is provided, allows you to start the search from a specific node.Feedback Function: May Not Use Operator# Student will not be able to use the addition operator prevent_operation("+") # Students can't use multiplication more than twice prevent_operation("*", at_most=2) # Students can't use the bitwise invert operator prevent_operation("~")
- Label: prevent_operation
- Response: You used the operator {name_message} on line {location.line}. You may not use that operator{capacity}.
- ensure_operation(operation: str, at_least=0, root=None) Feedback
- ensure_operator(operation: str, at_least=0, root=None) Feedback
Confirms that the operation IS in the code, generating feedback if it is not. This is a very simple check, simply looking for any
operation
calls inside the code. It does not check if the operation is actually called at runtime, or if that code is reachable!The
operation
should be provided as a string like"+"
or"<<"`
. All comparison, boolean, binary, and unary operations are supported.If
at_least
is provided, requires they use the operation at least that many times. If aroot
is provided, allows you to start the search from a specific node.Feedback Function: Must Use Operator# Student must use the addition operator ensure_operation("+") # Students must use multiplication at least twice ensure_operation("*", at_least=2) # Students must use the bitwise invert operator ensure_operation("~")
- Label: ensure_operation
- Response: You must use the operator {name_message}{capacity}.
- prevent_literal(literal: Any, at_most=0, root=None) Feedback
Confirms that the literal is not in the code, generating Feedback if it is. You can use literal any values including strings, integers, floats, booleans, lists, sets, tuples, dictionaries, and None.
If
at_most
is provided, allows them to use the literal at most that many times. If aroot
is provided, allows you to start the search from a specific node.Feedback Function: May Not Use Literal Value# Student will not be able to embed the literal 5 prevent_literal(5) # Students can't use 5 more than twice prevent_literal(5, at_most=2) # Students can't use the string "hello" prevent_literal("hello")
- Label: prevent_literal
- Response: You used the literal value {literal_message:python_expression} on line {location.line}. You may not use that value{capacity}.
- ensure_literal(literal: Any) Feedback
Confirms that the literal IS in the code, generating feedback if it is. You can use literal any values including strings, integers, floats, booleans, lists, sets, tuples, dictionaries, and None.
If
at_least
is provided, requires they use the literal at least that many times. If aroot
is provided, allows you to start the search from a specific node.Feedback Function: Must Use Literal Value# Student must use the literal 5 ensure_literal(5) # Students must use 5 at least twice ensure_literal(5, at_least=2) # Students must use the string "hello" ensure_literal("hello")
- Label: ensure_literal
- Response: You must use the literal value {literal_message}{capacity}.
- prevent_literal_type(literal_type: type, at_most=0, root=None) Feedback
Confirms that the literal type is not in the code, generating Feedback if it is. You can use the following literal types:
int
,float
,str
,bool
,list
,set
,tuple
,dict
, andNone
. Note that generics are not currently recognized!If
at_most
is provided, allows them to use the literal at most that many times. If aroot
is provided, allows you to start the search from a specific node.Feedback Function: May Not Use Type of Literal Value# Student will not be able to write any integer literals prevent_literal_type(int) # Students can't use integers more than twice prevent_literal_type(int, at_most=2) # Students can't use any strings prevent_literal_type(str)
- Label: prevent_literal_type
- Response: You used the literal value type {literal_type_message:python_expression} on line {location.line}. You may not use that type of value{capacity}.
- ensure_literal_type(literal_type: type, at_least=0, root=None) Feedback
Confirms that the literal type IS in the code, generating feedback if it is. You can use the following literal types:
int
,float
,str
,bool
,list
,set
,tuple
,dict
, andNone
. Note that generics are not currently recognized!If
at_least
is provided, requires they use the literal at least that many times. If aroot
is provided, allows you to start the search from a specific node.Feedback Function: Must Use Type of Literal Value# Student must use at least one integer literal ensure_literal_type(int) # Students must use integers at least twice ensure_literal_type(int, at_least=2) # Students must use at least one string ensure_literal_type(str)
- Label: ensure_literal_type
- Response: You must use a literal value of type {literal_type_message:python_expression}{capacity}.
- prevent_ast(ast_name: str, at_most=0, root=None) Feedback
Confirms that the AST type is not in the code, generating Feedback if it is. You should use the type of AST element you are looking for, provided as a string. Refer to the AST documentation for more information, or to GreenTreeSnakes.
If
at_most
is provided, allows them to use the literal at most that many times. If aroot
is provided, allows you to start the search from a specific node.Feedback Function: May Not Use Code# Student will not be able to write any integer literals prevent_ast("Num") # Students can't for loops more than twice prevent_ast("For", at_most=2) # Students can't use any function calls prevent_ast("Call") # Students can't use subscripts prevent_ast("Subscript")
- Label: prevent_ast
- Response: You used {name_message} on line {location.line}. You may not use that{capacity}.
- ensure_ast(ast_name: str, at_least=0, root=None) Feedback
Confirms that the AST type IS in the code, generating feedback if it is. You should use the type of AST element you are looking for, provided as a string. Refer to the AST documentation for more information, or to GreenTreeSnakes.
If
at_least
is provided, requires they use the literal at least that many times. If aroot
is provided, allows you to start the search from a specific node.Feedback Function: Must Use Code# Student must use at least one integer literal ensure_ast("Num") # Students must use for loops at least twice ensure_ast("For", at_least=2) # Students must use at least one function call ensure_ast("Call") # Students must use subscripts ensure_ast("Subscript")
- Label: ensure_ast
- Response: You must use {name_message}{capacity}.
- function_prints(function_name: str) Feedback
Confirms that the function prints something, generating feedback if it does not. This is a very simple check, simply looking for any
print
calls inside the function. It does not check if the function is actually called at runtime, or if that code is reachable!This is just a wrapper around
ensure_function_call()
.Feedback Function: Must Use Function# Student must have a function named "hello_world" with a print statement. function_prints("hello_world")
- Label: ensure_function_call
- Response: You must use the function {name_message}{capacity}.
- ensure_import(module_name: str, root=None) Feedback
Confirms that the module is imported, generating feedback if it is not. This is a very simple check, simply looking for any
import
orfrom
statements inside the code. It does not check if the module is actually used at runtime, or if that code is reachable! There is noat_least
parameter because it is assumed that the student will need to import the module at least once.Feedback Function: Must Import Module# Student must import the "math" module ensure_import("math")
- Label: ensure_import
- Response: You must import the module {name_message}.
- prevent_import(module_name: str, root=None) Feedback
Confirms that the module is not imported, generating feedback if it is. This is a very simple check, simply looking for any
import
orfrom
statements inside the code. It does not check if the module is actually used at runtime, or if that code is reachable!If a
root
is provided, allows you to start the search from a specific node.Feedback Function: May Not Import Module# Student must not import the "math" module prevent_import("math")
- Label: prevent_import
- Response: You may not import the module {name_message}.
- ensure_documented_functions(root=None) Feedback
Confirms that all functions are documented, generating feedback if they are not. Only proper docstrings are accepted, not comments. The docstring must be the first thing in the function, and must be a string literal. Students will be given a list of names of the functions that are not documented.
If a
root
is provided, allows you to start the search from a specific node.Feedback Function: Must Document Functionsensure_documented_functions()
- Label: ensure_documented_functions
- Response: You must document the following function{plural}: {names_message}.
- ensure_function(name: str, arity: int = None, parameters=None, returns=None, root=None, compliment=False) Feedback
Confirms that the function is defined, generating feedback if it is not. This checks a number of things about the function:
Whether or not the function exists, and whether it has been defined multiple times.
If found, then the function will move on to check the
arity
(number of parameters) if that was provided.If
parameters
is provided, then it will check that the parameters are of the correct types. The parameters should be a list of types, and can be given as strings or integers (generics are respected).If
returns
is provided, then it will check that the return type is correct. The return type should be a type, and can be given as a string or integer (generics are respected).
If
compliment
is a string, then it will use that to generate acompliment()
feedback if the function is found. Ifcompliment
isTrue
, then it will generate a default compliment.If
score
is provided, then it will usegive_partial()
to give partial credit.ensure_function('add', 2, [int, int], returns=int) ensure_function('double', parameters=[int]) ensure_function('clean_data', compliment="You defined the clean_data function!") ensure_function('move_forward', parameters=['Sprite', int], returns='Sprite')
- Feedback Function: Missing Function
- Label: missing_function
- Response: No function named {name_message} was found.
- Feedback Function: Duplicate Function Definition
- Label: duplicate_function_definition
- Response: The function {name_message} was defined multiple times, on lines {lines_message}.
- Feedback Function: Too Few Parameters
- Label: too_few_parameters
- Response: The function named {name_message} has fewer parameters ({found}) than expected ({expected}).
- Feedback Function: Too Many Parameters
- Label: too_many_parameters
- Response: The function named {name_message} has more parameters ({found}) than expected ({expected}).
- Feedback Function: Missing Parameter Type
- Label: missing_parameter_type
- Response: The function named {name_message} has a parameter named {parameter_message}, but that parameter does not have a type specified.
- Feedback Function: Invalid Parameter Type
- Label: invalid_parameter_type
- Response: The function named {name_message} has a parameter named {parameter_message} with an invalid type.
- Feedback Function: Wrong Parameter Type
- Label: wrong_parameter_type
- Response: The function named {name_message} has a parameter named {parameter_message} that is {actual_message}, but should be {expected_message}.
- Feedback Function: Wrong Return Type
- Label: wrong_return_type
- Response: The function named {name_message} was expected to return {expected_message}, but instead its header specifies that it returns {actual_message}.
- Feedback Function: Missing Return Type
- Label: missing_return_type
- Response: The function named {name_message} does not have a return type specified in its header.
- ensure_dataclass(example: dataclass, root=None, compliment=False) Feedback
- ensure_dataclass(name: str, fields: dict[str, Any], root=None, compliment=False) Feedback
Confirms that the dataclass is defined, generating feedback if it is not. This checks a number of things about the dataclass:
Whether or not the dataclass exists, and whether it has been defined multiple times.
If found, then the dataclass will move on to check the
fields
(names and types) if that was provided. The fields should be a dictionary of names to types, and can be given as strings or integers (generics are respected). Alternatively, you can provide the instructor version of the dataclass, and it will check that the student version has the same fields.
If
compliment
is a string, then it will use that to generate acompliment()
feedback if the dataclass is found. Ifcompliment
isTrue
, then it will generate a default compliment.If
score
is provided, then it will usegive_partial()
to give partial credit.ensure_dataclass('Person', {'name': str, 'age': int}) ensure_dataclass('Sprite', {'x': int, 'y': int}, compliment="You defined the Sprite dataclass!") @dataclass class Person: name: str age: int ensure_dataclass(Person)
- Feedback Function: Missing Dataclass
- Label: missing_dataclass
- Response: No dataclass named {name_message} was found.
- Feedback Function: Duplicate Dataclass Definition
- Label: duplicate_dataclass_definition
- Response: The dataclass {name_message} was defined multiple times, on lines {lines_message}.
- Feedback Function: Too Few Fields
- Label: too_few_fields
- Response: The dataclass named {name_message} has fewer fields ({found}) than expected ({expected}).
- Feedback Function: Too Many Fields
- Label: too_many_fields
- Response: The dataclass named {name_message} has more fields ({found}) than expected ({expected}).
- Feedback Function: Invalid Field Type
- Label: invalid_field_type
- Response: The dataclass named {name_message} has a field named {field_message} with an invalid type.
- Feedback Function: Unknown Field
- Label: unknown_field
- Response: The dataclass named {name_message} had a field named {field_message} but that field is not supposed to be there. Are you sure you got the name right?
- Feedback Function: Missing Field Type
- Label: missing_field_type
- Response: The dataclass named {name_message} has a field named {field_message}, but that field does not have a type specified.
- Feedback Function: Wrong Fields Type
- Label: wrong_fields_type
- Response: The dataclass named {name_message} has a field named {field_message} that is {actual_message}, but should be {expected_message}.
- Feedback Function: Name Is Not a Dataclass
- Label: name_is_not_a_dataclass
- Response: You defined `{name_message}`, but did not define it as a dataclass.
- Feedback Function: Dataclass Not Available
- Label: dataclass_not_available
- Response: You may have defined `{name_message}`, but it was not available to be called in the top-level scope. Perhaps you mistakenly defined it inside another dataclass or scope?
- Feedback Function: Dataclass Annotation Missing
- Label: missing_dataclass_annotation
- Response: You have defined {name_message}, but you did not include the {dc_annotation} annotation. Make sure you import and include the {dc_annotation} annotation directly before the class definition.
- ensure_prints_exactly(count: int) Feedback
Confirms that the student prints exactly the given number of times, generating feedback if they do not. This is just a wrapper around
ensure_function_call()
andprevent_function_call()
.ensure_prints_exactly(3)
- ensure_starting_code(code: str) Feedback
Confirms that the student’s code has the given code, generating feedback if it does not. This is most useful for providing some starting code that students are instructed to not mess with.
The given string of code will be parsed and checked with CAIT, so you can be a little flexible. It will not be a problem if the student introduces whitespace or comments, but changing variable names or something else will be detected.
Feedback Function: Don't Change Starting Codeensure_starting_code("import math") ensure_starting_code("def hello():\n print('hello')")
- Label: ensure_starting_code
- Response: You have changed or removed the starting code.
- prevent_embedded_answer(code: str) Feedback
Confirms that the student’s code does not have the given code, generating feedback if it does. This is most useful for checking to make sure that the student did not embed some exact literal solution.
The given string of code will be parsed and checked with CAIT, so you can be a little flexible. It will not be a problem if the student introduces whitespace or comments, but changing variable names or something else will be sufficient to beat this check.
Feedback Function: Don't Write Answer Directlyprevent_embedded_answer("print(3)") prevent_embedded_answer("def hello():\n print('hello')")
- Label: prevent_embedded_answer
- Response: You have embedded the answer directly in your code, instead of writing code to compute the answer.
- prevent_printing_functions(exception: str) Feedback
- prevent_printing_functions(exceptions: list[str]) Feedback
- prevent_printing_functions() Feedback
Confirms that the student’s code does not have any print statements in functions, generating feedback if it does. This is a common enough problem that it is worth checking for. You can provide a list of function names that are allowed to print (e.g.,
main
). You can also provide a single string.This does not actually check that
print
is called at runtime, and will not allow mundane uses of print (e.g., print-statement-debugging). It can also be defeated by things likesys.stdout.write
.Feedback Function: Do Not Print in Functionprevent_printing_functions() prevent_printing_functions('main') prevent_printing_functions(['main', 'log'])
- Label: prevent_printing_functions
- Response: The function {name} is printing on line {location.line:line}. However, that function is not supposed to print.
- ensure_functions_return() Feedback
- ensure_functions_return(exception: str) Feedback
- ensure_functions_return(exceptions: list[str]) Feedback
Confirms that the student’s functions return something, generating feedback if they do not. This is a common enough problem that it is worth checking for. You can provide a list of function names that do not have to return (e.g.,
main
). You can also provide a single string.This does not actually check that the function returns at runtime along every branch, and that the
return
statement is even reachable.Feedback Function: Must Return in Functionensure_functions_return() ensure_functions_return('main') ensure_functions_return(['main', 'save_to_file'])
- Label: ensure_functions_return
- Response: The function {name} is not returning. However, that function is supposed to have a return statement.
- only_printing_variables()
Confirms that the student’s code only prints variables, generating feedback if it does not. This is a narrow use case, to be sure.
Feedback Function: Print Variables, Not Valuesonly_printing_variables()
- Label: only_printing_variables
- Response: You printed something other than a variable on line {location.line:line}. Although that is not a normally an issue, we want you to practice printing variables in this problem.
- prevent_advanced_iteration(allow_while=False, allow_for=False, allow_function=None)
Confirms that the student’s code does not use any iteration, generating feedback if it does. By default, all forms of iteration that can be detected easily are blocked (
while
loops,for
loops). You can allow specific forms of iteration via the boolean flag parameters.By default, most built-in looping functions are blocked. You can override this list by providing a list of function names to allow. Otherwise, the following functions are blocked:
sum
,map
,filter
,any
,all
,reduce
,sorted
,reduce
,len
,max
,min
,getattr
,setattr
,eval
,exec
,iter
,next
.Surprisingly, does not block comprehensions of any kind. Unsurprisingly, does not block recursion.
Technically, this is just a wrapper around
prevent_function_call()
andprevent_ast()
.prevent_advanced_iteration() prevent_advanced_iteration(allow_while=True) prevent_advanced_iteration(allow_for=True) prevent_advanced_iteration(allow_function=['len', 'sum']) prevent_advanced_iteration(allow_function='sorted')
- files_not_handled_correctly(*filenames: str)
- files_not_handled_correctly(number_of_filenames: int)
Statically detects whether the files have all been opened and closed correctly. This is a very simple check, simply looking for corresponding
open
function calls andclose
method calls, or ifwith
was used (which counts as an implicitclose
). It does not check if the file is actually used, and that the files were opened and closed in the correct order or actually closed at runtime!If a
number_of_filenames
is provided, then it will check for that many files. If a list of strings is proivded, then it will check for those specific files.This function will also check if the student incorrectly uses
close
as a function oropen
as a method, providing cutom feedback.Feedback Function: Opened Without Arguments# Student must open and close exactly one file files_not_handled_correctly(1) # Student must open and close exactly two files files_not_handled_correctly(2) # Student must open and close the files "data.txt" and "output.txt" files_not_handled_correctly("data.txt", "output.txt")
- Label: open_without_arguments
- Response: You have called the `open` function without any arguments. It needs a filename.
Runtime
All of the runtime assertion checks do a lot of work to try to provide really nice feedback. They will try to provide a lot of information about what exactly they did when running the student code, so that the error is clearly in the students’ code, and not the instructor code.
- assert_equal(left, right) Feedback
- assert_equal(left, right, exact_strings=False, delta=DELTA)
The basic unit test, checks whether the left and right values are equal (uses
==
, notis
).The
exact_strings
parameter is whether to require that strings be exactly the same, for each character. IfFalse
(the default), then strings will be normalized (lowercased, trailing decimals chopped, punctuation removed, lines are flattened, and all characters are sorted).The
delta
controls how float values are compared - how close the values must be. If delta isNone
, then the default delta will be used (.001
).# This passes assert_equal(5, 5) # This fails assert_equal("five", 5) # This passes assert_equal([1,2,3], [1,2] + [3]) # Normally strings are matched imprecisely assert_equal("Hello, World!", "hello world") # Disable this for more precise matching assert_equal("Hello, World!", "hello world", exact_strings=True) # Floats are matched imprecisely assert_equal(1.0, 1.0001) # You might need a more precise delta assert_equal(1.0, 1.0001, delta=.0000001)
Typically, you will use this in conjunction with
call()
to check students’ functions.assert_equal(call("add", 2, 3), 5)
You do not have to call the function first, and you can even call it on either side if you want to do more complicated comparisons.
# This will work assert_equal(5, call("add", 2, 3)) # And so will this assert_equal(call("add", 2, 3), call("add", 3, 2))
In the case of more complicated calls, the feedback will be a little more complicated.
Student code failed instructor test. I ran the code: add(2, 3) add(3, 2) The value of the first result was: -1 The value of the second result was: 1 But I expected the first result to be equal to the second result
Note that you can override the “I ran the code” part by providing a
Feedback Function: Failed Instructor Testcontext
parameter. You can also override theexplanation
and theassertion
parts.- Label: assert_equal
- Response: Student code failed instructor test. {context_message}{assertion_message}{explanation}
- assert_not_equal(left, right) Feedback
- assert_not_equal(left, right, exact_strings=False, delta=DELTA)
Similar to
Feedback Function: Failed Instructor Testassert_equal()
, but checks that the left and right values are not equal (uses!=
, notis not
). See that function for more details.- Label: assert_not_equal
- Justification: Left and right were equal
- Response: Student code failed instructor test. {context_message}{assertion_message}{explanation}
- assert_less(left, right) Feedback
Checks that the left value is less than the right value (uses
Feedback Function: Failed Instructor Test<
). Seeassert_equal()
for more details.- Label: assert_less
- Justification: Left is not less than right
- Response: Student code failed instructor test. {context_message}{assertion_message}{explanation}
- assert_less_equal(left, right) Feedback
Checks that the left value is less than the right value (uses
Feedback Function: Failed Instructor Test<
). Seeassert_equal()
for more details.- Label: assert_less_equal
- Justification: Left is not less than or equal to the right
- Response: Student code failed instructor test. {context_message}{assertion_message}{explanation}
- assert_greater(left, right) Feedback
Checks that the left value is greater than the right value (uses
Feedback Function: Failed Instructor Test>
). Seeassert_equal()
for more details.- Label: assert_greater
- Justification: Left is not greater than right
- Response: Student code failed instructor test. {context_message}{assertion_message}{explanation}
- assert_greater_equal(left, right) Feedback
Checks that the left value is greater than or equal to the right value (uses
Feedback Function: Failed Instructor Test>=
). Seeassert_equal()
for more details.- Label: assert_greater_equal
- Justification: Left is not greater than or equal to the right
- Response: Student code failed instructor test. {context_message}{assertion_message}{explanation}
- assert_in(item, container) Feedback
Checks that the
item
is in thecontainer
(usesin
/not in
). Seeassert_equal()
for more details.Correctly handles asymmetry of function calls to give proper messages.
Feedback Function: Failed Instructor Test# "I expected the result to be in the: [1, 2, 3]" assert_in(call('guess_number'), [1, 2, 3]) # "I expected the result to contain: 7" assert_in(7, call('provide_options'))
- Label: assert_in
- Justification: Needle not in haystack
- Response: Student code failed instructor test. {context_message}{assertion_message}{explanation}
- assert_not_in(item, container) Feedback
Checks that the
item
is not in thecontainer
(usesin
/not in
).See
assert_not_in()
for more details.
- assert_contains_subset(needles, haystack) Feedback
Checks that the
needles
are all in thehaystacks
(usesin
/not in
). This is a little different fromassert_in()
because it will check that all of the needles are in the haystacks, not just one.Feedback Function: Failed Instructor Test# "I expected the result to contain: [1, 2, 3]" assert_contains_subset([1, 2, 3], call('provide_options')) # "I expected the result to be in: [4, 5, 6]" assert_contains_subset(call('guess_numbers'), [4, 5, 6])
- Label: assert_contains_subset
- Justification: Needles not in haystack
- Response: Student code failed instructor test. {context_message}{assertion_message}{explanation}
- assert_not_contains_subset(needles, haystack) Feedback
Checks that the
Feedback Function: Failed Instructor Testneedles
are not all in thehaystacks
(usesin
/not in
). Seeassert_contains_subset()
for more details.- Label: assert_not_contains_subset
- Justification: Needles in haystack
- Response: Student code failed instructor test. {context_message}{assertion_message}{explanation}
- assert_is(left, right) Feedback
Checks that the left value is the same as the right value (uses
Feedback Function: Failed Instructor Testis
). In other words, the two objects MUST be strictly equal, the actaul identical object and not just the same contents. This is actually unlikely to occur in most situations, given the way that Pedal executes student code in its own context, but is still sometimes necessary. Seeassert_equal()
for more details.- Label: assert_is
- Justification: Left is not identical to right
- Response: Student code failed instructor test. {context_message}{assertion_message}{explanation}
- assert_is_not(left, right) Feedback
Checks that the left value is not the same as the right value (uses
Feedback Function: Failed Instructor Testis not
). Seeassert_is()
for more details.- Label: assert_is_not
- Justification: Left is identical to right
- Response: Student code failed instructor test. {context_message}{assertion_message}{explanation}
- assert_is_none(value) Feedback
Checks that the value is
Feedback Function: Failed Instructor TestNone
(usesis None
). Seeassert_equal()
for more details.- Label: assert_is_none
- Justification: Value is not None
- Response: Student code failed instructor test. {context_message}{assertion_message}{explanation}
- assert_is_not_none(value) Feedback
Checks that the value is not
Feedback Function: Failed Instructor TestNone
(usesis not None
). Seeassert_equal()
for more details.- Label: assert_is_not_none
- Justification: Value is None
- Response: Student code failed instructor test. {context_message}{assertion_message}{explanation}
- assert_is_dataclass(value) Feedback
Checks that the value is a dataclass. Note that this is not a type check, but actually checking if the given object has the special
Feedback Function: Failed Instructor Test__dataclass_fields__
attribute. This is a little bit of a hack, but it is the best way to check if a value is actually a dataclass (or at least, it’s currently howis_dataclass
itself checks). Seeassert_equal()
for more details.- Label: assert_is_dataclass
- Justification: Value does not evaluate to a dataclass
- Response: Student code failed instructor test. {context_message}{assertion_message}{explanation}
- assert_is_not_dataclass(value) Feedback
Checks that the value is not a dataclass. Note that this is not a type check, but actually checking if the given object has the special
Feedback Function: Failed Instructor Test__dataclass_fields__
attribute. This is a little bit of a hack, but it is the best way to check if a value is actually a dataclass (or at least, it’s currently howis_dataclass
itself checks). Seeassert_equal()
for more details.- Label: assert_is_not_dataclass
- Justification: Value does not evaluate to a dataclass
- Response: Student code failed instructor test. {context_message}{assertion_message}{explanation}
- assert_true(value) Feedback
Checks that the value is (truthy)
Feedback Function: Failed Instructor TestTrue
. In other words, the result is converted to a boolean usingbool`, and then checked to see if it is ``True
. Seeassert_equal()
for more details.- Label: assert_true
- Justification: Value does not evaluate to true
- Response: Student code failed instructor test. {context_message}{assertion_message}{explanation}
- assert_false(value) Feedback
Checks that the value is (falsey)
Feedback Function: Failed Instructor TestFalse
. In other words, the result is converted to a boolean usingbool
, and then checked to see if it isFalse
. Seeassert_equal()
for more details.- Label: assert_false
- Justification: Value does not evaluate to false
- Response: Student code failed instructor test. {context_message}{assertion_message}{explanation}
- assert_length_equal(sequence, length) Feedback
Checks that the length of the sequence is equal to the given length. This function exists because the built-in
len
actually CANNOT be used in conjunction withcall
. Thelen
function in CPython must return an integer, or it will segfault. So to get around this, we have provided our version oflen
that will avoid this problem. Most of the time you won’t even notice that you are using our version oflen
, but if you do, then you can use thisassert_length_equal
function andcall
to check the length of its result more directly.See
Feedback Function: Failed Instructor Testassert_equal()
for more details on assertions, or This Post for more information about thelen
problem.- Label: assert_length_equal
- Justification: Sequence does not have length
- Response: Student code failed instructor test. {context_message}{assertion_message}{explanation}
- assert_length_not_equal(sequence, length) Feedback
Checks that the length of the sequence is not equal to the given length. Basically equivalent to
Feedback Function: Failed Instructor Testassert_not_equal(len(sequence), length)
, but will correctly provide context sincelen
cannot be used withcall
. Seeassert_length_equal()
for more details.- Label: assert_length_not_equal
- Justification: Sequence has length
- Response: Student code failed instructor test. {context_message}{assertion_message}{explanation}
- assert_length_less(sequence, length) Feedback
Checks that the length of the sequence is less than the given length. Basically equivalent to
Feedback Function: Failed Instructor Testassert_less(len(sequence), length)
, but will correctly provide context sincelen
cannot be used withcall
. Seeassert_length_equal()
for more details.- Label: assert_length_less
- Justification: Sequence length is less than
- Response: Student code failed instructor test. {context_message}{assertion_message}{explanation}
- assert_length_less_equal(sequence, length) Feedback
Checks that the length of the sequence is less than or equal to the given length. Basically equivalent to
Feedback Function: Failed Instructor Testassert_less_equal(len(sequence), length)
, but will correctly provide context sincelen
cannot be used withcall
. Seeassert_length_equal()
for more details.- Label: assert_length_less_equal
- Justification: Sequence length is less than or equal to
- Response: Student code failed instructor test. {context_message}{assertion_message}{explanation}
- assert_length_greater(sequence, length) Feedback
Checks that the length of the sequence is greater than the given length. Basically equivalent to
Feedback Function: Failed Instructor Testassert_greater(len(sequence), length)
, but will correctly provide context sincelen
cannot be used withcall
. Seeassert_length_equal()
for more details.- Label: assert_length_greater
- Justification: Sequence length is greater than
- Response: Student code failed instructor test. {context_message}{assertion_message}{explanation}
- assert_length_greater_equal(sequence, length) Feedback
Checks that the length of the sequence is greater than or equal to the given length. Basically equivalent to
Feedback Function: Failed Instructor Testassert_greater_equal(len(sequence), length)
, but will correctly provide context sincelen
cannot be used withcall
. Seeassert_length_equal()
for more details.- Label: assert_length_greater_equal
- Justification: Sequence length is greater than or equal to
- Response: Student code failed instructor test. {context_message}{assertion_message}{explanation}
- assert_is_instance(obj, cls) Feedback
Checks that the object is an instance of the given class using
isinstance
. If eitherint
orfloat
is provided, then they will allow the other to be used. To avoid this behavior, simply wrap the type in a tuple or list. Note that this function does not support generics. Seeassert_equal()
for more details.Feedback Function: Failed Instructor Test# This will pass assert_is_instance(5, int) # This will pass assert_is_instance(5, float) # This will fail assert_is_instance(5, [float])
- Label: assert_is_instance
- Justification: Object is not an instance of class
- Response: Student code failed instructor test. {context_message}{assertion_message}{explanation}
- assert_is_not_instance(obj, cls) Feedback
Checks that the object is not an instance of the given class using
isinstance
. If eitherint
orfloat
is provided, then they will allow the other to be used. To avoid this behavior, simply wrap the type in a tuple or list. Note that this function does not support generics. Seeassert_equal()
for more details.Feedback Function: Failed Instructor Test# This will fail assert_is_not_instance(5, int) # This will fail assert_is_not_instance(5, float) # This will pass assert_is_not_instance(5, [float])
- Label: assert_not_is_instance
- Justification: Object is an instance of class
- Response: Student code failed instructor test. {context_message}{assertion_message}{explanation}
- assert_type(value, expected_type) Feedback
Checks that the value is of the given type, more flexibly than
isinstance
. Basically, this uses Pedal’s type system, which allows for types to be provided as type objects (e.g.,int
,str
), with generics (list[int]
), as strings ("int"
,"str"
,"list[int]"
), and a few other ways. For more information about Pedal’s type system, see Pedal’s Type System. For more about assertions, seeassert_equal()
for more details.Feedback Function: Failed Instructor Test# This will pass assert_type(5, int) # This will pass assert_type(["Hello", "World"], list[str]) # This will fail assert_type([1, 2], list[str]) # This will pass assert_type([1, 2], "list[int]")
- Label: assert_type
- Justification: Value is not of type
- Response: Student code failed instructor test. {context_message}{assertion_message}{explanation}
- assert_not_type(value, expected_type) Feedback
Checks that the value is not of the given type, more flexibly than
isinstance
. Seeassert_type()
for more details.Feedback Function: Failed Instructor Test# This will fail assert_not_type(5, int) # This will fail assert_not_type(["Hello", "World"], list[str]) # This will pass assert_not_type([1, 2], list[str]) # This will fail assert_not_type([1, 2], "list[int]")
- Label: assert_not_type
- Justification: Value is of type
- Response: Student code failed instructor test. {context_message}{assertion_message}{explanation}
- assert_regex(pattern, text) Feedback
Checks that the regex matches the string, checking that
Feedback Function: Failed Instructor Testre.search
is not None. Seeassert_equal()
for more details.- Label: assert_regex
- Justification: Regex does not match text
- Response: Student code failed instructor test. {context_message}{assertion_message}{explanation}
- assert_not_regex(pattern, text) Feedback
Checks that the regex does not match the string, checking that
Feedback Function: Failed Instructor Testre.search
is None. Seeassert_equal()
for more details.- Label: assert_not_regex
- Justification: Regex matches text
- Response: Student code failed instructor test. {context_message}{assertion_message}{explanation}
- assert_output(execution, text) Feedback
- assert_output(execution, text, exact_strings=False) Feedback
Determine if the
execution
outputstext
. Uses the==
operator to do the final comparison. In this case, you can think of the output as a single string with newlines, as opposed to a list of strings (i.e., it is retrieved withget_raw_output()
). Seeassert_equal()
for more details.You can use the
student
variable from the Sandbox to get all of the output.# Accepts "Hello world" assert_output(student, "Hello, World!") # Only accepts "Hello, World!" assert_output(student, "Hello, World!", exact_strings=True)
Otherwise, the first argument can be a
call()
orrun()
result. This correctly checks that the desired text is in the output as a result of the given function call (and not for some other reason).assert_output(call('main'), "Hello world!")
If the
exact_strings
parameter is set to beFalse
, then output is first normalized following this strategy:Make strings lowercase
Remove all punctuation characters
Split the string by newlines into a list
Split each individual line by spaces (aka into words)
Remove all empty lines
Sorts the lines by default order
So the default check will be fairly generous about checking output; as long as all the lines are there (in whatever order), ignoring punctuation and case, the text will be found.
Feedback Function: Failed Instructor Test- Label: assert_output
- Justification: Did not print the output
- Response: Student code failed instructor test. {context_message}{assertion_message}{explanation}
- assert_not_output(execution, text) Feedback
- assert_not_output(execution, text, exact_strings=False) Feedback
Determine if the
Feedback Function: Failed Instructor Testexecution
does not outputtext
. Seeassert_output()
for more details.- Label: assert_not_output
- Justification: Printed the output
- Response: Student code failed instructor test. {context_message}{assertion_message}{explanation}
- assert_output_contains(execution, text) Feedback
- assert_output_contains(execution, text, exact_strings=False) Feedback
Determine if the
execution
outputstext
. Uses thein
operator to do the final comparison. The normalization forexact_strings
is more basic than forassert_output()
, since it will only do the lowercase conversion (punctuation is not removed).See
Feedback Function: Failed Instructor Testassert_output()
for more details.- Label: assert_output_contains
- Justification: Did not contain the printed output
- Response: Student code failed instructor test. {context_message}{assertion_message}{explanation}
- assert_not_output_contains(execution, text) Feedback
- assert_not_output_contains(execution, text, exact_strings=False) Feedback
Determine if the
Feedback Function: Failed Instructor Testexecution
does not outputtext
. Seeassert_output_contains()
for more details.- Label: assert_not_output_contains
- Justification: Contained the printed output
- Response: Student code failed instructor test. {context_message}{assertion_message}{explanation}
- assert_has_attr(obj, attr) Feedback
Determine if the
Feedback Function: Failed Instructor Testobj
has the attributeattr
using thehasattr
function. Seeassert_equal()
for more details.- Label: assert_has_attr
- Justification: Contained the attribute
- Response: Student code failed instructor test. {context_message}{assertion_message}{explanation}
- assert_has_variable(sandbox, variable_name) Feedback
Determine if the
sandbox
has the variablevariable_name
. This actually does check that the variable is defined at runtime, and not just that it is defined in the code. If you wish to check the top-level module of the student’s code, then you can use thestudent
variable from the Sandbox.Feedback Function: Failed Instructor Testassert_has_variable(student, "x")
- Label: assert_has_variable
- Response: Student code failed instructor test. {context_message}{assertion_message}{explanation}
- assert_has_function(sandbox, function_name) Feedback
Determine if the
sandbox
has the functionfunction_name
, and that the function iscallable
. If you wish to check the top-level module of the student’s code, then you can use thestudent
variable from the Sandbox.Feedback Function: Failed Instructor Testassert_has_function(student, "add")
- Label: assert_has_function
- Response: Student code failed instructor test. {context_message}{assertion_message}{explanation}
- ensure_coverage() Feedback
- ensure_coverage(at_least=.5) Feedback
Verifies that the most recent executed and traced student code has at least the given percentage of coverage (defaulting to half of the code covered). The ratio is calculated by dividing the number of lines executed by the total number of non-blank, executable lines. The coverage checker does not take into account certain line such as docstrings or comments.
Feedback Function: You Must Test Your Code# 50% coverage ensure_coverage() # 90% coverage ensure_coverage(.9) # 100% coverage ensure_coverage(1)
- Label: ensure_coverage
- Response: Your code coverage is not adequate. You must cover at least {at_least_message}% of your code to receive feedback. So far, you have only covered {coverage_message}%.
- ensure_called_uniquely(function_name) Feedback
- ensure_called_uniquely(function_name, at_least=1) Feedback
- ensure_called_uniquely(function_name, ignore=None, why_ignored='') Feedback
Verifies that the most recent executed and traced student code has
at_least
called the given function uniquely that number of times. In other words, it prevents students from calling the same function repeatedly WITHOUT changing the arguments. Students often try to bypass simpler checks (e.g., callassert_equal
three times) by calling the same function multiple times with the same arguments. This at least prevents the most obvious cheating, although you should still think about additional ways to check their tests.The
ignore
(set[tuple])
) andwhy_ignored
(str
) parameters are used to ignore certain sets of arguments (thetuple
are the arguments). This is useful if you have provided them with some tests, and you do not want those tests to count towards their total. Thewhy_ignored
is used to provide feedback to the student about why their test was ignored. For example, to say something like ````Feedback Function: You Must Test Your Code# Student must call the "add" function at least once ensure_called_uniquely("add") # Student must call the "add" function at least three times ensure_called_uniquely("add", 3) # We provide them with a few example tests ensure_called_uniquely("add", ignore={(1, 2), (2, 3)}, why_ignored=" because it was provided to you as an example.")
- Label: ensure_called_uniquely
- Response: You have not tested the function {function_name} enough. You should test it at least {at_least} times. Each time you test it, you should be using a new set of arguments. So far, you have called it {total_calls} times in total and {unique_calls} times distinctively{instructor_ignore_message}.
- ensure_function_callable(name) Feedback
Verifies that the most recent executed and traced student code has a function with the given name that is callable. This is often a little more intuitive to use than
assert_has_function()
.# Student must define a function called "add" ensure_function_callable("add")
- Feedback Function: Function Not Available
- Label: function_not_available
- Response: You may have defined `{name_message}`, but it was not available to be called in the top-level scope. Perhaps you mistakenly defined it inside another function or scope?
- Feedback Function: Name Is Not a Function
- Label: name_is_not_a_function
- Response: You defined `{name_message}`, but did not define it as a function.
- Feedback Function: Function Not Available
Equivalent Names
Despite the fact that PEP8 is pretty clear you should use snake_case instead
of camelCase, some folks are still just more comfortable with writing assertEqual
.
Therefore, we provide aliases of all the functions:
Original |
Alias |
assert_equal |
assertEqual |
assert_not_equal |
assertNotEqual |
assert_less |
assertLess |
assert_less_equal |
assertLessEqual |
assert_greater |
assertGreater |
assert_greater_equal |
assertGreaterEqual |
assert_length_equal |
assertLengthEqual |
assert_length_not_equal |
assertLengthNotEqual |
assert_length_less |
assertLengthLess |
assert_length_less_equal |
assertLengthLessEqual |
assert_length_greater |
assertLengthGreater |
assert_length_greater_equal |
assertLengthGreaterEqual |
assert_in |
assertIn |
assert_not_in |
assertNotIn |
assert_is |
assertIs |
assert_is_not |
assertIsNot |
assert_is_none |
assertIsNone |
assert_is_not_none |
assertIsNotNone |
assert_true |
assertTrue |
assert_false |
assertFalse |
assert_is_instance |
assertIsInstance |
assert_not_is_instance |
assertIsNotInstance |
assert_equal |
assertAlmostEqual |
assert_not_equal |
assertNotAlmostEqual |
assert_regex |
assertRegex |
assert_not_regex |
assertNotRegex |
assert_prints |
assertPrints |
assert_output |
assertOutput |
assert_not_output |
assertNotOutput |
assert_output_contains |
assertOutputContains |
assert_not_output_contains |
assertNotOutputContains |
assert_has_attr |
assertHasAttr |
assert_has_function |
assertHasFunction |
assert_has_variable |
assertHasVariable |
assert_type |
assertType |
assert_not_type |
assertNotType |