Command-Line Interface

In addition to being a library, Pedal also provides a command-line interface (CLI) for analyzing student code. This is useful for batch processing or on individual submissions (e.g., through a platform like GradeScope or VSCodeEdu). For specific integrations, see the Autograder Integrations section.

Command Line Modes

class SandboxPipeline(config)[source]: sandbox: Pipeline for running ONLY the student’s code, and then outputing the results to the console. There is no instructor control script logic, although the Source tool does check that the student’s code is syntactically correct. Otherwise, the students’ code is run in a Sandbox mode. This is useful if you just want to safely execute student code and observe their output.

class RunPipeline(config)[source]: run: Pipeline for running the instructor control script on a submission and generating a report file in the ini file format. This is a simple file format that has a lot of the interesting fields. The file is not actually dumped to the filesystem, but instead printed directly. So this is a good way to run students’ code in a sandbox and see what comes out.

class FeedbackPipeline(config)[source]: feedback: Pipeline for running the instructor control script on a submission and then printing the resolver output to the console. Often the most useful if you are trying to deliver the feedback without a grade.

class GradePipeline(config)[source]

grade: Pipeline for running the instructor control script on a submission and then outputing the grade to the console. This is useful for quickly grading a set of submissions. The instructor file, student data, and assignment are also all printed out in the following CSV format:

instructor_file, student_file, student_email, assignment_name, score, correct

class StatsPipeline(config)[source]: stats: Pipeline for running the instructor control script on a submission and then dumping a JSON report with all the feedback objects. This is useful for analyzing the feedback objects in a more programmatic way.

class VerifyPipeline(config)[source]

verify: Pipeline for running the instructor control script on a submission and then comparing the output to an expected output file. This is useful for verifying that the feedback is correct (at least, as correct as the expected output).

You can also use this pipeline to generate the output files, to quickly create regression “tests” of your feedback scripts.

class DebugPipeline(config)[source]: debug: Pipeline for running the instructor control script on a submission and then outputing the full results to the console. This is useful for debugging the instructor control script, as it will show the full output, error, all of the feedback objects considered, and the final feedback.

File Formats

Pedal can read and write several file formats. The most common are:

Python files (.py)
Folders containing Python files
Archive files (.zip, .tar.gz, etc.) containing Python files
JSON files (.json) that have Python code embedded in them
CSV files (.csv) that contain Python code

Pedal can also work with ProgSnap data dumps, containing code snapshots: - Zip files (.zip) - CSV files (.csv) - Sqlite files (.db) - this is often the fastest file format!

By far, the simplest option is just Python files or folders containing Python files. However, the other formats can be useful for more complex scenarios with additional metadata.

Pedal Job Files (.pedal) can be used to store the configuration for a Pedal job.

Command Line Parameters

The CLI is invoked with the pedal command. It accepts the following parameters:

Run instructor control script on student submissions.

usage: pedal [-h] [--alternate_filenames ALTERNATE_FILENAMES]
             [--output OUTPUT] [--config CONFIG] [--create_output]
             [--environment {blockpy,gradescope,jupyter,standard,vpl,terminal}]
             [--instructor_name INSTRUCTOR_NAME]
             [--progsnap_profile PROGSNAP_PROFILE]
             [--include_scripts INCLUDE_SCRIPTS] [--limit LIMIT]
             [--resolver RESOLVER] [--ics_direct] [--skip_tifa] [--skip_run]
             [--progsnap_events {run,edit,last,compile}] [--cache CACHE]
             [--threaded THREADED]
             [--log_level {DEBUG,INFO,WARNING,ERROR,CRITICAL}]
             [--gpt-api-key GPT_API_KEY]
             {run,feedback,grade,stats,sandbox,verify,debug} instructor
             [submissions]

Positional Arguments

mode

Possible choices: run, feedback, grade, stats, sandbox, verify, debug

What kind of Pedal analysis you’re running. See the description of each mode above.

instructor

The path to the instructor control script, or multiple scripts.

submissions

The path to the student submissions. Defaults to a folder named submissions adjacent to the instructor control script.

Default: “submissions”

Named Arguments

--alternate_filenames

A semicolon separated list of potential filenames to try if the main isn’t found.

Default: “”

--output, -o

The output file path for the result. Defaults to stdout.

Default: “stdout”

--config, -c

Uses the configuration file to get settings.

--create_output, -m

In verify mode, creates any missing outputs.

Default: False

--environment, -e

Possible choices: blockpy, gradescope, jupyter, standard, vpl, terminal

Sets the environment context for this script, which can run special setups and override tools as needed.

Default: “standard”

--instructor_name

Sets the name of the instructor file to something more friendly. If not given, then will default to the instructor filename.

--progsnap_profile

Uses the given profile’s default settings for loading in a ProgSnap2 dataset

Default: “blockpy”

--include_scripts

An optional filter to only include certain scripts

--limit

An optional limit to how many submissions are run. Mostly for testing purposes.

--resolver

Choose a different resolver to use (the name of a function defined in the instructor control script).

Default: “resolve”

--ics_direct

Give the instructor code directly instead of loading from a file.

Default: False

--skip_tifa

Skip using TIFA in the environment

Default: False

--skip_run

Skip automatically running student code in the environment

Default: False

--progsnap_events

Possible choices: run, edit, last, compile

Choose what level of event to capture from Progsnap event logs.

Default: “run”

--cache

Use the given directory to hold the cache. You can use “./” to use the current directory.

Default: False

--threaded

Run the instructor script in a separate thread to avoid timeout crashes.

Default: False

--log_level

Possible choices: DEBUG, INFO, WARNING, ERROR, CRITICAL

Set the logging level for Pedal.

Default: “ERROR”

--gpt-api-key

Set the GPT API key