AI grading tools and autograders for programming assignments

This page compares ten tools that grade programming assignments: Autolab¹, CodeGrade², CodeHS³, Codio⁴, Gradescope⁵, Graidable⁶, nbgrader⁷, Otter-Grader⁸, Vocareum⁹ and zyLabs¹⁰. Nine run tests or grading scripts on code the student submits. The tenth, Graidable, is for the paper exam: it grades code that students write by hand. Four of them (CodeHS, Codio, Graidable and Vocareum) say they use AI in grading. Pick by who you teach, by whether you want to run the software yourself, and by whether tests cover everything you grade.

The ten tools, in alphabetical order

Each profile says what the vendor’s or project’s own pages stated when we checked. Six are hosted products and three are open source. Plans and prices change, so read the vendor’s page before you decide.

Pick by what you teach

• If you teach a high school class and want the course material included, look at CodeHS. It says it is built for K-12, and its Free plan has autograders in the exercises.
• If you run a large university course with your own test suite, Autolab and Gradescope run the autograder you write, in any language. Gradescope’s programming assignments are on its Institutional plan.
• If you want tests without config files, CodeGrade has a block-based test builder, and Codio’s standard tests ask for the input and the output you expect.
• If your course is taught in Jupyter notebooks, nbgrader and Otter-Grader are open source. Otter-Grader also grades R.
• If you want the textbook and the labs in one product, zyLabs are part of zyBooks.
• If your labs need cloud accounts or a full Linux desktop, look at Vocareum.
• If the exam is on paper and students write code by hand, look at Graidable, the tool we make. It reads the scans and drafts points from your answer key.

If the same course has essays or paper exams, see AI essay graders compared and AI grading tools for handwritten math. The main comparison of AI grading tools covers every kind of grading.

What is the difference between test-based autograding and AI feedback on code?

A test-based autograder runs the program and checks what it does. AI feedback reads the code and comments on how it is written. They answer different questions, so most courses that use AI feedback still need the tests.

Tests give every student the same check, in seconds, and the result doesn’t depend on who grades. Their limit is that they don’t read the code. CodeHS’s help page says it plainly: an output test that expects 10 can’t tell whether the student added 5 and 5, multiplied 5 by 2 or just printed 10³.

Between the two sits a third kind of check that uses no AI: style checkers and linters. CodeGrade lists code quality tests that run tools such as PyLint and ESLint², and Codio’s advanced tests can run style checkers⁴. If “code quality” on your rubric means consistent style and no unused variables, a linter may be all you need.

AI feedback is a draft. CodeHS describes its AI suggestions as “always a starting point, not the final grade”³. We think that is the right way to treat any of them: run the tool on submissions you have already graded, and read its comments before a student does.

Which tools use AI to grade coding assignments?

Four of the ten say they do: CodeHS, Codio, Graidable and Vocareum.

• CodeHS: AI Grading suggests a score and feedback from a rubric. You adjust them, then finalize the assignment or mark it as needing work, and “nothing reaches the student until you approve it”. It is part of the paid plans³.
• Codio: its page for former GitHub Classroom users lists “LLM-powered rubric grading for code quality and style”⁴.
• Graidable: the AI drafts points and feedback for code written by hand on a paper exam, from your answer key or rubric, and you approve each score⁶.
• Vocareum: its site lists “AI-assisted grading” next to Bash scripts and API endpoints as a way to auto-grade⁹. The pages we read don’t describe it further, so ask for a demo.

Two more put AI in front of students instead. CodeGrade’s assistant answers students’ questions under rules you write for each assignment². zyLabs has AI Hints, and zyBooks says AI “plays no role in evaluating student work”¹⁰.

The pages we read for Autolab, Gradescope’s programming assignments, nbgrader and Otter-Grader describe grading by tests and by hand.

Which is better for a large university course, and which for a high school class?

For a large university course, start with the tools that run your own test suite and connect to your LMS. For a high school class, start with CodeHS, the one of the ten that describes itself as a platform for K-12 schools.

What the vendors say about large courses:

• Autolab puts grading jobs in a queue and hands each to an available container or virtual machine¹.
• CodeGrade says results come back in seconds “even for classes of 500+ students”².
• Gradescope says it runs “your autograder at scale” on its own servers⁵.
• Otter-Grader is designed for classes “at any scale” and grades in parallel Docker containers⁸.
• Vocareum says its engine can handle hundreds of submissions in minutes⁹.
• Codio says its auto-grading works “for classes of any size”⁴.

What they say about schools:

• CodeHS has courses for grades K-5, 6-8 and 9-12, and a Free plan with built-in autograders³.
• Autolab says its aim covers classes at the secondary level too. It is software you install and run yourself¹.
• CodeGrade’s free plan covers up to 50 students per course².
• Codio says it subsidizes K-12 student licenses⁴.

How do these tools handle AI-written code?

Mostly by showing you how the code was written. The vendors describe three approaches: recording the student’s work, limiting or flagging pasted code, and giving students an AI helper you can watch.

• Codio logs keystrokes. It says Behavior Insights lets teachers see “whether students have copied and pasted their work from tools like ChatGPT or elsewhere”, and Code Playback replays how a file was built⁴.
• zyBooks says it “cannot block external AI tools outright”. It offers Paste Limiting, Code Playback, a dashboard that flags outliers in time spent and pasted code, and lockdown browsers for exams¹⁰.
• CodeHS highlights pasted code in yellow in Code History. Its paid plans add a setting that turns off copy and paste in the editor³.
• CodeGrade lets you set what its AI assistant may do in each assignment, from guiding questions to full coding help, and shows you every conversation².

A similarity check is a different thing. It compares submissions with each other, so it finds two students who handed in the same code, wherever the code came from. Gradescope says its Code Similarity tool “does not automatically detect plagiarism”⁵, and zyBooks says the judgment call stays with the instructor¹⁰.

What happened to GitHub Classroom?

GitHub retired it on 28 August 2026. Accounts, repositories and organizations created through Classroom were not affected¹¹.

GitHub named two partners for the same workflow of handing out assignments, autograding and working in GitHub repositories. One is Codio, which is in the comparison above. The other is Classroom 50, which GitHub describes as a free, open-source alternative from the Fifty Foundation. It can autograde submissions with declarative tests or your own grading scripts, and it runs them with GitHub Actions¹¹.

We also looked for Replit’s Teams for Education. Its page on replit.com returned “Page not found” when we checked, so it is not in the comparison¹².

Questions people ask

Which autograders are free?

Three are open source, with no license fee: Autolab, nbgrader and Otter-Grader. You run them on your own machine or server.

Among the hosted tools:

• CodeGrade has a free plan for up to 50 students per course².
• CodeHS has a Free plan that includes autograders³.
• Codio gives instructors free accounts and new organizations a free trial⁴.
• Gradescope offers an Institutional Trial for one term⁵.

What do the paid tools cost?

• CodeGrade: $24 or $39 per student per course, plus $15 for the AI assistant².
• Codio: $48 per learner per semester or $90 per year in higher education⁴.
• Vocareum: $10 per monthly active user, plus resource fees for AI, virtual machines and cloud labs⁹.
• zyBooks: students in a class typically pay $69 to $90 for a zyBook¹⁰.
• CodeHS and Gradescope: by quote^3,5.

Do they send grades to Canvas or another LMS?

Most of the hosted tools do. CodeGrade, Codio, Gradescope, Vocareum and zyBooks describe LTI integrations with grade passback, and CodeHS has Google Classroom grade passback on its paid plans. Check the plan: CodeGrade puts LMS integration on Advanced, Gradescope on Institutional, and CodeHS puts LTI on its School and District plans. Autolab’s docs describe roster sync, and Otter-Grader’s docs name Canvas and Gradescope.

Does the teacher stay in control of the grade?

With tests, yes: you write the tests and decide what each one is worth. CodeGrade, Codio and Gradescope also describe grading by hand next to the autograder, with rubrics and comments in the code.

With AI, check before you buy. CodeHS says nothing reaches a student until the teacher approves it³. For the others, ask to see the review step.

What happens to student code and data?

With the open-source tools, submissions are graded wherever you run the software. For hosted tools, read the vendor’s privacy pages and ask what any AI feature sends to which provider.

zyBooks is specific about this. It says no personally identifiable student information is sent to AI systems, and that its AI provider is contractually barred from using customer data for training¹⁰.

How to test an autograder on your own assignment

1. Take an assignment you have already graded, with 15 to 30 submissions. Include one that doesn’t compile, one that loops forever, and one that prints the right answer in the wrong format.
2. Build the tests for it in the tool, and time yourself. The first assignment is the slow one, so this is the cost that matters.
3. Compare the tool’s score with yours, test by test. Totals hide errors that cancel out.
4. Look at each disagreement. Is it an output match that is too strict, a case your tests missed, or a judgment call?
5. Read what the student sees when a test fails. A message that names the failing case teaches something. A bare zero means an email to you.

If the tool has AI feedback, give it the same submissions and compare its rubric scores with yours, criterion by criterion.

Graidable is the one tool here for the paper exam. Students write code, math or explanations by hand, you scan the class set, and we draft a score and feedback for each question from your answer key. You review, change and approve every one before students see anything. For the math and science side of the same course, see AI grading tools for handwritten math.

Sources

1. Autolab Project, home page, docs (including the guide for lab authors and LTI course integration) and GitHub repository. Checked 8 October 2026. ↑

2. CodeGrade, home page, pricing, autograder, code plagiarism checker and AI assistant. Checked 8 October 2026. ↑

3. CodeHS, home page, plans, and the help articles CodeHS Autograders, Using the AI Grading App and Guide to cheat prevention and detection. Checked 8 October 2026. ↑

4. Codio, auto-grading, pricing, plagiarism detection, LMS integration, home page and page for GitHub Classroom users. Checked 8 October 2026. ↑

5. Gradescope, home page, pricing, autograder documentation, and the guides Grading a Programming Assignment, Generating a Code Similarity Report and LMS workflows. Checked 8 October 2026. ↑

6. Graidable, Pricing and the product itself; the tool is ours. Checked 8 October 2026. ↑

7. nbgrader, documentation (highlights, creating and grading assignments, FAQ) and GitHub repository. Checked 8 October 2026. ↑

8. Otter-Grader, documentation (including test files) and GitHub repository. Checked 8 October 2026. ↑

9. Vocareum, home page, pricing and higher education. Checked 8 October 2026. ↑

10. zyBooks, zyLabs, AI tools and content, academic integrity, home page, and the help articles Payment and Getting started with LTI. Checked 8 October 2026. ↑

11. GitHub, retirement announcement for GitHub Classroom (27 August 2026) and Export or migrate GitHub Classroom data. Checked 8 October 2026. ↑

12. Replit, replit.com/teams-for-education. Checked 8 October 2026. ↑