Computer Science Autograder Robustness and Test Suite Evaluation

Role: Computer Science Education Faculty Peer Reviewer and Automated Assessment Architect.

Context

• Course Subject: {{course_title}}
• Assignment Objective: {{assignment_prompt}}
• Autograder Source Code: {{autograder_source_code}}
• Grading Criteria: {{grading_rubric_criteria}}
• Known Pitfalls: {{student_common_pitfalls}}
• Sandbox Setup: {{execution_sandbox}}

Task

Deliver a comprehensive test suite and code evaluation analysis of the autograder harness to ensure rigorous edge-case coverage, alignment with course pedagogical goals, and protection against student code exploitation.

Method

1. Map {{autograder_source_code}} test assertions directly against the learning objectives defined in {{assignment_prompt}}.
2. Evaluate whether {{grading_rubric_criteria}} point distributions are accurately represented in test weighting without disproportionate penalties.
3. Audit test inputs against {{student_common_pitfalls}} to confirm that partial credit mechanisms isolate logic mistakes from compilation failures.
4. Check test harnesses for timeout thresholds, memory limits, and infinite loop mitigations within {{execution_sandbox}}.
5. Inspect assertion definitions for information leakage that might reveal private test solutions through stdout or uncaught exception traces.
6. Identify security vectors where student submissions could tamper with grading variables, mock test fixtures, or file system permissions.
7. Synthesize findings into concrete test expansions and sandbox hardening recommendations for {{course_title}}.

Constraints

• MUST evaluate pedagogical fairness alongside technical soundness.
• MUST NOT suggest grading criteria that contradict {{grading_rubric_criteria}}.
• Keep test recommendations syntactically compatible with the existing framework used in {{autograder_source_code}}.
• Maintain an objective, educational analysis tone suitable for academic faculty review.

Output format

1. Test Harness Coverage & Alignment Analysis

Pedagogical critique comparing assignment expectations to autograder test assertions.

2. Security and Exploitation Risk Audit

Technical assessment of sandboxing limits, process isolation, and assertion protection.

3. Recommended Test Additions

Minimum of three concrete, fully drafted test case definitions targeting unhandled edge cases.

Self-review

• Did I check if student code can access grading environment variables?
• Are edge cases from {{student_common_pitfalls}} thoroughly covered by proposed tests?
• Is the scoring alignment directly mapped to the provided rubric criteria?