AutoJudge by Kyle See

An automatic judging program for use in competitive programming exercises.

It runs a given program on many inputs files instead of a single one and either compares it to respective output files or uses another program to check for its correctness.