RM -RF: Reward Model for Run-Free Unit Test Evaluation

RM -RF: Reward Model for Run-Free Unit Test Evaluation proposes a novel lightweight reward model that predicts unit test quality without compiling or executing code by inferring three execution-derived signals directly from source and test code:

1. whether the augmented test suite would compile and run,
2. whether the new tests increase code coverage,
3. whether they improve mutation kill rate.

This work is motivated by the high computational cost of traditional compile-and-run validation in automated test generation, especially when using large language models (LLMs) for code tasks. RM-RF is trained on a multilingual dataset across Java, Python, and Go that pairs code + test files with execution-derived labels, and various model families and tuning regimes (zero-shot, fine-tuned, PEFT/LoRA) achieve ~0.69 F1 on the three targets.

Key contributions include:

• a scalable “run-free” reward model that reduces latency and infrastructure needs compared to execution-based evaluation,
• a curated dataset and methodology for comparative assessment of unit test quality signals,
• empirical analysis of RM-RF with different model sizes and fine-tuning strategies.

This approach can provide rapid feedback for large-scale test generation and RL-based code optimization, bridging a gap between high-fidelity execution feedback and scalable automated unit test evaluation.