# Job Submissions

A series of Perlmutter jobs can be submitted via the `submit.sh` shell script. This is a one-button method of launching parallel tests for a given list of models. 

## `submit.sh`

This script reads `../models.txt` or `../models_supervisor.txt` + `../models_coder.txt` and extracts the list of supervisor models and coder models to test. This script has a command-line input specifying the configuration mode using `--mode`.
* `--mode identical`: the default option. This mode reads from `../models.txt` and uses identical models for supervisor/coder
* `--mode pairwise`: This mode reads from `../models_supervisor.txt` + `../models_coder.txt` and constructs all pairwise combinations of supervisor/coder setups.

All of the different supervisor/coder configurations are then submitted as separate jobs. This allows each supervisor/coder pairing to run testing in parallel via the `run_tests.sh` script. To adjust the number of "trials" per test (number of times each test is run), just modify the variable `NUM_TESTS`. There is also a variable called `OUTDIR` that will let you specify the output directory for your tests.

## `run_tests.sh`
This script has 3 different input parameters:
* `supervisor`: the model to be used as supervisor
* `coder`: the model to be used as coder
* `NUM_TESTS`: the number of trials to run
* `OUTDIR`: the output directory for your tests (optional)

This script will just load the conda environment and call the final script of this chain, `test_models.py`. To adjust the slurm options, modify the header of this file (job time, account, qos, slurm output directory, etc). 

## `test_models.py`
This script parallelizes the testing for a given supervisor/coder setup. Each trial is broken down into 5 steps (summarize root, create_numpy, preprocess, scores, and categorization), and each step is run in parallel, taking advantage of the fact that each step is independent from all other steps. Furthermore, additional parallization is performed according to the number of total trials to be conducted. In the current configuration, 2 tests are run in parallel. You can modify the number of parallel tests my changing the `max_workers` in the argument of the `ProcessPoolExecutor`