feat: improve readibility & add best practices section

README.md

@@ -27,14 +27,14 @@ Training used a maximum sequence length of 131k tokens.
       <th colspan="4" style="background-color: #d4edda;">Maximum Number of Turns = 500</th>
     </tr>
     <tr>
-      <th style="background-color: #fff3cd;">Pass@1, %</th>
-      <th style="background-color: #fff3cd;">Pass@5, %</th>
-      <th style="background-color: #fff3cd;">Pass@1, %</th>
-      <th style="background-color: #fff3cd;">Pass@5, %</th>
-      <th style="background-color: #d4edda;">Pass@1, %</th>
-      <th style="background-color: #d4edda;">Pass@5, %</th>
-      <th style="background-color: #d4edda;">Pass@1, %</th>
-      <th style="background-color: #d4edda;">Pass@5, %</th>
+      <th style="background-color: #fff3cd;">Pass@1</th>
+      <th style="background-color: #fff3cd;">Pass@5</th>
+      <th style="background-color: #fff3cd;">Pass@1</th>
+      <th style="background-color: #fff3cd;">Pass@5</th>
+      <th style="background-color: #d4edda;">Pass@1</th>
+      <th style="background-color: #d4edda;">Pass@5</th>
+      <th style="background-color: #d4edda;">Pass@1</th>
+      <th style="background-color: #d4edda;">Pass@5</th>
     </tr>
   </thead>
   <tbody>
@@ -44,37 +44,37 @@ Training used a maximum sequence length of 131k tokens.
     <tr>
       <td><a href="https://huggingface.co/Qwen/Qwen3-30B-A3B-Instruct-2507">Qwen3-30B-A3B-Instruct-2507</a></td>
       <td>30B</td>
-      <td style="background-color: #fff3cd;text-align: center;">25.2 ± 0.7</td>
+      <td style="background-color: #fff3cd;text-align: center;">25.2</td>
       <td style="background-color: #fff3cd;text-align: center;">44.8</td>
-      <td style="background-color: #fff3cd;text-align: center;">11.8 ± 1.5</td>
+      <td style="background-color: #fff3cd;text-align: center;">11.8</td>
       <td style="background-color: #fff3cd;text-align: center;">24.4</td>
-      <td style="background-color: #d4edda;text-align: center;">25.7 ± 0.5</td>
+      <td style="background-color: #d4edda;text-align: center;">25.7</td>
       <td style="background-color: #d4edda;text-align: center;">44.2</td>
-      <td style="background-color: #d4edda;text-align: center;">14.2 ± 1.1</td>
+      <td style="background-color: #d4edda;text-align: center;">14.2</td>
       <td style="background-color: #d4edda;text-align: center;">26.5</td>
     </tr>
     <tr>
       <td><a href="https://huggingface.co/Qwen/Qwen3-Coder-30B-A3B-Instruct">Qwen3-Coder-30B-A3B-Instruct</a></td>
       <td>30B</td>
-      <td style="background-color: #fff3cd;text-align: center;"><strong>51.9</strong> ± 0.2</td>
+      <td style="background-color: #fff3cd;text-align: center;"><strong>51.9</strong></td>
       <td style="background-color: #fff3cd;text-align: center;"><strong>67.3</strong></td>
-      <td style="background-color: #fff3cd;text-align: center;"><strong>28.7</strong> ± 1.1</td>
+      <td style="background-color: #fff3cd;text-align: center;"><strong>28.7</strong></td>
       <td style="background-color: #fff3cd;text-align: center;"><strong>42.8</strong></td>
-      <td style="background-color: #d4edda;text-align: center;"><strong>50.0</strong> ± 0.5</td>
+      <td style="background-color: #d4edda;text-align: center;"><strong>50.0</strong></td>
       <td style="background-color: #d4edda;text-align: center;">63.0</td>
-      <td style="background-color: #d4edda;text-align: center;"><strong>28.1</strong> ± 1.5</td>
+      <td style="background-color: #d4edda;text-align: center;"><strong>28.1</strong></td>
       <td style="background-color: #d4edda;text-align: center;"><strong>38.7</strong></td>
     </tr>
     <tr style="background-color: #ebeced">
       <td style="color: black;">nebius/SWE-rebench-openhands-Qwen3-30B-A3B (Ours)</td>
       <td>30B</td>
-      <td style="background-color: #ffdf80;text-align: center;">49.7 ± 0.9<br/>(+24.5)</td>
+      <td style="background-color: #ffdf80;text-align: center;">49.7<br/>(+24.5)</td>
       <td style="background-color: #ffdf80;text-align: center;">65.4<br/>(+20.6)</td>
-      <td style="background-color: #ffdf80;text-align: center;">28.1 ± 1.5<br/>(+16.3)</td>
+      <td style="background-color: #ffdf80;text-align: center;">28.1<br/>(+16.3)</td>
       <td style="background-color: #ffdf80;text-align: center;">38.7<br/>(+14.3)</td>
-      <td style="background-color: #9df2b3;text-align: center;"><strong>50.3</strong> ± 0.7<br/>(+24.6)</td>
+      <td style="background-color: #9df2b3;text-align: center;"><strong>50.3</strong><br/>(+24.6)</td>
       <td style="background-color: #9df2b3;text-align: center;"><strong>68.3</strong><br/>(+24.1)</td>
-      <td style="background-color: #9df2b3;text-align: center;"><strong>28.1</strong> ± 1.0<br/>(+13.9)</td>
+      <td style="background-color: #9df2b3;text-align: center;"><strong>28.1</strong><br/>(+13.9)</td>
       <td style="background-color: #9df2b3;text-align: center;"><strong>38.7</strong><br/>(+12.2)</td>
     </tr>
     <tr>
@@ -83,9 +83,9 @@ Training used a maximum sequence length of 131k tokens.
     <tr>
       <td><a href="https://huggingface.co/zai-org/GLM-4.5-Air">GLM-4.5-Air</a></td>
       <td>106B</td>
-      <td style="background-color: #fff3cd;text-align: center;">58.2 ± 0.2</td>
+      <td style="background-color: #fff3cd;text-align: center;">58.2</td>
       <td style="background-color: #fff3cd;text-align: center;">73.5</td>
-      <td style="background-color: #fff3cd;text-align: center;">33.8 ± 1.2</td>
+      <td style="background-color: #fff3cd;text-align: center;">33.8</td>
       <td style="background-color: #fff3cd;text-align: center;">42.8</td>
       <td style="background-color: #d4edda;text-align: center;">-</td>
       <td style="background-color: #d4edda;text-align: center;">-</td>
@@ -98,25 +98,25 @@ Training used a maximum sequence length of 131k tokens.
     <tr>
       <td><a href="https://huggingface.co/Qwen/Qwen3-235B-A22B-Instruct-2507">Qwen3-235B-A22B-Instruct-2507</a></td>
       <td>235B</td>
-      <td style="background-color: #fff3cd;text-align: center;">45.2 ± 0.8</td>
+      <td style="background-color: #fff3cd;text-align: center;">45.2</td>
       <td style="background-color: #fff3cd;text-align: center;">65.9</td>
-      <td style="background-color: #fff3cd;text-align: center;">29.3 ± 2.4</td>
+      <td style="background-color: #fff3cd;text-align: center;">29.3</td>
       <td style="background-color: #fff3cd;text-align: center;">44.8</td>
-      <td style="background-color: #d4edda;text-align: center;">46.2 ± 0.4</td>
+      <td style="background-color: #d4edda;text-align: center;">46.2</td>
       <td style="background-color: #d4edda;text-align: center;">67.5</td>
-      <td style="background-color: #d4edda;text-align: center;">25.3 ± 1.9</td>
+      <td style="background-color: #d4edda;text-align: center;">25.3</td>
       <td style="background-color: #d4edda;text-align: center;">40.8</td>
     </tr>
     <tr>
       <td style="color: black;"><a href="https://huggingface.co/nebius/SWE-rebench-openhands-Qwen3-235B-A22B">nebius/SWE-rebench-openhands-Qwen3-235B-A22B</a> (Ours)</td>
       <td>235B</td>
-      <td style="background-color: #fff3cd;text-align: center;"><strong>59.9</strong> ± 0.1<br/>(+14.7)</td>
+      <td style="background-color: #fff3cd;text-align: center;"><strong>59.9</strong><br/>(+14.7)</td>
       <td style="background-color: #fff3cd;text-align: center;"><strong>73.9</strong><br/>(+8.0)</td>
-      <td style="background-color: #fff3cd;text-align: center;"><strong>35.1</strong> ± 1.0<br/>(+5.8)</td>
+      <td style="background-color: #fff3cd;text-align: center;"><strong>35.1</strong><br/>(+5.8)</td>
       <td style="background-color: #fff3cd;text-align: center;"><strong>46.9</strong><br/>(+2.1)</td>
-      <td style="background-color: #d4edda;text-align: center;"><strong>61.7</strong> ± 0.9<br/>(+15.5)</td>
+      <td style="background-color: #d4edda;text-align: center;"><strong>61.7</strong><br/>(+15.5)</td>
       <td style="background-color: #d4edda;text-align: center;"><strong>74.3</strong><br/>(+6.8)</td>
-      <td style="background-color: #d4edda;text-align: center;"><strong>34.2</strong> ± 1.5<br/>(+8.9)</td>
+      <td style="background-color: #d4edda;text-align: center;"><strong>34.2</strong><br/>(+8.9)</td>
       <td style="background-color: #d4edda;text-align: center;"><strong>44.8</strong><br/>(+4.0)</td>
     </tr>
     <tr>
@@ -125,9 +125,9 @@ Training used a maximum sequence length of 131k tokens.
     <tr>
       <td><a href="https://huggingface.co/zai-org/GLM-4.5">GLM-4.5</a></td>
       <td>355B</td>
-      <td style="background-color: #fff3cd;text-align: center;">64.4 ± 0.5</td>
+      <td style="background-color: #fff3cd;text-align: center;">64.4</td>
       <td style="background-color: #fff3cd;text-align: center;">76.2</td>
-      <td style="background-color: #fff3cd;text-align: center;">33.8 ± 1.7</td>
+      <td style="background-color: #fff3cd;text-align: center;">33.8</td>
       <td style="background-color: #fff3cd;text-align: center;">44.8</td>
       <td style="background-color: #d4edda;text-align: center;">-</td>
       <td style="background-color: #d4edda;text-align: center;">-</td>
@@ -137,21 +137,21 @@ Training used a maximum sequence length of 131k tokens.
     <tr>
       <td><a href="https://huggingface.co/Qwen/Qwen3-Coder-480B-A35B-Instruct">Qwen3-Coder-480B-A35B-Instruct</a></td>
       <td>480B</td>
-      <td style="background-color: #fff3cd;text-align: center;">64.7 ± 0.5</td>
+      <td style="background-color: #fff3cd;text-align: center;">64.7</td>
       <td style="background-color: #fff3cd;text-align: center;">75.8</td>
-      <td style="background-color: #fff3cd;text-align: center;">36.3 ± 1.6</td>
+      <td style="background-color: #fff3cd;text-align: center;">36.3</td>
       <td style="background-color: #fff3cd;text-align: center;">44.8</td>
-      <td style="background-color: #d4edda;text-align: center;">66.5 ± 0.4</td>
+      <td style="background-color: #d4edda;text-align: center;">66.5</td>
       <td style="background-color: #d4edda;text-align: center;">77.8</td>
-      <td style="background-color: #d4edda;text-align: center;">35.5 ± 1.4</td>
+      <td style="background-color: #d4edda;text-align: center;">35.5</td>
       <td style="background-color: #d4edda;text-align: center;">42.8</td>
     </tr>
   </tbody>
 </table>
 
-**Table 1.** Pass@1 with standard error of the mean and Pass@5 for OpenHands agent with the maximum number of turns set to 100
+**Table 1.** Pass@1 (averaged over 5 runs) and Pass@5 for OpenHands agent with the maximum number of turns set to 100
 (highlighted in <span style="background-color: #fff3cd; padding: 4px;">yellow</span>) and 500
-(highlighted in <span style="background-color: #d4edda; padding: 4px;">green</span>).
+(highlighted in <span style="background-color: #d4edda; padding: 4px;">green</span>). Metrics are reported in percentages.
 Deltas vs base models are shown in parentheses for fine-tuned models.
 
 We explicitly excluded all [SWE-bench Verified](https://huggingface.co/datasets/princeton-nlp/SWE-bench_Verified) and
@@ -168,6 +168,28 @@ For more details see our report in [Nebius blog](LINK-TO-BE-ADDED).
 
 ---
 
+# Best Practices
+
+1. **Deployment:**
+   * Use the following configuration to serve the model with vLLM:
+     ```bash
+     VLLM_USE_V1=1 vllm serve nebius/SWE-rebench-openhands-Qwen3-30B-A3B
+       --tensor-parallel-size 8
+       --served-model-name qwen_3_instruct_2507
+       --disable-log-requests
+       --enable-prefix-caching
+       --max-model-len 131072
+       --enable-auto-tool-choice
+       --tool-call-parser hermes
+     ```
+     Tested using `vllm/vllm-openai:v0.9.0` Docker image.
+
+2. **Sampling Parameters:**
+   * For optimal performance, we recommend `Temperature=0.7`, `TopP=0.8`, `TopK=20`, and `MinP=0`
+     that are consistent with the base model.
+  
+---
+
 # Citation
 
 ```