Add metrics table (#4)
Browse files- Add metrics table (cca6317a563180d3279808e6afcdad87fc599988)
- Re-order metrics table (c23181d3c4255737305d763db4d084533fc3ff74)
Co-authored-by: Adarsh <tidealwari@users.noreply.huggingface.co>
README.md
CHANGED
|
@@ -69,6 +69,67 @@ for i in output:
|
|
| 69 |
print(i)
|
| 70 |
```
|
| 71 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 72 |
## Training Data
|
| 73 |
|
| 74 |
- **Data Collection and Filtering:** Our code data is sourced from a combination of publicly available datasets (e.g., Code available on <https://github.com>), and additional synthetic data generated at IBM Quantum. We exclude code that is older than 2023.
|
|
|
|
| 69 |
print(i)
|
| 70 |
```
|
| 71 |
|
| 72 |
+
### Comparison of Qiskit models across benchmarks
|
| 73 |
+
|
| 74 |
+
<table
|
| 75 |
+
style="
|
| 76 |
+
display: inline-table;
|
| 77 |
+
border-collapse: separate;
|
| 78 |
+
border-spacing: 0;
|
| 79 |
+
font-family: Inter, -apple-system, 'Segoe UI', Roboto, 'Helvetica Neue', Arial, sans-serif;
|
| 80 |
+
box-shadow: 0 6px 18px rgba(12, 20, 29, 0.06);
|
| 81 |
+
border-radius: 12px;
|
| 82 |
+
overflow: hidden;
|
| 83 |
+
table-layout: auto;
|
| 84 |
+
box-sizing: border-box;
|
| 85 |
+
margin: 16px 0;
|
| 86 |
+
"
|
| 87 |
+
>
|
| 88 |
+
<thead>
|
| 89 |
+
<tr>
|
| 90 |
+
<th style="text-align:left; padding:12px 16px; background:linear-gradient(90deg,#f6f8fb,#eef3f9); color:#0b1220; font-weight:700; border-bottom:1px solid rgba(15,23,42,0.06);">
|
| 91 |
+
Model
|
| 92 |
+
</th>
|
| 93 |
+
<th style="text-align:center; padding:12px 16px; background:linear-gradient(90deg,#f6f8fb,#eef3f9); color:#0b1220; font-weight:700; border-bottom:1px solid rgba(15,23,42,0.06);">
|
| 94 |
+
QiskitHumanEval
|
| 95 |
+
</th>
|
| 96 |
+
<th style="text-align:center; padding:12px 16px; background:linear-gradient(90deg,#f6f8fb,#eef3f9); color:#0b1220; font-weight:700; border-bottom:1px solid rgba(15,23,42,0.06);">
|
| 97 |
+
QiskitHumanEval-Hard
|
| 98 |
+
</th>
|
| 99 |
+
<th style="text-align:center; padding:12px 16px; background:linear-gradient(90deg,#f6f8fb,#eef3f9); color:#0b1220; font-weight:700; border-bottom:1px solid rgba(15,23,42,0.06);">
|
| 100 |
+
HumanEval
|
| 101 |
+
</th>
|
| 102 |
+
</tr>
|
| 103 |
+
</thead>
|
| 104 |
+
<tbody>
|
| 105 |
+
<tr style="background:#f7fafc;">
|
| 106 |
+
<td style="padding:12px 16px; font-weight:700; color:#07102a;">Qwen2.5-Coder-14B-Qiskit</td>
|
| 107 |
+
<td style="padding:12px 16px; text-align:center; font-weight:700; font-family:ui-monospace, SFMono-Regular, Menlo, Monaco, monospace;">25.16</td>
|
| 108 |
+
<td style="padding:12px 16px; text-align:center; font-weight:700; font-family:ui-monospace, SFMono-Regular, Menlo, Monaco, monospace;">49.01</td>
|
| 109 |
+
<td style="padding:12px 16px; text-align:center; font-weight:700; font-family:ui-monospace, SFMono-Regular, Menlo, Monaco, monospace;">91.46</td>
|
| 110 |
+
</tr>
|
| 111 |
+
<tr style="background:#ffffff;">
|
| 112 |
+
<td style="padding:12px 16px; color:#0f172a;">mistral-small-3.2-24b-qiskit</td>
|
| 113 |
+
<td style="padding:12px 16px; text-align:center; font-family:ui-monospace, SFMono-Regular, Menlo, Monaco, monospace;">20.53</td>
|
| 114 |
+
<td style="padding:12px 16px; text-align:center; font-family:ui-monospace, SFMono-Regular, Menlo, Monaco, monospace;">40.39</td>
|
| 115 |
+
<td style="padding:12px 16px; text-align:center; font-family:ui-monospace, SFMono-Regular, Menlo, Monaco, monospace;">77.49</td>
|
| 116 |
+
</tr>
|
| 117 |
+
<tr style="background:#ffffff;">
|
| 118 |
+
<td style="padding:12px 16px; color:#0f172a;">granite-3.3-8b-qiskit</td>
|
| 119 |
+
<td style="padding:12px 16px; text-align:center; font-family:ui-monospace, SFMono-Regular, Menlo, Monaco, monospace;">14.56</td>
|
| 120 |
+
<td style="padding:12px 16px; text-align:center; font-family:ui-monospace, SFMono-Regular, Menlo, Monaco, monospace;">27.15</td>
|
| 121 |
+
<td style="padding:12px 16px; text-align:center; font-family:ui-monospace, SFMono-Regular, Menlo, Monaco, monospace;">62.80</td>
|
| 122 |
+
</tr>
|
| 123 |
+
<tr style="background:#fbfdff;">
|
| 124 |
+
<td style="padding:12px 16px; color:#0f172a;">granite-3.2-8b-qiskit</td>
|
| 125 |
+
<td style="padding:12px 16px; text-align:center; font-family:ui-monospace, SFMono-Regular, Menlo, Monaco, monospace;">9.93</td>
|
| 126 |
+
<td style="padding:12px 16px; text-align:center; font-family:ui-monospace, SFMono-Regular, Menlo, Monaco, monospace;">24.50</td>
|
| 127 |
+
<td style="padding:12px 16px; text-align:center; font-family:ui-monospace, SFMono-Regular, Menlo, Monaco, monospace;">57.31</td>
|
| 128 |
+
</tr>
|
| 129 |
+
</tbody>
|
| 130 |
+
</table>
|
| 131 |
+
|
| 132 |
+
|
| 133 |
## Training Data
|
| 134 |
|
| 135 |
- **Data Collection and Filtering:** Our code data is sourced from a combination of publicly available datasets (e.g., Code available on <https://github.com>), and additional synthetic data generated at IBM Quantum. We exclude code that is older than 2023.
|