|
|
--- |
|
|
license: apache-2.0 |
|
|
--- |
|
|
|
|
|
# ArcticSpeculator |
|
|
|
|
|
Build the fastest OSS vllm-based speculative decoding system for your own model, using [ArcticTraining](https://github.com/snowflakedb/ArcticTraining) and [ArcticInference](https://github.com/snowflakedb/ArcticInference)! |
|
|
|
|
|
<!--We compare the throughput (tokens/s) of existing vllm-based speculative decoding systems for Llama3.1-70B-Instruct on 8xH100 as below: |
|
|
|
|
|
| method | ShareGPT | HumanEval | |
|
|
|--------------------------------------|----------------|--------------| |
|
|
| VLLM V1 Baseline | 84.1 | 84.1 | |
|
|
| VLLM V1 Eagle | 102.2 | 112.0 | |
|
|
| VLLM V1 Eagle3 | 77.7 | 85.3 | |
|
|
| VLLM V0 MLP-Speculator (IBM) | 77.9 | 66.7 | |
|
|
| ArcticSpeculator | **172.4** | **203.7** | |
|
|
--> |
|
|
|
|
|
For more details about ArcticSpeculator and how to use it: |
|
|
|
|
|
* ❄️ [Using Arctic-Inference and Arctic-Training for improving real-world speculative decoding Performance (blog)]() |
|
|
* 🚀 [Getting started guide using ArcticTraining](https://github.com/snowflakedb/ArcticTraining/tree/mlp-variant-speculator/projects/mlp_variant_speculator) |
|
|
|
|
|
See all of the speculators we have released via our [Speculators Collection](https://huggingface.co/collections/Snowflake/speculators-6812b07f3186d13e243022e4) |
