purrgpt-community/Tiny-Purr-350M-GGUF

This model is a GGUF conversion of purrgpt-community/Tiny-Purr-350M-merged, created using llama.cpp via the ggml.ai's GGUF-my-repo space.
Refer to the original model card for more details.

Using with llama.cpp

Install llama.cpp (works on Mac and Linux):

brew install llama.cpp

You can invoke the model using either the CLI or the server.

CLI:

llama-cli --hf-repo purrgpt-community/Tiny-Purr-350M-GGUF --hf-file tiny-purr-350m-q4_k_m.gguf -p "The meaning to life and the universe is"

Server:

llama-server --hf-repo purrgpt-community/Tiny-Purr-350M-GGUF --hf-file tiny-purr-350m-q4_k_m.gguf -c 2048

Manual Setup:

  1. Clone llama.cpp from GitHub:
git clone https://github.com/ggerganov/llama.cpp
  1. Move into the llama.cpp folder and build with the LLAMA_CURL=1 flag. Add hardware-specific flags if needed (e.g., LLAMA_CUDA=1 for Nvidia GPUs on Linux):
cd llama.cpp
LLAMA_CURL=1 make
  1. Run inference via the main binary:
./llama-cli --hf-repo purrgpt-community/Tiny-Purr-350M-GGUF --hf-file tiny-purr-350m-q4_k_m.gguf -p "The meaning to life and the universe is"

or

./llama-server --hf-repo purrgpt-community/Tiny-Purr-350M-GGUF --hf-file tiny-purr-350m-q4_k_m.gguf -c 2048
Downloads last month
6
GGUF
Model size
0.4B params
Architecture
lfm2
Hardware compatibility
Log In to view the estimation

4-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for purrgpt-community/Tiny-Purr-350M-GGUF

Quantized
(3)
this model

Dataset used to train purrgpt-community/Tiny-Purr-350M-GGUF

Collection including purrgpt-community/Tiny-Purr-350M-GGUF