README.md

metadata

title: FilBench Leaderboard
emoji: 🥇
colorFrom: green
colorTo: indigo
sdk: gradio
app_file: app.py
pinned: true
license: apache-2.0
short_description: An Open LLM Leaderboard for Filipino
sdk_version: 5.19.0

HF Leaderboard Backend

This is a fork of the leaderboard demo from HuggingFace with some additional scripts for parsing the results from our evaluation runs.

Set-up and installation

To start development, clone the repository and install the dependencies. This assumes you have uv installed. Make sure to also install the pre-commit hooks so that formatting is uniform across the codebase.

git clone git@github.com:filbench/hf-leaderboard-backend.git
cd hf-leaderboard-backend
uv sync
source .venv/bin/activate
pre-commit install

To view the leaderboard locally, then run the following command:

gradio app.py

This will launch the leaderboard (by default in localhost:7860) in hot-reload mode, so you can see the changes as you edit the source code.

Updating the HuggingFace Space

All development should happen in this GitHub remote. If you want to update the HuggingFace space, you should add a remote pointing to the space and push all changes. You might need to add your machine's SSH key to HuggingFace.

git remote add hf git@hf.co:spaces/UD-Filipino/filbench-leaderboard
git push hf main

Cite

If you're using FilBench, please cite our work:

@inproceedings{miranda-etal-2025-filbench,
    title = "{F}il{B}ench: Can {LLM}s Understand and Generate {F}ilipino?",
    author = "Miranda, Lester James Validad  and
      Aco, Elyanah  and
      Manuel, Conner G.  and
      Cruz, Jan Christian Blaise  and
      Imperial, Joseph Marvin",
    editor = "Christodoulopoulos, Christos  and
      Chakraborty, Tanmoy  and
      Rose, Carolyn  and
      Peng, Violet",
    booktitle = "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2025.emnlp-main.127/",
    doi = "10.18653/v1/2025.emnlp-main.127",
    pages = "2496--2529",
    ISBN = "979-8-89176-332-6",
    abstract = "Despite the impressive performance of LLMs on English-based tasks, little is known about their capabilities in specific languages such as Filipino. In this work, we address this gap by introducing FilBench, a Filipino-centric benchmark designed to evaluate LLMs across a diverse set of tasks and capabilities in Filipino, Tagalog, and Cebuano. We carefully curate the tasks in FilBench to reflect the priorities and trends of NLP research in the Philippines such as Cultural Knowledge, Classical NLP, Reading Comprehension, and Generation. By evaluating 27 state-of-the-art LLMs on FilBench, we find that several LLMs suffer from reading comprehension and translation capabilities. Our results indicate that FilBench is challenging, with the best model, GPT-4o, achieving only a score of 72.23{\%}. Moreover, we also find that models trained specifically for Southeast Asian languages tend to underperform on FilBench, with the highest-performing model, SEA-LION v3 70B, achieving only a score of 61.07{\%}. Our work demonstrates the value of curating language-specific LLM benchmarks to aid in driving progress on Filipino NLP and increasing the inclusion of Philippine languages in LLM development."
}