| | --- |
| | license: mit |
| | widget: |
| | language: |
| | - en |
| |
|
| | datasets: |
| | - pytorrent |
| | --- |
| | |
| | # 🔥 RoBERTa-MLM-based PyTorrent 1M 🔥 |
| | Pretrained weights based on [PyTorrent Dataset](https://github.com/fla-sil/PyTorrent) which is a curated data from a large official Python packages. |
| | We use PyTorrent dataset to train a preliminary DistilBERT-Masked Language Modeling(MLM) model from scratch. The trained model, along with the dataset, aims to help researchers to easily and efficiently work on a large dataset of Python packages using only 5 lines of codes to load the transformer-based model. We use 1M raw Python scripts of PyTorrent that includes 12,350,000 LOC to train the model. We also train a byte-level Byte-pair encoding (BPE) tokenizer that includes 56,000 tokens, which is truncated LOC with the length of 50 to save computation resources. |
| |
|
| | ### Training Objective |
| | This model is trained with a Masked Language Model (MLM) objective. |
| |
|
| | ## How to use the model? |
| | ```python |
| | from transformers import AutoTokenizer, AutoModel |
| | |
| | |
| | tokenizer = AutoTokenizer.from_pretrained("Fujitsu/pytorrent") |
| | model = AutoModel.from_pretrained("Fujitsu/pytorrent") |
| | ``` |
| | ## Citation |
| | Preprint: [https://arxiv.org/pdf/2110.01710.pdf](https://arxiv.org/pdf/2110.01710.pdf) |
| | ``` |
| | @misc{bahrami2021pytorrent, |
| | title={PyTorrent: A Python Library Corpus for Large-scale Language Models}, |
| | author={Mehdi Bahrami and N. C. Shrikanth and Shade Ruangwan and Lei Liu and Yuji Mizobuchi and Masahiro Fukuyori and Wei-Peng Chen and Kazuki Munakata and Tim Menzies}, |
| | year={2021}, |
| | eprint={2110.01710}, |
| | archivePrefix={arXiv}, |
| | primaryClass={cs.SE}, |
| | howpublished={https://arxiv.org/pdf/2110.01710}, |
| | } |
| | ``` |
| |
|
