Create README.md

README.md

@@ -1,50 +1,97 @@
 ---
-tags:
-- generated_from_trainer
-model-index:
-- name: klue_roberta_base_for_legal
-  results: []
+language:
+- ko
+metrics:
+- accuracy
+library_name: transformers
 ---
 
-<!-- This model card has been generated automatically according to the information the Trainer had access to. You
-should probably proofread and complete it, then remove this comment. -->
+# KLUE Robeta-base for legal documents
 
-# klue_roberta_base_for_legal
+<!-- Provide a quick summary of what the model is/does. -->
 
-This model is a fine-tuned version of [klue/roberta-base](https://huggingface.co/klue/roberta-base) on an unknown dataset.
+- KLUE/Robeta-Base Model을 판결문으로 이뤄진 legal_text_merged02_light.txt 파일을 사용하여 재학습 시킨 모델입니다.
 
-## Model description
 
-More information needed
+## Model Details
 
-## Intended uses & limitations
+### Model Description
 
-More information needed
+<!-- Provide a longer summary of what this model is. -->
 
-## Training and evaluation data
+- **Developed by:** J.Park @ KETI
+- **Model type:** klue/roberta-base
+- **Language(s) (NLP):** korean
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
 
-More information needed
+### 학습 방법
 
-## Training procedure
+```python
 
-### Training hyperparameters
+from transformers import RobertaTokenizer, RobertaForMaskedLM
+from transformers import AutoModel, AutoTokenizer
 
-The following hyperparameters were used during training:
-- learning_rate: 5e-05
-- train_batch_size: 18
-- eval_batch_size: 8
-- seed: 1
-- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
-- lr_scheduler_type: linear
-- num_epochs: 5
+model = RobertaForMaskedLM.from_pretrained(base_model)
+tokenizer = AutoTokenizer.from_pretrained(base_tokenizer)
 
-### Training results
+from transformers import LineByLineTextDataset
 
+dataset = LineByLineTextDataset(
+    tokenizer=tokenizer,
+    file_path=fpath_dataset,
+    block_size=512,
+)
 
+from transformers import DataCollatorForLanguageModeling
 
-### Framework versions
+data_collator = DataCollatorForLanguageModeling(
+    tokenizer=tokenizer, mlm=True, mlm_probability=0.15
+)
 
-- Transformers 4.28.1
-- Pytorch 2.0.0+cu117
-- Datasets 2.12.0
-- Tokenizers 0.13.3
+from transformers import Trainer, TrainingArguments
+
+training_args = TrainingArguments(
+    output_dir=output_dir,
+    overwrite_output_dir=True,
+    num_train_epochs=5,
+    per_device_train_batch_size=18,
+    save_steps=100,
+    save_total_limit=2,
+    seed=1
+)
+
+trainer = Trainer(
+    model=model,
+    args=training_args,
+    data_collator=data_collator,
+    train_dataset=dataset
+)
+
+train_metrics = trainer.train()
+
+
+trainer.save_model(output_dir)
+trainer.push_to_hub()
+```
+
+### 학습용 configuration
+
+- number of epochs
+```bash
+  epochs = 50
+```
+  
+- JSON file
+```json
+[
+    {'basemodel'      : 'againeureka/klue_roberta_base_for_legal', 
+     'basetokenizer'  : 'klue/roberta-base', 
+     'trainmodel'     : 'againeureka/toulmin_classifier8_klue_roberta_base_retrained6', 
+     'batchsize'      : 92,
+     'epochs'         : epochs,
+     'push_to_hub'    : True,
+     'is_on'          : True,
+    },
+]
+```