Thamirawaran/llama2-7b-code-lora

LLaMA-2-7B CODE LoRA Adapter

This is a LoRA adapter for LLaMA-2-7B fine-tuned on code domain data.

Model Details

• Base Model: meta-llama/Llama-2-7b-hf
• Adapter Type: LoRA (Low-Rank Adaptation)
• Domain: Code
• Training Data: Python code instructions
• Training Examples: 2000 (1600 train, 200 val, 200 test)
• Epochs: 2

LoRA Configuration

• Rank (r): 16
• Alpha: 32
• Dropout: 0.05
• Target Modules: q_proj, v_proj, k_proj, o_proj

Performance Metrics (100 test examples)

• Loss: 0.573
• Perplexity: 1.773
• BLEU: 32.76

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

# Load base model
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-2-7b-hf",
    load_in_8bit=True,
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-hf")

# Load adapter
model = PeftModel.from_pretrained(model, "Thamirawaran/llama2-7b-code-lora")

# Generate
prompt = 'Write a Python function to sort a list'
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
outputs = model.generate(**inputs, max_length=256, temperature=0.7)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Training Details

• Trained with FYP_MDLE library
• 8-bit quantization during training
• Gradient accumulation: 16 steps
• Learning rate: 2e-4
• Warmup steps: 20

Citation

@misc{llama2-code-lora,
  author = {Team RAISE},
  title = {LLaMA-2-7B Code LoRA Adapter},
  year = {2026},
  publisher = {HuggingFace},
  howpublished = {\url{https://huggingface.co/Thamirawaran/llama2-7b-code-lora}}
}

License

Apache 2.0