FinLoRA: Financial Large Language Models with LoRA Adaptation

Overview

FinLoRA is a comprehensive framework for fine-tuning large language models on financial tasks using Low-Rank Adaptation (LoRA). This repository contains trained LoRA adapters for various financial NLP tasks including sentiment analysis, named entity recognition, headline classification, XBRL processing, RAG-enhanced models for CFA knowledge and FinTagging tasks, and APOLLO reasoning layers for advanced numerical calculations.

Model Architecture

Base Model: Meta-Llama-3.1-8B-Instruct (downloaded locally)
Adaptation Method: LoRA (Low-Rank Adaptation)
Quantization: 8-bit and 4-bit quantization support
Multi-Layer Support: RAG + APOLLO layered architecture
Local Usage: All models run locally without requiring Hugging Face online access
Tasks: Financial sentiment analysis, NER, classification, XBRL processing, CFA knowledge, FinTagging, numerical reasoning

Available Models

8-bit Quantized Models (Recommended)

sentiment_llama_3_1_8b_8bits_r8 - Financial sentiment analysis
ner_llama_3_1_8b_8bits_r8 - Named entity recognition
headline_llama_3_1_8b_8bits_r8 - Financial headline classification
xbrl_extract_llama_3_1_8b_8bits_r8 - XBRL tag extraction
xbrl_term_llama_3_1_8b_8bits_r8 - XBRL terminology processing
financebench_llama_3_1_8b_8bits_r8 - Comprehensive financial benchmark
finer_llama_3_1_8b_8bits_r8 - Financial NER
formula_llama_3_1_8b_8bits_r8 - Financial formula processing

RAG-Enhanced Models (Knowledge-Augmented)

cfa_rag_llama_3_1_8b_8bits_r8 - CFA knowledge-enhanced model with RAG
fintagging_combined_rag_llama_3_1_8b_8bits_r8 - Combined FinTagging RAG model
fintagging_fincl_rag_llama_3_1_8b_8bits_r8 - FinCL RAG-enhanced model
fintagging_finni_rag_llama_3_1_8b_8bits_r8 - FinNI RAG-enhanced model

APOLLO Models (Advanced Reasoning Layer)

apollo_cfa_rag_llama_3_1_8b_8bits_r8 - APOLLO reasoning layer for CFA tasks
apollo_fintagging_combined_llama_3_1_8b_8bits_r8 - APOLLO reasoning layer for FinTagging tasks

Note: APOLLO models are designed to be loaded on top of RAG models for enhanced numerical reasoning and calculation capabilities.

Bloomberg-Enhanced Models (Specialized Financial Tasks)

finlora_lora_ckpt_llama_8bit_r8 - Bloomberg FPB and FIQA specialized model
finlora_heads_llama_8bit_r8.pt - Bloomberg model weights (71MB)

Note: Bloomberg models are specialized for Financial Phrasebank (FPB) and Financial Question Answering (FIQA) tasks.

4-bit Quantized Models (Memory Efficient)

sentiment_llama_3_1_8b_4bits_r4 - Financial sentiment analysis
ner_llama_3_1_8b_4bits_r4 - Named entity recognition
headline_llama_3_1_8b_4bits_r4 - Financial headline classification
xbrl_extract_llama_3_1_8b_4bits_r4 - XBRL tag extraction
xbrl_term_llama_3_1_8b_4bits_r4 - XBRL terminology processing
financebench_llama_3_1_8b_4bits_r4 - Comprehensive financial benchmark
finer_llama_3_1_8b_4bits_r4 - Financial NER
formula_llama_3_1_8b_4bits_r4 - Financial formula processing

Quick Start

1. Installation

# Install dependencies
pip install -r requirements.txt

2. Local Model Setup

Important: This project uses locally downloaded models, not online Hugging Face models.

# The base Llama-3.1-8B-Instruct model will be automatically downloaded to local cache
# No internet connection required after initial setup
# All LoRA adapters are included in this repository

3. Basic Usage

from inference import FinLoRAPredictor

# Initialize predictor with 8-bit model (recommended)
predictor = FinLoRAPredictor(
    model_name="sentiment_llama_3_1_8b_8bits_r8",
    use_4bit=False
)

# Financial sentiment analysis
sentiment = predictor.classify_sentiment(
    "The company's quarterly earnings exceeded expectations by 20%."
)
print(f"Sentiment: {sentiment}")

# Entity extraction
entities = predictor.extract_entities(
    "Apple Inc. reported revenue of $394.3 billion in 2022."
)
print(f"Entities: {entities}")

4. Run Complete Test

# Test all models (this will download the base Llama model if not present)
python inference.py

# Test specific model
python -c "
from inference import FinLoRAPredictor
predictor = FinLoRAPredictor('sentiment_llama_3_1_8b_8bits_r8')
print('Model loaded successfully!')
"

Usage Examples

Financial Sentiment Analysis

predictor = FinLoRAPredictor("sentiment_llama_3_1_8b_8bits_r8")

# Test cases
test_texts = [
    "Stock prices are soaring to new heights.",
    "Revenue declined by 15% this quarter.",
    "The company maintained stable performance."
]

for text in test_texts:
    sentiment = predictor.classify_sentiment(text)
    print(f"Text: {text}")
    print(f"Sentiment: {sentiment}\n")

Named Entity Recognition

predictor = FinLoRAPredictor("ner_llama_3_1_8b_8bits_r8")

text = "Apple Inc. reported revenue of $394.3 billion in 2022."
entities = predictor.extract_entities(text)
print(f"Entities: {entities}")

XBRL Processing

predictor = FinLoRAPredictor("xbrl_extract_llama_3_1_8b_8bits_r8")

text = "Total assets: $1,234,567,890. Current assets: $456,789,123."
xbrl_tags = predictor.extract_xbrl_tags(text)
print(f"XBRL Tags: {xbrl_tags}")

RAG-Enhanced Models

# CFA RAG-enhanced model for financial knowledge
predictor = FinLoRAPredictor("cfa_rag_llama_3_1_8b_8bits_r8")

# Enhanced financial analysis with CFA knowledge
response = predictor.generate_response(
    "Explain the concept of discounted cash flow valuation"
)
print(f"CFA Response: {response}")

# FinTagging RAG models for financial information extraction
fintagging_predictor = FinLoRAPredictor("fintagging_combined_rag_llama_3_1_8b_8bits_r8")

# Extract financial information with enhanced context
entities = fintagging_predictor.extract_entities(
    "Apple Inc. reported revenue of $394.3 billion in 2022."
)
print(f"Enhanced Entities: {entities}")

APOLLO Models (Advanced Reasoning)

Important: APOLLO models are designed for advanced numerical reasoning and should be used for complex financial calculations.

# Load APOLLO model for advanced reasoning
apollo_predictor = FinLoRAPredictor("apollo_cfa_rag_llama_3_1_8b_8bits_r8")

# Financial calculations and reasoning
calculation = apollo_predictor.generate_response(
    "Calculate the present value of $10,000 received in 3 years with 5% annual discount rate"
)
print(f"APOLLO Calculation: {calculation}")

# Complex financial analysis
analysis = apollo_predictor.generate_response(
    "Analyze the impact of a 2% interest rate increase on a 10-year bond with 3% coupon rate"
)
print(f"APOLLO Analysis: {analysis}")

# Formula processing
formula_result = apollo_predictor.generate_response(
    "Solve: If a company has $1M revenue, 20% profit margin, and 10% growth rate, what's next year's profit?"
)
print(f"APOLLO Formula Result: {formula_result}")

Multi-Layer LoRA Architecture (RAG + APOLLO)

For maximum performance, you can combine RAG and APOLLO models:

# Step 1: Load RAG model for knowledge retrieval
rag_predictor = FinLoRAPredictor("cfa_rag_llama_3_1_8b_8bits_r8")

# Step 2: Load APOLLO model for reasoning (this will be layered on top)
apollo_predictor = FinLoRAPredictor("apollo_cfa_rag_llama_3_1_8b_8bits_r8")

# Use for complex financial reasoning tasks
complex_query = """
Given the following financial data:
- Revenue: $50M
- Cost of Goods Sold: $30M
- Operating Expenses: $15M
- Tax Rate: 25%

Calculate the net income and explain the calculation steps.
"""

response = apollo_predictor.generate_response(complex_query)
print(f"Multi-Layer Response: {response}")

Bloomberg-Enhanced Models (FPB & FIQA Specialized Tasks)

Important: Bloomberg models require special environment setup and are optimized for Financial Phrasebank (FPB) and Financial Question Answering (FIQA) tasks.

Environment Setup for Bloomberg Models

# 1. Create conda environment using the provided configuration
conda env create -f finlora_hf_submission/Bloomberg_fpb_and_fiqa/environment_contrasim.yml

# 2. Activate the environment
conda activate finenv

# 3. Navigate to the Bloomberg evaluation directory
cd finlora_hf_submission/Bloomberg_fpb_and_fiqa/

Testing Bloomberg Models on FPB and FIQA Datasets

# Run Bloomberg model evaluation
python trytry1.py

Configuration Notes for Testing:

Dataset Configuration: In trytry1.py, modify the EVAL_FILES line:

# Replace with your test datasets
EVAL_FILES = ["fiqa_test.jsonl", "fpb_test.jsonl"]

Model Path Configuration: For local testing, update the BASE_DIR in trytry1.py:

# For local Llama model deployment
BASE_DIR = "path/to/your/local/llama/model"

# For Hugging Face online model (original setting)
BASE_DIR = "d04e592bb4f6aa9cfee91e2e20afa771667e1d4b"

Model Components:
- ADAPTER_DIR: Points to the LoRA adapter (finlora_lora_ckpt_llama_8bit_r8)
- HEADS_PATH: Points to the model weights (finlora_heads_llama_8bit_r8.pt)

Bloomberg Model Usage Example

# Bloomberg models are specialized for FPB and FIQA tasks
# They provide enhanced performance on financial sentiment analysis
# and financial question answering compared to standard models

# The evaluation script automatically handles:
# - Model loading and configuration
# - Dataset processing
# - Performance metrics calculation
# - Memory management for large models

Local Model Management

Model Storage

Base Model: Downloaded to ~/.cache/huggingface/transformers/
LoRA Adapters: Stored in models/ directory
No Online Dependency: All models run locally after initial download

Model Loading Process

Base Model: Automatically downloaded on first use (~15GB)
LoRA Adapters: Loaded from local models/ directory
Quantization: Applied during loading (8-bit or 4-bit)
Device Detection: Automatically uses GPU if available, falls back to CPU

Performance Optimization

# For better performance on GPU
predictor = FinLoRAPredictor(
    model_name="sentiment_llama_3_1_8b_8bits_r8",
    use_4bit=False  # Use 8-bit for better performance
)

# For memory-constrained environments
predictor = FinLoRAPredictor(
    model_name="sentiment_llama_3_1_8b_4bits_r4",
    use_4bit=True   # Use 4-bit for memory efficiency
)

Evaluation

For Competition Organizers

This section provides guidance for evaluating the submitted models:

1. Quick Model Test

# Test if all models can be loaded successfully
python test_submission.py

2. Comprehensive Evaluation

# Run full evaluation on all models and datasets
python comprehensive_evaluation.py

# Check results
cat comprehensive_evaluation_results.json

3. Incremental Evaluation

# Run evaluation on missing tasks
python incremental_evaluation.py

# Check results
cat incremental_evaluation_results.json

4. APOLLO Model Testing

# Test APOLLO reasoning capabilities
python -c "
from inference import FinLoRAPredictor
apollo = FinLoRAPredictor('apollo_cfa_rag_llama_3_1_8b_8bits_r8')
result = apollo.generate_response('Calculate 15% of $1000')
print(f'APOLLO Test: {result}')
"

5. Bloomberg Model Testing (FPB & FIQA)

# Setup Bloomberg environment
conda env create -f finlora_hf_submission/Bloomberg_fpb_and_fiqa/environment_contrasim.yml
conda activate finenv

# Navigate to Bloomberg evaluation directory
cd finlora_hf_submission/Bloomberg_fpb_and_fiqa/

# Configure test datasets in trytry1.py:
# 1. Update EVAL_FILES = ["your_fiqa_test.jsonl", "your_fpb_test.jsonl"]
# 2. Update BASE_DIR for local model path or keep original for Hugging Face

# Run Bloomberg model evaluation
python trytry1.py

Project Structure

finlora_hf_submission/
├── models/                          # 8-bit LoRA model adapters (15 models)
│   ├── sentiment_llama_3_1_8b_8bits_r8/
│   ├── ner_llama_3_1_8b_8bits_r8/
│   ├── headline_llama_3_1_8b_8bits_r8/
│   ├── xbrl_extract_llama_3_1_8b_8bits_r8/
│   ├── xbrl_term_llama_3_1_8b_8bits_r8/
│   ├── financebench_llama_3_1_8b_8bits_r8/
│   ├── finer_llama_3_1_8b_8bits_r8/
│   ├── formula_llama_3_1_8b_8bits_r8/
│   ├── cfa_rag_llama_3_1_8b_8bits_r8/          # RAG-enhanced CFA model
│   ├── fintagging_combined_rag_llama_3_1_8b_8bits_r8/  # Combined RAG
│   ├── fintagging_fincl_rag_llama_3_1_8b_8bits_r8/     # FinCL RAG
│   ├── fintagging_finni_rag_llama_3_1_8b_8bits_r8/     # FinNI RAG
│   ├── apollo_cfa_rag_llama_3_1_8b_8bits_r8/           # APOLLO reasoning layer
│   ├── apollo_fintagging_combined_llama_3_1_8b_8bits_r8/ # APOLLO reasoning layer
│   └── xbrl_train.jsonl-meta-llama-Llama-3.1-8B-Instruct-8bits_r8/
├── Bloomberg_fpb_and_fiqa/          # Bloomberg specialized models for FPB & FIQA
│   ├── finlora_heads_llama_8bit_r8.pt
│   ├── finlora_lora_ckpt_llama_8bit_r8/
│   ├── environment_contrasim.yml    # Conda environment configuration
│   └── trytry1.py                   # Bloomberg model evaluation script
├── models_4bit/                     # 4-bit LoRA model adapters (8 models)
│   ├── sentiment_llama_3_1_8b_4bits_r4/
│   ├── ner_llama_3_1_8b_4bits_r4/
│   ├── headline_llama_3_1_8b_4bits_r4/
│   ├── xbrl_extract_llama_3_1_8b_4bits_r4/
│   ├── xbrl_term_llama_3_1_8b_4bits_r4/
│   ├── financebench_llama_3_1_8b_4bits_r4/
│   ├── finer_llama_3_1_8b_4bits_r4/
│   └── formula_llama_3_1_8b_4bits_r4/
├── testdata/                        # Evaluation datasets
│   ├── FinCL-eval-subset.csv
│   └── FinNI-eval-subset.csv
├── rag_system/                      # RAG system components
├── inference.py                     # Main inference script
├── comprehensive_evaluation.py      # Full evaluation script
├── incremental_evaluation.py        # Incremental evaluation
├── robust_incremental.py           # Robust evaluation
├── missing_tests.py                # Missing test detection
├── requirements.txt                 # Python dependencies
└── README.md                       # This file

Environment Requirements

Minimum Requirements (CPU Mode)

Python 3.8+
PyTorch 2.0+
8GB RAM
No GPU required

Recommended Requirements (GPU Mode)

Python 3.9+
PyTorch 2.1+
CUDA 11.8+ (for NVIDIA GPUs)
16GB+ GPU memory
32GB+ RAM

Installation Instructions

# 1. Clone or download this repository
# 2. Install dependencies
pip install -r requirements.txt

# 3. For GPU support (optional but recommended)
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118

# 4. Verify installation
python -c "import torch; print(f'PyTorch version: {torch.__version__}'); print(f'CUDA available: {torch.cuda.is_available()}')"

Troubleshooting

If you encounter memory issues:

Use 4-bit models instead of 8-bit models
Reduce batch size in inference
Use CPU mode if GPU memory is insufficient

If models fail to load:

Ensure all model files are present in the correct directories
Check that the base model (Llama-3.1-8B-Instruct) can be downloaded from HuggingFace
Verify internet connection for initial model download

Important Notes for Competition Organizers:

The base model (Llama-3.1-8B-Instruct) will be automatically downloaded from HuggingFace on first use (~15GB)
All LoRA adapters are included in this submission and do not require additional downloads
Models work in both CPU and GPU modes, with automatic device detection
APOLLO models provide enhanced reasoning capabilities for complex financial tasks
All models run locally without requiring ongoing internet connection

Model Details

Training Configuration

LoRA Rank: 8
LoRA Alpha: 16
Learning Rate: 1e-4
Batch Size: 4
Epochs: 3-5
Quantization: 8-bit (BitsAndBytes) / 4-bit (NF4)

Training Data

Financial Phrasebank
FinGPT datasets (NER, Headline, XBRL)
BloombergGPT financial datasets
Custom financial text datasets
APOLLO reasoning datasets for numerical calculations

License

This project is licensed under the MIT License - see the LICENSE file for details.

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

Contact

For questions and support, please open an issue in the repository.

Submission Summary

What's Included

17 Total Models: 15 8-bit models (9 original + 4 RAG-enhanced + 2 APOLLO) + 8 4-bit models
Complete Evaluation Results: Comprehensive and incremental evaluation results
RAG-Enhanced Models: CFA and FinTagging models with enhanced knowledge
APOLLO Reasoning: Advanced numerical reasoning and calculation capabilities
Cross-Platform Support: Works on CPU, GPU, and various memory configurations
Local Execution: All models run locally without online dependencies
Ready-to-Use: All dependencies specified, automatic device detection

Quick Start for Competition Organizers

Install dependencies: pip install -r requirements.txt
Test submission: python test_submission.py
Run evaluation: python comprehensive_evaluation.py
Test APOLLO reasoning: python -c "from inference import FinLoRAPredictor; apollo = FinLoRAPredictor('apollo_cfa_rag_llama_3_1_8b_8bits_r8'); print(apollo.generate_response('Calculate 10% of 500'))"

Test Bloomberg models (FPB & FIQA):

conda env create -f finlora_hf_submission/Bloomberg_fpb_and_fiqa/environment_contrasim.yml
conda activate finenv
cd finlora_hf_submission/Bloomberg_fpb_and_fiqa/
# Configure EVAL_FILES and BASE_DIR in trytry1.py
python trytry1.py

Check results: cat comprehensive_evaluation_results.json

Model Categories

Financial NLP: Sentiment, NER, Classification, XBRL processing
RAG-Enhanced: CFA knowledge and FinTagging with retrieval augmentation
APOLLO Reasoning: Advanced numerical calculations and financial reasoning
Memory Options: Both 8-bit and 4-bit quantized versions available

Acknowledgments

Meta for the Llama-3.1-8B-Instruct base model
Hugging Face for the transformers and PEFT libraries
The financial NLP community for datasets and benchmarks
APOLLO reasoning framework for enhanced numerical capabilities

Downloads last month: -; Downloads are not tracked for this model. How to track