small code and readme changes

README.md

@@ -47,10 +47,10 @@ LexVA follows a pure RAG approach without model finetuning:
 ### **Generation Model: Qwen3-14B**  
 Chosen for strong reasoning, thinking, and citation capability, as well as multilingual and long-context strengths.
 
-### **Retriever: Section-Level FAISS Index**
+### **Retriever: Section-Level**
 - Each statute section was embedded using the **Qwen3 Embedding** model.  
 - No internal chunking — the **complete statute text was the single retrieval unit**.  
-- Retrieval used vector similarity search (FAISS), typically with **top-k = 5**.
+- Vectors are stored in SQLite DB.
 
 ### **Synthetic QA Creation (used for evaluation)**
 - Qwen3-14B generated the natural-language questions and draft answers.
@@ -65,6 +65,8 @@ Chosen for strong reasoning, thinking, and citation capability, as well as multi
 
 No finetuning was performed; the evaluation tests the effectiveness of retrieval + prompting.
 
+Please note that prompts and answers are cached in the SQLite DB to speed up benchmarking tasks. 
+
 - **Developed by:** Dalila Cuevas Rodriguez
 - **Model type:** RAG Pipeline
 
@@ -80,6 +82,22 @@ LexVA is intended for:
 
 It is **not** intended for production legal advice or case outcome prediction.
 
+# Setup
+1. Clone this model repository using git `git clone https://huggingface.co/dcrodriguez/virginia-legal-rag-lexva`
+2. Optionally use a python 3.12 virtual environment
+3. Install the pip packages in `install.sh`
+4. Run `python setup.py`. This will download the dataset, load it into a SQLite DB, and generate all the document embeddings. 
+5. Run `python example.py`
+6. Write your own scripts using the chat_engine and retriever classes.
+
+Notes:
+- If you're running in an ephemeral environment like Google Colab, you should put your sqlite DB in your Google Drive. Change the relative path in `setup.py` to point to your drive, and then instantiate the chat engine like this:
+
+```python
+sqlite_path = "/content/drive/MyDrive/LexVA/va_code.db"
+chat_engine = LLMChatEngine(sqlite_path)
+```
+
 
 ## Bias, Risks, and Limitations
 
@@ -131,8 +149,25 @@ Use the code below to get started with the model.
 | Mistral 7B (no RAG)          | NA       | NA       | **0.855**                       | **0.224**         | **0.717**          |
 | Mistral 7B (with RAG)        | NA       | NA       | **0.946**                | **0.664**         | **0.790**                  |
 
+Cosine similarity measures how semantically similar two answers are by generating two embedding vectors. While it's quick to calculate, it ended up being a poor comparison metric for legal Q&A. You can see all the similarity values are within 8 percent. Answers generated without RAG were semantically similar, but contained many hallucinations and false statements. 
 
+The DeepEval benchmarks both use an LLM as a judge. I used GPT-5 Mini for these benchmarks as the judge model. Smaller local models had trouble with reasoning, and scores seemed inflated. 
+
+Faithfulness measures how correct the pipeline output is when given the relevant statute. The input is an actual output, and expected output, and retrieved context documents. It is a more standardized metric compared to GEval, but I think it struggled with RAG and non-RAG answers being similar despite citations being made up for non-RAG answers. Medium size models like Qwen 14B do a surprisingly good job answering these questions without RAG, but once again they make up most of their references, which makes it much less useful as a legal research tool. I think this is why Faithfulness metrics are as close as they are. Qwen3 14B scored higher than Mistral 7B which is expected.
+
+GEval was by far the most useful metric. The input is an actual output and an expected output. GEval supports custom grading criteria which was necessary to properly benchmark this pipeline. The evaluation steps are below. 
+
+```python
+evaluation_steps=[
+    "Check whether the facts in 'actual output' contradict any facts in 'expected output'.",
+    "Heavily penalize omission of important details from the expected output.",
+    "If a specific statute is referenced in 'expected output' but not 'actual output' lower the score, but numerically close references are ok. For example, 55.1-1243.2 is close to 55.1-1244.1, but is far from 19.2-308.",
+    "Additional statute references in actual output are ok",
+    "Minor stylistic differences or changes in wording are acceptable.",
+],
+```
 
+This benchmark was significantly more successful at identifying hallucinated citations. It evaluates both the claims made in the answer, as well as examines if the expected statute citations were used. You can see the largest delta between RAG and non-RAG answers using this benchmark. 
 
 ### Model Architecture and Objective
 

chat_engine.py

@@ -177,7 +177,7 @@ class LLMChatEngine:
             quantization_config=quant_config,
             device_map=device_map,     # spans GPU + CPU
             max_memory=max_memory,     # triggers offloading once GPU limit is hit
-            torch_dtype=torch.bfloat16 # or "auto", but be explicit if you like
+            dtype=torch.bfloat16 
         )
     
         # Precompute the token ids corresponding to "</think>" if possible

installs.sh

@@ -1,3 +1 @@
-pip install datasets numpy pandas tqdm
-pip install torch torchvision
-pip install transformers sentence-transformers
+pip install datasets numpy pandas tqdm torch torchvision transformers sentence-transformers deepeval beautifulsoup4 bitsandbytes