Added a prompt helper to help manage the tokens, reduced the summary size to 800

Memory issues solved

I now have more GPU, therefore I have now reduced the GPU utililisation to 0.8

Removed the quantization line - that's crashed vLLM.

Increase the max model length, and corrected the quantisation to awq_merlin.

Added a background wait to counter the problem

Changed max model length to 3600 to improve hte KV cache issue..

Reduced the time in the wait for vLLM fnuction.

Reduced max model length, and increased gpu utilisation to 0.9

Specified chat mode, and made sure that the message was streamed for a nice UI action.

Reduced the GPU utilization and specified the quantization method.

Added a memory buffer, and moved the wait llm function to the main bit for gradio.

Checking something

Update vllm_server.py

Changed the GPU utilisation to 0.95

I have added some server specifics because the gradio bit isn't starting up.

Upgraded the max model length to 8092.

Changed the model from FP4 to AWQ

Changed the model from Instruct FP4 to AWQ

Decreased the maximum model lenght to 3408.

Moved the embedding model to the CPU. This will allow me to have more space on the GPU for the LLM.

Changed the time from 5 to 20 seconds.

Switched the model from Llama 3.3-70B to Llama-3.3-70B-Instruct-FP4.

Switching to a pre-quantised version of llama 3.3-70B sourced from Nvidia.

Added missing library socket.

Rectified indentation error on line 47.

Added start server and wait for server

Updated vllm_server to include a wait for vllm portion that ensures that the model is up before the chat section loads.

Updated "--max-model-len" from 2048 to 8092 to increase the context window and reduce the negative token error.

Reduced the max-token-sizes to 2048.

Ensuring that it's Python 3.10

Set python to 3.10

Removed the vllm package specification to ensure a compatible installation.

Updated requirements

Changes added to move to vLLM

Uploading a vllm server file

Specified the provider (groq) anf edited max new tokens to 16k.

Increased the max new tokens and edited the system prompt to remove the in text citations. Reduced the temperature to 0.5 since there was a lot more faff in the response.

Forgot ot update to the new database.

Removed the sources display to update the look of the chatbot to something more friendly.

Uploading the updated database to change things up.

Updated the temperature to 0.8 , and the max new tokens to 8192. Removed the context window, might put it in later as 16000.

Updated max new tokens to 4096 to improve the cut offs.

Switched back to an inference provider for speed

I forgot a comma.

Added back in the tokenizer

Removed max_new_tokens

Added the HF_TOKEN collection

Updated HuggingFaceLLM parameters

Removed the provider=auto to fix type error