This is a repository for paper: Evaluating Open-Source LLMs in RAG Systems: A Benchmark on Diploma Theses Abstracts Using RAGAS
- Python 3.11 or higher
- Git
- OpenAI API key
-
Clone the repository
git clone https://github.com/margitantal68/rag_paper
-
Navigate to the project directory
cd rag_paper -
Create and activate a virtual environment
-
On Linux/macOS:
python3 -m venv venv source venv/bin/activate -
On Windows:
python -m venv venv venv\Scripts\activate
-
-
Set Up Elasticsearch
- Install Elasticsearch using Docker:
docker run -d -p 9200:9200 -e "discovery.type=single-node" -e "xpack.security.enabled=false" docker.elastic.co/elasticsearch/elasticsearch:8.9.0
- Install Elasticsearch using Docker:
-
Set Up Ollama
- Install Ollama and pull the required models
-
Install dependencies
pip install -r requirements.txt
This project requires an OpenAI API key. Follow these steps to set it up:
-
Obtain your OpenAI API key from OpenAI's website.
-
Copy the .env.example file in the project directory:
cp .env.example .env
-
Set the API key in the .env file:
OPENAI_API_KEY=your_api_key_here
-
Run the scripts in the following order:
- Create the Elasticsearch index
python theses_create_index.py
- Evaluate the Retriever
python theses_retrieval_evaluation.py
- Evaluate the Generation
python theses_rag_evaluation.py
theses_testset_creation_ragas_single_hop.py as it is not needed for the evaluation. The testset is already created and included in the repository theses\TESTSET\test_dataset.csv.
theses_testset_question_classification.py as it is not needed for the evaluation. The classification is already done and included in the repository theses\TESTSET\test_dataset.csv.