Ablageort im Repository (GitLab): Projekt2786-Evaluation Bundesbot
Diskussionsforum (Discourse): Projekt2786-Evaluation Bundesbot
Readme: Projekt2786-Evaluation Bundesbot
Beschreibung des Projektes:
PublicCode.YML: anzeigen
OSS Compliance: anzeigen
¶ Evaluation Bundesbot
-
Proof-of-concept
-
scripts for an automatic evaluation of an ITZBund Chatbot (Bundesbot) using a large language model (LLM)
-
features
- parsing the JSON-export of the Bundesbot
- generating new questions using an LLM
- new questions of a given topic
- variations of an existing question
- variations with typos
- translations
- evaluating the chatbot response
- retrieval of answers from the chatbot's API
- check compliance of answers with the JSON-export of the Bundesbot
- check whether chatbot response is meaningful using the LLM
-
overview:
¶ Setup
- python requirements
```
pip install dspy-ai scikit-learn matplotlib
``` - download JSON-Export from Chatbot Admin page
- CHATBOT-ADIMN-URL/admin/ui/admin/configuration/
- go to: bot > configuration > bot_configuration.json
- cloud symbol at the right upper corner
- access to an LLM, which can be
- an OpenAI API key (and sufficient credits) or
- a locally running open-source model, e.g. using vllm:
```
pip install vllm
python -m vllm.entrypoints.openai.api_server --model openchat/openchat-3.5-0106 --port 8000
```
¶ Parsing the JSON-export
-
functions: load.py
-
example:
```{python}
filename = "bot_configuration.json"load_chatbot_domains(filename=filename)
domains = ["kraftwerke", "gesundheit"]
load_chatbot_stories(filename=filename, select_domains=domains)
```
¶ Generating new questions with the help of an LLM
- generation classes: core.py
- generate new questions on a given topic
- generate question variations
- generate questions with mistakes (typos)
- generate translations
- demonstration of the generation classes: demo_generate_questions.ipynb
- building an SQLite-database of new questions: build_db.ipynb
- DB-file: `data/question_variations.db`
- table `questions` contains the existing questions from the JSON-Export
- table `variations` contains
- new questions on the topics of the existing questions
- new variations of the existing questions
- new variations of the existing questions with typos
- translations of the existing questions
¶ Evaluating the chatbot response
¶ Requesting the chatbot-API
- access with curl
```
curl --netrc-file password-file -X POST CHATBOT-URL/chat/http/ -H 'Content-Type: application/json; charset=utf-8' -d '{
"type": "text",
"data": {
"text": "Nachricht"
}
}'
```
¶ Check compliance of chatbot to JSON-export
- comparing the expected answer (as defined by the JSON-export) to the actual response by the chatbot
- demonstration: api_demobot.ipynb
- application to the SQLite-database: check_variations.ipynb
- analysis of the table `variations_evaluated`: analyse_variations.ipynb
¶ Check whether chatbot response is meaningful
- asking the LLM whether the response is answering the question
- demonstration: demo_evaluate_answers.ipynb
- application to the SQLite-database: evaluate_answers.ipynb
¶ Optimising the LLM-evaluation
- requires manually labelled data
- exploration of optimisation using DSPy: optimize_model.ipynb
- possible optimisation techniques: https://dspy-docs.vercel.app/docs/building-blocks/optimizers
- Automatic Few-Shot Learning
- Automatic Instruction Optimization
- Automatic Finetuning