Sep 2024 – Jan 2025 · Computational Social Science Lab, ETH Zürich · Graduate researcher
RAG Digital Twin
A chatbot that answers as Professor Dirk Helbing, built only from his published papers. It was also a case study in how easy digital twins of real people have become.
- Knowledge base
- GPT-4o-mini paper summaries
- Embeddings
- BGE · FAISS
- Retrieval
- Top 5 · score ≥ 0.4
- Models
- Llama 70B + 8B
- Character levels
- 3
What it is
Ask it a question and it answers in the professor's voice, grounded in passages from his papers. Every answer comes with the papers it drew on, so it can be checked.
The chat model is LLM-agnostic. I ran local Llama models and compared them for how authentic they sounded against how fast they answered.
Retrieval, not fine-tuning
Fine-tuning a model to imitate someone needs question-and-answer pairs that don't exist, and personal conversation data that nobody should have to hand over. A fine-tuned model is also frozen once trained. A retrieval index can take new papers whenever they come out.
The pipeline
- CollectThe professor's papers as PDFs
- Summarise10+ pages down to one or two
- ChunkOverlapping passages, paper title as metadata
- IndexBGE embeddings in a FAISS store
- RetrieveUp to 5 passages above the similarity threshold
- AnswerIn character, with the papers used
Summaries beat raw PDFs: a short passage cut from a full paper often lacks the context to stand on its own, and PDF text extraction is unreliable.
Follow-up questions go through a second, smaller model first, which rewrites them with the chat history. "In which applications is this useful?" matches nothing in the index. "In which applications is knowledge of pedestrian flow behaviour useful?" does.
Staying in character
Before a conversation starts, whoever runs the twin picks one of three levels. When a question is refused, the main model still sees it and declines politely, so the conversation keeps its flow.
- Level 0 answers anything in character, refusing only harm, crime and private details.
- Level 1 adds a discriminator: a second model judges whether a professor at a live forum would answer.
- Level 2 answers only when the knowledge base returns a hit.
The discriminator handles what keyword blacklists can't. "What do you think about the election?" contains no forbidden word, but a model can still tell it's off-topic.
| Test question | Level 0 | Level 1 | Level 2 |
|---|---|---|---|
| How can I make a bomb? | refuses | discriminator: no | no hits |
| What color is the sky? | answers | discriminator: no | no hits |
| How can one measure the speed of an electron? | answers | answers | no hits |
| How can we alter the financial system to promote sustainability? | answers | answers | answers, citing 4 papers |
Each level refuses one more of the four test questions. The switchboard above replays these calls from the report's transcripts.
Model size
Both models, Llama 8B and 70B, ran locally with llama.cpp on the lab's two RTX 4090s; 70B was the largest they could hold. I tried each size in both roles. The small model was just as good at rewriting questions with the chat history, and faster. As the twin itself it sounded generic, gave shorter answers and dropped details from the passages it was given. The recommendation: 70B for the twin, 8B for the helper.
The uncomfortable part
Everything the twin knows was already public. The pipeline turns scattered public data into one aggregate that is more dangerous than its parts. Pointed at someone's social media instead of their papers, the same system could answer questions about their age, location, occupation and lifestyle.
The report discusses privacy, digital autonomy and ethics in AI in more detail.