LLM Lingen

A modular, full-stack RAG (Retrieval-Augmented Generation) platform designed for academic research, enabling systematic comparison of how different LLM pipeline components affect answer quality, all without writing a single line of code.

Motivation

When researching how different LLM components affect answer quality, you don't want to rewrite your entire pipeline every time you swap out an embedder, try a different chunking strategy, or switch LLM providers. LLM Lingen solves this by making every stage of the RAG pipeline a swappable module: configurable through a web dashboard with immediate evaluation against standardized benchmarks.

High-Level Architecture

┌─────────────────────────────────────────────────────────────┐
│                        Frontend                              │
│  ┌─────────────────────┐    ┌─────────────────────────────┐ │
│  │  Legacy (HTML/JS)   │    │  React + TypeScript (Vite)  │ │
│  │  - Admin Dashboard  │    │  - Admin Dashboard          │ │
│  │  - User Chat UI     │    │  - Settings, Datasets,      │ │
│  │  - Jinja2 Templates │    │    Questions, Logs, Users   │ │
│  └────────┬────────────┘    └──────────┬──────────────────┘ │
│           │           REST API          │                    │
└───────────┼─────────────────────────────┼────────────────────┘
            │                             │
            ▼                             ▼
┌─────────────────────────────────────────────────────────────┐
│                    FastAPI Backend                            │
│                                                              │
│  ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌───────────────┐  │
│  │   Auth   │ │  Query   │ │   Chat   │ │  Evaluation   │  │
│  │  (JWT)   │ │  (RAG)   │ │ Sessions │ │  (Benchmark)  │  │
│  └──────────┘ └──────────┘ └──────────┘ └───────────────┘  │
│  ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌───────────────┐  │
│  │ Datasets │ │ Q-Sets   │ │ Settings │ │    Logging    │  │
│  └──────────┘ └──────────┘ └──────────┘ └───────────────┘  │
│                                                              │
│  ┌──────────────────────────────────────────────────────┐   │
│  │              Service Layer (Factories)                │   │
│  │  Every component is instantiated via Abstract Factory │   │
│  └──────────────────────────────────────────────────────┘   │
│                                                              │
└──────────────────────────┬───────────────────────────────────┘
                           │
                           ▼
┌─────────────────────────────────────────────────────────────┐
│                   SQLAlchemy / SQLite                         │
│  Users, Pools, Settings, Datasets, QuestionSets, Logs, etc. │
└─────────────────────────────────────────────────────────────┘

The RAG Pipeline

Each stage of the pipeline can be independently swapped at runtime through the admin dashboard:

PDF Upload → PDF Parser → Preprocessor → Chunker → Embedder → Vector Store
                                                                    │
User Query ──────────────────► Retrieval → Reranker → Prompt Builder → LLM → Response

Abstract Factory Pattern, The Core Design Decision

The defining architectural choice is the Abstract Factory Pattern applied consistently across every pipeline component. For academic research, you need to compare how different implementations of the same pipeline stage affect output quality.

The pattern is applied to 11 component families:

Component	Available Implementations
LLMs	Ollama, OpenAI, Anthropic, Cohere, Mistral, vLLM, HuggingFace Local
Embedders	OpenAI, Sentence-Transformers, Ollama
Vector Stores	FAISS, Qdrant, Milvus, PGVector, Elasticsearch
Chunkers	Simple (fixed-size), Recursive (structure-aware), Token-based
PDF Parsers	PyPDF2, pdfplumber, pdfminer
Pipelines	Simple Chat, Conversational RAG, Haystack-native
Preprocessors	Text cleaner/normalizer
Prompt Builders	Registry-based template system
Rerankers	Sentence-Transformer Cross-Encoder
Tokenizers	Whitespace, Regex, Character
Token Counters	HuggingFace tokenizer, tiktoken

The admin selects components through the settings UI. The backend resolves the correct implementation via the factory at runtime, no restart, no code change.

Question Generation & Evaluation

For any uploaded document, the system generates questions across 11 categories designed to test different aspects of LLM comprehension:

By format: Open questions, closed questions, multiple choice, fill-in-the-blank
By difficulty: Simple, medium, complex
By challenge type: Contextualized, ambiguous, erroneous (deliberately flawed), ethics/bias

Each question comes with a generated answer and the prompt used to produce it, creating a ground-truth dataset for benchmarking different pipeline configurations using metrics from scikit-learn and NLTK.

Haystack as the RAG Backbone

Haystack (farm-haystack)

The entire RAG pipeline is built on top of Haystack, the open-source NLP framework by deepset. Haystack is not just a utility, it is the central orchestration layer that ties the platform together:

Document dataclass: the universal data container used throughout the codebase to represent text chunks flowing through every pipeline stage.
Native pipeline abstractions: Haystack's composable pipeline API is used to wire retrieval, embedding, and generation stages together in a declarative way.
Built-in document store integrations: FAISS, Elasticsearch, and other vector stores are accessed through Haystack's document store interface, making it trivial to swap backends.
Retriever and Reader components: the core retrieval logic uses Haystack's retriever abstractions, which standardize how documents are fetched from vector stores.
Prompt node integration: LLM generation is routed through Haystack's prompt node, keeping prompt template management and model invocation consistent.

The rag_runner.py module implements the core RAG flow: given a user query and a set of context documents (as Haystack Document objects), it builds a prompt using the configured prompt template and sends it to the selected LLM. A second module, rag_runner_questionset.py, extends this for batch question generation: given a document, it generates questions across all 11 categories and their expected answers, which can then be used as ground truth for evaluation.

Multi-Tenant User System

The platform supports multiple user groups through a Pool system. Each Pool has a configurable user limit, an assigned LLM model, access to specific datasets, and a unique join token. Users authenticate via JWT and are assigned admin or user roles.

Logging & Observability

Every LLM generation is logged with full context: prompt template, rendered prompt, raw response, response time, token counts, and errors, essential for research reproducibility.

What This Enables

Upload academic papers as PDFs
Configure the RAG pipeline (chunker, embedder, vector store, LLM, etc.)
Generate a standardized question set across 11 categories
Run the question set and collect answers
Evaluate answer quality using automated metrics
Swap one component and re-run the same questions
Compare results across configurations, all through the web UI