Configure RAG with Ollama and Open WebUI on Linux

Last updated: Aug 5, 2026

RAG with Ollama and Open WebUI illustrated with an AI-themed Rags to Rags parody

In previous articles, I explained how to install Ollama to run local LLM models and how to install Open WebUI, a user-friendly interface for interacting with LLMs. In this tutorial, I will show how to implement Retrieval-Augmented Generation (RAG) with Ollama and Open WebUI. This allows a local LLM to use internal, private, or domain-specific information that was not included in its original training data while keeping the entire RAG workflow on premises.

By default, an LLM running through Ollama can rely only on its pretrained knowledge and on the information included in its current context. RAG makes it possible to add documents, procedures, technical notes, or company information to a searchable knowledge base. When a user asks a question, Open WebUI converts the query into an embedding, searches the vector database for the most relevant document chunks, and adds the retrieved content to the prompt sent to the LLM. This allows the model to generate answers grounded in your own data.

For this tutorial, I am using an NVIDIA Quadro P2000 graphics card with 5 GB of VRAM and the mistral:instruct LLM. However, I recommend using a more powerful graphics card with more VRAM if you want to run larger models or increase the model context size with the num_ctx parameter. A larger context window can accommodate more retrieved chunks and conversation history, but it also increases memory usage.

For the embedding model, I will use nomic-embed-text, a lightweight text embedding model available through Ollama. Its role is to convert text chunks into numerical vectors known as embeddings. Open WebUI stores these vectors, together with their associated chunks and metadata, in a vector database. When a question is submitted, the same embedding model converts the query into a vector so that Open WebUI can identify semantically similar chunks. This model accepts text input only.

You may wonder why RAG is useful when documents can already be attached directly to a conversation. Every LLM has a limited context window, which restricts the amount of text it can process in a single request. Loading an entire large document may consume too many tokens, cause useful content to be truncated, and leave less room for the conversation and generated answer. Instead, RAG divides documents into smaller chunks and retrieves only the sections considered most relevant to the current question. This reduces unnecessary token usage and makes it possible to search a knowledge base that is much larger than the model context window.

You may also wonder why we do not simply fine-tune the model with our own data. Fine-tuning is a different approach that modifies the model parameters and generally requires carefully prepared training data, suitable tools, technical expertise, and significant computing resources. For a fully local workflow, it also requires a model that supports the chosen fine-tuning method. Moreover, the model may need to be fine-tuned again when the information changes. RAG is therefore usually more practical for documents, procedures, inventories, and other information that must be updated regularly.

Understanding How RAG Works

The first step in a RAG workflow is to add the private, internal, or domain-specific information that we want the model to use when answering questions. In Open WebUI, the imported content is extracted and divided into smaller text chunks. Each chunk is then processed by the nomic-embed-text embedding model through Ollama, which converts it into a numerical vector.

These embeddings, together with their associated text chunks and metadata, are stored in a vector database. When a user asks a question, Open WebUI converts the query into an embedding and searches the database for the most semantically relevant chunks. The retrieved content is then added to the prompt sent to the LLM, allowing it to generate an answer grounded in the indexed knowledge base.

RAG document ingestion workflow showing documents imported into Open WebUI, split into chunks, converted into embeddings with nomic-embed-text through Ollama, and stored in a vector database — RAG document ingestion workflow showing text extraction, chunking, embedding generation, and vector indexing with Open WebUI and Ollama.

Once the document chunks and their embeddings have been indexed in the vector database, they can be retrieved when a user asks a question. Instead of relying only on its pretrained knowledge, the LLM receives the most relevant retrieved chunks as additional context and uses them to generate a more relevant and grounded answer.

RAG query workflow showing a user question sent through Open WebUI, relevant chunks retrieved from a vector database, added to the model context, and used by the LLM to generate the final answer — RAG query workflow showing how Open WebUI retrieves relevant chunks from the vector database and adds them to the model context before the answer is generated.

Install the RAG Embedding Model

On the server running Ollama, download the nomic-embed-text embedding model:

john@host:~$ ollama pull nomic-embed-text:latest

In Open WebUI, open the user menu, select Admin Panel, and then click Settings:

Open WebUI interface showing the Admin Panel option in the user menu and the Settings tab in the administration panel — Opening the Open WebUI administration panel and accessing its settings.

In the Documents settings, set Embedding Model Engine to Ollama, enter nomic-embed-text in the Embedding Model field, and then click Save:

Open WebUI Documents settings showing Ollama selected as the embedding model engine and nomic-embed-text configured as the embedding model — Configuring Ollama and nomic-embed-text as the embedding engine and model in Open WebUI.

Create a Knowledge Base in Open WebUI

The embedding model is now configured. The next step is to create a Knowledge Base in Open WebUI and add the documents or text content that will be indexed and used by the RAG workflow.

Click Workspace, then open the Knowledge tab:

Open WebUI interface showing the Workspace menu and the Knowledge tab — Opening the Knowledge section from the Open WebUI Workspace menu.

Click the Create button, enter a name and a short description for the new Knowledge Base, choose its access permissions, and then click Create Knowledge:

Open WebUI dialog for creating a Knowledge Base with the name STD.ROCKS website, a description of its purpose, private access permissions, and the Create Knowledge button — Creating a private Knowledge Base in Open WebUI by defining its name, purpose, and access permissions.

Click the + button, then select Add text content:

Open WebUI Knowledge Base showing the plus button menu with the Add text content option highlighted — Opening the Knowledge Base content menu and selecting **Add text content**.

Enter the text content that you want to add to the Knowledge Base, then click Save:

Open WebUI text editor containing information about the STD.ROCKS website with the Save button highlighted — Adding text content to the Open WebUI Knowledge Base and saving it for indexing.

Return to New Chat, click the + button, select Attach Knowledge, and choose the Knowledge Base you created. You can also type # in the prompt field and select the Knowledge Base from the list:

Open WebUI New Chat interface showing the plus menu, the Attach Knowledge option, and the STD.ROCKS website Knowledge Base selected — Attaching the STD.ROCKS Knowledge Base to a new Open WebUI conversation.

Ask a question related to the attached Knowledge Base. Open WebUI retrieves relevant content and provides it to the LLM as context for its answer:

Open WebUI conversation showing a question about the best rock band, an answer generated by mistral:instruct, and one retrieved source from the attached STD.ROCKS Knowledge Base — Open WebUI retrieves a relevant source from the attached Knowledge Base and provides it to the LLM as context for the generated answer.

RAG Configuration and Troubleshooting

Reindex the Knowledge Base

🚨 Warning: After changing the embedding model, reindex all existing Knowledge Base documents. Embeddings generated by different models are not compatible with each other.

If you change the embedding model or modify document indexing settings, you should reindex your existing Knowledge Base content so that previously added documents are processed again with the updated configuration.

In Open WebUI, open the user menu, select Admin Panel, and then click Settings:

In the Documents settings, scroll down to the Reindex Knowledge Base Vectors section, then click Reindex to rebuild embeddings for the existing indexed content:

Open WebUI Documents settings showing the Reindex Knowledge Base Vectors section with the Reindex button highlighted — Reindexing existing Knowledge Base vectors from the Open WebUI Documents settings.

Increase the Context Size

If your retrieved RAG content is large, or if your conversation already contains a lot of text, the LLM may not be able to fit all of the useful context into its context window. In that case, some retrieved chunks, instructions, or parts of the conversation may be truncated, which can reduce answer quality. To improve this, you can increase the num_ctx parameter, which defines the model context size.

💡 Note: Before increasing num_ctx, check the maximum context length supported by your model and make sure your system has enough available memory. A larger context size usually requires more VRAM or RAM. On systems with an NVIDIA GPU, you can monitor memory usage with the nvidia-smi command. Increasing num_ctx provides more room for retrieved chunks, conversation history, and generated output, but it does not automatically increase the number of retrieved chunks. Retrieval also depends on settings such as Top K and Chunk Size, described below.

In Open WebUI, open Admin Panel, and then click Settings. In the Models settings, edit the model used for RAG, and configure a custom value for the num_ctx Ollama parameter:

Open WebUI model settings showing the num_ctx Ollama parameter configured as a custom value of 8192 — Increasing the `num_ctx` value in the Open WebUI model settings to allow more context to be sent to the LLM.

Optimize RAG Retrieval

You can adjust several settings to improve the quality of your RAG workflow. The most important parameters are Chunk Size, Chunk Overlap, and Top K. Their optimal values depend on the structure of your documents, the embedding model, and the context window available to the LLM.

Consider the following parameters when tuning your RAG configuration:
- Chunk Size: defines the maximum size of each text chunk before it is converted into an embedding and indexed in the vector database. Depending on the selected Text Splitter, this value is measured in characters or tokens. Larger chunks preserve more surrounding context, but they may reduce retrieval precision and consume more of the LLM context window. Smaller chunks can improve retrieval precision, but they may lose important surrounding information.
- Chunk Overlap: defines how much content is repeated between consecutive chunks. This helps preserve meaning when a sentence, paragraph, or procedure crosses a chunk boundary. A common starting point is approximately 10% of the Chunk Size, such as a Chunk Size of 1000 with an overlap of 100.
- Top K: defines the maximum number of relevant chunks retrieved from the vector database and added to the model context. A value between 3 and 5 is generally a reasonable starting point for models with a limited context window. Increasing this value provides more retrieved content, but it also consumes more tokens and may introduce less relevant information.

💡 Note: There is no universal configuration that works best for every Knowledge Base. Test several values with representative questions, and reindex existing documents after changing the chunk size, chunk overlap, text splitter, or embedding model.

In the Documents settings, configure Chunk Size, Chunk Overlap, and Top K, then click Save:

Open WebUI Documents settings showing Chunk Size set to 1000, Chunk Overlap set to 100, and Top K set to 3 — Configuring the Chunk Size, Chunk Overlap, and Top K retrieval settings in Open WebUI.

Bypass RAG and Attach a File

💡 Note: This method can require a much larger context window than a standard RAG workflow. Before using it, make sure your model supports a large enough num_ctx value and that your system has enough available memory.

In some cases, the RAG workflow may not return the expected answer. This can happen for several reasons, such as suboptimal chunking, an unsuitable top_k value, embedding limitations, or insufficient context size. If your document is small enough to fit into the model context window, you can bypass retrieval and attach the original file directly to the conversation instead.

This approach can be useful when you want the LLM to read the full document rather than only a few retrieved chunks. However, it also consumes more input tokens and is limited by the maximum context size supported by the model.

Return to New Chat, click the + button, select Attach Files, and choose the file you want to send to the model:

Open WebUI New Chat interface showing the plus menu, the Attach Files option, and the STD.ROCKS.txt file selected — Attaching a file directly to a new Open WebUI conversation instead of using a Knowledge Base.

Click the attached file, then enable Using Entire Document so that Open WebUI sends the full file content to the model instead of relying on chunk retrieval:

Open WebUI file preview showing the attached STD.ROCKS.txt document with the Using Entire Document option enabled — Enabling **Using Entire Document** so the full file content is sent to the model context.

Conclusion

As we have seen, Retrieval-Augmented Generation (RAG) is an effective way to provide a local LLM with retrieved context from private, internal, or domain-specific documents without modifying the model itself. By combining Ollama, Open WebUI, an embedding model such as nomic-embed-text, and a vector database, it is possible to build a fully local knowledge base that can provide more relevant and better-grounded answers.

However, RAG is not a perfect solution and may require some tuning. Retrieval quality depends on several factors, including document structure, text extraction, Chunk Size, Chunk Overlap, Top K, the embedding model, and the context size available to the LLM. In some cases, testing different values and reindexing the Knowledge Base may be necessary before obtaining consistent results.

Despite these limitations, RAG remains a practical and flexible approach for providing a local LLM with frequently updated information at query time. Better hardware, especially a graphics card with more VRAM, can also make it easier to run larger models and use a wider context window.