Installation

This chapter describes the installation of OTOBO AI. It guides you through the setup process for both the services and the OTOBO package. For further configuration, refer to the specific use case.

There are two components to OTOBO AI:

The services otobo-ai-services.

The otobo-ai OTOBO package.

These installation instructions illustrate their installation.

otobo-ai-services Docker Compose Stack

The otobo-ai-services are provided as self-contained docker compose stack. There are two docker containers:

rag

A Chroma database for vector data and webservice to interface with OTOBO

postgres

A database holding prepared data

The repository may be found here: https://github.com/RotherOSS/otobo-ai-services

First, set up the docker compose project:

cd /opt
git clone git@github.com:RotherOSS/otobo-ai-services.git
cd otobo-ai-services

Create a .env file in the root directory to configure environment variables:

cp .docker_compose_env_ai .env

Edit the .env file to set your desired configuration options.

Use Docker Compose to run the configured services:

docker compose up --detach

OTOBO-AI Package

The package can simply be installed via the OTOBO Package Manager on OTOBO version >= 11.0. See the OTOBO Admin Guide: Package Manager for details on package management in OTOBO.

Configure the package settings in the system configuration. See the OTOBO Admin Guide: System Configuration for general instructions. The package manual includes all relevant information. You may just search OTOBOAI:: to find all relevant settings. These are the basic steps:

Select data for import into the RAG stack (what your answers are based on).

Configure the auth token into the webservice.

Setup the answer template.

Continue depending on your use case with Chat or Retrieval Augmented Generation (RAG).

Retrieval Augmented Generation (RAG)

The RAG system allows you to automatically generate answers to customer requests based on the information available in your OTOBO. This document details the configuration and usage of the RAG system.

Workflow

At first, all tickets, FAQ and external documentation are imported in the RAG system. During this step, the information is scrubbed, chunked, and sent to a small LLM to obtain a numeric vector that describes the presented information. This procedure is called embedding. Every vector describes a point in high dimensional space. Similar information will be close to each other, while unrelated information will be far apart. All the vectors and documents are stored in the otobo-ai-services. This allows for fast selection of documents that address the same topic.

If a new article is created, generate an answer:

An embedding is requested for the new article. The resulting vector is used to identify related documents.

The related documents are injected into the prompt for a larger LLM to generate an answer based on the provided information.

The generated answer is stored in a Dynamic Field at the article.

The dynamic field can be referenced in an Answer Template.

If the user chooses this Answer Template, the generated answer of the LLM will be visible in the compose window.

Setup

OTOBO-AI supports multiple RAG definitions, each with its own input data selection, prompt and output queue. While the RAG definition is stored in the otobo-ai-services docker compose project, input and output is configured in the OTOBO package. These setups are called pipelines. You may have multiple separate pipelines, for example one for each department or product you are supporting. Pipelines allow for complete separation of concerns.

┌───────────────┐                          ┌─────────┐
│ ticket search ┼───┐                  ┌───► queue 1 │
└───────────────┘   │                  │   └─────────┘
   ┌────────────┐   │    ┌────────┐    │   ┌─────────┐
   │ faq search ┼───┼────►  RAG   ┼────┼───► queue 2 │
   └────────────┘   │    └───▲────┘    │   └─────────┘
   ┌────────────┐   │    ┌───┼────┐    │   ┌─────────┐
   │ doc folder ┼───┘    │ prompt │    └───►    ...  │
   └────────────┘        └────────┘        └─────────┘
      INPUT                                  OUTPUT

RAG Pipeline in OTOBO

To set up such a pipeline, you need to create a RAG definition.

RAG Definition

Under /opt/otobo-ai-services/rag_examples, example RAG definitions are provided. As a basic template the default RAG definition is provided. This RAG definition supports Tickets (t), FAQ (f) and Documentation (d). Copy the RAG description to your RAG definition folder:

cp -r rags_examples/default rags

All RAG definitions placed here are exposed at the web service. You may tune it to your liking (see Tuning), or create a new one! You should at least tailor the prompts/rag_prompt.txt to your use case. Template text is provided in this file.

Note

Docker mounts the local ./rags directory into the container as src/rags, enabling external customization. However, a restart of the container is required for the changes to take effect.

Input & Output Configuration

Next, you need to configure the input and output of the RAG definition in the OTOBO package. Navigate to the system configuration. For the first RAG definition, the following settings need to be configured:

OTOBOAI::Document::SearchRestrictions###0100-Default

Location of the documentation to be imported into the RAG system.

OTOBOAI::FAQ::SearchRestrictions###0100-Default

FAQ search restrictions to select the FAQ articles to be imported into the RAG system.

OTOBOAI::Ticket::SearchRestrictions###0100-Default

Ticket search restrictions to select the tickets to be imported into the RAG system.

Search restrictions are able to use any field and restriction from the classic search function (refer to the User Guide for details). You may use the same configuration suffixed ...###0200-Custom1 for a second RAG definition, and so on.

To configure which pipeline generate answers for a queue, configure the system setting OTOBOAI::QueueToRAGMapping.

Note

Only text information is indexed by the RAG system and used for answer generation. No documents, pictures, or attachments will be processed.

Initial Data Import

Place documentation under the path configured in the system setting OTOBOAI::Document::SearchRestrictions relative to your OTOBO installation.

Note

Pandoc may be helpful with data conversion. Beware of index files, they match on every topic and are relatively dense. Beware of files like a license, they are never a match.

And import all the data from command line:

# run --help for an overview
# importing may take some time depending on the data size and LLM throughput
bin/otobo.Console.pl Maint::AI::Import

It is advisable to exclude a portion of the data from import. The import command supports the parameters --sample-nth and --sample. A generated sample may serve as evaluation set for tuning (see Tuning).

You may verify the import using the command line from the rag container:

docker compose exec rag python src/cli.py

Incremental Data Ingestion

The OTOBO-AI package supports the automatic synchronization of newly added or updated tickets and FAQ items via incremental ingestion. More precisely, OTOBO listens to a pre-defined set of Ticket and FAQ related Events, which in turn re-query the SearchRestrictions when triggered.

This means that the configuration of OTOBOAI::Document::SearchRestrictions and Ticket::EventModulePost###8300-OTOBOAIIncrementalSynchronization are closely coupled. For correct incremental ingestion, please make sure that your specified Events fully cover your SearchRestrictions, per data source.

For a list of ticket related events, refer to the Developer Manual. The event configuration supports regular expressions, as seen in the default setting:

TicketCreate|Ticket.*Update|TicketMerge|TicketDelete|ArticleCreate|ArticleUpdate|ArticleEdit

The FAQ event configuration is done by FAQ::EventModulePost###8300-OTOBOAIIncrementalSynchronization. Supported events are covered by the default setting:

FAQCreate|FAQUpdate|FAQDelete

Attention

This functionality is disabled by default. It also relies on a running OTOBO Daemon, as it handles the asynchronous ingestion calls.

On OTOBO 11.0 the FAQ event configuration requires the optional Elasticsearch-FAQ package to provide event handling. For 11.1 and later, the FAQ event configuration is supported by default.

Answer Generation

You may generate answers either prompted by a customer request or manually on-demand.

On-Demand Answers

There are two options to obtain on-demand answers, either the command line or a new widget in the agent overview. The widget is served by the default RAG. If you handle sensitive information with the default RAG, add a Group to the setting DashboardBackend###0160-OTOBOAIDashboard to restrict access to the widget.

The command line may be reached from the command line interface of OTOBO:

docker compose exec web ./bin/otobo.Console.pl Maint::AI::GetAnswer "Hello Roboto!"

Automatic Answer Suggestions

However, OTOBO may query the RAG for an answer suggestion whenever a new article is created. Upon retrieval, the answer is stored in a dynamic field, configured in OTOBOAI::DynamicFieldOTOBOAI. It may be referenced from an answer template. The pre-configured answer template is OTOBOAI Answer Template. Assign it to the queues you want to offer RAG answer suggestions to the agents. Refer to the Admin Manual for more information. If an agent chooses this answer template, the AI answer is revealed, may be edited by the agent and sent out to the customer.

Note

You may also assign the dynamic field to the AgentTicketZoom to display the answer at the article and have it accessible without the need to lock the ticket.

Chat

OTOBO may interface with an external chat system. The connection is established via the OTOBO webservices and an MCP (Model Context Protocol) server. The following diagram illustrates the connection between OTOBO, the external chat system, and the MCP server:

                     ┌───────┐
link:         ┌──────┼ OTOBO ◄──────┐
 - session_id │      └───────┘      │  webservices
 - ...        │                     │
           ┌──▼───┐              ┌──┼──┐
           │ CHAT ┼──────────────► MCP │
           └──────┘   session_id └─────┘
                      [...]

Dataflow within the OTOBO AI chat setup

The OTOBO-AI package generates a session_id for the current user and the IP if the MCP and allows to pass it as a link parameter. OTOBO keeps session_ids bound to IP addresses as per system configuration SessionCheckRemoteIP. Further, it allows to create a link to an external chat service and displays it beside every article. It may pass the ticket_id and the article_id as link parameters to the external chat service. This allows an agent to jump into an AI chat out of the ticket context and to pick up the context within the chat. The chat calls the MCP server to interface with the OTOBO webservices, pre-configured by the OTOBO-AI package.

MCP Server

The OTOBO MCP server serves as a proxy to the OTOBO webservices. It relies on the session_id passed by the chat service to authenticate as the user at the OTOBO webservices. This allows to keep the access management consistent with the configuration in OTOBO, i.e., a user may only access resources (e.g., tickets) which are accessible to them in OTOBO.

The MCP server is developed in its own repository otobo-ai-mcp. However, its images are published to Docker Hub and may be used in the OTOBO Docker Compose setup. Example configuration is provided in the docker-compose/otobo-ai_mcp.yml file of the otobo-ai-services. To activate the MCP server, you need to add the file to the COMPOSE_FILE variable in the .env file and fill out the required variables in the MCP labeled section.

The MCP server is not exposed to the outside by default. It is meant to serve as an internal interface between the chat service and the OTOBO webservices. If you want to expose the MCP server to the outside, you need to configure a reverse proxy in your OTOBO Docker Compose setup to handle SSL and strongly control access by a firewall.

Note

If you use the MCP server inside a chat service, simply ask the LLM about the capabilities of the MCP server.

Chat Service

The chat service is interchangeable. However, it needs to be able to pick up the GET parameters, session_id and the different context ids provided by the OTOBO-AI package and place them to the chat context. It is recommended to separate the chat sessions of different users. Hence, the chat service should be able to handle user authentication. Users may be managed by the chat service itself or by an external identity provider like LDAP or OIDC.

Within the otobo-ai-services OpenWebUI is provided as a reference implementation of a chat service. You may find the Docker Compose file in the repository at docker-compose/otobo-ai_openwebui.yml. The service is pre-configured to share most of the variables declared in the .env (see: Installation). To activate the service you need to add the file to the COMPOSE_FILE variable in the .env file and fill out the required variables in the chat labeled section.

# after changes to the .env file, reinitialize the Docker Compose setup:
docker compose down && docker compose up --detach

We may use NGINX of your OTOBO Docker Compose setup to handle SSL and pose as a reverse proxy to the OpenWebUI service on port 9000. To do so, you need to configure a custom override. Navigate to your OTOBO Docker Compose project, by default in /opt/otobo-docker. Place a file named otobo-override-nginx-openwebui.yml in this folder.

touch /opt/otobo-docker/docker-compose/otobo-override-nginx-openwebui.yml

Place the following content in the file:

services:
  nginx:
    ports:
      - "9000:9000"

This will expose the port 9000 of the NGINX container to the outside. Further, you need to override the NGINX configuration template used to generate the configuration to serve OTOBO. Please refer to the OTOBO Installation Guide for details. Add this block to the end of the exposed template:

server {
    listen 9000 ssl;
    listen [::]:9000 ssl;
    http2 on;

    include snippets/ssl-params.conf;
    ssl_certificate     ${OTOBO_NGINX_SSL_CERTIFICATE};
    ssl_certificate_key ${OTOBO_NGINX_SSL_CERTIFICATE_KEY};

    server_name  localhost;

    client_max_body_size 1G;

    location / {
        proxy_http_version 1.1;

        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $host;

        proxy_set_header X-Forwarded-Host $host:$server_port;
        proxy_set_header X-Forwarded-Server $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_pass       http://chat:8080/;

        proxy_read_timeout 300s;
        proxy_send_timeout 300s;
    }
}

Now we need to add the file otobo-override-nginx-openwebui.yml to the end of the COMPOSE_FILE variable in the .env file. Then we may reinitialize the NGINX container to apply the changes:

docker compose down nginx && docker compose up --detach

After start, you may access the OpenWebUI service at https://<your-otobo-host>:9000. For user management setup, see the documentation of OpenWebUI. Using an OIDC service that is able to handle Single Sign-On (SSO) provides the most seamless experience for the user.

Model configuration heavily depends on the LLM API used (see also the OpenWebUI documentation). At the time of writing, the best result is achieved, by:

Setting the model to be used to public so it can be freely accessed.

Set a system prompt that refers to the MCP tool but does not respond with a welcome message in the first response. Even in multiple scenarios we were unable to get the LLM to use the MCP from the system prompt.

You should uncheck everything besides file context, status updates, the built-in tool memory, and the OTOBO MCP. This will provide you with a simple setup; you can expand it as needed.

Note

LibreChat may also be used as a chat service. While it is more capable out-of-the-box, to handle a larger amount of users, it is more complex to maintain.

Tuning

Every OTOBO is different. And so is the information processed by it. To account for this fact, this setup is as flexible as possible. There are a lot of options when tuning this setup. If you would like to dig into the answer generation process, it is recommended to hook a Langfuse to your RAG setup. This chapter addresses the various tuning options available.

Prompt

Prompt templates can be adjusted very freely. They are placed at the rags folder in the folders of the otobo-ai-services. There are two prompts, one for answer generation, which is discussed here, and one for answer evaluation, which is discussed below. They have significant impact on the form of the generated answer.

A prompt is composed of a system part. It allows you to define a general setting and role for the LLM to act in. It is introduced by:

<|begin_of_text|><|start_header_id|>system<|end_header_id|>

It is followed by the user part. This part is composed of the context, a specific task and the question, injected to the prompt by {question}, to answer. It is introduced by:

<|eot_id|><|start_header_id|>user<|end_header_id|>

The context is composed by these tags: {faqs}, {docs}, {ticket_data}, and {ticket_chunks}. Tickets are presented by question-answer-pairs (first customer article in and last agent article out) as well as chunks. During import, data is cut into overlapping chunks. This facilitates topical specificity within the chunks. However, on close inspection of the prompts, one may find the chunks too small or too large. Chunk size might be fit by setting the variables OTOBO_AI_EMBEDDING_CHUNK_SIZE and OTOBO_AI_EMBEDDING_CHUNK_OVERLAP in your .env.

Given a question, the most relevant pieces by type are selected. However, this list has to be cut off. Only the most n relevant hits are considered. Please refer to the graph.py of your rag definition.

The format of the external documentation may also be considered. Different data format might be processed differently.

The prompt closes with a tag for the answer:

<|eot_id|><|start_header_id|>assistant<|end_header_id|>

Models

There are two models at play, one that provides the embeddings and hence is responsible for the decision on what information is deemed most relevant to the question at hand. The other one is for answer generation and extracts an answer from the presented documents and presents it in the form requested in the prompt. Both models have to be capable of handling the language(s) you want to process. They may be configured in the .env file of the otobo-ai-services.

The main characteristics of the embedding models, aside from benchmarks, are context size and dimensions. The bigger the information pieces to process are, the bigger the context size needs to be. With an increase in nuances in the whole information pool, the dimensions need to increase to account for the differences.

At the time of writing, it is generally recommended to use an instruction-tuned model in order to obtain answers faithful to the prompt given. Furthermore, a multi-modal model should be considered. Please keep in mind, that this model has to support a quite large context window because it needs to consume all relevant documents at once.

The model itself may be tuned by temperature, which is the models risk affinity to the possibility to hallucinate. It may be specified in the .env. In order to obtain faithful answers, lower values are recommended.

Note

At the time of writing, images are not considered for processing.

Evaluation

In order to evaluate the impact of changes to the quality of system responses, Roboto facilitates scoring of answers generated for an evaluation set. An evaluation set is a portion of the ticket data, randomly chosen and deliberately not imported (see Retrieval Augmented Generation (RAG)). You may generate a sample during import like this:

bin/otobo.console.pl Maint::AI::Import --sample-nth 20 --sample ~/var/ai/sample.json

Sample size should depend on the amount of tickets to import, but should range from 20% (hard validation) to 5% (plausibility check). This evaluation set might even be excluded from reimport by passing the --reference argument. It seems reasonable to keep a sample file at ~/var/ai/sample.json. If the evaluation set is ever imported, evaluation will be structurally biased and hence useless.

In order to evaluate Roboto’s performance on an evaluation set, answers are generated for the tickets in the evaluation set and these answers are evaluated with respect to a set of freely chosen criteria. These evaluation criteria are defined in the eval_prompt.txt of your RAG definition. The difference to the RAG prompt is the data description and format definition as the task.

Note

The model chosen for evaluation needs to follow prompt instructions very closely.

Run the evaluation like this:

bin/otobo.console.pl Maint::AI::Scoring \
  --sample /opt/otobo/var/ai/sample.json \
  --result /opt/otobo/var/ai/score-baseline.csv \
  --json /opt/otobo/var/ai/score-baseline.json

You may use jq and a spreadsheet application for analysis.

Index