OTOBO AI Manual

This is the OTOBO AI (codename “Roboto”) Manual. It serves as a reference to help administrators manage and configure the OTOBO AI capabilities effectively.

Hint

Roboto is currently in closed beta. If interested, feel free to contact us.

Overview

Roboto implements a RAG (Retrieval-Augmented Generation) system i.e., it is able to generate response suggestions for tickets based on the data from within your OTOBO.

There are two components that interface OTOBO with an LLM:

The services otobo-ai-services.

The otobo-ai OTOBO package.

The OTOBO package is the interface between OTOBO and the services. The services hold the data in a format that allows finding similar data related to a request efficiently, and interface with the LLM.

                      ┌╌╌╌╌╌╌╌╌╌╌╌╌┐
┌───────────────┐     ┆  Langfuse  ┆
│ OTOBO         │     └╌╌╌╌╌┬╌╌╌╌╌╌┘
│   ┌──────────┐│  ┌────────┴──────────┐  ┌───────┐
│   │ otobo-ai─┼│──┼ otobo-ai-services ┼──┼  LLM  │
│   └──────────┘│  └───────────────────┘  └───────┘
└───────────────┘

Workflow

At first, all tickets, FAQ and external documentation are imported. During this step, the information is scrubbed, chunked, and sent to a small LLM to obtain a numeric vector that describes the presented information. This procedure is called embedding. Every vector describes a point in high dimensional space. Similar information will be close to each other. All the vectors and documents are stored in the otobo-ai-services. This allows for fast selection of documents that address the same topic.

If a new article is created, generate an answer:

An embedding is requested for the new article. The resulting vector is used to identify related documents.

The related documents are injected into the prompt for a larger LLM to generate an answer based on the provided information.

The generated answer is stored in a Dynamic Field at the article.

The dynamic field can be referenced in an Answer Template.

If the user chooses this Answer Template, the generated answer of the LLM will be visible in the compose window.

Contents

Installation

otobo-ai-services Docker Compose Stack

OTOBO-AI Package

Usage

Data Import

Answer Generation

Tuning

Prompt

Models

Evaluation

This work is copyrighted by ROTHER OSS GmbH (https://otobo.io), Oberwalting 31, 94339 Leiblfing, Germany

Terms and Conditions Rother OSS: Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.3 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license is included in the section entitled “COPYING”.

Published by: Rother OSS GmbH, (https://otobo.de), Oberwalting 31, 94339 Leiblfing, Germany.

Authors: Rother OSS GmbH (https://otobo.de).

Installation

There are two components that interface OTOBO with an LLM:

The services otobo-ai-services.

The otobo-ai OTOBO package.

These installation instructions illustrate their setup.

otobo-ai-services Docker Compose Stack

The otobo-ai-services are provided as self-contained docker compose stack. There are two docker containers:

otobo-ai

A Chroma database for vector data and webservice to interface with OTOBO

postgres

A database holding prepared data

The repository may be found here: https://github.com/RotherOSS/otobo-ai-services

First, set up the docker compose project:

cd /opt
git clone git@github.com:RotherOSS/otobo-ai.git
cd otobo-ai

Example RAG definitions are provided under rag_examples. The simple_rag is for standalone development. If you use this setup with OTOBO, choose the tfd_rag1. It supports Tickets (t), FAQ (f) and Documentation (d). Copy the RAG description to your RAG definition folder:

cp -r rags_examples/tfd_rag1 rags

All RAG definitions placed here are exposed at the web service.

Note

Docker mounts the local ./rags directory into the container as src/rags, enabling external customization. However, a restart of the container is required for the changes to take effect.

You may tune it to your liking (see Tuning), or create a new one! You should at least tailor the prompt to your use case.

Create a .env file in the root directory to configure environment variables:

cp .docker_compose_env_ai .env

Edit the .env file to set your desired configuration options.

Use Docker Compose to build and run the server:

docker compose up --build --detach

OTOBO-AI Package

The package can simply be installed via the OTOBO Package Manager on OTOBO version >= 11.0. See the OTOBO Admin Guide: Package Manager for details on package management in OTOBO.

Configure the package settings in the system configuration. See the OTOBO Admin Guide: System Configuration for general instructions. The package manual includes all relevant information. You may just search OTOBOAI:: to find all relevant settings. These are the basic steps:

Select data for import into the RAG stack (what your answers are based on).

Configure the auth token into the webservice.

Setup the answer template.

Continue to Usage.

Usage

In order for Roboto to generate answers, you need to import the relevant data. After import answers to articles may be generated and the database may be queried individually.

Data Import

With the OTOBO-AI package installed, three data sources may be configured: tickets, FAQ and documents. Tickets and FAQs import selection is guided by their search functions, i.e., you may use any field and restriction from the classic search function. Place documentation under the path configured in the system setting OTOBOAI::Document::SearchRestrictions relative to your OTOBO installation. And import all the data from command line:

# run --help for an overview
bin/otobo.Console.pl Maint::AI::Import

Note

Currently plain text documentation is preferred. Pandoc may be helpful with data conversion. Beware of index files, they match on every topic and are relatively dense. Beware of files like a license, they are never a match.

It is advisable to exclude a portion of the data from import. The import command supports the parameters --sample-nth and --sample. A generated sample may serve as evaluation set for tuning (see Tuning).

You may verify the import using the command line from the otobo-ai container:

docker compose exec otobo-ai python src/cli.py

Answer Generation

You may generate answers either prompted by a customer request or manually on-demand.

On-Demand Answers

There are two options to obtain on-demand answers, either the command line or a new widget in the agent overview. The command line may be reached from the CLI of OTOBO:

docker compose exec web ./bin/otobo.Console.pl Maint::AI::GetAnswer "Hello Roboto!"

This option is especially useful if service management requires deep domain knowledge that is not externalized or easily accessible.

Automatic Answer Suggestions

However, OTOBO may query the LLM for an answer suggestion whenever it creates a new article. Upon retrieval, the answer is stored in a dynamic field, configured in OTOBOAI::DynamicFieldOTOBOAI. It may be referenced from an answer template. The pre-configured answer template is OTOBOAI Answer Template, and might need to be tweaked to adhere to your use-case. If an agent chooses this answer template, the LLM answer is revealed, may be edited by the agent and sent out to the customer.

Tuning

Every OTOBO is different. And so is the information processed by it. To account for this fact, this setup is as flexible as possible. There are a lot of options when tuning this setup. If you would like to dig into the answer generation process, it is recommended to hook a Langfuse to your RAG setup. This chapter addresses the various tuning options available.

Prompt

Prompt templates can be adjusted very freely. They are placed at the rags folder in the folders of the otobo-ai-services. There are two prompts, one for answer generation, which is discussed here, and one for answer evaluation, which is discussed below. They have significant impact on the form of the generated answer.

A prompt is composed of a system part. It allows you to define a general setting and role for the LLM to act in. It is introduced by:

<|begin_of_text|><|start_header_id|>system<|end_header_id|>

It is followed by the user part. This part is composed of the context, a specific task and the question, injected to the prompt by {question}, to answer. It is introduced by:

<|eot_id|><|start_header_id|>user<|end_header_id|>

The context is composed by these tags: {faqs}, {docs}, {ticket_data}, and {ticket_chunks}. Tickets are presented by question-answer-pairs (first customer article in and last agent article out) as well as chunks. During import, data is cut into overlapping chunks. This facilitates topical specificity within the chunks. However, on close inspection of the prompts, one may find the chunks too small or too large. Chunk size might be fit by setting the variables OTOBO_AI_EMBEDDING_CHUNK_SIZE and OTOBO_AI_EMBEDDING_CHUNK_OVERLAP in your .env.

Given a question, the most relevant pieces by type are selected. However, this list has to be cut off. Only the most n relevant hits are considered. Please refer to the graph.py of your rag definition.

The format of the external documentation may also be considered. Different data format might be processed differently.

The prompt closes with a tag for the answer:

<|eot_id|><|start_header_id|>assistant<|end_header_id|>

Models

There are two models at play, one that provides the embeddings and hence is responsible for the decision on what information is deemed most relevant to the question at hand. The other one is for answer generation and extracts an answer from the presented documents and presents it in the form requested in the prompt. Both models have to be capable of handling the language(s) you want to process. They may be configured in the .env file of the otobo-ai-services.

The main characteristics of the embedding models, aside from benchmarks, are context size and dimensions. The bigger the information pieces to process are, the bigger the context size needs to be. With an increase in nuances in the whole information pool, the dimensions need to increase to account for the differences.

At the time of writing, it is generally recommended to use an instruction-tuned model in order to obtain answers faithful to the prompt given. Furthermore, a multi-modal model should be considered. Please keep in mind, that this model has to support a quite large context window because it needs to consume all relevant documents at once.

The model itself may be tuned by temperature, which is the models risk affinity to the possibility to hallucinate. It may be specified in the .env. In order to obtain faithful answers, lower values are recommended.

Note

At the time of writing, images are not considered for processing.

Evaluation

In order to evaluate the impact of changes to the quality of system responses, Roboto facilitates scoring of answers generated for an evaluation set. An evaluation set is a portion of the ticket data, randomly chosen and deliberately not imported (see Usage). You may generate a sample during import like this:

bin/otobo.console.pl Maint::AI::Import --sample-nth 20 --sample ~/var/ai/sample.json

Sample size should depend on the amount of tickets to import, but should range from 20% (hard validation) to 5% (plausibility check). This evaluation set might even be excluded from reimport by passing the --reference argument. It seems reasonable to keep a sample file at ~/var/ai/sample.json. If the evaluation set is ever imported, evaluation will be structurally biased and hence useless.

In order to evaluate Roboto’s performance on an evaluation set, answers are generated for the tickets in the evaluation set and these answers are evaluated with respect to a set of freely chosen criteria. These evaluation criteria are defined in the eval_prompt.txt of your RAG definition. The difference to the RAG prompt is the data description and format definition as the task.

Note

The model chosen for evaluation needs to follow prompt instructions very closely.

Run the evaluation like this:

bin/otobo.console.pl Maint::AI::Scoring \
  --sample /opt/otobo/var/ai/sample.json \
  --result /opt/otobo/var/ai/score-baseline.csv \
  --json /opt/otobo/var/ai/score-baseline.json

You may use jq and a spreadsheet application for analysis.

Index