AI&human

A simple guide to your own AI assistant


von hannu

Introduction

I am an exchange student from Finland. I came to Tralios IT GmbH in Germany for my on-the-job training. My main task at Tralios was to learn about different AI models and how to utilize them locally for the company. This blog is my small guide to people who want to try out AI models locally for their company or for personal use.

Here are a few examples of what you can do with the models

[siteorigin_widget class=“SiteOrigin_Widget_Tabs_Widget”][/siteorigin_widget]

[siteorigin_widget class=“WP_Widget_Custom_HTML”][/siteorigin_widget]

What You’ll Learn

  • How to set up a basic AI environment on your computer.
  • How to utilize AI models like Llama and Mistral running on your computer.
  • Why this is useful for personal and business purposes.

How does it work?

Imagine a package that has everything you need to work with inside it. That’s what we are trying to achieve here with AI. We are taking different AI models and putting them into a package that’s ready to use, without needing complicated setups.

The Tools We’re Using

  • Ollama: Is used to download and run your AI models in different environments.
  • OpenWebUI: Is for using the AI models in a web interface like a browser. There you can chat with the AI models directly and use multiple models at the same time.
  • NVIDIA Container Toolkit: This tool is used to utilize your graphics card (GPU) in the AI processing, making the AI model processing even faster.
  • Docker: Is like a program running in the background handling all your packages.

Why should you try this?

  • Privacy: You’re in control of your data. You don’t need to share information to other AI companies.
  • Customization: You can try out different AI models and customize them to to suit your specific needs.
  • Cost-Effective: You can run AI models from your own hardware instead of paying for cloud-services.
  • Future of AI: AI will be a part of our future and having information about how it works might prove useful.

Important Notes Before You Start

  • Operating System: You’ll need Ubuntu 24.04 installed.
  • Graphics card: You’ll need an NVIDIA graphics card. We recommend at least 4-8GB of VRAM (VRAM is your graphics card memory). You can find a list of supported GPUs here
  • Compatibility: This guide is made using an NVIDIA graphics card so it might not work on other brand graphics cards.
  • Updates: AI technology advances quickly so this guide might not be 100% up to date.

[siteorigin_widget class=“FT_Tabs”][/siteorigin_widget]

[siteorigin_widget class=“WP_Widget_Custom_HTML”][/siteorigin_widget]

How to get and use different models in OpenWebUI

Here we learn how to download models from ollama to your openwebui website. Here is a link to ollama models. Try to learn a bit about what the different models can do by browsing the models.

When you click a model name on the site, you can see what they are designed for and what parameter sizes are available.
Parameter count b is simply a batch size meaning the bigger the b number is the bigger your request length can be. It also means that the bigger the number the more data it has to work with, because the bigger ones have been trained with more data. When the parameter count is higher it means it needs more from your hardware and if your hardware is not enough for the model it will be significantly slower.

After finding your desired model go back to your openwebui page.

In the upper left-corner there should be an arrow pointing down, click that, and write the desired model into the search field.
Note: It has to be precise like ( Gemma3:4b or llama3.2:3b). Click pull the model from ollama.com to initiate the download. The first time you pull a model, it will take a significant amount of time and disk space. After the download you can select the model and ask something from it.

Tips for the future

  • Stopping/Restarting Services: To stop or restart the services we navigate to the containing folder. ``

In order to stop the server run:

cd /opt/openwebui-ollama
docker compose down

In order to restart the server run:

cd /opt/openwebui-ollama
docker compose restart
  • Updating services: You might need to update the service from time to time. There should be a blue text notification about updating in the lower right corner of your webui website. To update the container, we navigate to the containing folder, download the updated images with a command and then shut it down and start it again.
cd /opt/openwebui-ollama
docker compose pull
docker compose down
docker compose up --detach

Summary about what you’ve done

Now you’ve learned how to install the necessary tools for creating your own AI environment.

Now you can use what you’ve learned for yourself or your company and try out different AI models for various situations.

Perhaps you need help setting it up or just don’t want to set it up alone, just contact our team we will do our best to help you.

Note: There is a follow-up article to this blog post.