How to Set Up Your Own AI at Home

#ai#homelab

Running Ollama, OpenWebUI, and Stable Diffusion on a CPU-only homelab server — installation, configuration, and integration.

I wanted to know what my R720 could do with AI workloads before spending anything on a GPU. Dual 6-core Xeons, 192GB of RAM, no accelerator at all. Three pieces: Ollama for inference, Stable Diffusion for images, OpenWebUI in front of both.

It all works. It is also 0.46 tokens per second, which is the number that ended the experiment — the full measurement is at the bottom.

Ollama

Ubuntu 22.04 VM on Proxmox, 20 cores and 64GB allocated.

Terminal window
curl -fsSL https://ollama.com/install.sh | sh
ollama pull llama3
ollama run llama3

That drops you straight into an interactive session. If it answers at all, the install is done.

Stable Diffusion

Setup follows the AUTOMATIC1111 WebUI repo. Dependencies first:

Terminal window
# Debian/Ubuntu
sudo apt install wget git python3 python3-venv libgl1 libglib2.0-0


# Red Hat
sudo dnf install wget git python3 gperftools-libs libglvnd-glx


# openSUSE
sudo zypper install wget git python3 libtcmalloc4 libglvnd
Terminal window
wget -q https://raw.githubusercontent.com/AUTOMATIC1111/stable-diffusion-webui/master/webui.sh
./webui.sh

The default arguments assume a CUDA device and the first run dies on the torch check. Four flags in webui-user.sh fix that:

Terminal window
export COMMANDLINE_ARGS="--lowvram --precision full --no-half --skip-torch-cuda-test"

--lowvram reduces memory pressure, --precision full and --no-half avoid half-precision maths CPUs handle badly, and --skip-torch-cuda-test skips a check that would fail regardless.

It binds to localhost by default, which is fine until OpenWebUI needs to reach it:

Terminal window
./webui.sh --listen

That puts it on 0.0.0.0:7860.

Stable Diffusion Web Interface

OpenWebUI

The easiest part of the whole setup — a ChatGPT-style front end over both of the above.

Terminal window
# Ollama on the same host
docker run -d -p 3000:8080 --add-host=host.docker.internal:host-gateway \
  -v open-webui:/app/backend/data --name open-webui --restart always \
  ghcr.io/open-webui/open-webui:main


# Ollama on a different server
docker run -d -p 3000:8080 -e OLLAMA_BASE_URL=https://example.com \
  -v open-webui:/app/backend/data --name open-webui --restart always \
  ghcr.io/open-webui/open-webui:main


# With an Nvidia GPU
docker run -d -p 3000:8080 --gpus all --add-host=host.docker.internal:host-gateway \
  -v open-webui:/app/backend/data --name open-webui --restart always \
  ghcr.io/open-webui/open-webui:cuda

Ollama connects at http://ip_server_openwebui:3000/admin/settings/ → Models → Manage Ollama Models. Stable Diffusion connects under Images → AUTOMATIC1111 Base URL; press refresh before saving, because the field accepts an unreachable URL without complaint and only fails later when you ask for an image.

OpenWebUI Admin Settings

Stable Diffusion Integration

OpenWebUI Interface

What CPU-only actually costs

Llama3 on dual Xeon E5-2620 v2s, one question, 457 tokens of answer:

Metric Value
Response Token/s 0.46
Prompt Token/s 1.99
Total Duration 1072376.46 ms (~17 min 52 sec)
Load Duration 61347.1 ms
Prompt Eval Count 33
Prompt Eval Duration 16571.72 ms
Eval Count 457
Eval Duration 994411.07 ms

Eighteen minutes for one answer. It’s a working setup, not a usable one — as a way to learn the stack it’s fine, and as something anyone would sit in front of it isn’t. If your plan is to use this daily, buy the GPU first and skip this post.

I put a Tesla P40 in the R720 afterwards. Getting it through to the VM is its own problem, covered in the Proxmox GPU passthrough guide.

References