01
A private AI for an office team, on GPUs they already had
- Problem
- The team wanted an AI assistant for chat and coding, but company code and documents shouldn't go to a public service. No single workstation had enough GPU memory for a 27B model with a long context.
- What we did
- We split a 27-billion-parameter model across four existing GPU workstations, three NVIDIA and one AMD, with llama.cpp RPC, built from source for CUDA, ROCm/HIP and Vulkan. People use it through Open WebUI for chat and through an OpenAI-compatible API in their coding tools. Deployment runs on Bash and systemd with health checks and rollback. The RPC port is firewalled, the service has its own account and the API needs a key.
- Result
- Load test with 10–16 people at once: 0 errors, about 22 tokens per second each, 262K-token context.
- Stack
llama.cpp RPC · CUDA · ROCm/HIP · Vulkan · Open WebUI · Linux · systemd · Bash
Diagram: users connect to Open WebUI or the API, which send requests to llama-server. llama-server spreads the model across four GPU workers over llama.cpp RPC: three NVIDIA and one AMD. Tokens are streamed back.
↑ answers streamed back
llama-server
llama-server loads the 27B model and puts a part of it into each worker's GPU memory. For every token it coordinates the workers over the network. It runs under its own service account.
This splits one model across four machines; it isn't load balancing. Every request uses all four GPUs, because each holds part of the same model.
What it's used for
Most questions are about code. Because the model runs in the building, people can paste internal code without it leaving the network.
Developer
This deploy script restarts the inference server but never checks that it came back up. How do I wait for it to be healthy, with a timeout?
systemctl restart llama-server
echo "deploy finished"Local model · 27B
Poll llama-server's /health endpoint after the restart. Exit as soon as it answers, and fail with a non-zero code if it's still down after 60 seconds, so the deployment can roll back:
systemctl restart llama-server
for attempt in $(seq 1 30); do
if curl -fsS http://127.0.0.1:8080/health > /dev/null; then
echo "llama-server is healthy"
exit 0
fi
sleep 2
done
echo "llama-server not healthy after 60 s" >&2
exit 1Measured in the load test
- 4
- 27B
- 262K
- 10–16
- 0
- ≈22