How to Deploy AI Agents on an Ubuntu Server

AI agents can be deployed on an Ubuntu server in several ways, depending on how the application and its underlying language model are managed. You can deploy using containerized environments or run the application directly on the server, and you can access the language model through an external API or host it locally. The chosen setup affects how the application is installed, configured, and managed.

This guide presents three methods for deploying an AI agent on an Ubuntu server. Read the tutorial or watch the video below for a complete walkthrough.

Prerequisites

A machine running Ubuntu (You can use any compatible Ubuntu server or deploy one through phoenixNAP Bare Metal Cloud).
An internet connection.
An LLM provider or local LLM runtime.

AI Agent Deployment Options

This guide covers three methods for deploying an AI agent on an Ubuntu server. Choose the method that best fits your deployment requirements:

Method 1: Docker Compose - uses Docker to package the AI agent and its dependencies into a container. This approach provides an isolated and portable deployment.
Method 2: Python Virtual Environment and systemd - runs the AI agent directly on the Ubuntu server in a Python virtual environment and manages it as a systemd service. This approach does not require containers.
Method 3: Ollama and systemd - runs the AI agent as a systemd service and uses Ollama as a local LLM backend. This approach does not require an external API key because the model runs directly on the server.

Note: The AI agent in this guide is a simple example to demonstrate deployment methods. The same deployment approaches can be adapted to more advanced AI agent applications.

Method 1: Deploy an AI Agent with Docker Compose

Docker Compose simplifies the deployment by defining the AI agent and its configuration in a single Compose file. In this method, the agent runs in a Docker container and uses OpenRouter to access a language model.

Docker and Docker Compose must already be installed on the server. For instructions, see the how to install Docker on Ubuntu guide.

1. Create the Project Directory

First, create a dedicated directory for the AI agent files. This directory stores the application code, dependency definitions, Docker configuration, and environment variables used during deployment.

1. Create a directory for the application using mkdir:

mkdir ~/ai-agent

2. Navigate to the directory using the cd command:

cd ~/ai-agent

2. Create the AI Agent Application

Next, create the Python application that provides the AI agent's API. The application exposes health and chat endpoints and sends user requests to the OpenRouter API.

1. Create the application with:

nano agent.py

2. Add the following content:

import os

from fastapi import FastAPI, HTTPException
from openai import OpenAI
from pydantic import BaseModel

app = FastAPI(title="AI Agent")


class ChatRequest(BaseModel):
    message: str


@app.get("/health")
def health():
    return {"status": "ok"}


@app.post("/chat")
def chat(request: ChatRequest):
    api_key = os.getenv("OPENROUTER_API_KEY")

    if not api_key:
        raise HTTPException(
            status_code=500,
            detail="OPENROUTER_API_KEY is not configured",
        )

    client = OpenAI(
        base_url="https://openrouter.ai/api/v1",
        api_key=api_key,
    )

    response = client.chat.completions.create(
        model="openrouter/free",
        messages=[
            {
                "role": "system",
                "content": "You are a helpful AI assistant.",
            },
            {
                "role": "user",
                "content": request.message,
            },
        ],
    )

    return {"response": response.choices[0].message.content}

3. Save the file and exit.

3. Create the Python Dependencies File

The application requires FastAPI to provide the API, Uvicorn to run the application, and the OpenAI Python library to communicate with the OpenRouter API. Define these dependencies in a requirements.txt file so Docker can install them when building the image:

1. Create the file with nano:

nano requirements.txt

2. Add:

fastapi
uvicorn[standard]
openai

3. Save the file and exit.

4. Create the Dockerfile

The Dockerfile defines how Docker builds the AI agent image. It specifies the Python base image, installs the required dependencies, copies the application into the image, and starts the API server.

1. Create the file with:

nano Dockerfile

2. Paste the following:

FROM python:3.12-slim

WORKDIR /app

COPY requirements.txt .

RUN pip install --no-cache-dir -r requirements.txt

COPY agent.py .

EXPOSE 8000

CMD ["uvicorn", "agent:app", "--host", "0.0.0.0", "--port", "8000"]

3. Save the file and exit.

5. Create a Docker Ignore File

Create a .dockerignore file to prevent unnecessary files from being copied into the Docker build context. This keeps the build context smaller and prevents local configuration files from being included in the image.

1. Create .dockerignore with:

nano .dockerignore

2. Paste the following:

.env
.venv
__pycache__
*.pyc
.git

3. Save and exit the file.

6. Configure the API Key

The AI agent needs an API key to authenticate with OpenRouter. Store the key in an environment file instead of adding it directly to the application code or Docker image.

1. Create an environment file:

nano .env

2. Add your OpenRouter API key:

OPENROUTER_API_KEY=[YOUR_API_KEY]

Replace [YOUR_API_KEY] with the actual key.

3. Restrict access to the file to secure it:

chmod 600 .env

7. Create the Docker Compose Configuration

Create a Compose file to define how Docker runs the AI agent. The configuration builds the application image, passes the API key to the container, and exposes the agent on port 8000.

1. Create the compose.yaml file with:

nano compose.yaml

2. Paste the following content:

services:
  ai-agent:
    build: .
    container_name: ai-agent
    env_file:
      - .env
    ports:
      - "8000:8000"
    restart: unless-stopped

3. Save the file and exit.

8. Deploy the AI Agent

With the application and Docker configuration in place, build the image and start the AI agent as a container. Docker Compose reads the configuration from compose.yaml and starts the service with the required environment variables.

Run the following command to deploy the app:

docker compose up -d --build

Building the AI agent with Docker compose.

9. Verify and Test the Deployment

Finally, verify that the container is running and that the AI agent responds to requests. The health endpoint confirms the application is available before you test an actual AI request.

1. Verify the deployment with:

docker compose ps

The ai-agent service should show a running status.

2. Check the health endpoint:

curl http://localhost:8000/health

A successful response is {"status":"ok"}:

3. Lastly, send a test prompt to the AI agent. For example:

curl -X POST http://localhost:8000/chat \
-H "Content-Type: application/json" \
-d '{"message":"Explain Docker in one sentence."}'

The endpoint should return a response generated by the configured LLM:

Method 2: Deploy AI Agent as a systemd Service

The second method runs the AI agent directly on the Ubuntu server instead of inside a container. The application runs in a Python virtual environment and uses a systemd service to start and manage the agent automatically.

Follow the steps below to deploy the AI agent with Python virtual environment and systemd:

1. Install Python and Required Packages

First, install Python and the packages required to create and manage a virtual environment. These packages provide the environment needed to run the AI agent directly on the server.

1. Update the system package index to ensure you get the latest available versions:

sudo apt update

2. Install Python and required packages:

sudo apt install -y python3 python3-pip python3-venv

These packages provide Python, pip, and the functionality required to create a Python virtual environment.

2. Create the Project Directory

Create a dedicated directory for the AI agent files. This directory contains the application code and Python dependency definitions used by the deployment.

1. Use the mkdir command to create the directory:

mkdir ~/ai-agent

2. Navigate to the directory using cd:

cd ~/ai-agent

3. Create the AI Agent Application

Next, create the Python application that provides the AI agent's API. The application exposes health and chat endpoints and sends user requests to the OpenRouter API.

1. Create the same agent.py application shown in Method 1:

nano agent.py

2. Paste the following:

import os

from fastapi import FastAPI, HTTPException
from openai import OpenAI
from pydantic import BaseModel

app = FastAPI(title="AI Agent")


class ChatRequest(BaseModel):
    message: str


@app.get("/health")
def health():
    return {"status": "ok"}


@app.post("/chat")
def chat(request: ChatRequest):
    api_key = os.getenv("OPENROUTER_API_KEY")

    if not api_key:
        raise HTTPException(
            status_code=500,
            detail="OPENROUTER_API_KEY is not configured",
        )

    client = OpenAI(
        base_url="https://openrouter.ai/api/v1",
        api_key=api_key,
    )

    response = client.chat.completions.create(
        model="openrouter/free",
        messages=[
            {
                "role": "system",
                "content": "You are a helpful AI assistant.",
            },
            {
                "role": "user",
                "content": request.message,
            },
        ],
    )

    return {"response": response.choices[0].message.content}

3. Save the file and exit.

4. Define Python Dependencies

Define the Python packages required by the application in a requirements.txt file. Dependencies include FastAPI for the API, Uvicorn to run the application, and the OpenAI Python library to communicate with OpenRouter.

1. Create requirements.txt:

nano requirements.txt

2. Use the same contents as in Method 1:

fastapi
uvicorn[standard]
openai

5. Create a Python Virtual Environment

Create a Python virtual environment to isolate the AI agent's dependencies from the system Python installation. This allows the application to use its own package versions without affecting other Python applications on the server.

1. Create the virtual environment with:

python3 -m venv .venv

2. Activate the virtual environment:

source .venv/bin/activate

Activating a virtual environment with python.

3. Install the application dependencies:

pip install -r requirements.txt

The virtual environment keeps the application's Python dependencies isolated from the system Python installation.

6. Configure the API Key

The AI agent requires an API key to authenticate with OpenRouter. Store the key in a separate environment file so it is not included in the application code or service configuration.

1. Create a dedicated directory for the environment configuration:

sudo mkdir -p /etc/ai-agent

2. Create the environment file:

sudo nano /etc/ai-agent/agent.env

3. Add:

OPENROUTER_API_KEY=[YOUR_API_KEY]

Replace [YOUR_API_KEY] with the actual key.

4. Restrict access to the file because it contains a sensitive API key:

sudo chmod 600 /etc/ai-agent/agent.env

7. Create the systemd Service

Create a systemd service to manage the AI agent as a background service. The service runs the Uvicorn server from the virtual environment and loads the API key from the environment file.

1. Create the service unit:

sudo nano /etc/systemd/system/ai-agent.service

2. Add:

[Unit]
Description=AI Agent Service
After=network.target

[Service]
User=ubuntu
WorkingDirectory=/home/ubuntu/ai-agent
EnvironmentFile=/etc/ai-agent/agent.env
Environment=PYTHONUNBUFFERED=1

ExecStart=/home/ubuntu/ai-agent/.venv/bin/uvicorn agent:app --host 0.0.0.0 --port 8000

Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target

The service runs the application using the Uvicorn executable inside the virtual environment and automatically restarts it if the process stops.

Important: The example uses the ubuntu user and /home/ubuntu paths. If your server uses a different Linux username, replace ubuntu with the appropriate username and update the corresponding paths.

8. Enable and Start the Service

After creating the service, reload the systemd configuration so it recognizes the new service. Then enable and start the AI agent.

1. Reload systemd:

sudo systemctl daemon-reload

2. Enable the service and start it:

sudo systemctl enable --now ai-agent

3. Verify its status:

sudo systemctl status ai-agent --no-pager

The service should show an active (running) status:

9. Test the Deployment

Check the service status to confirm that the AI agent is running correctly. Then use the health endpoint to verify that the API responds before sending an actual request to the AI agent.

1. Check the health endpoint:

curl http://localhost:8000/health

A successful response is {"status":"ok"}:

2. Send a test prompt:

curl -X POST http://localhost:8000/chat \
-H "Content-Type: application/json" \
-d '{"message":"Explain Linux in one sentence."}'

The AI agent should return a response generated by the configured LLM:

Sending a test prompt to an AI agent on Ubuntu.

Method 3: Deploy an AI Agent with a Local LLM Backend

The third method uses Ollama to run the LLM locally on the Ubuntu server. This eliminates the need for an external LLM API key and allows the AI agent to communicate with a model running directly on the server.

The resources required for local inference depend on the model being used. The example below uses llama3.2:3b as a relatively small model suitable for demonstrating the deployment.

1. Install Python and Required Packages

First, install Python and the packages required to create a virtual environment. The Python environment runs the AI agent application independently from the system Python installation.

1. Update the package index with:

sudo apt update

2. Install Python and its dependencies:

sudo apt install -y python3 python3-pip python3-venv

2. Install Ollama

Next, install Ollama on the Ubuntu server. Ollama is an open-source software platform that lets you download, manage, and run large language models (LLMs) directly on your own computer. It provides the local runtime that the AI agent uses to communicate with the language model.

1. Install Ollama using its installation script:

curl -fsSL https://ollama.com/install.sh | sh

2. Verify the installation:

ollama --version

3. Enable and Start Ollama

Enable and start the Ollama systemd service so the AI agent can access the local LLM backend.

Run the following command:

sudo systemctl enable --now ollama

Verify that the service is running:

sudo systemctl status ollama --no-pager

The service should show an active (running) status:

4. Download the Local LLM

Download the llama3.2:3b model. Ollama stores the model locally and makes it available to applications through its local API.

1. Download the model with:

ollama pull llama3.2:3b

2. Verify that the model is available:

ollama list

5. Create the AI Agent Directory

Create a dedicated directory for the AI agent files. The directory contains the application code and Python dependency definitions used by the deployment.

1. Create the directory:

mkdir ~/ai-agent

2. Navigate to the new directory:

cd ~/ai-agent

6. Create the AI Agent Application

Next, create the Python application that provides the AI agent's API. Instead of connecting to an external provider, the application sends requests to the OpenAI-compatible API provided by the local Ollama instance.

1. Create agent.py:

nano agent.py

2. Add the following:

from fastapi import FastAPI
from openai import OpenAI
from pydantic import BaseModel

app = FastAPI(title="AI Agent")


class ChatRequest(BaseModel):
    message: str


@app.get("/health")
def health():
    return {"status": "ok"}


@app.post("/chat")
def chat(request: ChatRequest):
    client = OpenAI(
        base_url="http://localhost:11434/v1/",
        api_key="ollama",
    )

    response = client.chat.completions.create(
        model="llama3.2:3b",
        messages=[
            {
                "role": "system",
                "content": "You are a helpful AI assistant.",
            },
            {
                "role": "user",
                "content": request.message,
            },
        ],
    )

    return {"response": response.choices[0].message.content}

7. Define Python Dependencies

Define the Python packages required by the application in a requirements.txt file. These dependencies provide the API framework, application server, and client used to communicate with Ollama.

1. Create the requirements.txt file:

nano requirements.txt

2. Add the following content:

fastapi
uvicorn[standard]
openai

8. Create the Virtual Environment

Create a Python virtual environment to isolate the AI agent's dependencies from the system Python installation.

1. Run:

python3 -m venv .venv

2. Then, activate the environment:

source .venv/bin/activate

3. Install the dependencies:

pip install -r requirements.txt

9. Create the systemd Service

Create a systemd service to manage the AI agent as a background service. The service starts the application from the Python virtual environment and declares Ollama as a required dependency.

1. Create the service:

sudo nano /etc/systemd/system/ai-agent.service

2. Add the following content:

[Unit]
Description=AI Agent Service
After=network.target ollama.service
Requires=ollama.service

[Service]
User=ubuntu
WorkingDirectory=/home/ubuntu/ai-agent
Environment=PYTHONUNBUFFERED=1

ExecStart=/home/ubuntu/ai-agent/.venv/bin/uvicorn agent:app --host 0.0.0.0 --port 8000

Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target

The After and Requires directives make Ollama a dependency of the AI agent service, ensuring that the local LLM backend is available when the agent starts.

Important: Replace ubuntu and the corresponding /home/ubuntu paths if your server uses a different Linux username.

10. Enable and Start the AI Agent

Reload the systemd configuration to register the new service:

sudo systemctl daemon-reload

Enable and start the service:

sudo systemctl enable --now ai-agent

11. Test the Deployment

Check the service status and use the health endpoint to confirm that the AI agent is running correctly. Then send a request to verify that the agent can communicate with the local language model.

1. Verify the service status:

sudo systemctl status ai-agent --no-pager

2. Check the health endpoint:

curl http://localhost:8000/health

A successful response is {"status":"ok"}.

3. Send a test prompt:

curl -X POST http://localhost:8000/chat \
-H "Content-Type: application/json" \
-d '{"message":"Explain what a Linux server is in one sentence."}'

The response is generated by the llama3.2:3b model running locally through Ollama.

4. Finally, verify that Ollama is actively running the model:

ollama ps

The output shows the model Ollama is currently loading.

Conclusion

This article explained three methods for deploying an AI agent on an Ubuntu server: Docker Compose, a systemd service with a Python virtual environment, and a local LLM backend using Ollama. The first two methods use an external LLM provider through an API, while the Ollama-based method runs the language model directly on the server.

For more information about running containerized workloads on Ubuntu, see our Docker attach guide.

Was this article helpful?

YesNo