Prefect Server

This server was set up to be a centralized workflow monitoring and management interface.

Installation

Submitted the following ticket to have IS set up the VM

Request_Submitted_By: kgomes@mbari.org
 - VM_Name: prefect
 - VM_Purpose: This is another VM that will be a part of the monitoring testing 
   Travis and I are working on. It will run Prefect which will allow developers
   to connect up to workflows to make sure things are healthy.
 - VM_Expiration: 5 years
 - VM_Support: IS_Supported
 - VM_Support_Alt_Adm: 
 - VM_OS: Ubuntu 24.04 LTS
 - CPU: 2
 - CPU_Addl_Reason: 
 - RAM: 4
 - RAM_Addl_Reason: 
 - GPU_Extra: 
 - GPU: NO
 - Disk_Extra: 
 - Network: (SHORE) Internal
 - Resource_Priority: Medium
 - Resource_Priority_Reason: 
 - Conf_Firewall: 
 - Conf_Share_Access: 
 - Conf_vCenter_Access: 
 - Conf_Desktop_Extra: 
 - Conf_Desktop: NO
 - Conf_Logins: ad+nfshome
 - Conf_Docker: YES
 - Conf_Docker_Extra: 
 - Conf_WebServer: nginx-docker
 - Conf_ACME_SSLcert: YES
 - Conf_ACME_SSLcert_Reload: 
 - Conf_sudo: kgomes and tbaraki
 - VM_Comments: Hey again Peter,

This is probably exactly the same kind of setup that I put in for the 'monitoring' VM.

Kevin

And Peter set it up and responded:

The VM has been created with all that you specified. Your and Travis's users 
have full sudo permissions. The ACME SSL certs are located in `/etc/ssl/acme-certs/`.

Take a look around and let me know if anything is missing.

I created the necessary directories in /opt by running sudo mkdir -p /opt/prefect/{nginx,data,postgres}
Changed things over to the docker group by running sudo chown -R kgomes:docker /opt/prefect
Set permission using sudo chmod -R 2775 /opt/prefect

Then I created the /opt/prefect/docker-compose.yml file with the following contents:

services:

  postgres:
    image: postgres:16
    restart: unless-stopped
    environment:
      POSTGRES_USER: prefect
      POSTGRES_PASSWORD: xxxxxxxxxx
      POSTGRES_DB: prefect
    volumes:
      - /opt/prefect/postgres:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U prefect"]
      interval: 10s
      timeout: 5s
      retries: 5
    networks:
      - internal

  prefect:
    image: prefecthq/prefect:3-latest
    restart: unless-stopped
    command: prefect server start
    environment:
      PREFECT_SERVER_API_HOST: "0.0.0.0"
      PREFECT_API_DATABASE_CONNECTION_URL: "postgresql+asyncpg://prefect:xxxxxxxx@postgres:5432/prefect"
      PREFECT_UI_API_URL: "https://prefect.shore.mbari.org/api"
      PREFECT_SERVER_CORS_ALLOWED_ORIGINS: "https://prefect.shore.mbari.org"
      PREFECT_SERVER_METRICS_ENABLED: "true"
    volumes:
      - /opt/prefect/data:/root/.prefect
    depends_on:
      postgres:
        condition: service_healthy
    expose:
      - "4200"
    networks:
      - internal

  nginx:
    image: nginx:alpine
    restart: unless-stopped
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - ./nginx/nginx.conf:/etc/nginx/nginx.conf:ro
      - /etc/ssl/acme-certs:/etc/ssl/acme-certs:ro
    depends_on:
      - prefect
    networks:
      - internal

networks:
  internal:
    driver: bridge

Then I created the /opt/prefect/nginx/nginx.conf file:

events {
    worker_connections 1024;
}

http {
    upstream prefect_server {
        server prefect:4200;
    }

    # Redirect all HTTP to HTTPS
    server {
        listen 80;
        server_name prefect.shore.mbari.org;
        return 301 https://$host$request_uri;
    }

    server {
        listen 443 ssl;
        server_name prefect.shore.mbari.org;

        ssl_certificate     /etc/ssl/acme-certs/prefect.shore.mbari.org-chain.crt;
        ssl_certificate_key /etc/ssl/acme-certs/prefect.shore.mbari.org.key;
        ssl_protocols       TLSv1.2 TLSv1.3;
        ssl_ciphers         HIGH:!aNULL:!MD5;
        ssl_session_cache   shared:SSL:10m;
        ssl_session_timeout 10m;

        # Larger buffer for Prefect's API responses
        proxy_buffer_size       16k;
        proxy_buffers           4 32k;
        proxy_busy_buffers_size 64k;

        location / {
            proxy_pass         http://prefect_server;
            proxy_set_header   Host              $host;
            proxy_set_header   X-Real-IP         $remote_addr;
            proxy_set_header   X-Forwarded-For   $proxy_add_x_forwarded_for;
            proxy_set_header   X-Forwarded-Proto $scheme;

            # WebSocket support — Prefect UI uses it for live updates
            proxy_http_version 1.1;
            proxy_set_header   Upgrade    $http_upgrade;
            proxy_set_header   Connection "upgrade";

            proxy_read_timeout 300s;
        }
    }
}

Then I created the systemd file /etc/systemd/system/prefect.service

[Unit]
Description=Prefect Server Stack
After=docker.service
Requires=docker.service

[Service]
Type=oneshot
RemainAfterExit=yes
User=kgomes
Group=docker
WorkingDirectory=/opt/prefect

ExecStart=/usr/bin/docker compose up -d
ExecStop=/usr/bin/docker compose down

[Install]
WantedBy=multi-user.target

Then I enabled and started it

sudo systemctl daemon-reload
sudo systemctl enable prefect
sudo systemctl start prefect

Now I can see the Prefect server at https://prefect.shore.mbari.org/dashboard and it starts up automatically on reboot. COOL!

Note

The way this is set up, in order to look at the docker container status or logs, you need to run sudo, so sudo docker ps and sudo docker logs xxxxx.