Setup

Step 1: Prepare /opt/deployment Directory

Ensure the /opt/deployment directory exists and is writable

sudo mkdir -p /opt/deployment
sudo chown $USER:$USER /opt/deployment
chmod 755 /opt/deployment
ls -ld /opt/deployment 

Step 2: Install

Run the production installation script:

sudo just prod-install

This will: - Store all k3s data in /opt/deployment/argo-workflows/k3d/ - Configure GPU support with NVIDIA device plugin - Set up local storage provisioner

Step 3: Setup Environment

Add to your shell profile (~/.bashrc, ~/.zshrc, or ~/.config/fish/config.fish):

export KUBECONFIG=/opt/deployment/argo-workflows/k3d/kubeconfig/k3s.yaml

Then reload your shell or run:

source ~/.bashrc  # or your shell's config file

Step 4: Deploy Argo Workflows and Configuration

Use Ansible to deploy the full production setup:

You will be prompted for the sudo password

just prod-setup

This deploys: - Argo Workflows - NVIDIA RuntimeClass - Local storage configuration - Shared workflow templates

Step 5: Build Images

Build Docker images:

just build-all 

Step 6: Verify Installation

Run the verification script:

just prod-verify

Expected output should show: - ✓ k3s running - ✓ GPU available (count > 0) - ✓ Images imported - ✓ Storage class configured - ✓ Argo Workflows running

If the output matches the expected results, congratulations! You are ready to use your singler-server Kubernetes cluster 🚀

Usage

Submit Workflows

GPU Inference Workflow

# Or directly with argo
argo submit -n default workflows/ultralytics-inference.yml --watch

# With custom parameters
argo submit -n default workflows/ultralytics-inference.yml \
  -p model-name=yolo11x.pt \
  -p confidence-threshold=0.4 \
  --watch

Monitor Workflows

# List workflows
argo list -n default

# Get workflow details
argo get <workflow-name> -n default

# View logs
argo logs <workflow-name> -n default

# View latest logs
argo logs -n default @latest

# Watch workflow progress
argo watch <workflow-name> -n default

Check GPU Usage

During workflow execution:

# List GPU-enabled pods
kubectl get pods -n default -o json | \
  jq '.items[] | select(.spec.containers[].resources.limits."nvidia.com/gpu") | .metadata.name'

# Check GPU usage in a pod
kubectl exec -n default <pod-name> -- nvidia-smi

Image Management

Building Images

just build-all 

Verify Images in k3s

# List images in k3s
k3s crictl images | grep -E '(inference|detector|tator|tool)'

# Verify specific image
k3s crictl images | grep 'ultralytics'

Storage Management

Persistent Volume Claims

Production uses local hostPath storage at /opt/deployment/argo-workflows/k3d/storage/.

# List PVCs
kubectl get pvc -n default

# Describe PVC
kubectl describe pvc <pvc-name> -n default

# Check actual storage location
ls -lh /opt/deployment/argo-workflows/k3d/storage/

Accessing Workflow Data

# Find the PVC directory for a workflow
kubectl get pvc -n default -o json | \
  jq -r '.items[] | "\(.metadata.name): \(.spec.volumeName)"'

# Data is stored in subdirectories of /opt/deployment/argo-workflows/k3d/storage/
ls -lh /opt/deployment/argo-workflows/k3d/storage/

Backup Workflow Data

# Backup a specific PVC's data
sudo tar -czf workflow-backup-$(date +%Y%m%d).tar.gz \
  -C /opt/deployment/argo-workflows/k3d/storage/ <pvc-directory>

Maintenance

Check System Status

# Full production verify
just prod-verify

# Check GPU
kubectl get nodes -o json | jq '.items[].status.capacity."nvidia.com/gpu"'

# Check Argo
kubectl get pods -n default

Clean Up Old Workflows

# Delete completed workflows older than 7 days
argo delete -n default --older 7d --completed

# Delete all workflows
just clean-workflows

Troubleshooting

GPU Not Detected

Check NVIDIA device plugin:

# Check device plugin pods
kubectl get pods -n kube-system -l name=nvidia-device-plugin-ds

# Check device plugin logs
kubectl logs -n kube-system -l name=nvidia-device-plugin-ds

# Verify GPU on host
nvidia-smi

Workflow Pod Pending

Check pod events:

kubectl describe pod <pod-name> -n default

Common issues: - Insufficient GPU: GPU already in use - Image not found: Image not available in k3s - PVC pending: Storage provisioner issue

Image Not Found

Verify image is available:

# List local Docker images
docker images | grep -E '(inference|detector|tator|tool)'

# List images in k3s
k3s crictl images | grep -E '(inference|detector|tator|tool)'

Storage Issues

Check storage provisioner:

# Check local-path provisioner
kubectl get pods -n kube-system -l app=local-path-provisioner

# Check storage class
kubectl get storageclass

# Check PVC status
kubectl get pvc -n default

k3s Service Won't Start

Check logs:

sudo journalctl -u k3s -n 100 --no-pager

# Check containerd config
cat /var/lib/rancher/k3s/agent/etc/containerd/config.toml.tmpl

Configuration

Environment Variables

Production configuration is in ansible/group_vars/prod.yml:

# GPU settings
gpu_enabled: true
gpu_runtime_class: nvidia

# Storage
storage_class: local-path
storage_base_path: /opt/deployment/argo-workflows/k3d/storage

# Resources (adjust based on your hardware)
inference_gpu_request: "1"
inference_memory_request: 8Gi
inference_cpu_request: "4"

Workflow Parameters

Edit workflow files or pass parameters at submission:

# Edit default in workflows/ultralytics-inference.yml
# Or pass at runtime:
argo submit -n default workflows/ultralytics-inference.yml \
  -p model-name=yolo11x.pt \
  -p confidence-threshold=0.4 \
  -p iou-threshold=0.45

Security Considerations

File Permissions

• k3s config: /opt/deployment/argo-workflows/k3d/ should be readable by your user
• Kubeconfig: /opt/deployment/argo-workflows/k3d/kubeconfig/k3s.yaml mode 0644
• Storage: /opt/deployment/argo-workflows/k3d/storage/ should be writable by k3s

Network Access

k3s API server runs on port 6443. For remote access, configure firewall:

# Allow k3s API (if needed)
sudo ufw allow 6443/tcp

Quick Reference

Common Commands

# Status
just prod-verify              # Verify setup
kubectl get nodes -o wide     # Cluster nodes/status

# Build & Deploy
just build-all-non-tator      # Build non-Tator images
just build-all-tator          # Build Tator images
just deploy-inference-gpu     # Deploy GPU workflow

# Submit Workflows
just submit-inference-gpu     # Submit GPU inference
just submit-ultralytics       # Submit Ultralytics GPU inference (example)

# Monitor
argo list -n default            # List workflows
argo logs -n default @latest    # Latest logs
kubectl get pods -n default     # List pods

File Locations

Component	Location
k3s binary	/opt/deployment/bin/k3s
k3s config	/opt/deployment/argo-workflows/k3d/k3s-config.yaml
Kubeconfig	/opt/deployment/argo-workflows/k3d/kubeconfig/k3s.yaml
k3s data	/opt/deployment/argo-workflows/k3d/data/
Containerd config	/opt/deployment/argo-workflows/k3d/data/agent/etc/containerd/
Storage	/opt/deployment/argo-workflows/k3d/storage/
Systemd service	/etc/systemd/system/k3s.service
Logs	sudo journalctl -u k3s -f

Comparison: Dev vs Production

Development Workflow (Minikube)

# Start minikube
just minikube-start

# Build images
just build-all

# Deploy
just deploy-inference-cpu

# Submit CPU inference
just submit-inference-cpu

Production Workflow (k3s)

# One-time setup
just prod-setup

# Build images
just build-all-non-tator
just build-all-tator

# Deploy
just deploy-inference-gpu

# Submit GPU inference
just submit-inference-gpu

Support

For issues or questions: 1. Check logs: journalctl -u k3s -f 2. Run verification: just prod-verify 3. Check workflow events: kubectl describe workflow <name> -n default 4. Review this documentation

Additional Resources

• k3s Documentation
• Argo Workflows
• NVIDIA Device Plugin
• Local Path Provisioner

🗓️ Updated: 2026-08-25