Operator Manual

Operational guidance for running and maintaining a PRISM MEV instance.

Health and Monitoring

The providers health endpoint: GET /api/v1/providers/health
Integrate with your monitoring stack (Prometheus/Grafana) to scrape metrics if exposed by your deployment.

Before deploying to production, ensure the following minimums and preparation:

Infrastructure (HelioHost VPS)
- Recommended: Ubuntu 22.04 LTS (or compatible), 2 vCPU, 4–8 GB RAM, 40+ GB SSD
- Static public IPv4, root or sudo access
- Open inbound ports: 80/tcp and 443/tcp for Coolify (reverse proxy & TLS)
- Outbound egress to Solana RPC endpoints (HTTPS 443, WebSocket 443)
Domain & DNS
- A/AAAA record(s) for your Coolify panel and app subdomains pointing to the VPS IP
- If using Cloudflare, set SSL mode to "Full" and allow HTTP challenge on port 80 for initial certificate issuance
Software
- Docker Engine and Docker Compose Plugin installed
- Coolify (self‑hosted PaaS) installed on the VPS
Security & Secrets
- Production RPC URLs and API keys ready (do not hardcode; store as Coolify secrets)
- Any private keys required by the app are stored securely (hardware/remote signer preferred); never commit secrets
- Enforce least privilege for integrations; rotate credentials on schedule
Operational Readiness
- Define SLOs for latency and availability
- Monitoring and alerting destinations determined (Pager/SMS/Email)
- Backup and rollback strategy confirmed (Coolify backups, VPS snapshots)

High‑level flow using Coolify’s official installation and UI. Adjust to match your environment and organization policies.

Install Docker
- Follow official Docker install for your OS; verify:
```
docker --version && docker compose version
```
Install Coolify
- Follow Coolify’s official installation guide to run it via Docker/Compose on ports 80/443.
- After start, access the Coolify dashboard via your configured domain/IP and create the admin account.
Configure DNS and TLS
- Point your chosen domain/subdomain(s) to the VPS IP via A/AAAA records.
- In Coolify, enable TLS for the domain; ensure port 80 is reachable for HTTP challenge.
Create an Application in Coolify
- Source: connect to your Git repository for prism-rust or use a container image if prebuilt.
- Choose build strategy (Dockerfile/Buildpacks). If using Dockerfile, ensure it exists at repo root or specify location in Coolify.
- Set the service’s internal/external ports to match your app’s exposed port.
Configure Environment Variables & Secrets
- Add RPC endpoints, provider tokens, and any feature flags as Coolify environment variables/secrets.
- Do not commit credentials to the repository.
Deploy
- Trigger the initial build and deployment from the Coolify UI.
- Verify health:
  - GET /api/v1/providers/health responds 200
  - Application logs show no startup errors
Verification Checklist
- App reachable over HTTPS with valid certificate
- Metrics/observability (if enabled by your deployment) are scraping and visible
- Coolify shows deployment healthy and container(s) running

Health Checks
- Monitor GET /api/v1/providers/health for liveness.
- Track error rates and latency in your monitoring stack if metrics are enabled.
Log Review
- Use Coolify’s container log viewer for recent logs.
- For deeper analysis, aggregate logs to your central system.
Deployment Management
- Use Coolify to trigger rolling redeployments from main/trunk after reviews pass.
- Keep an annotated change log to correlate deploys with metrics.
Docs Hygiene
- make docs-validate on each PR affecting docs.
- make docs-build to ensure publishability.

Upgrades
- Application: Redeploy from updated repo or pull new image tag via Coolify.
- Coolify platform: Follow Coolify’s upgrade notes; schedule maintenance window.
- Docker Engine: Upgrade during off‑peak; restart Coolify afterward.
Secrets & Keys
- Rotate API keys on a fixed cadence; update Coolify secrets and redeploy.
- Validate access logs for unused credentials and revoke promptly.
Backups
- Use Coolify’s backup features if configured, and/or VPS snapshots.
- Test restores periodically in a staging environment.
Capacity Management
- Monitor CPU/RAM/disk. Clean unused images/volumes during maintenance:
```
docker system df
docker image prune -a
docker volume ls
```

Roll Back a Bad Deploy
- In Coolify, redeploy the previous successful commit or image tag.
- If necessary, scale replicas to zero to halt traffic, then restore to last known good.
Disable External Traffic Quickly
- Temporarily change DNS to a maintenance page or remove the A record.
- If using Cloudflare, enable maintenance mode / block rules for the app subdomain.
Credential Compromise
- Revoke affected keys immediately; rotate secrets in Coolify and redeploy.
- Audit logs and increase monitoring thresholds.
Resource Exhaustion (CPU/RAM/Disk)
- Scale down workload or temporarily disable non‑critical services.
- Prune Docker artifacts to reclaim space; expand disk if needed.

Symptom: HTTPS certificate not issued; browser shows invalid certificate.
- Ensure port 80 is open for HTTP challenge.
- Verify DNS A/AAAA records point to the correct VPS IP and have propagated.
- If using Cloudflare, set SSL mode to "Full" (not "Flexible"); temporarily disable "Always Use HTTPS" until issuance succeeds.

Symptom: Coolify or app fails to start; errors about ports 80/443 already in use.
- Stop or remove other services (e.g., Apache/Nginx) binding 80/443.
- Verify with:
```
sudo lsof -i :80 -i :443
```

Symptom: Builds killed (OOM) or fail intermittently.
- Add swap (e.g., 2–4 GB) on small VPS instances.
- Use multi‑stage Docker builds or remote builders to reduce memory usage.

Symptom: no space left on device, image pulls/builds fail.
- Clean Docker artifacts:
```
docker system prune -a
docker volume ls
```
- Expand VPS disk or move Docker data to larger volume as needed.

Symptom: Coolify reports healthy, but app URL 404/502.
- Check service port mapping in Coolify matches container’s exposed port.
- Confirm health endpoint GET /api/v1/providers/health returns 200.
- Inspect reverse proxy (Traefik) logs from Coolify.

Symptom: Timeouts reaching Solana RPC/WebSocket endpoints.
- Verify firewall/eGRess policies allow HTTPS/WSS to provider domains.
- Check DNS resolution on the VPS; ensure NTP time sync is healthy.

Symptom: Coolify UI/agent not responding.
- Restart Coolify containers via Docker and recheck logs.
- Ensure VPS resources are sufficient; review Docker and system logs.

Coolify official documentation (installation, upgrades, backups)
Docker Engine and Compose official documentation
Your organization’s runbooks for secrets rotation, backup, and incident response