Operations

Self-hosting

Run webfetch on your own infrastructure — Docker, Kubernetes, BYOK for provider auth, observability, and scaling notes.

Everything in webfetch is self-hostable. The same binaries that power webfetch Cloud run on your infrastructure with zero functional difference. This page covers the operational details.

#What you're hosting

  • webfetch — a Node/Bun binary. No daemon, no config server.
  • webfetch-server — a stateless HTTP server on 127.0.0.1:7600 by default.
  • getwebfetch-mcp — an MCP stdio server. Used inline by IDEs; also runs standalone.

All three share webfetch-core, so behavior is identical across surfaces.

#Prerequisites

  • Bun 1.1+ or Node 20+.
  • (Optional) Playwright + headless Chromium for the browser fallback: bunx playwright install chromium.
  • (Optional) Redis for shared cache in a horizontally-scaled deployment.

#Run the HTTP server

#Bare metal

git clone https://github.com/ashlrai/webfetch
cd webfetch
bun install
UNSPLASH_ACCESS_KEY=... \
PEXELS_API_KEY=... \
bun run --cwd packages/server start

On first boot the server generates ~/.webfetch/server.token, prints it to stdout, and opens http://127.0.0.1:7600/auth/display unless you pass --no-open. Every API request, including /health and /providers, must send Authorization: Bearer <token>.

#Docker

docker run -d --name webfetch \
  -p 7600:7600 \
  -e UNSPLASH_ACCESS_KEY=... \
  -e PEXELS_API_KEY=... \
  -v webfetch-home:/root/.webfetch \
  ghcr.io/ashlrai/webfetch:latest server --host 0.0.0.0 --no-open

#Docker Compose

services:
  webfetch:
    image: ghcr.io/ashlrai/webfetch:latest
    command: ["server", "--host", "0.0.0.0", "--no-open"]
    ports: ["127.0.0.1:7600:7600"]
    environment:
      WEBFETCH_CACHE_DIR: "/var/cache/webfetch"
      UNSPLASH_ACCESS_KEY: "${UNSPLASH_ACCESS_KEY}"
      PEXELS_API_KEY: "${PEXELS_API_KEY}"
      SPOTIFY_CLIENT_ID: "${SPOTIFY_CLIENT_ID}"
      SPOTIFY_CLIENT_SECRET: "${SPOTIFY_CLIENT_SECRET}"
    volumes:
      - webfetch-cache:/var/cache/webfetch
      - webfetch-home:/root/.webfetch
    restart: unless-stopped
    healthcheck:
      test: ["CMD-SHELL", "TOKEN=$$(cat /root/.webfetch/server.token); curl -fsS -H \"authorization: Bearer $$TOKEN\" http://127.0.0.1:7600/health >/dev/null"]
      interval: 30s

volumes:
  webfetch-cache:
  webfetch-home:

#Kubernetes (Helm hint)

A first-party Helm chart is on the roadmap. In the meantime, a minimal Deployment + Service works:

apiVersion: apps/v1
kind: Deployment
metadata: { name: webfetch }
spec:
  replicas: 2
  selector: { matchLabels: { app: webfetch } }
  template:
    metadata: { labels: { app: webfetch } }
    spec:
      containers:
        - name: webfetch
          image: ghcr.io/ashlrai/webfetch:latest
          args: ["server", "--host", "0.0.0.0", "--no-open"]
          ports: [{ containerPort: 7600 }]
          envFrom: [{ secretRef: { name: webfetch-secrets } }]
          resources:
            requests: { cpu: "100m", memory: "256Mi" }
            limits:   { cpu: "1",    memory: "1Gi" }
          readinessProbe:
            exec: { command: ["sh", "-c", "TOKEN=$(cat /root/.webfetch/server.token); curl -fsS -H \"authorization: Bearer $TOKEN\" http://127.0.0.1:7600/health >/dev/null"] }
            initialDelaySeconds: 5
          volumeMounts:
            - { name: cache, mountPath: /var/cache/webfetch }
            - { name: home, mountPath: /root/.webfetch }
      volumes:
        - { name: cache, emptyDir: {} }
        - { name: home, emptyDir: {} }
---
apiVersion: v1
kind: Service
metadata: { name: webfetch }
spec:
  selector: { app: webfetch }
  ports: [{ port: 7600, targetPort: 7600 }]

For a horizontally-scaled deployment, put a shared Redis behind WEBFETCH_CACHE_REDIS_URL so every replica sees the same cache.

#Bring your own keys

Every provider key is a plain environment variable. See the CLI reference for the full list. Providers without a configured key skip silently — no error, no warning unless you pass --verbose. Start with the free-tier keys (Unsplash, Pexels, Pixabay, Flickr, Europeana, Smithsonian) and add paid ones (Brave, SerpAPI) when you hit coverage gaps.

#The cache

Downloaded bytes land in $WEBFETCH_CACHE_DIR (default: ~/.webfetch/cache) keyed by SHA-256. The cache is append-only; webfetch never deletes. Budget 10–50 GB depending on your image flow — find $WEBFETCH_CACHE_DIR -atime +90 -delete is a reasonable LRU policy if disk pressure appears.

For shared-cache Kubernetes deployments, use a PVC or mount an object-storage bucket (S3, GCS) via FUSE.

#Observability

The server writes one JSON log line per request:

{
  "t": "2026-04-13T12:00:00Z",
  "endpoint": "/search",
  "duration_ms": 420,
  "ok": true,
  "providers_called": 5,
  "candidates_returned": 12,
  "licenses": { "CC_BY": 8, "CC_BY_SA": 4 }
}

Ship to your logging stack. Useful queries:

  • error_rate — count(ok=false) / count(*) over 5m.
  • p95_duration — alert if > 5s sustained.
  • provider_failure_rate — per-provider skip/error rate; tracks provider health.

No telemetry is sent anywhere. If you want webfetch Cloud's dashboards without Cloud, wire these logs into Grafana with the sample dashboards in docs/grafana/.

#Security posture

  • Outbound traffic only: image provider APIs on port 443.
  • Inbound: port 7600, bearer auth required for /health, /providers, and every POST endpoint. Only /auth/display is public.
  • No filesystem writes outside $WEBFETCH_CACHE_DIR.
  • No code execution — every response is JSON.

See SECURITY.md in the repo for our disclosure policy.

#Upgrading

Pull the latest image or git pull + rebuild. State is in two places:

  • The cache directory (keep it).
  • ~/.webfetchrc (keep it).

No migrations are needed between minor versions. Major versions (0.x → 1.x) will ship an explicit migration note in the Changelog.

#Scaling rules of thumb

  • One CPU core sustains ~5 concurrent searches with the default provider set.
  • Memory is flat around 200 MB per replica; add ~50 MB per concurrent Chromium (browser tier).
  • The upstream provider rate limits are the real bottleneck. webfetch providers --verbose shows your current per-provider quota usage.
  • For global traffic, deploy one replica per region and pin clients geographically — image CDNs serve better locally-routed traffic.

#Enabling the browser fallback

Not recommended on self-hosted deployments unless you need it. It's resource-intensive and requires Playwright + Chromium:

docker run -d --name webfetch \
  -e WEBFETCH_ENABLE_BROWSER=1 \
  -v /dev/shm:/dev/shm \
  ghcr.io/ashlrai/webfetch:latest server --host 0.0.0.0 --no-open

Install Playwright/Chromium in a derived image before enabling the browser provider. See Browser layer for what this costs and what it enables.