Run Whisper Speech-to-Text API on an Ubuntu VPS

Transcription is a simple way to add useful AI without sending customer audio to a third-party API. If you need private speech-to-text for support calls, voice notes, or media uploads, you can run Whisper on a VPS. Then expose a small HTTP endpoint behind TLS.

This walkthrough uses Ubuntu, systemd, and Nginx. It also includes practical CPU/GPU sizing so you don’t buy the wrong box.

The setup is single-node by design. When volume grows, keep the same client integration. Then move the service to a bigger VM (or a GPU host).

What you’ll build (and what it’s good for)

You’ll deploy a Whisper-based transcription service. It accepts an audio file (WAV/MP3/M4A) and returns JSON with the transcript.

Common uses:

Private transcription for uploaded audio on your site (podcasts, interviews, course content).
Voice notes inside an internal tool (sales, support, ops) with your own retention rules.
Search pre-processing: store transcripts in your database and index them.

The goal is an operator-friendly service. No Kubernetes.

You get predictable logs. Nginx sits at the edge for HTTPS and request control.

Prerequisites and realistic resource sizing

Decide up front whether you’ll run CPU-only or plan for GPU acceleration. Both work.

The right choice depends on file length, concurrency, and latency expectations.

CPU-only sizing (common for most site owners)

Minimum: 4 vCPU, 8 GB RAM (fine for light volume and smaller models).
Comfortable: 8 vCPU, 16 GB RAM (handles more concurrent jobs with fewer timeouts).
Disk: 30+ GB SSD/NVMe. Uploads and temporary WAVs grow faster than you expect.

CPU-only transcription is slower, but it’s straightforward.

If you expect long recordings or steady volume, plan for a GPU or add a queue.

GPU sizing (for throughput and lower latency)

VRAM: 8–16 GB VRAM is a sensible starting point for medium-to-large models with reasonable batch sizes.
System RAM: 16–32 GB recommended to avoid cache pressure under load.
Reality check: a GPU speeds up inference, but you still need CPU for decoding, request handling, and TLS.

If you’re weighing options, a HostMyCode VPS works well for CPU-only transcription APIs.

If your main app already has spikes, put transcription on its own VM. That keeps it from stealing resources from your web tier.

Other prerequisites

Ubuntu server with a public IP
A domain/subdomain pointed to your VPS (for example: stt.example.com)
Basic Linux admin comfort: editing files, restarting services, reading logs

If DNS isn’t set up yet, create an A/AAAA record at your registrar or DNS provider.

HostMyCode also provides domains and DNS management via domains.

Step 1: Prepare the Ubuntu VPS

Update packages and install the basics:

sudo apt update
sudo apt -y upgrade
sudo apt -y install nginx certbot python3-certbot-nginx ffmpeg curl unzip

Why ffmpeg? Real uploads are messy. Converting everything to a clean 16kHz mono WAV avoids many edge-case failures.

Set a simple firewall policy. If you’re using UFW:

sudo ufw allow OpenSSH
sudo ufw allow 'Nginx Full'
sudo ufw --force enable
sudo ufw status

If you want a more complete baseline after the service is up, follow Set Up UFW Firewall on Ubuntu for a Web Server.

Step 2: Create a dedicated user and directories

Don’t run this as root. Create a system user and a consistent filesystem layout:

sudo useradd --system --home /opt/whisper-api --create-home --shell /usr/sbin/nologin whisper
sudo mkdir -p /opt/whisper-api/{bin,models,work,logs}
sudo chown -R whisper:whisper /opt/whisper-api

This layout keeps things predictable.

Use /opt/whisper-api/bin for executables, /opt/whisper-api/models for model files, and /opt/whisper-api/work for temporary jobs.

Step 3: Install a Whisper server (Faster-Whisper + HTTP API)

This build uses a practical pairing:

Faster-Whisper for efficient inference on CPU or GPU
A small FastAPI app so clients get a stable HTTP interface

Install Python tooling:

sudo apt -y install python3-venv python3-pip

Create a virtual environment owned by the whisper user:

sudo -u whisper python3 -m venv /opt/whisper-api/venv
sudo -u whisper /opt/whisper-api/venv/bin/pip install --upgrade pip wheel

Install dependencies (CPU-only baseline):

sudo -u whisper /opt/whisper-api/venv/bin/pip install \
  fastapi uvicorn python-multipart pydantic \
  faster-whisper

If you’re running NVIDIA GPUs, you’ll usually add a GPU-enabled runtime for the inference backend. Driver/CUDA combinations vary.

Validate behavior on CPU first. Then switch to GPU.

For a GPU ops pattern that transfers well, see Run vLLM on a GPU VPS With OpenAI-Compatible API (different workload, similar operational approach).

Create the API app

Create /opt/whisper-api/app.py:

sudo -u whisper nano /opt/whisper-api/app.py

Paste this minimal, production-leaning API.

It accepts an upload, normalizes audio to 16kHz mono WAV using ffmpeg, transcribes it, then returns JSON:

import os
import uuid
import subprocess
from fastapi import FastAPI, UploadFile, File, HTTPException
from faster_whisper import WhisperModel

APP_NAME = "whisper-api"
WORK_DIR = os.environ.get("WHISPER_WORK_DIR", "/opt/whisper-api/work")
MODEL_NAME = os.environ.get("WHISPER_MODEL", "small")
DEVICE = os.environ.get("WHISPER_DEVICE", "cpu")  # cpu or cuda
COMPUTE_TYPE = os.environ.get("WHISPER_COMPUTE_TYPE", "int8")

app = FastAPI(title=APP_NAME)

model = WhisperModel(
    MODEL_NAME,
    device=DEVICE,
    compute_type=COMPUTE_TYPE,
)


def _run(cmd: list[str]):
    p = subprocess.run(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
    if p.returncode != 0:
        raise RuntimeError(p.stderr.strip())


@app.get("/health")
def health():
    return {"status": "ok", "model": MODEL_NAME, "device": DEVICE, "compute_type": COMPUTE_TYPE}


@app.post("/v1/transcribe")
async def transcribe(file: UploadFile = File(...), language: str | None = None):
    if not file.filename:
        raise HTTPException(status_code=400, detail="Missing filename")

    job_id = str(uuid.uuid4())
    in_path = os.path.join(WORK_DIR, f"{job_id}-in")
    wav_path = os.path.join(WORK_DIR, f"{job_id}.wav")

    os.makedirs(WORK_DIR, exist_ok=True)

    data = await file.read()
    if len(data) < 10:
        raise HTTPException(status_code=400, detail="Empty upload")

    with open(in_path, "wb") as f:
        f.write(data)

    try:
        # Normalize to Whisper-friendly WAV
        _run([
            "ffmpeg", "-y",
            "-i", in_path,
            "-ac", "1",
            "-ar", "16000",
            "-vn",
            wav_path
        ])

        segments, info = model.transcribe(wav_path, language=language, vad_filter=True)

        text_parts = []
        for s in segments:
            text_parts.append(s.text)

        text = "".join(text_parts).strip()
        return {
            "id": job_id,
            "language": info.language,
            "duration": info.duration,
            "text": text
        }
    except Exception as e:
        raise HTTPException(status_code=500, detail=str(e))
    finally:
        for p in (in_path, wav_path):
            try:
                if os.path.exists(p):
                    os.remove(p)
            except Exception:
                pass

Model choice: small is a solid default for CPU. If you have headroom, try medium.

Bigger models use more RAM and usually increase latency.

Step 4: Run it locally and verify basic behavior

Start the API bound to localhost only:

sudo -u whisper /opt/whisper-api/venv/bin/uvicorn app:app \
  --app-dir /opt/whisper-api \
  --host 127.0.0.1 --port 8000

In another SSH session, verify the health endpoint:

curl -s http://127.0.0.1:8000/health | jq .

Then test a transcription (replace sample.mp3 with your file):

curl -s \
  -F "file=@sample.mp3" \
  "http://127.0.0.1:8000/v1/transcribe" | jq .

If something breaks, it’s usually one of these:

ffmpeg errors: run the same conversion manually and read stderr.
RAM pressure: a small VPS may get OOM-killed. Check dmesg -T | tail -n 50.

Step 5: Create a systemd service (autostart, logs, restarts)

Stop the foreground server. Then create a unit file:

sudo nano /etc/systemd/system/whisper-api.service

Use the unit below. It runs as whisper.

It binds to localhost and restarts on failure:

[Unit]
Description=Whisper Speech-to-Text API
After=network.target

[Service]
Type=simple
User=whisper
Group=whisper
WorkingDirectory=/opt/whisper-api
Environment=WHISPER_MODEL=small
Environment=WHISPER_DEVICE=cpu
Environment=WHISPER_COMPUTE_TYPE=int8
Environment=WHISPER_WORK_DIR=/opt/whisper-api/work
ExecStart=/opt/whisper-api/venv/bin/uvicorn app:app --app-dir /opt/whisper-api --host 127.0.0.1 --port 8000
Restart=on-failure
RestartSec=2

# Basic hardening
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/opt/whisper-api/work

[Install]
WantedBy=multi-user.target

Enable and start it:

sudo systemctl daemon-reload
sudo systemctl enable --now whisper-api
sudo systemctl status whisper-api --no-pager

Tail logs:

sudo journalctl -u whisper-api -f

Step 6: Put Nginx in front (TLS, size limits, timeouts)

Don’t expose Uvicorn directly to the internet. Put Nginx in front.

Nginx terminates TLS, enforces upload limits, and gives you clean controls for rate limiting and timeouts.

Create the Nginx site

Create /etc/nginx/sites-available/stt.example.com (replace the domain):

sudo nano /etc/nginx/sites-available/stt.example.com

Paste this config:

server {
  listen 80;
  server_name stt.example.com;

  # Allow larger audio uploads. Adjust to your needs.
  client_max_body_size 50m;

  location / {
    proxy_pass http://127.0.0.1:8000;
    proxy_http_version 1.1;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;

    # Transcription can take time on CPU.
    proxy_connect_timeout 10s;
    proxy_send_timeout 300s;
    proxy_read_timeout 300s;
  }
}

Enable the site and test Nginx:

sudo ln -s /etc/nginx/sites-available/stt.example.com /etc/nginx/sites-enabled/stt.example.com
sudo nginx -t
sudo systemctl reload nginx

Add HTTPS with Let’s Encrypt

Issue a certificate:

sudo certbot --nginx -d stt.example.com

Certbot will update your Nginx config and set up renewals.

For TLS troubleshooting and renewal gotchas, keep this guide nearby: TLS setup guide tutorial (2026): Configure HTTPS with Let’s Encrypt on a VPS.

Step 7: Secure the endpoint (don’t ship an open transcription box)

An unauthenticated transcription endpoint won’t stay quiet for long. People will throw giant files at it. You’ll pay for the CPU time.

Option A: Simple API key header (good baseline)

You can enforce a shared key at the edge. Generate one:

openssl rand -hex 32

Edit your Nginx server block and add this near the top (inside server { }):

set $allow_key 0;
if ($http_x_api_key = "PUT_YOUR_RANDOM_KEY_HERE") { set $allow_key 1; }
if ($allow_key = 0) { return 401; }

Reload Nginx:

sudo nginx -t
sudo systemctl reload nginx

Test with the header:

curl -s https://stt.example.com/health \
  -H "X-API-Key: PUT_YOUR_RANDOM_KEY_HERE"

This isn’t “full auth.” It does block drive-by abuse.

It also works well for server-to-server calls.

Option B: Restrict access by network (best for internal apps)

If only your backend servers should call the API, whitelist their IPs in Nginx:

allow 203.0.113.10;  # your app server
allow 198.51.100.22; # your office IP
deny all;

For private admin access patterns, a VPN is often cleaner than juggling IP lists: Set Up WireGuard VPN on Ubuntu VPS for Admin Access.

Rate limiting (so one client can’t starve your CPU)

Even with an API key, limit request rates.

Add a zone in /etc/nginx/nginx.conf inside the http {} block:

limit_req_zone $binary_remote_addr zone=sttlimit:10m rate=5r/m;

Then, in your server block inside location /:

limit_req zone=sttlimit burst=10 nodelay;

For a deeper playbook (including how to avoid blocking legitimate users), see Nginx Rate Limiting Tutorial (2026).

Step 8: Production tuning: model choice, quantization, and timeouts

You’ll see the trade-offs quickly once real audio shows up. The API can be “working” and still fail on realistic files.

Model size vs. cost

Smaller models: faster, lower RAM, but more errors on accents or noisy audio.
Larger models: better accuracy, but higher CPU time and memory use.

Start with small. If quality is the complaint, try medium.

As you test, watch CPU time and memory during peak traffic.

Quantization settings (compute_type)

The systemd unit uses WHISPER_COMPUTE_TYPE=int8 to keep CPU costs down.

If you see odd behavior, try int8_float16 or float16 (especially on GPU).

Quantization is a trade. Performance improves, but accuracy and stability can shift.

Timeouts you should actually set

Long audio on CPU often exceeds default proxy timeouts. You already bumped Nginx to 300 seconds.

But 30–60 minute files can still run longer.

That’s usually a signal to do one of the following:

Enforce a max duration (application side), or
Move to async processing with a queue and return a job ID.

Step 9: Monitoring and quick diagnostics

For this service, the two failure modes are simple: CPU saturation and memory pressure.

Watch live usage

sudo apt -y install htop
htop

What to watch:

Cores pinned at 100% during transcribes (normal under load).
Swap activity (usually means latency spikes are coming).
OOM kills (confirm via dmesg).

Check systemd and Nginx logs

sudo journalctl -u whisper-api --since "30 min ago" --no-pager
sudo tail -n 200 /var/log/nginx/error.log

If you want alerts for CPU/disk and “service down,” build a small monitoring stack. It’s a good fit for bursty AI workloads: Server Monitoring Setup Guide Tutorial (2026).

Step 10: Integrate with a website or app (example client calls)

From your application server, send audio to the endpoint. Example with curl:

curl -s \
  -H "X-API-Key: PUT_YOUR_RANDOM_KEY_HERE" \
  -F "file=@voice-note.m4a" \
  "https://stt.example.com/v1/transcribe" | jq -r .text

If you run WordPress and want transcription for uploaded media, don’t call the endpoint from the browser.

Send requests server-side (or via a plugin hook). That keeps your API key out of frontend code.

Common pitfalls (the stuff that burns time)

Exposing port 8000 publicly: keep Uvicorn on 127.0.0.1. Only Nginx should face the internet.
Upload size surprises: Nginx can reject large files by default. Set client_max_body_size deliberately.
Long audio on small VPS: CPU-only transcription for 60-minute files can be painfully slow. Consider chunking or async jobs.
Disk fills up: temporary WAV files are large. Monitor /opt/whisper-api/work and keep cleanup in mind.

Where HostMyCode fits (infrastructure choices)

For most private transcription workloads, a CPU VPS with headroom is the cleanest place to start.

If your app is already running elsewhere, keep speech-to-text separate. Then you can scale or throttle it without touching the web stack.

Start with a HostMyCode VPS for a dedicated transcription endpoint.
If you’d rather not handle patching and service babysitting, consider managed VPS hosting and focus on integration.

If you want private speech-to-text without handing customer audio to third parties, HostMyCode is a practical place to host it. Start with a HostMyCode VPS, and switch to managed VPS hosting if you want to offload routine server maintenance while keeping the transcription stack self-hosted.

FAQ: Run Whisper Speech-to-Text on a VPS

Can I run Whisper on a small 2 vCPU VPS?

You can, but you’ll hit timeouts and memory pressure quickly. For real usage, 4 vCPU / 8 GB RAM is a safer floor, especially with concurrent requests.

Should I use GPU from day one?

Not necessarily. CPU-first keeps the setup simpler and lets you validate API behavior and security. Move to GPU when you need higher throughput or lower latency on longer files.

How do I stop users from uploading huge files?

Set client_max_body_size in Nginx, enforce a max duration in your application, and add rate limiting. If usage grows, switch to async jobs and a queue.

Is an API key in Nginx “secure enough”?

It’s a solid baseline for server-to-server calls if you keep the key off the frontend and rotate it periodically. For multi-user products, implement real auth (JWT/OAuth) in the application layer.

How do I back up anything here?

The service is stateless if you don’t store transcripts. If you do store transcripts, back up the database and any object storage. Treat audio uploads as user data and enforce retention rules.

Summary

You now have a private speech-to-text endpoint. Whisper inference runs under systemd and sits behind Nginx with HTTPS and basic abuse controls.

Start with CPU, measure real usage, then decide whether GPU or asynchronous processing is the next upgrade.

If you want a clean environment to run Whisper on a VPS without risking your main site’s stability, deploy it on a separate HostMyCode VPS. Then scale it independently as transcription load grows.