GPU Inference

Deploy AI models with an OpenAI-compatible REST API. RailPush schedules your inference server on dedicated NVIDIA GPUs, handles TLS, load balancing, and autoscaling -- you ship the model, we run it.

Supported Frameworks

Framework	Description
vLLM	High-throughput LLM inference engine. OpenAI-compatible /v1/chat/completions and /v1/completions endpoints.
Text Generation Inference (TGI)	HuggingFace TGI for fast transformer inference with continuous batching.

Available GPUs

GPU	VRAM	Best for
a100-40gb	40 GB HBM2	7B--30B models, high throughput
a100-80gb	80 GB HBM2e	65B--70B models, large batches
l40s	48 GB GDDR6	Balanced cost/performance, 13B--34B
rtx-4090	24 GB GDDR6X	Smaller models, dev/staging

Deploy a vLLM Inference Service

Create inference service via API

curl -X POST https://railpush.com/api/v1/inference \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "llama3-8b",
    "framework": "vllm",
    "model": "meta-llama/Meta-Llama-3-8B-Instruct",
    "gpu_type": "a100-40gb",
    "hf_token": "<your-hf-token>"
  }'

# Response: { "id": "...", "endpoint": "https://llama3-8b.railpush.com" }

Query the Endpoint

curl

# OpenAI-compatible chat completions
curl https://llama3-8b.railpush.com/v1/chat/completions \
  -H "Authorization: Bearer <your-railpush-api-key>" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Meta-Llama-3-8B-Instruct",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

Python (OpenAI SDK)

from openai import OpenAI

client = OpenAI(
    base_url="https://llama3-8b.railpush.com/v1",
    api_key="<your-railpush-api-key>",
)

completion = client.chat.completions.create(
    model="meta-llama/Meta-Llama-3-8B-Instruct",
    messages=[{"role": "user", "content": "What is RailPush?"}],
)
print(completion.choices[0].message.content)

Drop-in OpenAI replacement

Any library or tool that speaks the OpenAI API works with RailPush inference endpoints out of the box -- LangChain, LlamaIndex, Vercel AI SDK, and more.