Armada AI

LLM Inference Planning & Optimization

Engine-independent predictions of KV-cache capacity, latency, throughput and serving cost from spec sheets, with optional GPU-profile calibration.

Built the planner and used its predictions to choose serving configurations, then deployed them. Spec-sheet KV-cache capacity predictions were exact on 25/25 dense and MoE configurations.

Decisions

Compares GPU and configuration choices for a selected model and workload: weight quantization, KV-cache quantization, tensor parallelism, eager execution and automatic prefix caching. Quantization work includes AWQ, GPTQ and weight/activation precision configurations (WxAy). The planner recommends settings; I carry out deployment afterward. It does not automatically deploy workloads.

Predictions

The planner is engine-independent. Models KV-cache requirements, time to first token (TTFT), time per output token (TPOT) and token throughput before downloading a model or running a trial GPU deployment. Covers dense, mixture-of-experts and hybrid model families, including mixed attention patterns and attention/state-space architectures.

From inference planning to deployment

Workflow overview

Planning tool

  1. Input

    Specify inputs

    Model, GPU, workload, GPU cost/hour; optional GPU profile

  2. Process

    Predict

    KV cache, TTFT, TPOT and throughput

  3. Check

    Compare choices

    Hardware and inference settings

  4. Output

    Recommend

    Configuration choices and estimated token costs

Engineering handoff

RecommendSelected configuration

My engineering work

  1. Input

    Selected configuration

    Model, GPU and serving settings

  2. Output

    Deploy workload

    Hands-on deployment after planning

Spec-sheet predictions need no GPU access; a GPU profile can optionally calibrate predictions. The planner recommends settings using the supplied GPU cost per hour. I choose the serving configuration and deploy it afterward.

Validation against measured GPU runs

KV-cache capacity predictions from spec sheets were exact on 25/25 dense and MoE configurations (15 dense, 10 MoE). For hybrid architectures, maximum concurrency (N_max) was within 2% on 22 of 24 configurations. Neither result uses GPU-profile calibration.

For dense models, spec-sheet-only versus GPU-profiled mean absolute percentage error (MAPE) was 15.9% versus 3.6% for time to first token (TTFT; n=174), 13.8% versus 5.2% for batched time per output token (TPOT; n=242), and 7.2% versus 4.4% for single-user TPOT (n=134). Spec-sheet-only predictions require no GPU access.

Profiling takes roughly a few minutes on the target GPU and measures the gap between spec-sheet and actual hardware behavior. Some evaluation runs were also used to calibrate the GPU profile, so profiled errors are partly in-sample. They do not establish held-out generalization. MAPE is an average, not a worst-case bound; individual errors can exceed it. Profiling does not improve every metric or architecture.

Current limitations

Throughput prediction is the weakest measured area. The 256-user evaluation has only 3 to 6 samples per architecture family, so throughput accuracy is not yet well established. Serving-cost estimates also inherit uncertainty from predicted throughput.

Deployment

I used the planner’s predictions to choose serving configurations, then deployed those configurations. Deployment is my engineering work; the planner itself recommends settings.

Estimated economics

Estimates prefill and decode costs per million tokens from the user-supplied hourly GPU cost and predicted throughput. These are planning estimates, distinct from measured serving costs.

Tools

KV-cache modeling · LLM inference · Quantization · Tensor parallelism · Automatic prefix caching