WorkflowStack AI
WorkflowsIndustriesToolsGuidesAI QuizBlogEnterprise
Get Free Workflows
WorkflowStack AI

Practical AI workflows for SMB operators and enterprise teams. No fluff. No hype. Just what ships.

Library

  • All Workflows
  • Industries
  • Enterprise
  • Tools
  • Guides

Company

  • About
  • Blog
  • Newsletter
  • Contact

Stay Updated

Weekly workflow ideas for operators and enterprise teams.

Get Free Workflows

© 2026 Blueteem LLC. All rights reserved.

Privacy PolicyTerms of Service
HomeBlogBaseten vs Modal vs Together AI (2026): When to Leave a Token API for Your Own GPUs
November 10, 2026

Baseten vs Modal vs Together AI (2026): When to Leave a Token API for Your Own GPUs

Baseten runs dedicated model deployments, Modal sells serverless GPU seconds, and Together AI and Groq sell hosted open-model tokens. Here is what each costs and the volume at which your own GPU starts to pay.

The short answer

Stay on a per-token API (Together AI, Groq or Baseten's Model APIs) until one model's traffic can keep a GPU busy. gpt-oss-120B costs $0.15/$0.60 per million tokens on Together. Against that, a Baseten H100 at about $6.50 an hour breaks even near 18 billion tokens a month at a 3:1 input-to-output mix, or roughly 6,900 tokens per second sustained. Pick Baseten for managed dedicated deployments, Modal for bursty per-second GPU jobs, and Together for the widest hosted open-model catalog.

Three kinds of inference, not three versions of one product

People type "Baseten vs Modal" as if they were the same thing. They sell different units:

  • Dedicated or custom-model inference (Baseten). You deploy a model, whether open-weight, fine-tuned or your own, onto GPUs that Baseten runs, autoscales and bills per minute per replica. Baseten also sells per-token Model APIs for popular open models.
  • Serverless GPU compute (Modal). You write Python functions, Modal runs them on GPUs, and you pay per second while containers run. Inference is one use among many, alongside batch jobs, fine-tuning and sandboxes. You own the serving code.
  • Hosted open-model APIs (Together AI, Groq). You call a model by name and pay per token. Together also rents dedicated endpoints and raw GPU clusters. Groq runs a short public catalog on its own hardware.

The query "Bedrock vs Baseten" belongs in the same frame. Amazon Bedrock also sells open models per token: gpt-oss-120b is $0.15 input and $0.60 output per million tokens in US East (N. Virginia). It is a token API, not a dedicated deployment. OpenRouter, by contrast, hosts no models: it routes requests to other providers' APIs and charges a fee when you buy credits. For the managed-cloud side, see our OpenRouter vs Amazon Bedrock vs Google Vertex AI comparison.

Prices and plan names are as published on each vendor's own pricing page; last verified 10 November 2026.

Three kinds of inference, not three versions of one product
VendorBilling unitPublished entry priceWhat it's for
BasetenPer minute per running replica (dedicated); per 1M tokens (Model APIs)NVIDIA T4 $0.01052/min; H100 80 GB $0.10833/min (about $6.50/hr); gpt-oss-120B API $0.10 in / $0.50 outManaged dedicated deployments of open, fine-tuned or custom models
ModalPer second (GPU, CPU, memory) plus plan feeNVIDIA T4 $0.000164/s; H100 SXM5 $0.001097/s; Starter $0/month with $30/month free credits; Team $250/month with $100/month creditsServerless GPU functions, batch jobs, inference you code yourself
Together AIPer 1M tokens (serverless); per GPU-hour (dedicated, clusters)gpt-oss-120B $0.15 / $0.60; dedicated HGX H100 $5.49/GPU-hourHosted open-model APIs, plus dedicated endpoints and GPU clusters
GroqPer 1M tokensGPT OSS 120B $0.15 / $0.60; GPT OSS 20B $0.075 / $0.30Fast hosted inference on a short public catalog

GPU prices, side by side

These are list rates for the same GPU class, converted to hourly where the vendor bills per second or per minute:

  • NVIDIA H100 (80 GB). Baseten dedicated: $0.10833/min (Baseten's own hourly figure is $6.50). Together dedicated inference: $5.49/hour. Modal: $0.001097/s, or $3.95/hour. Together GPU clusters (no managed serving): $3.99/hour on demand, $1.99/hour preemptible.
  • NVIDIA B200. Baseten: $0.16633/min (about $9.98/hour). Together dedicated: $8.99/hour. Modal: $0.001736/s, or $6.25/hour.
  • NVIDIA A100 80 GB. Baseten: $4.00/hour. Modal: $0.000694/s, or $2.50/hour.

Modal's figures are GPU-only; CPU and memory are billed per second on top. Comparing these naively will mislead you, for three reasons:

  • Modal's rates are base prices, and GPU functions are preemptible. Modal Functions "are subject to preemption by default", and its docs say the nonpreemptible option "is not supported for GPU Functions". Pinning a region multiplies GPU, CPU and memory prices by 1.15x (broad, e.g. us) to 1.75x (narrow, e.g. us-west).
  • Baseten's rate includes the managed serving layer. That covers autoscaling, replica management and deployment tooling. Modal provides building blocks, and your team writes and maintains the server.
  • Together's $5.49 is a dedicated inference endpoint, and its $3.99 is a cluster GPU. These are different products with different amounts of work left for you.

Per-token prices for the same open models

All three API sellers publish per-token prices for overlapping open-weight models:

Per-token prices for the same open models
ModelBaseten Model APIsTogether AI serverlessGroq
gpt-oss-120B$0.10 / $0.50$0.15 / $0.60$0.15 / $0.60
GLM-5.3$1.40 / $4.40$1.40 / $4.40Not listed
DeepSeek V4.1 Flash$0.30 / $1.20$0.30 / $1.20Not listed
Kimi K3.1 Pro$2.00 / $10.00$2.00 / $10.00Not listed

Prices are input / output per million tokens. Baseten and Together list identical input and output prices on several models (cached-input rates differ), so on those models the decision comes down to rate limits, latency and contract terms, not price. Groq's production self-serve list is short. Its models page prices GPT OSS 120B and 20B, plus Whisper. Llama 3.3 70B and Llama 3.1 8B are marked "Enterprise" with "Contact Sales" pricing. Groq lists GPT OSS 120B speed as 500 tokens per second (vendor-reported). Groq's Batch API is priced "at a 50% cost discount compared to synchronous API pricing" (gpt-oss-120b is supported). Together's batch docs promise "up to 50%" savings, but the discounted list currently names one model (openai/whisper-large-v3); "Models not listed run at standard rates", and dedicated inference is excluded.

Get one new AI workflow every week

Practical playbooks, prompts, and tool stacks — for the SMB operators and enterprise teams actually shipping AI.

Worked example: the break-even between tokens and a dedicated GPU

The comparison everyone needs is when an always-on GPU becomes cheaper than paying per token. Assumptions:

  • Model: gpt-oss-120B. Its Hugging Face model card says it runs "on a single 80GB GPU (like NVIDIA H100 or AMD MI300X)", so one H100 is a fair unit.
  • Traffic mix: 3 input tokens for every 1 output token.
  • Token API: Together serverless at $0.15 / $0.60. Blended cost = (3 × $0.15 + 1 × $0.60) ÷ 4 = $0.2625 per million tokens.
  • Dedicated GPU: one Baseten H100 running 24/7, at $0.10833/min × 60 × 730 hours ≈ $4,745/month.

Break-even volume: $4,745 ÷ $0.2625 per million ≈ 18.1 billion tokens a month.

Break-even throughput: 18.1 billion ÷ 2,628,000 seconds in a 730-hour month ≈ 6,900 tokens per second, every second of the month.

Now add utilization. Real traffic is not flat. If your GPU is actually busy 40% of the time, it has to process 6,900 ÷ 0.4 ≈ 17,200 tokens per second while busy to match the token API. No vendor on this list publishes a comparable aggregate-throughput figure for gpt-oss-120B on one H100, so benchmark your own prompts before you sign anything.

The same arithmetic on the other GPUs, all against $0.2625 per million:

  • Together dedicated H100 ($5.49/hour): about $4,008/month, break-even near 15.3 billion tokens, or about 5,800 tokens/second sustained.
  • Modal H100 ($3.95/hour, base rate, billed only while running): break-even is about 4,200 tokens per second during billed seconds. Modal bills only while containers run, so a bursty workload can come out ahead even at low monthly volume. Pinning a region raises that to about 4,800 tokens per second (1.15x) or 7,300 (1.75x). Modal also bills the idle window it keeps a container warm after traffic stops ("you will be billed for any resources used while the container is idle").
  • Against Baseten's own Model API ($0.10 / $0.50, blended $0.20 per million): the bar for a dedicated Baseten H100 rises to roughly 9,000 tokens per second sustained.

A team running 2 billion tokens a month (1.5 billion in, 0.5 billion out) pays $225 + $300 = $525/month on Together's serverless API, or $150 + $250 = $400/month on Baseten's Model API, against about $4,745 for a dedicated H100. At that volume the token API wins by roughly 9 to 12 times.

When a team should leave the token API anyway

Price is not the only reason to move to a dedicated deployment:

  • No API serves your model. A fine-tune, distilled checkpoint or proprietary model has to be hosted somewhere. Baseten's dedicated deployments and Modal are built for this.
  • You need isolation. Together describes its dedicated endpoints as "Single-tenant GPU instances" with "Guaranteed performance (no sharing)".
  • Rate limits, not price, are the bottleneck. Shared APIs cap you. On Groq's Developer plan, GPT OSS 120B is listed at 250K tokens per minute and 1K requests per minute.
  • Your cloud commitment can pay for it. Modal's Enterprise plan lists "Use committed spend on AWS and GCP" through cloud marketplaces.

When not to use each one

When not to use Baseten: your traffic is low or spiky and you would keep min_replica at 1 or more, which means paying for idle minutes. Scaling to zero avoids that, but Baseten's docs say "starting and loading a new replica is billable" and that a request can wait through the cold start. The pricing page publishes no plan tiers or free credits, so ask sales about volume terms.

When not to use Modal: you want a managed endpoint without writing serving code. It is also a poor fit when you need GPU capacity that cannot be preempted (Modal does not offer non-preemptible GPU functions), or when you need a specific region, which multiplies GPU, CPU and memory prices by 1.15–1.75x. The Starter plan is capped at 10 concurrent GPUs.

When not to use Together AI: you need H200 or B300 dedicated capacity at a published price (both say "Contact us"), or you are counting on batch discounts for a model that is not on its discounted list.

When not to use Groq: you need a broad self-serve model catalog. Its production list is short, and its Llama models are listed as Enterprise, "Contact Sales".

Frequently asked questions

Is Modal cheaper than Baseten for GPU inference?

At list GPU rates, yes: $0.001097 per second on Modal (about $3.95 an hour) against $0.10833 per minute on Baseten (about $6.50 an hour). But Modal bills CPU and memory on top, adds 1.15–1.75x if you pin a region, runs GPU functions as preemptible, and leaves the serving layer for you to build.

When should I move from a per-token API to a dedicated GPU?

Move when one model's sustained traffic would keep the GPU busy enough to beat per-token pricing. Other reasons are needing a model no API serves, single-tenant isolation, or relief from shared rate limits. On gpt-oss-120B at $0.15/$0.60 per million tokens and a 3:1 input-to-output mix, a Baseten H100 at about $6.50 an hour breaks even only around 18 billion tokens a month.

What is the difference between Baseten and OpenRouter?

Baseten hosts models: it runs your deployments on its GPUs and sells per-token APIs for some open models. OpenRouter hosts nothing. It routes your requests to other providers' APIs and charges a fee when you buy credits.

Is Together AI cheaper than Baseten for open-source models?

It depends on the model. On gpt-oss-120B, Baseten's Model API lists $0.10/$0.50 per million tokens against Together's $0.15/$0.60. On GLM-5.3, DeepSeek V4.1 Flash and Kimi K3.1 Pro, the two list identical input and output prices.

Does Modal have a free tier?

Modal's Starter plan has no monthly base fee and includes $30 a month in free compute credits. You pay standard per-second rates beyond that. The Team plan is $250 a month and includes $100 in credits.

Should I use Amazon Bedrock or Baseten for open models?

Use Bedrock if you want open models as a per-token API inside your AWS account (gpt-oss-120b is $0.15/$0.60 per million tokens in US East). Use Baseten if you need a dedicated deployment of a model you control, such as a fine-tune or custom weights.

Next step

Before you move a production model to your own GPU, set up an agent evaluation and regression harness so you can prove the self-hosted version matches the API's quality. Then read the Baseten, Modal and Together AI profiles, or browse the full AI infrastructure category.

Related Workflows

Predictive Maintenance Alerting

Monitor equipment sensor data and production metrics to predict failures before they happen — reducing downtime and emergency repair costs.

View workflow

Insurance Eligibility Verification Agent for Dental & Med Spa

A browser-using agent checks tomorrow's patients against each payer portal overnight and writes benefits back to the chart, so the front desk starts the day with exceptions instead of a hold queue.

View workflow

AI Spend Guardrails & Cost Allocation (FinOps for Agents)

Put a gateway in front of every model call so each token is tagged to a team and a use case, routed to the cheapest adequate model, capped by budget, and alerted on before the vendor cuts you off.

View workflow

Keep Reading

November 10, 2026

OpenRouter vs Amazon Bedrock vs Google Vertex AI (2026): Router, Cloud, or Gateway?

OpenRouter is a hosted router, Bedrock and Vertex AI are cloud model platforms, and LiteLLM is a gateway you run yourself. Here is what each costs on the same Claude workload, and which one your cloud contract probably picks for you.

October 1, 2026

Agentforce vs. Copilot Studio vs. Gemini Enterprise vs. OpenAI Agents API vs. Claude Managed Agents: Five Billing Units for One Enterprise Agent

Salesforce bills actions or conversations, Microsoft bills credits and switches you off at 125%, Google bills seats, OpenAI bills tokens plus a sandbox, Anthropic bills tokens plus session-hours. One 900-conversation-a-day agent, priced on each where the published numbers allow.

March 19, 2026

Vapi vs Bland vs Retell (2026): AI Voice Agent Pricing, Limits and Real Monthly Cost

Vapi, Retell and Bland price AI phone agents in three different ways. We read each vendor's own pricing page and worked out what 2,000 inbound minutes a month actually costs on each.

Found this helpful?

Get weekly AI workflow ideas in your inbox.