Cheapest way to serve a 7B model at low, spiky traffic?

Kaito Mercer

Founder & CEO · Zero-Trust Architecture

Idle GPU cost is eating us. Considering scale-to-zero with a cold-start penalty.

Amara Okonkwo

ML Systems Engineer · GPU Kernels

Batch aggressively and accept the latency floor. We cut spend by two thirds moving to scale-to-zero.

SIGN IN_ to join and answer →