GPU rents are going up while token prices keep falling. Again.

In my opinion, that shows demand is still running ahead of supply.

Silicon Data's H100 index was $2.86 an hour on 21 Oct, up 10% in a week. Its token index is at $0.96 per million tokens, down from $2.07 in May.

Cheaper tokens don't mean less compute. The token index is falling mainly because more everyday work is being routed to cheaper models. More work at a lower price per token still needs GPUs underneath it, and the GPU-hour is the part getting more expensive.

The way I see it, every big jump in how useful models are shows up as inference demand first, and rental prices follow.

I think it's notable that a GPU-hour keeps getting more expensive while a token keeps getting cheaper. The bears tend to quote only the second number.