The "cold start" problem is making it difficult for early-stage AI startups to enter the market. What is it, and why is it an issue?

This problem refers to the time it takes for a server to boot up and be ready to serve tasks. During this initialization stage, the physical server is reserved but not yet usable, costing money without generating value. This can be a problem for AI applications since models are often several gigabytes in size, taking many minutes to load into memory.

At VISOID, it takes us around 10 seconds to generate an image with a rented GPU hosted in the cloud. However, before we can do that, the server has to boot up, and the model loaded, which takes an additional ~120 seconds. At a cost of $0.00140/second to rent this GPU, this means that it costs us $0.0140 to generate the image and another $0.168 just to boot up the server. This can become expensive quickly!

The cost is one thing, but it also results in a terrible user experience, forcing the user to wait several minutes for the server to boot up before their image is generated.

A solution is to keep the server running at all times. For larger companies with more traffic, this is fine because of the higher utilization rate. They can justify the operational cost and minimize the impact of the cold start problem. For a small startup with limited resources and few users, this is not an option as it would be too expensive. They have to shut down the server in periods of low activity to save costs and with that, go through the whole cold start process again the next time the server is needed.

You end up with a divide where existing players in the market have access to relatively "cheaper" GPUs and a better user experience, while newcomers have to pay a high price in the early phases while they grow their user base.

There are companies working on solving this technologically, but there is still some way to go before this issue is truly solved. This is only further exacerbated by the GPU shortage, which is driving up prices and making it very expensive for early-stage AI startups to enter the market.