Yesterday, on October 1st, the "prompt generation" feature in our app suddenly stopped working. This feature uses the Gemini API to generate a short text, and out of nowhere, was now returning a "401 Unauthorized" error. This was extremely perplexing, since we had not deployed any changes to the app for a couple of days. To make it even more confusing, other features also utilizing the Gemini API were not impacted and they all use the same API key.

After over 12 hours of feature downtime, and a few hours of debugging exploring different paths to solving the issue, one of the developers on our team discovered that Netlify has a new feature called "AI gateway". When was it released? On October 1st! Hah! Very suspicious...

This "AI gateway" feature automatically injects API keys as environment variables for various LLM providers and routes LLM traffic through their own gateway, and then they charge you via their own credits system. That by itself, seems like a questionable product in my opinion, but even worse, it's enabled by default for all Netlify users.

They say they don't override environment variables that are already set, so as not to mess with already existing apps. What they didn’t fully consider are apps where variables are set during build time instead of runtime (I can say many things about why we do it like this, but that is for another blog post).

Because of this, it looks like we have not set the GEMINI_API_KEY environment variable and they automatically inject their own API key instead. With their own API key injected, it overrides our build variable, which again results in the 401 error.

They have a setting to disable the "AI gateway", but for whatever reason, disabling it did not work for us. The solution in the end was to include our own GEMINI_API_KEY as an environment variable, so that Netlify would not inject their own.

Why did it not break the rest of the features depending on Gemini? The broken feature utilized Google’s genai SDK, while the rest did not. Most likely the SDK has some kind of interaction with the environment variables that caused things to break, or the Netlify tampers with the functionality of the SDK.

Thanks, Netlify...