We Need To Talk About Vercel

Recently there's been some discussion around Vercel and its hefty markup over AWS Lambda or its surprise runaway costs.¹ That said, there's an argument to be made that the development experience value add is worth the pretty penny. And indeed for some, that may be the case.

However, as I've spent more time working with the Vercel Content Delivery Network (CDN) product, I've encountered technical deficiencies which have been met with a customer support response that not only leaves me skeptical of the proffered value add, but has left me to question the product altogether.

Some Background

I've been working on remotejobs.org, which is largely static but updates periodically throughout the day. I'll spare readers the technical details here but what's important is to understand that this is implemented via Astro's Server-Side Rendering and is intended to be cached by the CDN.

To do so, we make use of the s-maxage and stale-while-revalidate directives of the Cache-Control header. The goal here is to have the surrogate, i.e. the CDN network appliance, cache the content for a set amount of time and then to lean on the stale-while-revalidate directive to serve cached content while fresh data is fetched in the background.

As of writing the Vercel docs give this description of stale-while-revalidate:

This allows you to serve content from the Edge cache while simultaneously updating the cache in the background with the response from your Serverless Function.

Offering some relevant use-cases and overlap with what we want:

Some situations where stale-while-revalidate is of great value:

• Your content changes frequently but it takes a significant amount of time to regenerate. For example, an expensive database query or upstream API request.
• Your content changes infrequently but you want to have the flexibility to update it (to fix a typo, for example) and don't wait for the cache to expire.

Fortunately Vercel makes configuring such a header easy via the vercel.json configuration:

{
    "headers": [
        {
            "source": "/(.*)",
            "headers": [
                {
                    "key": "Cache-Control",
                    "value": "public, max-age=0, s-maxage=3600, stale-while-revalidate=3900"
                }
            ]
        }
    ]
}

Mission accomplished.

Not So Easy After All

Only it doesn't work the way it's advertised.

Bizarrely, Vercel's network appliance mangles the Cache-Control header and indeed does not respect the configured values as it should.² For some resources, things work as expected, but for e.g. our index page the s-maxage and stale-while-revalidate directives are both missing.

In fact, only resources like JavaScript and CSS see the correct header value with everything else being configured such that Cache-Control has a value of public, max-age=0. Not what we wanted or configured.

Looking for Support

After messing around with this for some time, ensuring we were manually setting the headers on our origin, I had no luck and reached out to Vercel's support.

Over the course of a couple of weeks I was told by their support team:

1. This is not a real issue, try these various configuration tweaks (none worked and they would later admit they could easily reproduce the issue independently).
2. This is a real issue, but it's Astro's fault (it wasn't, as the Astro team would trivially demonstrate) and I should do free work for Vercel to poll their community regarding how widespread the issue is.
3. Vercel's support staff is only trained to handle frontend issues, therefore I can't expect helpful responses (Vercel sells a CDN product, so this is surprising).³
4. The internal team is working on an RFC to make documentation changes (odd, but okay; the issue still remains).
5. I would receive an update within one week (now a month later, I have had no update whatsoever and the issue persists).

I was also told that because I haven't graduated to a high-enough service tier, my issue wasn't important. I find this a strange response, because other customers, who might be paying lots of money to use Vercel's product, would benefit from the CDN operating as it's advertised.

On March 9th, 2023, Vercel's support team promised to follow up "next week". They have not done so and the issue remains an issue.

"Please keep an eye out for a message from our team next week."

Where We Go from Here

CDNs are hard. There's no doubt a lot going on in the background here that we can't see. For instance, later on in the support thread, Vercel's support said this:

"After catching up with our CDN team, it's worth sharing that we'll be reviewing docs on s-maxage/stale-while-revalidate/stale-while-error and how they are consumed by our proxy. Slight overshare but there's an active RFC internally to review this behaviour in its entirety."

While this is a somewhat odd response, it demonstrates that Vercel is aware of these problems and that they can be difficult to tackle.⁴

However, for me the bigger issue is how Vercel's frontline support team responds and behaves. I was repeatedly told my issue wasn't real, a fairly blatant attempt to gaslight me, and then when it had been confirmed multiple times that I should just go away already.

Now I'm clearly not worth enough for Vercel to care about. Fair enough. They're right that I won't independently bring them large bags of money in the form of recurring revenue. But being so blunt about that seems shortsighted at best: Should I ever be in the position to bring more business to Vercel, I now won't.

Contrast this approach with Cloudflare or AWS: Even as a free or essentially free customer, both platforms invest in their customers because it's part of a larger customer acquisition strategy.

Good support is not just a resource to fallback to.

Take AWS as an example, which has for years invested heavily in customer success. AWS offers a wide array of resources, including dedicated support teams, structured programs, and ongoing training. Without such things in place, AWS wouldn't just be more difficult to use, but would also likely have a far less commanding hold of the cloud market. While Vercel may not be at AWS scale, even in the beginning Amazon has had a reputation for being "customer obsessed".

Given where Vercel is today, I'll be moving on. Most of remotejobs.org is already hosted on a VPS and so in my case I'll move the web pieces there with a CDN like Cloudflare in front of it.

Addendum

I reached out for comment before publishing this piece, but Vercel did not provide one. However, there's since been active discussion on Hacker News with Vercel's CEO, Guillermo Raunch, explaining that this behavior is in place to protect users:

"Vercel strips them because (1) at the time this RFC didn't exist and (2) most of the time you we found customers don't want to cache on the browser side or proxying CDNs, which makes purging and reasoning about cache staleness very difficult."

[...]

'Customers want to deploy and see their changes instantly. They want to their customers "go to our blog to see the news" and not have to second guess or fear the latest content will be there.'

Footnotes

1. Note that the latter was due how the deployment process was structured, however billing limits or quotas would have prevented such a thing from happening in the first place; such misconfigurations can and do happen and shouldn't bankrupt a small business. ↩

2. I spent some time trying various things, ordering the directives differently, duplicating them, etc. Some combinations produce different results, but none I could find consistently set the correct header value. ↩

3. To clarify, this was directed at a specific support team member, who I won't name; presumably other support team members have different backgrounds and may be better equipped to field these issues. However, that's in fact the issue: if you sell a CDN product, it's important your staff is trained in supporting it. ↩

4. It could make some sense to update the docs in the meantime at least. ↩