Hey everyone,
Today, I will talk about a cache poisoning vulnerability that led to DOS on a company customer service chat.
First of all, lets discuss on what is a web Cache and how it works (credit to https://portswigger.net/web-security/web-cache-poisoning).
How does a web cache work?
To understand how web cache poisoning vulnerabilities arise, it is important to have a basic understanding of how web caches work.
If a server had to send a new response to every single HTTP request separately, this would likely overload the server, resulting in latency issues and a poor user experience, especially during busy periods. Caching is primarily a means of reducing such issues.
The cache sits between the server and the user, where it saves (caches) the responses to particular requests, usually for a fixed amount of time. If another user then sends an equivalent request, the cache simply serves a copy of the cached response directly to the user, without any interaction from the back-end. This greatly eases the load on the server by reducing the number of duplicate requests it has to handle.
Cache keys
When the cache receives an HTTP request, it first has to determine whether there is a cached response that it can serve directly, or whether it has to forward the request for handling by the back-end server. Caches identify equivalent requests by comparing a predefined subset of the request's components, known collectively as the "cache key". Typically, this would contain the request line and Host header. Components of the request that are not included in the cache key are said to be "unkeyed".
If the cache key of an incoming request matches the key of a previous request, then the cache considers them to be equivalent. As a result, it will serve a copy of the cached response that was generated for the original request. This applies to all subsequent requests with the matching cache key, until the cached response expires.
Crucially, the other components of the request are ignored altogether by the cache.
A cache poison attack will usually happen when we are successfully make the cache server, cache an unintended response. in our example, instead of caching the chat configuration, it will cache a 404 page, and that will lead to a DOS.
- for a more in-depth information visit https://portswigger.net/web-security/web-cache-poisoning
The Vulnerability
After understanding the basics of what is cache, cache poisoning and unkeyed cache keys, lets dive in the vulnerability.
After surfing the application, and understanding it, the behavior of how it retrieves customer service chat configuration led me to think on a theoretical way to exploit it through cache poisoning, specifically because of how the customer service chat was called and handled.
A company domain is redacted.com, the user browser makes a request at the background to chat.redacted.com/v1/chat/config?region=us,
Only after receiving response from that endpoint with chat configuration information in the response, the customer service chat will be loaded to the browser (at redacted.com).
So, theoretically, if we can manage to deny access somehow to that endpoint, we could DOS the chat service.
After Understanding this basic interaction, we now know that if somehow we can manage to disrupt the response from the chat.redacted.com/v1/start?region=us endpoint, we can disrupt the customer service chat.
First things first, we will use a cache buster lilbullet=xyz, so we wont cause real harm to customers using the service (Since the query parameter is a part of the cache key), We will try our tests on chat.redacted.com/v1/start?region=us&lilbullet=xyz.
After playing for a while with the endpoint and testing it manually, I managed to find an unkeyed parameter ‘user-agent-via’ which is causing the cache server cache a 404 response.
Since its an unkeyed cache key the cache server will serve the same regular request as if the ‘user-agent-via’ header was not sent due to being an unkeyed key. Serving the “poisoned” response (404 Not Found) denying the chat configuration from being sent and load the customer service chat at the client side.
Now that the response got cached, a normal user that tries to load the configuration for the customer service chat will be served with the 404 Not Found response, and the chat would not be able to load without its configuration.
Upon analyzing further, it appears that the response was only being cached for 5 second without any suggestive cache headers that the response was cached. Additionally due to how web cache servers work, it will only cache on the geographically nearest cache server to the attacker.
💡Even if no suggestive cache hit headers appear, the cache server might still cache the response.
After understanding these principles, it was much easier to explain the triager why he couldn’t see the issue at first place, and had to revisit the vulnerability after trying to reproduce the POC.
First of all, for the vulnerability to happen, we had to resend the request that cause the 404 response cache every 5 second, an easy way for that is just running a python script with the proper request in a loop.
Second of all, we had to confirm that it was indeed poisoned through geographically nearby countries by using a VPN or a dedicated VPS. Since the nearest CDN servers send to the host the cache response.
A good example and explanation could be found here: https://cpdos.org/paper/Your_Cache_Has_Fallen__Cache_Poisoned_Denial_of_Service_Attack__Preprint_.pdf
And third, theoretically if we would want to cache the 404 response to all cache servers, we could rent cheap 5$ VPSs across the globe and run the script that causes the customer service chat DOS on each of them eventually causing all the cache servers serve the 404 response. A customer service chat DOS chaos.
After confirming the vulnerability, I submitted a report, this was triaged pretty fast although needed more details and was fixed pretty quickly after that.