Upgrade Header Smuggling
- Do you work in a cybersecurity company? Do you want to see your company advertised in HackTricks? or do you want to have access to the latest version of the PEASS or download HackTricks in PDF? Check the SUBSCRIPTION PLANS!
A normal HTTP connection typically lasts only for the duration of a single request. However, H2C or “http2 over cleartext” is where a normal transient http connection is upgraded to a persistent connection that uses the http2 binary protocol to communicate continuously instead of for one request using the plaintext http protocol.
The second part of the smuggling occurs when a reverse proxy is used. Normally, when http requests are made to a reverse proxy, the proxy will handle the request, process a series of routing rules, then forward the request onto the backend and then return the response. When a http request includes a
Connection: Upgrade header, such as for a websocket connection, the reverse proxy will maintain the persistent connection between the client and server, allowing for the continuous communication needed for these procotols. For a H2C Connection, the RFC requires 3 headers to be present:Upgrade: h2c
HTTP2-Settings: AAMAAABkAARAAAAAAAIAAAAA
Connection: Upgrade, HTTP2-Settings
So where is the bug? When upgrading a connection, the reverse proxy will often stop handling individual requests, assuming that once the connection has been established, its routing job is done. Using H2C Smuggling, we can bypass rules a reverse proxy uses when processing requests such as path based routing, authentication, or the WAF processing provided we can establish a H2C connection first.
Note from the explanation of the vulnerability that the proxy server needs to forward the Upgrade header, and sometimes the Connection header also needs to be successfully forwarded.
By default, the following services do forward Upgrade and Connection headers during proxy-pass, thereby enabling h2c smuggling out-of-the-box.:
- HAProxy
- Traefik
- Nuster
By default, these services do not forward both Upgrade and Connection headers during proxy-pass, but can be configured in an insecure manner (by passing unfiltered Upgrade and Connection headers):
- AWS ALB/CLB
- NGINX
- Apache
- Squid
- Varnish
- Kong
- Envoy
- Apache Traffic Server
The original blog post points out that not all servers will forward the required headers for a compliant H2C connection upgrade. This means load balancers like AWS ALB/CLB, NGINX, and Apache Traffic Server amongst others will prevent a H2C connection by default. However, at the end of the blog post, he does mention that “not all backends were compliant, and we could test with the non-compliant
Connection: Upgrade variant, where the HTTP2-Settings value is omitted from the Connection header.”Note that even if the
proxy_pass URL (the endpoint the proxy forwards the connection) was pointing to a specific path such as http://backend:9999/socket.io the connection will be stablished with http://backend:9999 so you can contact any other path inside that internal endpoint abusing this technique. So it doesn't matter if a path is specified in the URL of proxy_pass.Using the tools https://github.com/BishopFox/h2csmuggler and https://github.com/assetnote/h2csmuggler you can try to bypass the protections imposed by the proxy establishing a H2C connection and access proxy protected resources.
Similar to previous technique, this one instead of creating a HTTP2 tunnel to an endpoint accessible via a proxy, it will create a Websocket tunnel for the same purpose, bypass potential proxies limitations and talk directly to the endpoint:
We have backend that exposes public WebSocket API and also has internal REST API not available from outside. Malicious client wants to access internal REST API.
On the first step client sends Upgrade request to reverse proxy but with wrong protocol version inside header
Sec-WebSocket-Version. Proxy doesn't validate Sec-WebSocket-Version header and thinks that Upgrade request is correct. Further it translates request to the backend.On the second step backend sends response with status code
426 because protocol version is incorrect inside header Sec-WebSocket-Version. However, reverse proxy doesn't check enough response from backend (including status code) and thinks that backend is ready for WebSocket communication. Further it translates request to the client.Finally, reverse proxy thinks that WebSocket connection is established between client and backend. In reality there is no WebSocket connection - backend refused Upgrade request. At the same time, proxy keeps TCP or TLS connection between client and backend in open state. Client can easily access private REST API by sending HTTP request over the connection.
It was found that following reverse proxies are affected:
- Varnish - team refused to fix described issue.
- Envoy proxy 1.8.0 (or older) - in newer versions upgrade mechanism has been changed.
- Others - TBA.
The majority of reverse proxies (e.g. NGINX) check status code from backend during handshake part. This makes attack harder but not impossible.
Let's observe second scenario. We have backend that exposes public WebSocket API and public REST API for health checking and also has internal REST API not available from outside. Malicious client wants to access internal REST API. NGINX is used as reverse proxy. WebSocket API is available on path
/api/socket.io/ and healthcheck API on path /api/health.Healthcheck API is invoked by sending POST request, parameter with name
u controls URL. Backend reaches external resource and returns status code back to the client.On the first step client sends POST request to invoke healthcheck API but with additional HTTP header
Upgrade: websocket. NGINX thinks that it's a normal Upgrade request, it looks only for Upgrade header skipping other parts of the request. Further proxy translates request to the backend.On the second step backend invokes healtcheck API. It reaches external resource controlled by malicious users that returns HTTP response with status code
101. Backend translates that response to the reverse proxy. Since NGINX validates only status code it will think that backend is ready for WebSocket communication. Further it translates request to the client.
Note how this scenario is much more complex to exploit as you need to be able to contact some endpoint that returns status code 101.
Finally, NGINX thinks that WebSocket connection is established between client and backend. In reality there is no WebSocket connection - healthcheck REST API was invoked on backend. At the same time, reverse proxy keeps TCP or TLS connection between client and backend in open state. Client can easily access private REST API by sending HTTP request over the connection.
The majority of reverse proxies should be affected by that scenario. However, exploitation requires existence of external SSRF vulnerability (usually considered low-severity issue).
- Do you work in a cybersecurity company? Do you want to see your company advertised in HackTricks? or do you want to have access to the latest version of the PEASS or download HackTricks in PDF? Check the SUBSCRIPTION PLANS!