A fresh look on reverse proxy related attacks

In recent years, several researches have been published about attacks deliberately or directly related to reverse proxies. While implementing various reverse-proxy checks on the scanner, I started analyzing implementations of reverse proxies.

Initially, I wanted to analyze how both reverse proxies and web servers parse requests, find out inconsistencies in the process between them and use this knowledge for some kind of bypasses. Unfortunately, I was stuck with analyzing web servers and application servers due to too many possible variations. For example, Apache web server behaves differently depending on how you connect it with PHP. Also, an implementation of a web application, framework or middleware used by a web application can influence the requests parsing process as well. In the end I realized that some attacks are still little-known or completely unknown.

The goal of this research is to portray the bigger picture of potential attacks on a reverse proxy or the backend servers behind it. In the main part of the article, I will show some examples of vulnerable configurations and exploitation of attacks on various reverse proxies, but the second goal of the research is to share the raw data about various implementations of reverse proxies so you can find your ways/tricks (depending on a backend server in each specific situation).

Terms

Actually, the research is not only about reverse proxies, but also about load balancers, cache proxies, WAFs and other intermediate servers between a user and web application which parses and forwards requests. However I haven’t found a good term which correctly describes such a server and is well-known in the community, so I will use “reverse proxy” even when I talk about load balancers or cache proxy. I will call a web application behind a reverse proxy a back-end server. Be aware that a backend server is so-called an origin server (this will make sense when we start talking about caching).

What is reverse proxy?

How proxies work

The basic idea of a reverse proxy is quite simple. It’s an intermediate server between a user and a back-end server. The purpose of it can be quite different: it can route requests depending on the URL to various backends or it can just be there “to protect” against some attacks or simply to analyze traffic. The implementations can be different too, but the main sequence of steps is quite the same.

A reverse proxy must receive a request, it must process it, perform some action on it and forward to a backend.

Processing of a request consists of several main steps:

1. Parsing
   When a reverse proxy receives a request, it must parse it: to get a verb, a path, a HTTP version, host header and other headers and body.

   GET /path HTTP/1.1
   Host: example.com
   Header: something

   Everything may look quite simple, but if you dive into details, you will see implementations are different.

   Some examples:
   – If a reverse supports Absolute-URI, how will it parse it? Does Absolute-URI have a higher priority than Host header?:

   GET http://other_host_header/path HTTP/1.1
   Host: example.com

   – URL consists of `scheme:[//authority]path[?query][#fragment], and browsers don’t send #fragment. But how must a reverse proxy handle #fragment?
   Nginx throws fragment off, Apache returns a 400 error (due to # in the path), some others handle it as a usual symbol.

   – How does it handle symbols which must be URL-encoded?

   GET /index.php[0x01].jsp HTTP/1.1

2. URL decoding
   Due to standards, symbols with a special meaning in the URL must be URL-encoded (%-encoding), like the double quote (") or “greater than” sign (>). But practically, any symbol can be URL-encoded and sent in a path part. Many web servers perform URL-decoding while processing a request, so next requests will be treated in the same way by them.

   GET /index.php HTTP/1.1

   GET %2f%69%6e%64%65%78%2e%70%68%70 HTTP/1.1

3. Path normalization
   Many web servers support path normalization. Main cases are well-known:

   /long/../path/here -> /path/here
   /long/./path/here -> /long/path/here

   But what about /..? For Apache, it’s an equivalent of  /../, but for Nginx it means nothing.

   /long/path/here/.. -> /long/path/ - Apache
   /long/path/here/.. -> /long/path/here/.. - Nginx

   The same with // (“empty” directory). Nginx converts it to just one slash /, but, if it’s not the first slash, Apache treats it as a directory.

   //long//path//here -> /long/path/here - Nginx
   //long/path/here -> /long/path/here - Apache
   /long//path/here -> /long//path/here - Apache

   Here are some additional (weird) features which are supported by some web servers. For example: support of path parameters – /..;/ is valid for Tomcat and Jetty or traversal with backslash (\..\).

b) Applying rules and performing actions on a request

Once a request is processed, the reverse proxy can perform some actions on the request due to its configuration. Important to note that in many cases, rules of a reverse proxy are path (location) based. If the path is pathA, do one thing, if pathB – do another.

Depending on the implementation or on the configuration, a reverse proxy applies rules based on a processed (parsed, URL-decoded, normalized) path or on an unprocessed path (rare case). It’s also important for us to note if it is case-sensitive or not. For example, will the next paths be treated equally by a reverse proxy?:

/path1/ == /Path1/ == /p%61th1/ == /lala/../path1/

C) Forwarding to a back-end

The reverse proxy has processed a request, found appropriate rules for it and performed necessary actions. Now it must send (forward) it to a backend. Will it send the processed request or initial request? Obviously, if it has modified the request, then it sends the modified version, but in this case, it must perform all the necessary steps, for example, to perform URL-encoding of special symbols. But what if the reverse proxy just forwards all requests to only one backend, maybe forwarding the initial request is a good idea?

As you can see all these steps are quite obvious and there are not so many variations. Still, there are differences in implementations, which we, as attackers, can use for our goals.

Therefore, the idea of all attacks described below is that a reverse proxy processes a request, finds and applies rules and forwards it to a backend. If we find an inconsistency between the way a reverse proxy processes a request and the way a backend server processes it, we are then able to create such a request(path) which is interpreted like one path by the reverse proxy and a completely different path by the backend. So, we will be able to bypass or to forcefully apply some rules of the reverse proxy.

Here are some examples

Nginx

Nginx is a well-known web server, but is also very popular as a reverse proxy. Nginx supports Absolute-URI with an arbitrary scheme and higher priority than a Host header. Nginx parses, URL-decodes and normalizes a request path. Then it applies location-based rules depending on the processed path.

But it looks like Nginx has two main behaviors and each of them has its own interesting features:

• With trailing slash

  location / {
  proxy_pass http://backend_server/;
  }

  In this configuration, Nginx forwards all requests to the `backend_server`. It sends the processed request to the backend, meaning that Nginx must URL-encode the necessary symbols. The interesting thing for an attacker is that Nginx doesn’t encode all the symbols which browsers usually do. For example, it doesn’t URL-encode ' " < >.
  Even if there is a web application (back-end server) which takes a parameter from a path and which is vulnerable to XSS, an attacker cannot exploit it, because modern browsers (except dirty tricks with IE) URL-encode these symbols. But if there is Nginx as a reverse proxy, an attacker can force a user to send a URL-encoded XSS payload in the path. The Nginx decodes it and sends the decoded version to the backend server, which makes exploitation of XSS possible.

  Browser -> http://victim.com/path/%3C%22xss_here%22%3E/ -> Nginx -> http://backend_server/path/<"xss_here">/ -> WebApp

• Without trailing slash

  location / {
  proxy_pass http://backend_server;
  }

  The only difference between this config and the previous one is the lack of the trailing slash. Although seemingly insignificant, it forces Nginx to forward an unprocessed request to the backend. So if you send /any_path/../to_%61pp#/path2, after processing of the request, Nginx will try to find a rule for `/to_app`, but it will send /any_path/../to_%61pp#/path2 to the backend. Such behavior is useful to find inconsistencies.

Haproxy

Haproxy is a load balancer (with HTTP support). It doesn’t make much sense to compare it to Nginx, but it will give you an idea of a different approach.

Haproxy makes minimal processing of a request. So there is no “real” parsing, URL-decoding, normalization. It doesn’t support Absolute-URI either.

Therefore, it takes everything (with few exceptions) between a verb and HTTP version (GET !i<@>?lala=#anything HTTP/1.1) and, after applying rules, forwards it to a backend server. However it supports path-based rules and allows it to modify requests and responses.

How proxies are used

While I was working on this research, analyzing various configurations of reverse proxies, I came to the conclusion that we can both bypass and apply rules of a reverse proxy. Therefore, to understand the real potential of reverse proxy related attacks, we must have a look at their abilities.

First of all, a reverse proxy has access to both a request and a response (including those which it sends/receives from a backend server). Secondly, we need a good understanding of all the features which a reverse proxy supports and how people configure them.

How can a reverse proxy handle a request?:

1. Routing to endpoint. It means that a reverse proxy receives a request on one path (/app1/), but forwards the request to a completely different one (/any/path/app2/) on a backend. Or it forwards the request to a specific backend depending on a Host header value.
2. Rewriting path/query. This is similar to the previous one, but usually involves different internal mechanisms (regexp)
3. Denying access. When a reverse proxy blocks a request to a certain path.
4. Headers modification. In some cases, a reverse proxy may add or change headers of the request. It could be a cool feature for an attacker, but it’s hard to exploit with a black box approach.

How can a reverse proxy handle a response?:

1. Cache. Many reverse proxies support caching of response.
2. Headers modification. Sometimes a reverse proxy adds or modifies response headers (even security related), because it cannot be done on a backend server
3. Body modification. Reverse proxies will sometimes modify the body too. Edge Side Includes (ESI) is an example of when this can happen.

All this is important for to see more potential attacks, but also understand that in many cases we don’t need to bypass, but apply rules. Which leads to a new type of attacks on reverse proxies – proxy rules misusing.

Server-Side attacks

Bypassing restriction

The most well known case about reverse proxy related attacks.
When someone restricts access (3. Denying access), an attacker needs to bypass it.

Here is an example.
Let’s imagine that there are Nginx as a reverse-proxy and Weblogic as a backend server. Nginx blocks access to an administrative interface of Weblogic (everything that starts with /console/).

Configuration:

location /console/ {
deny all;
return 403;
}

location / {
proxy_pass http://weblogic;
}

As you can see, `proxy_pass` here is without trailing slash, which means that a request is forwarded unprocessed. Another important thing to bypass the restriction is that Weblogic treats `#` as a usual symbol. Therefore, an attacker can access the administrative interface of Weblogic by sending such a request:

GET /#/../console/ HTTP/1.1

When Nginx starts processing the request, it throws off everything after #, so it skips the /console/ rule. It then forwards the same unprocessed path (/#/../console/) to the Weblogic, the Weblogic processes the path and after path normalization, we are left with/console/.

Request Misrouting

It’s about “1. Routing to endpoint” and, in some cases, “2. Rewriting path/query”.
When a reverse proxy forwards requests only to one endpoint, it can make an illusion that an attacker cannot reach other endpoints on a backend or that it cannot reach a completely different backend.

Example 1.
Let’s have a look at similar combinations: Nginx+Weblogic. In this case, Nginx proxies requests only to a certain endpoint of Weblogic (http://weblogic/to_app). So only requests, which come to a path /to_app on Nginx, are forwarded to the same path on Weblogic. In this situation, it may look like Weblogic’s administrative interface (console) or other paths are not accessible for an attacker.

location /to_app {
proxy_pass http://weblogic;
}

In order to misroute requests to other paths, we need to know two things again. Firstly, the same as in the example above – `proxy_pass` is without a trailing slash.
Secondly, Weblogic supports “path parameters” (https://tools.ietf.org/html/rfc3986#section-3.3). For example, /path/to/app/here;param1=val1, and param1 will be accessible in a web app through API.

I think many are aware about this feature (especially after the Orange Tsai’s presentation from BlackHat in the context of Tomcat. Tomcat allows to perform really “weird” traversals like /..;/..;/. But Weblogic treats path parameters differently, as it treats everything after the first ; as a path parameter. Does it mean that this feature is useless for an attacker?
Nope. Let’s have a look at this “magic” which allows accessing any path on Weblogic in this configuration.

GET /any_path_on_weblogic;/../to_app HTTP/1.1

When Nginx receives such a request, it normalizes the path. From /any_path_on_weblogic;/../to_app it gets /to_app which successfully applied to the rule. But Nginx forwards /any_path_on_weblogic;/../to_app and Weblogic, during parsing, treats everything after ; as a path parameter, so Weblogic sees /any_path_on_weblogic. If it’s necessary, an attacker can go “deeper” by increasing the amount of /../` after `;.

Example 2.
This one is about a “bug” of Nginx. But this “bug” is just a consequence of how Nginx works (so will not be fixed)
A rule location /to_app means that all paths which start with /to_app (prefix) fall under the rule. So, /to_app, /to_app/, /to_app_anything (including special symbols) fall under it. Also, everything after this prefix(/to_app) will be taken and then concatenated with value in proxy_pass.
Look at the next config. Nginx, after processing `/to_app_anything`, will forward the request to http://server/any_path/_anything

location /to_app {
proxy_pass http://server/any_path/;
}

If we put both features together, we will see that we can go to any path one level higher on almost any backend. We just need to send:

GET /to_app../other_path HTTP/1.1

Nginx applies /to_app rule, gets everything(../other_path) after the prefix, concatenates it with a value from `proxy_pass`, so it forwards http://server/any_path/../other_path to a backend. If the backend normalizes the path, we can reach a completely different endpoint.
Actually, this trick is similar to a well-known alias trick. However, the idea here is to show an example of possible misusing of reverse proxy’s features.

Example 3.
As I mentioned before, it’s a common case when a reverse proxy routes requests to different backends depending on the Host header in a request.
Let’s have a look at Haproxy configuration which says that all requests with example1.com in the Host header must be proxied to a backend example1_backend – 192.168.78.1:9999.

frontend http-in
acl host_example1 hdr(host) -i example1.com
use_backend example1_backend if host_example1
backend example1_backend
server server1 192.168.78.1:9999 maxconn 32

Does such a configuration mean that an attacker cannot access other virtual hosts of a backend server? It may look like that, but an attacker can easily do it. Because, as mentioned above, Haproxy doesn’t support Absolute URI, but most web-servers do. When Haproxy receives Absolute URI, it forwards this unprocessed Absolute URI to a backend. Therefore, just by sending next request, we can easily access other virtual hosts of the backend server.

GET http://unsafe-value/path/ HTTP/1.1
Host: example1.com

Is it possible to force a reverse proxy to connect to an arbitrary backend server? I’d say that in most cases (Nginx, Haproxy, Varnish), this cannot be done, but Apache (in some configurations/versions) is vulnerable to it. As Apache “parses” a host value from ProxyPass, we can send something like GET @evil.com HTTP/1.1, so Apache sees a value `http://backend_server@evil.com` and sends the request to `evil.com` (SSRF). Here you can see an example of such vulnerability.

Client-Side attacks

If we have a look at reverse proxy features again, we can see that all response-related have a potential for client-side attacks. It doesn’t make them useless. I’d say otherwise. But client-side attacks have additional limitations to possible inconsistencies between the reverse proxy and the web server, as the browser process a request before sending it.

Browser processing

In a client-side attack, an attacker needs to force a victim’s browser to send a special request, which will influence a response, to a server. But the browser follows the specifications and processes the path before sending it: ^The browser parses the URL (e.g. throws off a fragment part), URL-encodes all the necessary symbols (with some exceptions) and normalizes a path. Therefore, to perform such attacks, we can only use a “valid” request which must fit into the inconsistency between three components (browser, reverse proxy, backend server).

Of course, there are differences in browser implementations, plus some features which still allows us to find such inconsistencies:

• For example, Chrome and IE don’t decode `%2f`, so a path like that /path/anything/..%2f../ will not be path normalized.
• Older versions of Firefox didn’t URL-decode special symbols before normalization, but now it behaves in a similar way to Chrome.
• There is information that Safari doesn’t URL-decode a path, so we can force it to sent such a path /path/%2e%2e/another_path/.
• Also, IE, as usual, has some magic: it doesn’t process a path when it’s redirected with Location header.

Misusing Header modification

A common task for reverse proxy is to add, delete or modify headers from a response of a backend. In some situations, it’s much easier than modification of the backend itself. Sometimes it involves modification of security-important headers. So as attackers, we may want to force a reverse proxy to apply such rules to wrong responses (from wrong backend locations) and then use it for attacks on other users.

Let’s imagine that we have Nginx and Tomcat as a backend. Tomcat, by default, sets header `X-Frame-Options: deny`, so a browser cannot open it in an iframe. For some reason, a part of the web application (/iframe_safe/) on the Tomcat must be accessible through iframe, so Nginx is configured to delete the header `X-Frame-Options` for this part. However, there is no potential for clickjacking attacks on iframe_safe. Here is the configuration:

location /iframe_safe/ {
proxy_pass http://tomcat_server/iframe_safe/;
proxy_hide_header "X-Frame-Options";
}
location / {
proxy_pass http://tomcat_server/;
}

However, as attackers, we can make a request which falls under the iframe_safe rule, but it will be interpreted by Tomcat as a completely different location. Here it is:

<iframe src="http://nginx_with_tomcat/iframe_safe/..;/any_other_path">

A browser doesn’t normalize such a path. For Nginx it falls under the iframe_safe rule. Since Tomcat supports path parameters, after path normalization, it will get `/any_other_path`. Therefore, in such a configuration, any path of Tomcat can be iframed, so an attacker can perform clickjacking attacks on users.

Of course, with a similar approach, other security-related headers (e.g. CORS, CSP, etc) might be misused too.

Caching

Caching is one of the most interesting, with a good potential for various attacks, but is still a little-known feature of reverse proxies. Recently, cache-related attacks have gotten more attention in some awesome researches including Web Cache Deception and Practical Web Cache Poisoning. In my research, I’ve been focusing on caching too: I wanted to analyze various implementations of cache. As a result, I’ve got several ideas on how to improve both cache deception and cache poisoning attacks.

How it works
There are several factors on cache of a reverse proxy which help us with understanding attacks.

The idea of caching is quite simple. In some situations, a reverse proxy stores a response from a backend in the cache and then returns the same response from the cache without accessing the backend. Some reverse proxies support caching by default, some require configuration. Generally, a reverse proxy uses as a key of cache, a concatenation of Host header value with unprocessed path/query from a request.

To decide if it is Ok to cache a response or not, most reverse proxies check Cache-Control and Set-Cookie headers from a response of a backend. Reverse proxies don’t store responses with Set-Cookie at all, but Cache-Control, as it describes a caching policy and requires additional parsing. Format of Cache-control header is quite complex, but basically, it has several flags which allows caching or not, and sets for how long a response can be cached.

Cache-Control header may look like these:

Cache-Control: no-cache, no-store, must-revalidate

Cache-Control: public, max-age=31536000

The first example forbids caching by a reverse proxy, the second – allows it. The absence of a Cache-Control header usually means that a reverse proxy is allowed to store a response.

Many web servers, application servers and frameworks set Cache-Control headers automatically and correctly. In most cases, if a web app uses session in an script, it will set Cache-Control headers which restricts caching, so usually programmers don’t need to think about it. However, in some situations, for example, if a web application uses its own session mechanism, Cache-Control header can be set incorrectly.

Attacks
A commonly used feature of a reverse proxy cache is “aggressive caching” (it’s not really an official term, but describes the idea). In some cases (for example, a backend can be too strict about caching and doesn’t allow to cache anything) an administrator, instead of changing the backend, changes rules of a reverse proxy, so it starts caching responses even with Cache-Control header which restricts caching. Usually such rules have some limitations. For example, to cache only responses of certain extensions (.jpg, .css, .js), or from specific paths (/images/).

If a reverse proxy has a path-based rule which allows aggressive caching, an attacker can create such a path which falls into the rule but will be interpreted as a completely different path by a backend server.

As an example let’s take Nginx+Tomcat again. Next rule intends to force Nginx to cache all the responses from the /images directory of Tomcat.

location /images {
proxy_cache my_cache;
proxy_pass http://tomcat_server;
proxy_cache_valid 200 302 60m;
proxy_ignore_headers Cache-Control Expires;
}

As attackers, we can misuse this rule to perform a web cache deception attack. All we need to do is to force a victim user to open the next URL (using img, for example):

<img src="http://nginx_with_tomcat.com/images/..;/index.jsp">

A victim’s browser then sends a request (with authentication cookies). Nginx sees /images, so forwards the request to Tomcat and then caches a response (it doesn’t care about Cache-Control headers). Again, for Tomcat, a path after normalization is completely different – /index.jsp. In this way an attacker can force Nginx to cache any page of Tomcat. To read this cached response, the attacker just needs to access the same path (/images/..;/index.jsp) and Nginx returns the victim’s sensitive data (e.g. csrf token).

In some way, it’s just a variation web cache deception, but not only.

Let’s think about a cache poisoning attack. The attack relies on finding unkeyed values from a request which can significantly (from a security point of view) influence a response, but at the same time, this response must be cached by a reverse proxy, so Cache-Control header must be permissive. If we mix everything together, we will be able to find more ways to exploit cache poisoning attacks.

Let’s imagine the situation. There is Nuster (it’s a cache proxy based on Haproxy) and a web application. The web application has a self-XSS vulnerability (which works only in an attacker’s account) in /account/attacker/. Nuster is configured to cache all the responses from /img/ directory on the web application:

nuster cache on
nuster rule img ttl 1d if { path_beg /img/ }

The attacker just needs to create a special URL (/img/..%2faccount/attacker/), so Nuster applies an “aggressive caching” rule, still, the web app returns a response of self XSS (it sees ‘/account/attacker/`). The response with an XSS payload will be cached by Nuster (with the key: Host + /img/..%2faccount/attacker/), so the attacker will be able to misuse this cache to XSS attack other users of the web application.From the self-XSS, we’ve got a usual XSS.

Conclusion

I have showed several examples of vulnerable configurations for each attack type. But exact cases are not so important. I wanted to give a fresh look on reverse proxy related attacks. If we know how a reverse proxy works, how it processes a request and what is the difference compared to a backend server, we (as attackers) will be able to reach more endpoints or perform more sophisticated attacks on users.

Regarding protections against such attacks, I see no “silver bullet” here (until we have a really good standard/specification on how to handle a request/path), but I think this project could help defenders as well. If you know your proxy and its limitations, you will be able to change its configuration accordingly.

Due to my desire to share my thoughts and explain stuff, the article has become very big. Still, I had to skip a bunch of tricks, you could see them in the presentation here. And the most important point of this research – “raw” results. The research is not finished yet. I will fulfill it step by step with other software. Push requests are really appreciated.

While preparing this research, I found several other kinds of similar ones, including – https://github.com/irsdl/httpninja. Through a combination of our projects, it’s possible to almost get a matrix of possible inconsistencies.

Frequently asked questions