“HTTPS everywhere” is not a luxury anymore. It is a necessity. Thankfully, obtaining an SSL certificate has become easier too, with initiatives such as Let’s Encrypt, GeoTrust, Positive SSL, StartSSL. Even cloud based services such as Cloudflare and Amazon AWS provide free SSL certificates to their customers.
Here is setting some context to help the reader appreciate the discussion:
We host our rails applications on Amazon AWS. We generally use three different environments - development, staging and production. Development environment is generally local to a developer while staging and production are hosted on the cloud. There is a minor difference in the way we configure our staging and production environments. Our staging environment typically contains a single machine instance hosting our application. This single instance is exposed to internet directly (has a public IP). On the other hand, our production environment typically contains a cluster of instances for the sake of horizontal scaling. These instances typically do not have a public IP and hence not exposed to internet directly. We put this cluster behind an internet-facing Elastic Load Balancer (ELB).
We use chef-solo to manage our cloud infrastructure as well as to deploy code to various environments.
The Problem Statement:
Using Let’s Encrypt in a clustered setup is tricky, since you need to make one of the instances stateful, in the sense, one instance needs to be given the responsibility of obtaining and renewing SSL certificate from Let’s Encrypt. All other instances need to copy this certificate every time its renewed. This requirement unnecessarily complicates the setup and also takes away some amount of flexibility. Also, Let’s Encrypt does not issue wildcard certificates and the validity of a certificate is just 90 days
The certificates provisioned from the other provider, Amazon AWS, can only be installed on an ELB. Hence is best suited for our clustered setup, namely production. An added advantage is that Amazon can issue wildcard certificates. We could always add an ELB to our staging environment (even though we will never have more than one instance), but that costs extra money for no reason.
This leaves us with these options
We went ahead with this choice. Using chef to manage our setup came handy.
We first configured our Staging environment and everything worked as expected.
However, the same application, in production environment, started throwing
CSRF detected Error whenever an OAuth2 callback happened. This was really strange. Our application integrated with two different OAuth providers, and the problem was consistent with both these providers.
What’s the issue?
The only difference between our Staging and Production setups was the ELB.
A closer look would reveal that the rails application had no way to know if the callback was made on a http:// URL or a https:// URL, because it sees only HTTP (due to offloading). Was this the reason rails was unhappy?
OAuth2, by design, does not accept plain HTTP callbacks (unless it is to localhost).
How do we move forward?
PoC to prove the theory
Just to confirm what we think is the cause, we enabled HTTPS on NGINX (like we did in our staging environment). This was in addition to HTTPS on the Load balancer. We reconfigured the Load Balancer to NOT offload HTTPS but forward the request as-is to NGINX.
What do we have now? The
CSRF detected errors are gone. Application behaves just like it should.
This confirmed our theory.
But the question now is, how do we achieve our desired configuration of offloading HTTPS at the ELB ? Is it just not possible ?
We have been using X-Forwarded-For header while reverse proxying to unicorn so that our rails application knows the client IP address (rather than the IP address of the Load Balancer). We need this for logging and tracking.
Could there be something on similar lines to tell the rails application that the request was not on HTTP but on HTTPS?
Sure there is. We had to set a header in our reverse proxy configuration:
X-Forwarded-Proto to https
For NGINX, we do it like this:
proxy_set_header X-Forwarded-Proto https;
Voila, Rails is happy and things are back to normal!
Rails bothers about SSL only at two places,
1. At environment config, force_ssl.
2. At external included Gem like Omniauth.
In Rails environment config.
config.force_ssl = true
This does the trick, but doesn’t seem like a good idea to enable this option in Rails because, we offload https at NGINX. For Rails, request came in http, so it does a permanent redirect to https, which ends in a infinite loop.
Our stack trace gave a clue that error might be inside
actionview-22.214.171.124/lib/action_view/rendering.rb:30 → process
actionpack-126.96.36.199/lib/action_controller/metal.rb:196 → dispatch
actionpack-188.8.131.52/lib/action_controller/metal/rack_delegation.rb:13 → dispatch
actionpack-184.108.40.206/lib/action_controller/metal.rb:237 → block in action
actionpack-220.127.116.11/lib/action_dispatch/routing/route_set.rb:74 → dispatch
actionpack-18.104.22.168/lib/action_dispatch/routing/route_set.rb:43 → serve
actionpack-22.214.171.124/lib/action_dispatch/journey/router.rb:43 → block in serve
actionpack-126.96.36.199/lib/action_dispatch/journey/router.rb:30 → each
actionpack-188.8.131.52/lib/action_dispatch/journey/router.rb:30 → serve
actionpack-184.108.40.206/lib/action_dispatch/routing/route_set.rb:817 → call
omniauth-1.3.1/lib/omniauth/strategy.rb:186 → call!
omniauth-1.3.1/lib/omniauth/strategy.rb:164 → call
As we dug inside the Gem and found out that Omniauth looks at these headers
def ssl? request.env['HTTPS'] == 'on' || request.env['HTTP_X_FORWARDED_SSL'] == 'on' || request.env['HTTP_X_FORWARDED_SCHEME'] == 'https' || (request.env['HTTP_X_FORWARDED_PROTO'] && request.env['HTTP_X_FORWARDED_PROTO'].split(',') == 'https') || request.env['rack.url_scheme'] == 'https' end
This is where we found that setting up X_FORWARDED_PROTO to https should fix our problems.
Initially, this X_FORWARDED_PROTO was set to $scheme. Which will be http for production as https is offloaded at ELB.
Now, by setting X_FORWARDED_PROTO to https, we are making sure that redirects are happening on https.