Thursday, May 1, 2014

2006 Called and Wants its Apache Config Back

Summary

Upgrade Apache, stop using Rewrite Rules and go back to ProxyPass.

Details

This is a quicky but hopefully it will save someone else from my pain. We upgraded our apache instance from 2.2 to 2.4 in the last few months (with some, but not a ton of pain). The biggest change IMO was the way permissions work, moving away from Order allow,deny syntax and towards Require syntax. I followed the upgrade guide and everything started normally and Rewrite rules were all good so I did the lazy admin thing and wrapped it up.

My papertrail alerts notified me with this log, on a completely blocked off site:

Sadly, this error is actually just a canary. I discovered that someone in Poznan [1] has been using our server as a Proxy [3]. Guh.

I validated this by setting my browser proxy for that subomain on for our http, not https domain (https didn't work) and then attempting to browse the internet [2]. Which worked. Moar guh. You could see in the logs that I was using our server as a passthrough, which is bad juju. I have no idea how or why, since we have ProxyRequests off and we were using rewrite with [L,P] and I was set on a traditional tried and true recommended Plone rewrite method. I used ProxyPass rules then I could NOT use that server as a proxy, but if I used Rewrite with [P] then I could.

I honestly don't understand 100%, but it has to do with the fact that Require all granted is needed to get to the rewrite rules, and because we are using apache like a proxy using Require all denied and then Require local just doesn't work (tail the auth logs at trace3 and you'll see something like client denied by server configuration: proxy:http://127.0.0.1 where that proxy prefix makes things "not work as expected"(tm). The docs recommend using mod_remoteip to get around this limitation but I tried unsuccessfully many variations of this and just got nowhere.

During this long ass investigation, I also learned that using rewrite rules are not the recommended way to do things anymore anyways and furthermore are less performant because rewrites don't use Apache's worker pool. That's obviously no good.

It would be easy enough to fix if we didn't have a lot of domain shit going on (who doesn't these days), which turns out to be pretty annoying and hard to resolve. I come to you with this config snippet, which should for just about anyone who wants to update to 2.4.


If you use puppetlabs apache module this will look like:
I still have not figured out a sexy one off solution for managing multiple subdomains this way and rewriting them to plone sites so you would need to rely on your trusty musty config management tool to do that for you (which you are using right???).

I know I usually use this blog for more definitive advice but this is actually NOT one of those. I am extremely surprised at the ease which I set my own server up as a proxy and the pain it took to undo that. This post is part alternative method, part warning, and part documentation as to why I'm switching over from Rewrite to ProxyPass again.

[1] To do this, look in your apache access logs and then perform a whois on the requesters ip.
[2] You can also do this with telnet, but I got mixed results that way. I found the browser was just easier to jerry rig the test case.
[3] Note that this doesn't necessarily mean that your server is compromised, but rather that some baddies can proxy through your servers and do baddie things.  More importantly, your service host can find out and slap you with something if you are lucky or suspend your sever if you are not. DO NOT FEED THE BADDIES! Unless you are a sympathizer. Then honeypot up!

6 comments:

  1. Can you further elaborate:
    "I validated this by setting my browser proxy for that subdomain on for our http, not https domain (https didn't work) and then attempting to browse the internet " most specifically "... subdomain on for our http ..."
    Thanks!

    ReplyDelete
    Replies
    1. Hi David -

      Sure. In Chrome, I opened up advanced settings then went to proxy, which in mac takes you to the system settings. There I used my domain and port 80 as a web proxy and applied the settings. This basically means fetch all requests through that proxied url, usually used if you work for a company with a tight firewall or are from a country with an oppressing government. Then, surf normally and watch your apache logs explode with the content you are proxying. Note that browsing http sites for me didn't work, only https. Also if the browser already had a bunch of stuff cached it was a false negative so be sure to try new sites. If you are testing, try both http and https as a proxy, and the same for the sites you are visiting. Hope that helps!

      Delete
    2. Hey Liz,

      So, you're saying that if I make the changes in proxy settings as described (using Firefox and checking box, "use this proxy server for all protocols", then go back and try to browse to a previously un-browsed-to-website, I will get that website IF my server is proxying? So, since I invariably get my home page, that means that my server isn't proxying? My logs give a "500" (instead of the 200 you mention above) so that confirms a good thing, I think.

      Thanks for posts like these for those of us who use tech but more marginally.

      Delete
    3. Hi David -

      500 is an internal server error. Is that causing your plone to be angry? It should return a 404 and/or just always return your sites root or something like that.

      Delete
    4. Hi Liz,

      The 500 appears in the logs only, the browser itself returns the site root. I think it's pretty happy!

      Delete
  2. Useful post - thanks! Any suggestions how I could amend the example you have given so that the Plone site points to a sub-folder within Plone rather than the root?

    ReplyDelete