Summary
Upgrade Apache, stop using Rewrite Rules and go back to ProxyPass.Details
This is a quicky but hopefully it will save someone else from my pain. We upgraded our apache instance from 2.2 to 2.4 in the last few months (with some, but not a ton of pain). The biggest change IMO was the way permissions work, moving away from Order allow,deny syntax and towards Require syntax. I followed the upgrade guide and everything started normally and Rewrite rules were all good so I did the lazy admin thing and wrapped it up.
My papertrail alerts notified me with this log, on a completely blocked off site:
Sadly, this error is actually just a canary. I discovered that someone in Poznan [1] has been using our server as a Proxy [3]. Guh.
I validated this by setting my browser proxy for that subomain on for our http, not https domain (https didn't work) and then attempting to browse the internet [2]. Which worked. Moar guh. You could see in the logs that I was using our server as a passthrough, which is bad juju. I have no idea how or why, since we have ProxyRequests off and we were using rewrite with [L,P] and I was set on a traditional tried and true recommended Plone rewrite method. I used ProxyPass rules then I could NOT use that server as a proxy, but if I used Rewrite with [P] then I could.
I honestly don't understand 100%, but it has to do with the fact that Require all granted is needed to get to the rewrite rules, and because we are using apache like a proxy using Require all denied and then Require local just doesn't work (tail the auth logs at trace3 and you'll see something like client denied by server configuration: proxy:http://127.0.0.1 where that proxy prefix makes things "not work as expected"(tm). The docs recommend using mod_remoteip to get around this limitation but I tried unsuccessfully many variations of this and just got nowhere.
I honestly don't understand 100%, but it has to do with the fact that Require all granted is needed to get to the rewrite rules, and because we are using apache like a proxy using Require all denied and then Require local just doesn't work (tail the auth logs at trace3 and you'll see something like client denied by server configuration: proxy:http://127.0.0.1 where that proxy prefix makes things "not work as expected"(tm). The docs recommend using mod_remoteip to get around this limitation but I tried unsuccessfully many variations of this and just got nowhere.
During this long ass investigation, I also learned that using rewrite rules are not the recommended way to do things anymore anyways and furthermore are less performant because rewrites don't use Apache's worker pool. That's obviously no good.
It would be easy enough to fix if we didn't have a lot of domain shit going on (who doesn't these days), which turns out to be pretty annoying and hard to resolve. I come to you with this config snippet, which should for just about anyone who wants to update to 2.4.
If you use puppetlabs apache module this will look like:
I still have not figured out a sexy one off solution for managing multiple subdomains this way and rewriting them to plone sites so you would need to rely on your trusty musty config management tool to do that for you (which you are using right???).
I know I usually use this blog for more definitive advice but this is actually NOT one of those. I am extremely surprised at the ease which I set my own server up as a proxy and the pain it took to undo that. This post is part alternative method, part warning, and part documentation as to why I'm switching over from Rewrite to ProxyPass again.
[1] To do this, look in your apache access logs and then perform a whois on the requesters ip.
[2] You can also do this with telnet, but I got mixed results that way. I found the browser was just easier to jerry rig the test case.
[3] Note that this doesn't necessarily mean that your server is compromised, but rather that some baddies can proxy through your servers and do baddie things. More importantly, your service host can find out and slap you with something if you are lucky or suspend your sever if you are not. DO NOT FEED THE BADDIES! Unless you are a sympathizer. Then honeypot up!
I know I usually use this blog for more definitive advice but this is actually NOT one of those. I am extremely surprised at the ease which I set my own server up as a proxy and the pain it took to undo that. This post is part alternative method, part warning, and part documentation as to why I'm switching over from Rewrite to ProxyPass again.
[1] To do this, look in your apache access logs and then perform a whois on the requesters ip.
[2] You can also do this with telnet, but I got mixed results that way. I found the browser was just easier to jerry rig the test case.
[3] Note that this doesn't necessarily mean that your server is compromised, but rather that some baddies can proxy through your servers and do baddie things. More importantly, your service host can find out and slap you with something if you are lucky or suspend your sever if you are not. DO NOT FEED THE BADDIES! Unless you are a sympathizer. Then honeypot up!
