Phusion server security report

Executive summary: our web host Linode has been compromised and the responsible hacker group appears to claim to have had access to one of the Phusion servers, which prompted us to start a full investigation. Until now, no evidence of third party access has been found, and no tampering of the Phusion Passenger Enterprise files have been found. In spite of this, we are taking precautionary action and we urge customers to verify their Phusion Passenger Enterprise installations through the instructions at the bottom of this message.

Dear users and customers,

About 3 weeks ago, our web host Linode issued several public statements[1][2] claiming that one of their customers was the subject of an attack by a group called HTP. From what we’ve been able to read from HTP[3] a few hours ago, we believe that SwiftIRC and/or nmap was the target Linode was referring to.

In Linode’s initial statement[1], they also mentioned that law officials were aware of the attack and that Linode had found no evidence of other customer data being compromised. We too hadn’t noticed any suspicious activity on our servers and weren’t notified by Linode about being the attacked target which led us to believe that this initial statement held true.

A few hours ago however, a statement released by HTP was brought to our attention wherein they claimed otherwise[3]. In particular, the statement appears to claim that HTP has had root access to one of the Phusion servers and this immediately prompted us to start a new investigation of our own. Up to this point, we have found no evidence that they have had access to our data, but we are checking our systems several times over to minimize the possibility of having missed a potential attack vector on the first few passes. We have also contacted Linode to get a clarification on their first statement[1] in light of new events that seem to point to nmap’s server to have indeed been compromised. Pending this response, we didn’t want to take any risks in waiting to notify our customers of the current situation.

The absence of evidence after all doesn’t necessarily mean that the server has not been accessed: even though we feel we have taken all the necessary steps to ensure maximum security on our servers, we remain scrutinous of our systems’ integrity at all times. There are after all a myriad of components that comprise a server, and each of them could be a potential attack vector as long as fault free software is something developers in general can only hope to aspire to. More specifically, as long as erring is human, we can only hope to minimize these chances rather than believing we can prevent them completely 100% of the time. Zero day exploits can always occur at any time and the best thing we can do is to be as transparent about this to our customers as we can. To that end, we’d like to notify our customers that we are moving our services to another web host and will be reinstalling our servers as a precaution.

If HTP has indeed compromised our systems without us being able to tell, then we would be interested in learning how and would encourage them to contact us (info@phusion.nl). We value security and transparency over pride and are extremely committed towards serving our customers. It is also the reason why we are informing our customers about this in an open manner several hours after seeing HTP’s claim despite not being able to verify this claim to be accurate ourselves.

We would also like to take this opportunity to encourage all Phusion Passenger users – that is, open source users and Enterprise customers alike – to make use of the PGP digital signatures that we employed since February this year.[4] Checking the signature of your Phusion Passenger download against the corresponding key helps minimize the chances of the downloaded software being tampered with. We have already manually reviewed the Phusion Passenger Enterprise source code and have found no evidence of suspicious activity. For your own safety however, we would always recommend you to take proper caution when downloading and installing software from the internet. The PGP digital signatures are provided to aid in that aspect and we would highly recommend you to use this at all times.

Having said this, if our servers actually were accessed, then it’s possible that the attackers temporarily inserted compromised gems and tarballs and removed them later. We therefore urge our Enterprise customers to verify the integrity of their Phusion Passenger Enterprise installations. Instructions can be found at the end of this message.

In any case, Phusion has not, does not and will not store customer creditcard information on any of its servers. All credit card information is stored on servers of third party, PCI-DSS compliant payment gateways, e.g. FastSpring and Paypal. Phusion also does not store customer passwords in plain text; all customer passwords are stored in BCrypt format.

The open source version of Phusion Passenger is hosted on another server, namely GitHub, and we have also found no suspicious activity in its repository.

We understand that after reading all this, you might have concerns with regards to your own server’s integrity. Even though we have found no evidence of suspicious activity on our own servers or in Phusion Passenger’s code base, we feel that we should still encourage you to remain scrutinous of your own servers’ integrity and take the steps you deem necessary in maximizing its security.

Needless to say, we remain committed in being transparent towards our customers and will continue in keeping them up to date of any of our findings concerning this matter. If you have any questions, please feel encouraged to contact support@phusion.nl.

With warm regards,
Hongli Lai
Ninh Bui

References:

Instructions for verifying Phusion Passenger Enterprise installations

We have generated SHA-1 hashes of all Phusion Passenger Enterprise files inside the gems and tarballs. You can use these hashes to verify your installed Phusion Passenger files. If anything is amiss or if you require further assistance, please contact support@phusion.nl.

Install GnuPG. Debian users can apt-get install gnupg, OS X users can use GPG Tools: https://gpgtools.org/
Login to the Customer Area: https://www.phusionpassenger.com/orders
Scroll down to the “Files” section.
Download the “sha1sums.txt” and “sha1sums.txt.asc” files that pertain to the version of Phusion Passenger Enterprise that you’re currently running. Ensure that both files are in the same directory.
Import the Phusion Software Signing PGP key: http://www.modrails.com/documentation/Users%20guide%20Apache.html#_importing_the_phusion_software_signing_key Name: Phusion Software Signing (software-signing@phusion.nl) Short key ID: 0x0A212A8C Long key ID: 0x2AC745A50A212A8C Fingerprint: D5F0 8514 2693 9232 F437 AB72 2AC7 45A5 0A21 2A8C
Set this key to trusted: gpg –edit-key software-signing@phusion.nl Then in the GPG prompt, type: trust Choose: 5 = I trust ultimately In the GPG prompt, type: save
Verify the downloaded sha1sums.txt against its signature: gpg –verify sha1sums.txt.asc You should see: Good signature from “Phusion Software Signing software-signing@phusion.nl“
Copy sha1sums.txt to your server.
On your server, find out where the Phusion Passenger files are by running: passenger-config –root
Run: cd
Run: sha1sum -c /path-to/sha1sums.txt –quiet