I was hacked.

[FuguTabetai]

So my server was hacked recently, and I wanted to see what I could do to look for problems. I found http://www.chkrootkit.org/ which was a great way to check for root kits. Also, running rpm -Va and examining that output was great - it uses rpm to check for changes in the installed packages managed by rpm.

It looks like the only thing the hacker was able to do was put some files in /tmp and run those. So I mounted /tmp as a separate partition with the noexec parameter set (as per directions at http://www.webhostingtalk.com/archive/thread/326561-1.html

Since it looks like the attackers got in through phpbb (which I like in general, and don't really blame for the break-in for reasons I'll get to below) I updated to the latest version of phpbb and added some mod_rewrite rules which prevent attacks trying to use the highlight/search functionality that gets php to eval a string. Information on that is here:
http://www.phpbb.com/phpBB/viewtopic.php?t=249010&postdays=0&postorder=asc&start=0

The reason the break in happened is because my apache user could:

1. Write to /tmp
2. Execute files from /tmp
3. Run some executables that it doesn't really need

so I addressed 2 by mounting it without exec permissions, although it looks like you can still invoke an interpreter on a file in there (e.g., /bin/bash badscript.sh or /bin/perl attack.pl) - that is something I'll have to look into...

I think my apahce user needs to be able to write to /tmp sometimes, so that isn't something I want to disable.

I did change /usr/bin/gcc /usr/bin/perl /usr/bin/wget to 700 mode though, which should prevent attempts to compile new binaries, use perl for scripts that are uploaded to /tmp if they get in there, and use wget (a key component to how I was hacked) to grab other scripts.

Anyway, there is some information on what might be an exploit in phpbb that allows uploading scripts and then possibly running them (if you have a /tmp and user that can run interpreters for your web-account) here:
http://www.phpbb.com/phpBB/viewtopic.php?t=308400

Hope that helps. If anyone else has good advice for these sorts of situations, I would appreciate it. I actually just bought "Hacking Linux" (or some big book like that) to learn about this stuff, but I didn't bring it with me (currently on vacation)!

[valor]

Hi!

Sorry to hear that. I don't use PhpBB but I've some comments and questions for you.

- The noexec partition could by bypassed just by instead of do
./mybackdoor do . ./mybackdoor

- Ask rimuhosting to load a /backup/ partition for you with the last backup they have from you, then compare both filesystems to see what have changed and lookup documentation of every file with a change and identify if the change is normal or not. Otherwise I think you should not trust your own server and you should call for restore to the last known secure backup and inmediatly login to the box and take the propper actions.

- How you remounted tmp partition? Does rimu proovide partitioning of vps filesystem if customer ask at startup? I've everything on one partition, I didn't knew that!. That's great.

- I would not suggest to chmod 700 perl. Even if you don't have perl CGI's your system probably uses perl scripts for some tasks. And there's a chance that for security some program/script don't run as root and therefore it will fail.
If you set 750 and you chown perl to a group that webserver user is not part of, it will be enough to keep out access from people hacking through your web server.

- I would suggest you reading about apache module mod_security (http://www.modsecurity.org/) and also the book "Practical UNIX and Internet Security" from Oreilly.

- I don't know how much resources do you have but if you are really paranoid and your server load is not too much, one thing you can do to limit intrusion damage to the sytem is create a jail for Apache and put only the required minimal files to have everything you need working (php, etc). Then if an attacker get through your webserver he will be at least one step away from have a real shell, indeed it's possible and not that much difficult to avoid a jail but at least most script kiddies would not even know what's going on.
For more info on chrooting apache see this article:
http://www.linux.com/article.pl?sid=04/05/24/1450203


My two grams of sand...

valor

[RedOut]

PHPBB is evil, too many holes, and too many appologists, even if you stay current you'll get hit, the exploits are out there and running rampant weeks before they acknowledge it...

I just bit the bullet and bought IPB to replace it.

I'd seriously suggest SMF or something else because every version of PHPBB has turned out to be an open door to your system, and I don't expect the new version to be any different. They need to accept help from the community, because the developers frankly, aren't very good.

Sorry for coming off so against this software, but I've used it for years now, and have been cracked too. I've followed the development of the next version (for years as well) and things are not good.

The best thing you can do to secure a machine with PHPBB on it, is to uninstall that software.

[FuguTabetai]

Hi!
- The noexec partition could by bypassed just by instead of do
./mybackdoor do . ./mybackdoor

I'm not sure I understand; the problem is that the apache process was able to write files to /tmp, and then could run them from there. Mounting /tmp as noexec nosuid should take care of that, and I've got apache set up so it shouldn't be able to write in other places. Just do ../mybackdoor shoulldn't work since the hacker shouldn't be able to write to /. Of course, I'm not sure; I am not a linux/unix expert (just a home user, and I've got a few books.)


- Ask rimuhosting to load a /backup/ partition for you with the last backup they have from you, then compare both filesystems to see what have changed and lookup documentation of every file with a change and identify if the change is normal or not. Otherwise I think you should not trust your own server and you should call for restore to the last known secure backup and inmediatly login to the box and take the propper actions.


I'll look into setting up the back stuff. I don't have old snapshots to compare against. I'll probably re-install the OS and make a move to the Xen services, but for now it should be fine. All the stuff I run is hobbyist and not too important, and as long as I'm not causing problems with network / CPU / disk usage, I think things are ok. I agree that I can't be sure that my machine isn't compromised, but it looks ok, and it has been behaving, and I'm on vacation, so that's good enough. :)


- How you remounted tmp partition? Does rimu proovide partitioning of vps filesystem if customer ask at startup? I've everything on one partition, I didn't knew that!. That's great.

Well, I did of course re-mount tmp and added a line to /etc/fstab so it would come up on boot. It isn't a separate partition - it uses the loopback to mount a file as a new partition, but I think that it effectively looks like a new partition. Of course, this is all just coming from what little I've read on the subject. I've got some books I'll check up with when I get back home.


- I would not suggest to chmod 700 perl. Even if you don't have perl CGI's your system probably uses perl scripts for some tasks. And there's a chance that for security some program/script don't run as root and therefore it will fail.
If you set 750 and you chown perl to a group that webserver user is not part of, it will be enough to keep out access from people hacking through your web server.

That is a good point. I don't think I'm using Perl in any cron jobs, but I'll check. I just wanted something I could do quickly to prevent similar sorts of exploits. I want to look into either chrooting apache, or putting it in a different group with access to only necessary programs. I'm really only doing all this for fun though, so it wasn't something I thought about when I set up my server. Overall, getting hacked has been a fun learning experience (but one I don't want to repeat.)


- I would suggest you reading about apache module mod_security (http://www.modsecurity.org/) and also the book "Practical UNIX and Internet Security" from Oreilly.

Looks good. I've got the "Hacking Linux Exposed" from McGraw-Hill at home to read. I didn't bring it on vacation with me since I didn't think I would need it!


- I don't know how much resources do you have but if you are really paranoid and your server load is not too much, one thing you can do to limit intrusion damage to the sytem is create a jail for Apache and put only the required minimal files to have everything you need working (php, etc). Then if an attacker get through your webserver he will be at least one step away from have a real shell, indeed it's possible and not that much difficult to avoid a jail but at least most script kiddies would not even know what's going on.
For more info on chrooting apache see this article:
http://www.linux.com/article.pl?sid=04/05/24/1450203
valor
Yes, that looks interesting. does running a jail put much stress on the systems? Shoot, I should have spent some time talking to my roomate before I finish my phd - he's doing his phd in computer security. Me, I'm just a natural language processing guy, so this isn't really my cup of tea... :)

thanks for the advice though.

[FuguTabetai]

PHPBB is evil, too many holes, and too many appologists, even if you stay current you'll get hit, the exploits are out there and running rampant weeks before they acknowledge it...

Well, I'll look at it as a challenge: how can I secure my server with a relatively unsecured cgi running? :)

I've seen a lot of people moving away from phpbb recently, but they have been trying to respond quickly to many of the exploits, and I actually agree that some of the things that are being exploited are not their problem. In general, I think the webserver should not be able to write files that can then be executed; that isn't really a phpbb fault. There are other things like that, so I think it will be fun to learn how to harden the server in general.

And I've been using phpbb for a while, so "the devil you know..."

[Brandon]

One thing that I have done to try and stop automated scripts like the phpBB hack is to create a .wgetrc file that has bogus proxy servers in it. If apache runs as user "nobody" and "nobody"'s home directory is /, then create /.wgetrc with the contents:

http_proxy = http://bogus.baddomain.com:19999/
ftp_proxy = http://bogus.baddomain.com:19999/

This way, when the script tries to use wget, it will try to go through a proxy server that doesn't exist, and will fail.

Of course, there are lots of ways around this, but all of the automated phpBB hacks that I've seen asume that wget works, so this effectively stops it.

[rkn]

FuguTabetai - can i ask what script was used to hack your site? I recently encountered a similair problem on a site hosted elsewhere involving phpbb, and havent found any similair cases yet and im stil trying to pinpoint exactly what happened.

[jamieflournoy]

I'm not sure I understand; the problem is that the apache process was able to write files to /tmp, and then could run them from there.

The problem isn't /tmp permissions, the problem is that you somehow allowed an attacker to take control of Apache on your server.

Mounting /tmp as noexec nosuid should take care of that, and I've got apache set up so it shouldn't be able to write in other places. Just do ../mybackdoor shoulldn't work since the hacker shouldn't be able to write to /. Of course, I'm not sure; I am not a linux/unix expert (just a home user, and I've got a few books.)

You misread his post (easy to do): he said dot space dot slash, you read dot dot slash.

../somescript.sh is different from . ./somescript.sh.

. is shorthand for source, as in "source ./somescript.sh". It treats somescript.sh as a text file full of command lines to execute. "source somescript.pl" would fail if you were running Bash because your current Bash shell would try to execute the perl code in somescript.pl line by line.

So if a hacker managed to get a shell and then ran "source /tmp/somescript.sh" it wouldn't matter if somescript.sh were executable or not, since it's not being executed, just read as a file full of text, that the current shell interprets as commands to execute.

But this is academic. This is one of innumerable local exploits that the attacker can use to gain root access. Bottom line, if an attacker can trick a network-facing application into executing local commands in a local shell, you're doomed. Give up trying to plug individual local exploits; it's futile. (Once the Greeks are inside the walls of Troy, they win.)

Focus on removing the network facing vulnerability first. That means either replacing PHPBB or patching it so that it never, ever takes data straight from the user and evals it or puts it right on the command line.

[FuguTabetai]

FuguTabetai - can i ask what script was used to hack your site? I recently encountered a similair problem on a site hosted elsewhere involving phpbb, and havent found any similair cases yet and im stil trying to pinpoint exactly what happened.
I believe that they used one of the vulnerabilities in PHPBB's search to execute arbitrary code. It has since been patched. This was quite a while ago though, so maybe there are new vulnerabilities, but I haven't heard of anything.