Help dealing with malware

[rmisaki]

I'm using Ubuntu 20.04 for running CyberPanel on my webserver, on the last few days, a security flaw found in Cyberpanel led to attacks in thousands of Cyberpanel machines, and mine didn't escape that x. x
On mine they installed kinsing malware which was using all the CPU, so I found out about it and was trying to remove it, those were the major happenings after that:

- When trying to remove Kinsing, all files got suddenly encrypted by C3rb3r encryptor, so I rolled back to a backup from a day earlier
- I managed to remove kinsing, but something called zzxx64 was still running and reinstalling on the /var/ folder
- I removed that one as well, lots of random websites spawned up in the public_html folders, like public_html/library, public_html/gig, etc
- They had the audacity to even register those on Google Search console under random e-mails, that's how I noticed those, so I went into each public_html folder and deleted those as well
- They also added a sysdb.php and wp-inc.php file with virus in each public_html folder, I deleted those as well
- Changed root password a few times and then I was unable to log in with password for some reason
- Added an SSH key and disabled password auth for SSH
- Had problems with dovecot getting too many connections, so I had to change it into performance mode and also configured fail2ban to block the IPs trying to brute-force the SSH and the e-mails (doesn't seem to have worked for the e-mails though)
- Ran clamscan multiple times
- Ran rkhunter and chkrootkit
- It says I might have an LMK rootkit

So after I did everything I could think of to protect and save my server, a few issues still remain:

- Apt installing anything says there is problems with initramfs-tools with this message:

Setting up initramfs-tools (0.136ubuntu6.7) ...
update-initramfs: deferring update (trigger activated)
Processing triggers for initramfs-tools (0.136ubuntu6.7) ...
update-initramfs: Generating /boot/initrd.img-5.4.0-198-generic
E: /usr/share/initramfs-tools/hooks/fsck failed with return 1.
update-initramfs: failed for /boot/initrd.img-5.4.0-198-generic with 1.
dpkg: error processing package initramfs-tools (--configure):
 installed initramfs-tools package post-installation script subprocess returned error exit status 1
Errors were encountered while processing:
 initramfs-tools

- I'm still having multiple IPs trying to log in into SSH in random ports and probably on e-mail as well, but I don't know how to verify or block for email, the ufw firewall and the VPS firewall doesn't seem to be blocking those attempts

- chkrootkit says there's 200 something hidden files and processes and that I could have a LMK rootkit, CPU goes all the way to 100% all the time, except when I try to see which process is using that much CPU with top or htop, then CPU drastically reduces (temp fix: have a screen with top running all the time)

CPU graphs (while running top):


I closed the screen that was running top command and it didn't come back to 100% cpu, but I'm sure there might be something avoiding detection...

Please help, I don't know what else can I do x. x

[Irihapeti]

I'm not a super server nerd, and hopefully one of them will show up shortly. But I have run my own servers, and in a case like this I'd strongly advise erasing the whole thing and reinstalling. There are a lot of known problems which may or may not have been successfully dealt with, plus who knows what unknown/concealed items might be lurking. I'd also not want to reinstall CyberPanel but use SSH with the commandline instead. It's not as pretty as the various control panels and it has something of a learning curve, but SSH is a lot more robust. Not totally foolproof, of course, but then nothing is if you want to have an online server.

Probably not what you wanted to hear, but much more likely to get you where you want to be in the long run. I'll now stand back and let others offer their contributions.

[stephan4]

Just removing malware is not enough. You have to figure out the root cause why it were able to infect your system. Otherwise, it will infect you again sooner or later.
That said, even a backup does not fully help if the vulnerability is still present.

I suggest, rebuilt from scratch and harden your server.

[rmisaki]

Yeah, rebuilding from scratch and hardening the server, even ditching CyberPanel for SSH would definitely be better solutions, problem is I can't really have too much downtime because there are costumer's websites on the server and I'm also way to busy to deal with too many manual configurations right now
I intend on planning a bit and migrating to a new server, but I can't really do that right now

Edit: Forgot to mention, the reason the server got infected was because of a security breach found in CyberPanel, it got patched and I updated it, but the harm was already done, the breach was exploited really quickly after it was found

[rmisaki]

OK!
Found out why top and htop didn't show up the reason for 100% CPU

/usr/bin/.local/bin was added to path and those were the contents of this directory:


drwxr-xr-x 2 root lscpd 4096 Nov 3 17:23 .
drwxr-xr-x 3 root lscpd 4096 Oct 30 20:51 ..
-rwxr-xr-x 1 root lscpd 15784 Nov 3 15:43 crontab
-rwxr-xr-x 1 root nogroup 113 Nov 3 15:43 df
-rwxr-xr-x 1 root lscpd 15928 Nov 3 15:43 htop
-rwxr-xr-x 1 root lscpd 15504 Nov 3 15:43 ldd
-rwxr-xr-x 1 root root 15560 Nov 3 15:43 lsof
-rwxr-xr-x 1 root nogroup 116 Nov 3 15:43 mount
-rwxr-xr-x 1 root lscpd 15544 Nov 3 15:43 strace
-rwxr-xr-x 1 root lscpd 15896 Nov 3 15:43 top

So I couldn't find it, the virus files were under /usr/bin/wbin

I managed to clear the stuff, now I hope nothing else remains x. x