supakaity

So we have some bad news. Some news I had hoped we’d never have to tell you.

Unfortunately the blahaj zone instances have been hacked.

I want to be brutally honest with you and not hide anything, so you know what’s going on, so that we can impart some knowledge to other admins out there, and so that anyone else in this situation can go and check their servers to make sure that they’re safe. I feel like trying to hide what happened does a disservice to the wider community in many ways and as an employee of other companies I’ve been frustrated when told to hush it up for exactly these reasons. We should treat this as a learning and teaching opportunity, and sure it sucks, but the users deserve to know exactly what their exposure is too.

Let me lay out from what I suspect and what I have gathered the timeline and path of attack was.

Our Peertube instance had come under attackvia a SQL injection vulnerability in peertube versions prior to 8.1.6. This vulnerability was disclosed and patched in PeerTube 8.1.6 on 20th of May, but not until the exploit was live in the wild, and hit many peertube instances. The attacker installed a plugin called peertube-plugin-google-analytics-js, which we discovered and removed. We upgraded to 8.1.6 which fixed the security hole, and then to 8.1.8 which cleaned up an exploit path. Our peertube instance runs in docker, we don’t require email addresses or real names from users when they sign up, and no account information was accessed. We thought that was the end of it. It seems however that the attacker also minted themselves a root OAuth token, and kept it without revealing it.

For the next 20 days, nothing obvious happened, they’d gotten a foothold but done nothing but plan and wait.

Forensically, the earliest found evidence of them testing the waters was on 11 June at 17:03 UTC they used Postgres's archive_command to write a marker file: id > /tmp/cve20253_pwned.txt – This was the first sign I could find of them probing.

I suspect that in the intervening days before this they were probing to see how they can get more access to the server. The way I THINK (but cannot confirm) they did this it via Postgres pgcrypto CVE-2026-2005, which can allow someone who has QUERY access to the server to exploit their way from an unprivileged peertube only user to the postgres superuser via a buffer overflow. We did not have pgcrypto installed in the peertube database, but I overlooked that someone could connect to the main postgres database if they say had a nodejs plugin running.

Then on the 15 June at 12:17:46 UTC they began their real attack. They first dropped a script at /data/pgsql/ukabegesay (a folder writeable by their postgres user) which called out to 217.60.195.71:9001 to give them remote shell access to the server.

Then they transfered a file to /tmp/exp which was linux kernel CVE-2026-43500, nicknamed ‘Dirty Frag’, an RxRPC local privilege escalation. I had not patched these internal servers that nobody should have access to against this. Rebooting DB servers causes downtime, and in my hubris – I thought nobody should (nay COULD) be on my servers except me, right? I was running linux kernel 6.8.0-60 which was patched against Dirty Frag in 6.8.0-124 earlier in May.

They had 44 minutes of fun rifling around my server. I can see they looked at a bunch of different files through my system through the command history. Who knows what else they did or touched on that server. Then my drive filled up several hundred gigs of WAL logs (because they modified the archive command and forgot to undo it, the backups weren't happening and the WAL files that should have been getting backed up and cleaned out weren't).

It was at this point that I started to try to login to my server, and found I couldn't because my ssh authorised_keys file had been overwritten, and the other servers in the cluster had a rogue atchive command causing the drives to fill, and a sinking feeling in the pit of my stomach. It was not going to be a fun few days for me.

Because once a server is compromised, you can't trust it anymore. It's scorched earth. Wipe the machines, change passwords, access keys, copy and sanitise the data, rotate secrets, basically set anything they touched or had access to up from scratch again. Doing this while juggling around almost a terabyte of data from host to host, running forensics, collecting evidence, figuring out the jigsaw puzzle, working out what they could have had access to, running worst case scenarios.

So what do you need to know as a user? I'm sure you've been through hacks before and so all the “change your password, be careful of people trying to scam you” advice applies here. But what about us specifically?

Well we operate a no-email-address needed policy where we can, so if you either used throw-aways or opted to not supply an email at all, they didn't get any of that information. Passwords are encrypted in the database, so if you had a reasonably complex, unguessable or random password that's never been compromised before, likely you're safe, although you should change it regardless. Any personal information you provided (Real names, emails, dates, information) is possibly in the hacker's possession. You should assume that 2FA (TOTP) is likely compromised, and you should reenroll if you were using it, though without access to valid passwords, there is little the attacker can do with the 2FA codes. In theory, the attacker could also have accessed DMs and private posts on Sharkey. In theory, matrix/synapse data may also have been made available to the attacker, but this would only consist of non encrypted content, and encrypted content would remain out of their reach.

We'll do our best to cancel any tokens and login sessions, so you'll have to login on every device again on all of our services. We are also rotating activity key pairs on all of our services to avoid the possibility of impersonation attacks. Rotating these keys is a manual process though, so our instances will slowly start to come back online one at a time, once we have had the chance to secure them.