On December 10, 2021, a vulnerability within the Log4j2 library was discovered. Due to how widespread Log4j2 is, I had to mitigate the damage as quickly and efficiently as possible.
What is Log4j?
Log4j is a library in Java used for logging actions done within a program. These actions can include error messages, user input, responses, and warnings and are often used to refer back to what caused a given issue or determine how to optimize a program. In fact, there’s likely a good chance that you have used a logging library like Log4j before, whether realizing it or not. Due to the nature of Java and Log4j, modifications to the code are often encouraged for your use case. Such modifications, or forks, can include the ability to parse links, anonymize users, enable compatibility, and back up logs.
Log4j 2 specifically is a fork of Log4j to include patches for vulnerabilities, performance improvements, better API support, and filtering, just to name a few. Because it was based on Log4j, Log4j 2 is meant to be a drop-in replacement for its predecessor with little to no additional work done on the programmer’s end.
Step 0 – Determining Potential Roadblocks
The main issue with attempting to mitigate these types of issue is how Java and, by extent, Log4j is implemented. Most modern day implementations of Java is in a matter beyond our control, often compiled into executable that your operating system can understand without installing dependencies. While irrelevant to my workflows, Ubiquiti’s controller software, IPMI for various computers, and even car stereos are based off of Java as examples. Due to the closed nature of them, the best that I can do is update what I can and pray that the vulnerability has been taken care of. Ubiquiti actually made the update known and made that exploit fix one of, if not the first line, in the change log.
What I can control, however, is what software I used and patch the issue in any open source programs that I use. That being said, all of my outside facing servers have automatic updates set up and configured with a set-it-and-forget-it mindset. While for severe issues such as this necessitate manually checking, it serves well for the occasional performance update, less severe security issues, and generalized bug fixes. The main things I was concerned about were programs that I haven’t used in extended periods of time.
Step 1 – Removing Java
The computers and servers that I had a keen eye on were the following, with some obfuscation put in place for privacy concerns
- My desktop running Fedora Linux 35 – KDE Plasma spin
- My laptop from Framework running Windows 11
- TrueNAS Scale Beta Server – Based off of Debian 10 Buster
- Proxmox 7.4 Server – Based off of Debian 11 Bullseye
- Raspberry Pi running Raspberry Pi OS Bullseye 64 bit serving as a remote backup server
- Ubuntu Server 20.04 on a Linode server that’s used as a project server
- The server running this website using WordPress on a Linode server
Furthermore, I had various client computers and servers I needed to get upgraded immediately, but due to privacy concerns I’m electing to not share their hardware and software setup.
Because most of my hardware had at one point used Java, I first needed to check to see if Java was installed and if it was determine what version it was.
$ java -versionThis revealed a lot of information. If I had Java installed, the version that was used most of the time was a relatively recent patch of 1.8. Ideally, I should’ve been running 17 regardless. For my Linux based hardware, I needed to remove any variant of Java potentially installed.
# apt-get purge *java*
# dnf remove *java*As for Windows, I simply opened up Settings -> Apps and search for Java and uninstalled anything that came up. Furthermore, I ended up ensuring I had the latest versions of Visual Studio Code and Eclipse installed because, even though they used the development kit of whatever language you want to use, I didn’t want to take any chances.
Step 2: Forcing Updates to be Installed
For programs I was unsure of what language programs were using, I forced updates to be checked on everything. I’ve been using winget on Windows since it was first announced as a first-party package manager for Windows. While it’s still flawed to this day, it covers well over 90% of my programs that I use.
PS C:\Windows\System32> winget update --allWinget was able to update most of the programs that needed to be updated. The ones that weren’t covered I went and manually installed the latest version. Linux machines, on the other hand, were significantly easier to update due to the nature of how programs are installed.
# apt-get update
# apt-get upgrade -y
# dnf update -yI ran apt-get on Debian and Ubuntu based servers and dnf on Fedora systems to make sure I was fully updated.
Step 3: Installing Java
Due to the nature of the issue, I repeatedly checked for updates over the span of a week, as multiple different variants of this issue came up after the root bug was discovered. By about late December, I had decided that I was updated enough. I ended up installing the latest version of Java JDK 17 that previously had and was known to need Java to ensure I had the least bug prone version.
# apt-get install java-jdk-latest
# dnf install java-openjdk-latestAs for Windows being concerned, I ended up going to Oracles website and downloading the latest version of the Java 17 JDK.
What Took So Long?
This has been sitting in my drafts since the issue was discovered and I even started writing it as I was taking care of security issues. It led to a philosophical question: is security through obscurity good security or masking bad security? Ultimately I decided that it can be a good measure, but more often than not it shows that you’re not confident in your security measures. I also publish all of my works to GitHub, so for me to not have my efforts publicized and available to others shows a lack of confidence in my efforts. Furthermore, another question came up: was my mitigation effective? Or did I uninstall and reinstall Java for no reason? Aside from the peace of mind, I have noticed that several of my programs that I use regularly had Log4j patches pushed, including the core Java program itself.
Conclusion
I learned from this that it can be easy to become complacent with your updates, especially with zero-day vulnerabilities like this. Furthermore, this tested my remote management skills to ensure I was even able to still make connections to my machines regardless of where I was.