Finding Log4j instances in the runtime environment and tracking completed fixes at a Fortune 100 company
Time is a funny thing. It’s hard to believe that it’s already been a little over a month since Log4Shell, a zero-day vulnerability in the Java Log4j logging tool, was publicly disclosed on December 9, 2021. The next day I was contacted by one of our clients, a Fortune 100 company, for assistance in finding and remediating Log4j instances among the millions of assets it manages. At the start of the crisis, they estimated that it would take 2-3 months to discover instances of Log4j in their environment, and several additional months to remediate vulnerable instances.
I am pleased to share that with the help of the Balbix platform, the team has significantly reduced this time frame from months to weeks and will adhere to their internal mandate that all identified instances be fixed by the 20th January. Their very relieved Application Security Manager in charge of the project said confidently, “I should have some free time on January 20.”
I am writing to highlight the ongoing challenges security teams face in discovering and remediating vulnerabilities in their environment. I also want to share how the Balbix platform provides automation to help security teams overcome these challenges in general, but also how some of our specific features were able to help this client (and others) reduce their risk of Log4j vulnerabilities. To quote our client, “we need tools like Balbix that can look at devices but also see into the software stack.”
Launch of the initial search
As any security specialist familiar with Log4j knows, finding Log4j instances is complicated by the fact that it may exist as (1) library resource, (2) JAR file in a third-party application, or ( 3) Integrated into a custom application. When Log4Shell was released to the public, our client identified 2700 instances of Log4j that resided as an independent application (not a JAR file) in their environment. Within hours, it was determined that none were vulnerable to the exploit. Good news! However, rather than feeling reassured, the team was skeptical that their search was over. They assumed they had vulnerable instances elsewhere, an assumption that would quickly prove correct.
The team came up with a two-part plan to find the hidden instances. They would run their network scanner and use a scripting mechanism they had previously installed on all servers to scan their drives for the Log4j name or file hash to identify the presence of Log4J.
Unfortunately, the results of the network analysis proved tedious to use. A quick review showed an equal number of false positives and real instances of Log4j. The parsing mechanism was also problematic because it took too long. For example, a server scan was only 40% complete after 48 hours of execution. Additionally, despite running the search at an appropriate “good value” to avoid affecting other compute-intensive resources, the security team received complaints from IT and users. The security team estimated that it would take 2-3 months depending on these tools to discover their remaining Log4j instances.
Rapid discovery of instances using Balbix to examine running processes
In the meantime, our Balbix team has set out to help the client find running instances of Log4j. We already had a huge amount of information about the client’s environment, as providing a continuously updated asset inventory is a fundamental part of the Balbix platform.
We had most of the information we needed. Since Balbix determines asset types, we knew which devices were servers. For these servers, we also knew:
- what software was deployed
- the asset owner
- the person responsible for IT change management
- and hundreds of other attributes.
What was missing was additional fidelity regarding third-party applications running on these servers.
Our engineering team was able to make some quick improvements to collect this data. For example, we ingested data on their Linux systems using (1) RPM package manager for Red Hat distributions and (2) Debian package manager for Ubuntu and some other Linux distributions, to discover Log4j packages dependent. To detect cases where Log4j was not installed via package managers, as is often the case with custom applications, Balbix successfully used live process analysis to analyze libraries, then correlate and deduplicate the results. The information we collected included the affected application, the version of the application, where it was installed, and the full path and version of the Log4j components.
Within hours, Balbix was able to begin identifying new Log4j instances, ultimately identifying Log4j embedded in over 1,000 distinct application versions across more than 60,000 assets, including 450 application versions across approximately 5,000 critical assets. The security team told us that “Balbix data is highly accurate and actionable – 4x to 8x better than information obtained from other sources.” In addition, the false positive rate was less than 0.1%.
Rapid prioritization and correction
Balbix was then able to leverage other attributes we had pre-mapped for the servers to provide the business context of the affected assets in order to prioritize remediation. As a result, our client’s security team quickly prioritized remediation actions based on factors such as:
- If the server was business critical
- Whether on the Internet
- If it contained customer data
With a list of assets to be patched in hand, the application security manager at the customer then normally contacts each application owner to ask them to deploy the required patches. To determine if the task is complete, the AppSec team will then need to continue contacting the app owner to ask if the update is complete, or they will need to run a scanner. It would be a laborious and iterative process.
Instead, they were able to save a lot of effort by tracking mitigated apps in Balbix, as Balbix continuously updates the status of each asset, including software version and Log4j mitigations (including changes environmental variables or removing classes from the classpath). The team still needed to follow up with app owners who hadn’t completed the remediation in a timely manner, but the to-do list was much shorter and made them feel confident about meeting their mandatory deadline. resolution of all identified instances by January 17.
For each instance, Balbix has also provided the appropriate patch version for each instance within the platform (see summary table below). The client noted that “the fact that we are able to detect mitigations is the icing on the cake, which helps us reduce the cost of communicating with thousands of application teams.”
The inventory also included a list of known vulnerabilities for each asset. This data proved particularly valuable as new Log4j CVEs were identified within days of the first disclosure. There was no need to run the scans again to identify new vulnerabilities published by Apache. Application owners were able to combine their response into a single set of actions to address all Log4j CVEs, and the security team was able to track all Log4j CVEs in a single report.
Post mortem: advantages
Looking back, the most significant benefit our client saw was the ability to reduce time. By having an accurate inventory of assets in place at the time of the Log4Shell notification, the team was able to work with Balbix to quickly discover Log4J instances and track completed updates. Having asset-level detail also enhanced the value of their network scanners when used in combination with Balbix. As noted by our client, “Accuracy, specificity, and actionability were the key factors in tracking this vulnerability and remediation/mitigation actions. With the help of Balbix, [we] Reduces end-to-end risk mitigation work time from months to weeks.
The team was also able to reduce the skill level of responders. With much of the discovery and tracking done through automation, senior team members were able to delegate most of the remaining tasks to their junior teammates.
Overall, the response to Log4Shell highlighted the benefits of automating CVE management – discovery, prioritization and remediation. Specifically, it showed the power to correlate all relevant information. The Log4j remediation prioritization effort would have been considerably more difficult if some of the attributes of the assets were not available. For example, what would the team have done if the inventory did not list servers connected to the Internet?
The team used the analogy of a fishing boat on the water to emphasize the importance of their better visibility. Many tools can tell them what lies on the surface of the water, they said, but only Balbix could show them what lived below the surface. Continuing the metaphor, they reasoned that Balbix could identify every fish, every octopus, all the way to the bottom.
Finally, the Log4Shell incident reinforced the fact that security posture management is an area of executive visibility. As part of their response to Log4Shell, the team had to provide a daily report to the CISO, and the CISO, in turn, had to report to the CTO. With this kind of visibility, vulnerability management reporting is one part of cybersecurity you definitely want to master. As we complete the Log4j project, our client has confirmed that reporting is another area where Balbix can provide key support: “We will be leaning heavily on Balbix when the teams come back to full strength. I want to make sure I don’t end up in front of the headlights while we work out our communications with the officers.
