Last year (2021) was a tough year for the software supply chain. And in December, the year’s parting gift was the discovery of a critical, zero-day vulnerability in Apache Log4j. Log4j is a popular open source logging library integrated into Apache Struts 2, Solr, Druid and Flink, all of which are used in innumerable commercial applications. As news of the vulnerability broke, attackers immediately began exploiting the Log4j vulnerability, which allows unauthenticated remote code execution (no credentials required) to gain control of vulnerable servers.
In this interview, Mike Manrod, CISO, and Christian Taillon, IT security engineer at Grand Canyon Education explain why this vulnerability is so dangerous and how Software Bills of Materials (SBOMs) can reduce risks from open source components to better protect the software supply chain.
Q: Can you describe some of the exploits occurring through the Log4j vulnerability reported in December?
Christian: Log4j’s jndi lookup functionality is intended to resolve information in records to make logs easier to use. Researchers found a way to exploit that functionality to run their own code in the environment. They realized that attackers could use it to redirect exploited systems to share and disclose sensitive information about the environment such as passwords and tokens. More interesting is the ability to execute code against a server and instruct it to do whatever you want it to do, such as load a crypto miner, or deploy a backdoor and sell the infected servers on the dark web. Ransomware groups, in particular, are already exploiting this vulnerability.
Q: What makes this vulnerability particularly noteworthy?
Mike: A remote execution vulnerability is much more difficult to detect and patch when it’s related to a software component of many products, as opposed to a single piece of software, particularly if it’s open source. A component can be in practically any software and in many different forms and versions, which makes it difficult to detect and patch.
Q: Who, in your mind, bears responsibility for losses due to Log4j exploits that occur through vulnerable third-party applications?
Christian: Everyone involved has responsibility, starting with the developer who may be working free on weekends creating this open source package that is getting thousands or millions of downloads a week. There’s an implied ‘ought’ in there to take necessary steps to develop securely. There is no liability or required responsibility on the part of developer to legally secure the code in the shared software licenses, however. In the Apache license agreements, the developer isn’t even responsible for creating a patch and the software is offered ‘as is.’
Mike: Most of the responsibility falls on third-party software vendors that are using free, open source components from shared libraries, integrating them with 100 different open source components in their product, and then they’re selling that product for money. They [third-party software vendors] have the biggest responsibility to make sure that all the pieces they put together for their product are documented, secure, updated and patched.
In the case of Log4j, I was horrified at how terrible the support from one of our third-party vendors was when we were trying to fix this right before Christmas. They missed some of the versions of Log4j in their own products, and even suggested we rip out Log4j but they couldn’t tell us if it would break the products we were using.
Q: Let’s talk about controls around the code libraries, especially when you also consider the Java developer who purposefully sabotaged his own apps in his popular shared library.
Mike: If I were a bad guy right now, I’d chart the most common libraries and components downloaded most frequently. Then I’d look for vulnerabilities such as remote code execution. We know this is already happening to some degree and we can expect to see more of it.
Christian: The good news is that there are a lot of efforts around securing the libraries and packages that developers rely on. One of the companies leading the charge is GitHub, which has a utility called Dependabot that looks at a project’s dependencies and presents the developer with information about vulnerabilities that are inherited from their currently imported libraries or packages.
Q: This type of tracking sounds similar to the Software Bill of Materials (SBOM). How are SBOM reports aggregated and actualized into better, more reliable code across the supply chain?
Christian: To support the supply chain, SBOMs can be used to track vulnerabilities and map code dependencies. Then solutions need to take these dependences and aggregate the repair options. For example, out of 23 vulnerabilities, 18 can be fixed by upgrading to the latest version, saving the developer time sifting through 50 libraries to find this information. That makes updating commercial software or adding a new feature easier, more secure, and less costly because you know the vulnerabilities and can fix them before you go into production. Code and application scanners should work alongside the developers as they prepare to make new pull requests in support of the CI/CD pipeline.
