How to Secure Your Software Supply Chain – Practical Lessons To Protect Your App

Feross Aboukhadijeh at UXDX APAC. Video: https://youtu.be/XG2xI896oRs

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

Introduction

[00:00:00] Hello and welcome to my talk, How to Secure Your Software Supply Chain: Practical Lessons to Protect Your App.

[00:00:09] My name is Feross and I'm the founder and CEO of Socket, an open source supply chain security company. I'm also an instructor at Stanford University where I teach the web security course, as well as the creator of hundreds of open source projects which are downloaded 500 million times per month, including Standard JS, a popular community JavaScript style guide, and WebTorrent, the first browser BitTorrent client. In the past I also used to be a board member of the Node.js Foundation, as well as a consultant for the Silicon Valley TV show, which was a pretty fun job.

[00:00:42] A little bit about my company, Socket. Socket is a cybersecurity platform that protects companies from software supply chain attacks. We're auditing every open source package to detect supply chain attacks such as malware, typosquats, hidden code and misleading packages, and to block it in real time. Customers include thousands of top tech organizations.

The ua-parser-js story

[00:01:07] Now let's get started. Let me tell you a story. On January 13th 2012, over 10 years ago, a developer named Faisal Salman published a new project to GitHub. It was called ua-parser-js. It was a user agent string parser. Lots of people found it useful, and over the next 10 years Faisal continued to develop the package with help from many open source contributors, publishing 54 versions. The package just continued to grow in popularity. Eventually it grew to 7 million downloads per week and it was used by over 3 million GitHub repositories.

[00:01:48] Let me tell you a story. On October 5th 2021, on a notorious Russian hacking forum, this post appeared. A hacker was selling the password to an npm account that controlled a package with over 7 million weekly downloads. If you hadn't made the connection, that's the same package as before.

[00:02:07] So two weeks later ua-parser-js was compromised and three malicious versions of this package were published. Malware was added that would execute immediately whenever anyone installed one of these malicious versions.

[00:02:23] What I want to do now is actually jump into that malware and give you a peek to see what it did. This is the package.json file that makes up the package, and I'll draw your attention to the new version here, as well as the pre-install script on this line. This script is just simply running a file called preinstall.js, which is where the malware lives. So if you open up that file, this is what it looks like.

[00:02:48] The first thing you'll notice is that different code is running on each platform, so Mac, Windows and Linux have different code paths there. Mac users are lucky and nothing actually happens on that platform. But on Windows this file is executed here, preinstall.bat, and on Linux a similar file, preinstall.sh, is executed. So let's open up those files and dig in.

[00:03:12] The first thing you'll notice is that the user's country of origin is fetched based on their IP address, and if they come from Russia, Ukraine, Belarus or Kazakhstan, then the program actually just exits and nothing happens. This is actually pretty common in malware and it's done presumably because the attackers actually live in one of these countries and they're attempting to not antagonize their local law enforcement.

[00:03:41] If you're not in one of these countries the malware continues, and you'll see here it's looking to see, using pgrep, whether the application is actually running. If it is, then it'll end, otherwise it'll actually download the malware and then prepare it for execution and execute it. If you look at the arguments here you'll notice this is actually a cryptocurrency miner, specifically the Monero miner. What this is going to do is use up the resources on whatever machine is unlucky enough to install this, whether it's your personal laptop, your build server, or maybe even your production server.

[00:04:20] On Windows the malware is very similar. It follows a very similar process of downloading this miner, but it also downloads an additional file here, this DLL file. After starting up the miner it will also register that DLL, which turns out to be pretty nefarious. That particular file will actually steal passwords from 100 different programs on the machine, as well as anything in the Windows credential manager, which includes the user's passwords.

[00:04:51] So this is pretty bad and obviously not something you want to run on your machine. The aftermath of this was that the maintainer apologized and sent a message to npm support to get this package taken down. It was published for a total of four hours, and anyone who installed the package during this time period was compromised. That means any software builds that were done in projects were compromised, anyone who updated to this version or merged a pull request to this new version would have been compromised. So, a very unfortunate incident.

[00:05:26] This was big news in the JavaScript world. You might have even heard about this back in October. A lot of libraries that are used by big companies, created by big companies like Facebook, were actually affected because they depended upon ua-parser-js.

[00:05:42] This is actually just the tip of the iceberg though. This one incident drew a lot of attention to this issue in October, but there's been many incidents since. In November there was another similar incident with a cryptocurrency miner. In January there was an incident with a maintainer sabotaging his own packages. In March and April there have been incidents with protestware, which is basically people sabotaging packages in order to make political statements. So this is really just the tip of the iceberg. We've seen 150 different packages removed for security reasons in just the last 30 days alone. This trend seems to be accelerating and attackers are really taking advantage of the trust in the open source ecosystem.

Why is this happening now?

[00:06:24] One question you might ask is, why is this happening now? I think it comes down to really four reasons.

[00:06:31] The first reason is that 90% of an app's code comes from open source. Open source has enabled teams to build powerful applications in days or weeks instead of months or years. Open source works because anyone can inspect the code, anyone can contribute, anyone can publish a package, and open source communities are trusting by default. Good contributors are rewarded with recognition and eventually publish rights. This is also the reason why your npm node modules folder is often one of the heaviest objects in the universe.

[00:07:11] The other reason is we have lots of transitive dependencies. The way we write software has really changed in the last decade. We use dependencies a lot more liberally, and that leads to thousands of dependencies in most projects. Let's just take one example here. Let's look at Discord, a popular chat application. Discord is an Electron app and it's built on a massive amount of open source. As you can see here, it's around 19,000 total packages, with code contributions from 300,000 different contributors from 206 different countries. So just a mind-boggling amount of open source in an app like Discord.

[00:07:51] This is aided by the way that the packages are written in modern times. A 2019 paper found that installing an average npm package introduces an implicit trust on 79 third-party packages and 39 maintainers, creating a really surprisingly large attack surface. We actually at Socket made this visualization here to kind of give you a sense for what a dependency actually looks like. This is the webpack dependency, that's very, very common, it's probably building your front end server right now if you have one. What you see here is, every gray box is a package and every purple box is a file within a package. What you'll see here is we're sort of peeling back the layers, taking off the gray, opening up the gray box to sort of see the files inside and also the nested packages within that. As you can see here, there's just a lot of code and a lot of different packages to make up webpack.

[00:08:53] The other reason is, no one reads the code today. Basically what we're doing is kind of crazy: we're downloading code from the internet written by unknown individuals, that we haven't read, that we execute with full permissions on our laptops and servers where we keep our most important data. When you think of it like that, it's actually a miracle that the system works at all and that it's continued to work for so long.

[00:09:22] Another reason why people don't read code is that npm doesn't make it easy. If you open up the website, you just go to this Explore tab here, you can't actually even see the code that you're going to be downloading when you're selecting a package. So developers actually have a pretty hard time making good decisions and actually reading the code here. They often have to resort to clicking on the GitHub link and going to GitHub to read the code.

[00:09:48] But attackers can publish different code to npm and GitHub. Often the code is different in the associated GitHub repository, and attackers actually know this and will take advantage of this fact. npm does not guarantee that the code on GitHub matches the code on npm.

[00:10:06] So no one is looking at the code. You might think that's okay. Linus Torvalds has this quote about how, given enough eyeballs, all bugs are shallow. To some extent this is true. But if everyone is relying on someone else to read the code, then who is finding the malware?

[00:10:30] So this is maybe why, on average, a malicious package is available for 209 days before it's publicly reported. This is from a 2020 research paper, and I personally find this number very shocking. Another paper in 2021 found similar results, including that 20% of malware persists in package managers for over 400 days and has more than 1,000 downloads.

[00:10:57] Finally, the last reason why this is happening now is that popular tools give a false sense of security. It's very common to use vulnerability scanning tools to tell you whether open source is safe or not, but scanning for known vulnerabilities is not enough. The entire security industry is obsessed with scanning for known vulnerabilities, an approach which is too reactive to stop an active supply chain attack. Vulnerabilities can take weeks or months to be discovered, and in today's culture of fast development a malicious dependency can be updated, merged and running in production in days or even hours. This really isn't enough time for a vulnerability report to be created and to make its way into the vulnerability scanning tools that a lot of teams use.

[00:11:42] So really what it comes down to is, supply chain attacks and vulnerabilities are very different and they need different solutions. Vulnerabilities are accidentally introduced by open source maintainers, and it's sometimes okay to ship these to production if they're low impact. Supply chain attacks on the other hand are intentionally introduced by an attacker, and it's never okay to ship such code to production. It's never okay to ship malware to production. You must catch it before you install it or depend on it in your application. So vulnerability scanners will not catch the next supply chain attack.

Attack vectors: how the attacker tricks you

[00:12:18] Now that we kind of understand why supply chain attacks are happening now, let's dig a little bit into how a supply chain attack actually works. There's kind of two types of things I want to talk about. There's attack vectors as well as attack tactics. Vectors are how the attacker tricks you, and tactics are what the code actually does when it runs. So let's start with attack vectors, how the attacker tricks you.

[00:12:48] The first and most common attack technique is hijacked packages. These are often the source of a lot of the headlines that you see in the news around supply chain attacks and open source packages that have been compromised. This is the same thing that happened in the example that we started this talk with, ua-parser-js.

[00:13:09] Packages get hijacked for a number of reasons. It could be that the maintainer chose a weak password, that's what happened in the case of ua-parser-js. It could be the maintainer themselves gives access to a malicious actor by mistake. Maintainers can become malicious themselves, that actually happened in January. Maintainers can also use their packages to protest. And maintainers could also just get malware on their laptops. All of this is exacerbated by the fact that npm doesn't enforce 2FA, though this is starting to somewhat improve in recent times.

[00:13:46] The next tactic is typosquatting. Typosquatting is a pretty nefarious trick. If you look at these two packages here, one of them is real and one of them is fake, but you might be hard-pressed to guess which is real and which is fake. I'll just tell you: the first one is real, the second one's fake. If you were to make the mistake of installing the fake package, you would get greeted with a nice supply chain attack.

[00:14:14] What you have here is the contents of the fake package, and as you'll note, the line here says that a script is going to run automatically. If we open up that file to see what's inside, we're greeted with a completely obfuscated file. Even without being able to decipher what this code does, it's very clear this is not something that you should be running on your machine or on production. This is not going to do anything that you want to do.

[00:14:43] So let's talk about the next tactic, dependency confusion. Dependency confusion is closely related to typosquatting, but instead of relying on the user making a mistake about the specific dependency that they install, this attack works when a company publishes packages to their own internal private npm registry and they use a name that hasn't been registered on the public registry. An attacker can come along and register a package with the same name but on the public registry, and then later some internal tools may get confused and use the public version of the package instead of the internal version. So that's why it's called the dependency confusion attack.

[00:15:27] There's been plenty of examples of this recently, many companies affected. Just looking through the recently deleted npm packages, we found a bunch of likely dependency confusion attacks. Most of these had malicious code in them, and all these packages have names which appear to conflict with likely internal package names. So you can see here all kinds of organizations were affected, including really large companies, the federal government, and there's even more on this page here. So this is really quite a widespread problem.

[00:16:00] This also isn't helped by the fact that npm recently had an issue where they lost data on these private package names and leaked them, and so that made attackers' jobs even easier to pull off this attack, because they could just pull a list.

Tactics: what the attack code does

[00:16:18] All right, so now let's talk about tactics. What does the attack code do? Once this code's running on your machine, what's it actually going to do? The most common thing by far is install scripts. Most malware is in an install script. This is because, if you're an attacker, why wouldn't you want your code to run automatically the moment that the user installs the package? A 2022 paper found almost 94% of malicious packages had at least one install script, and unfortunately install scripts have legitimate uses, so it's not an easy solution to just disable them. We've seen already many examples of this, but this single line here is enough to cause malware to automatically execute.

[00:17:02] Now let's talk about the second tactic: stealing data. This is very common in a lot of the malware that we see on npm. If we take that last example again and we dive into these scripts here that it's attempting to run, this is a very, very frequent example of a piece of malware that you'll see on npm. You'll see it's making an HTTP request, and the data that it's sending from your system is going to this domain here, and the data that's actually being sent is process.env, which is your environment variables. That means your tokens, your keys, whatever environment variables are in the environment are going to get exfiltrated by this script here.

[00:17:46] There's different techniques that attackers like to use. Sometimes an HTTP request can get blocked by a firewall, and so there's also a DNS technique. This uses a DNS lookup to exfiltrate the data, it puts it into the subdomain of the URL and it sends the environment variables off to the attacker. The attacker can of course use this data however they like. They can post it online, they can use it to break into your systems, or whatever they feel like.

[00:18:14] Finally, the last tactic is to delete or to ransom your data. Some examples of this are the recent protestware that we've seen, peacenotwar and node-ipc. We'll actually take a look at that example here. This is a piece of malware written by some activists who wanted to make a statement about the war in Ukraine, and so they included this code in their package. If you take a close look at it, it's a little bit obfuscated here, but the key line is down here where it's basically going to iterate through all the files on your machine and write garbage data over them, so effectively deleting all your files. So, not a friendly piece of software, not something that you want to be running on any machine that's yours.

How to protect your app: choose better dependencies

[00:19:03] So how can you protect your app? This is where we talk about actual potential solutions to these problems. The first thing you can do is to choose better dependencies. If you ship code to production, you are responsible for it.

[00:19:21] We really need a mindset shift as developers. Node, or the process that's running your code, doesn't care whether that code was written by you as the developer or whether it came from open source. This mindset of, it's not my problem because it's open source, needs to change, because ultimately it's all going to run in the same program and Node doesn't care who wrote the code. It's all part of your application once it's bundled up and running in production. So really we need a mindset shift around this.

[00:19:52] If you look at the actual open source licenses that are part of all the open source packages that you consume, the most popular license, the MIT license, literally says this. It says open source is provided as is, without warranty of any kind, in no event shall the author be liable for any claim, damages or liability. This is actually the truth. You're ultimately responsible for the open source code that you've taken, that you run on your production servers.

[00:20:20] So we need to do a better job picking dependencies. How do we do this? Most of us aren't going to look at the actual code, so we look at heuristics, we look at signals that give us an indication as to whether an open source package is quality and trustworthy. So we look at things like, first of all, does it get the job done? Does it have an open source license so we can use it? Does it have good docs? Does it have a lot of downloads and GitHub stars? Does it have recent commits? Does it have types? And does it have tests?

[00:20:52] This is good stuff to look at, but it doesn't really dig deep into the package and it's not really going to give you an indication about whether a package has malware in it or not. Most of the time this is going to work, but occasionally, in those rare cases where a package is compromised that you've been depending on, it might check all these boxes but you still could get affected. So we really need to dig a little deeper.

[00:21:15] I'm going to show you an example of a tool. This is the tool that we build at Socket that can actually dig into the contents of a package and tell you what it does. There's probably other tools out there, but this is a good example of what you can do with really good static analysis. So here we have a package, bufferutil, and you can see that it's going to run code automatically on installation and it's going to run some native code. You can see that right there on the package page.

[00:21:40] This package actually turns out to be totally benign, but what about something sketchy? What about a package that's doing something kind of weird, what would that look like? So here's a package, it's quite popular, and it puts a little overlay on the page, it's a React-like overlay component, so it's just a web component. But if you dig into its dependencies you'll find that it's actually doing quite a lot of interesting things. You'll see here that it has code that runs automatically, it has telemetry, which means it's sending tracking code back to the maintainer, so it's tracking who installs the package, and it does a bunch of other things like run shell commands and access the network. So this is worth digging into a little bit further, and if you click in here you can learn a little bit more about what this package is actually doing.

[00:22:27] Now let's take a look at an actual piece of malware. Here's a piece of malware. If you were to look it up on Socket you'd see that we've detected that it's going to run code on installation and that it accesses the network. If you click into the alerts there you'll actually get jumped straight to the line of code where it actually does the given behavior. You'll see here that the package is accessing environment variables on this line here, and you'll see here it's just sending it off to some server on the internet. So, typical kind of data theft attack. So make sure to take a close look at the dependencies that you use.

Update cadence and automated auditing

[00:23:02] The next tip is to update dependencies at the right cadence. A lot of us are using bots like Dependabot to stay on the latest version of code, and that's usually a good security practice. However, the quicker that you update your dependencies, the fewer eyeballs that have had a chance to actually look at the code. So there's really a trade-off here. How quickly should you update?

[00:23:32] This is a really tough question. If you update too slowly then you're exposed to known vulnerabilities, and that's not good. If you update too quickly though, you're exposed to supply chain attacks, because now you're running code that no eyeballs have looked at, no one has seen yet. So there's no good answer here, and there's really just trade-offs all around. But this is something to at least think about as a team and kind of come up with a policy for what you want to do here.

[00:24:05] Okay, so the third thing you can do: use automation to audit every dependency. How closely should you actually audit a dependency? Again there's kind of a trade-off here. You can do a full audit. Full audit means read every single line of code of every dependency in your project. If you do this it's thorough, it's the best-in-class thing you can do, but it's a lot of work. It's also slow and time-consuming and therefore expensive, because it takes a lot of time to properly audit that much code.

[00:24:37] On the other hand, a lot of teams are doing nothing, and if you do that then you're vulnerable to supply chain attacks. You're just running code that you haven't even audited or looked at in any way and hoping that it's good. So this is risky, and it can be expensive in a different way. It could be expensive in terms of PR cost to the company, or in terms of breaches.

[00:24:56] So really the happy medium here is to lean on automation. What we recommend is using static analysis to audit every dependency and detect indicators of packages doing suspicious things, such as using privileged APIs like the file system or the network, or containing obfuscated code. If you detect packages doing these things then you can manually audit just those packages, just the most suspicious packages, and that way you can spend security team resources on the highest impact tasks instead of doing it all or nothing. And if you do this and you put the security information directly in pull requests so that developers can see it, then you empower developers to solve security issues before they're deployed into production, and you can find these attacks before they affect your users.

Socket, and a mindset shift

[00:25:51] So our final recommendation, and I know I'm biased here, is to consider using Socket to do this. You can use the tool, as we showed earlier, to research packages. This is a free tool, free to the community, anyone can use this. You can look up packages and all this data is open and accessible. Feel free to do this when you're considering which packages to use, or when you're auditing an application.

[00:26:16] But we also have the ability to monitor pull requests for bad dependencies. This gives you an active way to monitor for bad dependencies before they're merged in. So this is what it looks like. It's a bot, it will come to your pull request and leave a comment telling you what issues are present in this dependency, and give the developer that information that they need to really make an informed decision about this dependency. If you're interested in this you can install it for free at socket.dev.

[00:26:50] I want to end on a kind of positive note here and just say that I'm hopeful[?] for supply chain security. I think developers and security practitioners and really entire teams need to change the way they think about dependencies in order to solve the software supply chain security crisis that we're in right now. We need a mindset shift around dependencies and we need to think of them as part of our apps, and they are part of our apps. We need to really understand the threats that are out there and what steps we can take as teams to protect ourselves from this emerging threat.

[00:27:28] And with that I'll end. Thank you for having me here, I appreciate it. My contact info is here on the slide if you're interested in reaching out, and also just a shout out that we're hiring developers at Socket. So, thanks.