How to Secure Your Software Supply Chain – Practical Lessons To Protect Your App

Feross Aboukhadijeh at UXDX USA. Video: https://www.youtube.com/watch?v=CalQC0oa9o8

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

Introduction

[00:00:00] Hello, and welcome to my talk, "How to Secure Your Software Supply Chain: Practical Lessons to Protect Your App." My name is Feross, and I'm the founder and CEO of Socket, an open source supply chain security company. I'm also an instructor at Stanford University, where I teach the web security course, as well as the creator of hundreds of open source projects, which are downloaded 500 million times per month, including StandardJS, a popular community JavaScript style guide, and WebTorrent, the first browser BitTorrent client. In the past I also used to be a board member of the Node.js Foundation, as well as a consultant for the Silicon Valley TV show, which was a pretty fun job.

[00:00:44] A little bit about my company, Socket. Socket is a cybersecurity platform that protects companies from software supply chain attacks. We're auditing every open source package to detect supply chain attacks, such as malware, typosquats, hidden code and misleading packages, and to block them in real time. Customers include thousands of top tech organizations. Now let's get started.

The story of ua-parser-js

[00:01:09] Let me tell you a story. On January 13th, 2012, over 10 years ago, a developer named Faisal Salman published a new project to GitHub. It was called ua-parser-js. It was a user agent string parser, and lots of people found it useful. Over the next 10 years, Faisal continued to develop the package with help from many open source contributors, publishing 54 versions. The package just continued to grow in popularity. Eventually it grew to 7 million downloads per week, and it was used by over 3 million GitHub repositories.

[00:01:48] Let me tell you another story. On October 5th, 2021, on a notorious Russian hacking forum, this post appeared. A hacker was selling the password to an npm account that controlled a package with over 7 million weekly downloads. If you hadn't made the connection, that's the same package as before. Two weeks later, ua-parser-js was compromised, and three malicious versions of this package were published. Malware was added that would execute immediately whenever anyone installed one of these malicious versions.

[00:02:23] What I want to do now is jump into that malware and give you a peek at what it did. This is the package.json file that makes up the package, and I'll draw your attention to the new version here, as well as the preinstall script on this line. This script is simply running a file called preinstall.js, which is where the malware lives. If you open up that file, this is what it looks like. The first thing you'll notice is that different code runs on each platform, so Mac, Windows and Linux have different code paths. Mac users are lucky, and nothing actually happens on that platform. But on Windows this file, preinstall.bat, is executed, and on Linux a similar file, preinstall.sh, is executed. So let's open up those files and dig in.

[00:03:12] The first thing you'll notice is that the user's country of origin is fetched based on their IP address, and if they come from Russia, Ukraine, Belarus or Kazakhstan, then the program just exits and nothing happens. This is actually pretty common in malware, and it's done presumably because the attackers live in one of these countries and are attempting not to antagonize their local law enforcement. If you're not in one of these countries, the malware continues, and you'll see here it's using pgrep to check whether the application is already running. If it is, it'll end. Otherwise it'll download the malware, prepare it for execution, and execute it.

[00:03:59] If you look at the arguments here, you'll notice this is a cryptocurrency miner, specifically the Monero miner. What this is going to do is use up the resources on whatever machine is unlucky enough to install it, whether it's your personal laptop, your build server, or maybe even your production server.

[00:04:20] On Windows the malware is very similar. It follows a very similar process of downloading this miner, but it also downloads an additional file here, this DLL file, and after starting up the miner it will also register that DLL, which turns out to be pretty nefarious. That particular file will actually steal passwords from 100 different programs on the machine, as well as anything in the Windows Credential Manager, which includes the user's passwords. So this is pretty bad, and obviously not something you want to run on your machine.

[00:04:54] The aftermath of this was that the maintainer apologized and sent a message to npm support to get the package taken down. It was published for a total of four hours, and anyone who installed the package during this time period was compromised. That means any software builds that were done in projects were compromised; anyone who updated to this version, or merged a pull request to this new version, would have been compromised. A very unfortunate incident.

[00:05:26] This was big news in the JavaScript world. You might even have heard about it back in October. A lot of libraries that are used by big companies, and created by big companies like Facebook, were actually affected, because they depended upon ua-parser-js.

[00:05:42] This is actually just the tip of the iceberg, though. This one incident drew a lot of attention to the issue in October, but there have been many incidents since. In November there was another similar incident with a cryptocurrency miner. In January there was an incident with a maintainer sabotaging his own packages. In March and April there have been incidents with protestware, which is basically people sabotaging packages in order to make political statements. So this is really just the tip of the iceberg. We've seen 150 different packages removed for security reasons in just the last 30 days alone. This trend seems to be accelerating, and attackers are really taking advantage of the trust in the open source ecosystem.

Why this is happening now

[00:06:24] One question you might ask is, why is this happening now? I think it comes down to four reasons. The first reason is that 90% of an app's code comes from open source. Open source has won. Open source has enabled teams to build powerful applications in days or weeks instead of months or years. Open source works because anyone can inspect the code, anyone can contribute, anyone can publish a package. Open source communities are trusting by default: good contributors are rewarded with recognition and eventually publish rights. This is also the reason why your node_modules folder is often one of the heaviest objects in the universe.

[00:07:11] The other reason is that we have lots of transitive dependencies. The way we write software has really changed in the last decade. We use dependencies a lot more liberally, and that leads to thousands of dependencies in most projects. Let's take one example: Discord, a popular chat application. Discord is an Electron app, and it's built on a massive amount of open source. As you can see here, it's around 19,000 total packages, with code contributions from 300,000 different contributors from 206 different countries. Just a mind-boggling amount of open source in an app like Discord.

[00:07:49] This is aided by the way packages are written in modern times. A 2019 paper found that installing an average npm package introduces an implicit trust in 79 third-party packages and 39 maintainers, creating a surprisingly large attack surface. At Socket, we made this visualization to give you a sense of what a dependency actually looks like. This is the webpack dependency, which is very, very common. It's probably building your front end right now, if you have one. Every gray box is a package, and every purple box is a file within a package. What you'll see here is that we're peeling back the layers, opening up the gray boxes to see the files inside, and also the nested packages within them. As you can see, there's just a lot of code and a lot of different packages that make up the webpack package.

[00:08:52] The next reason is that no one reads the code today. What we're doing is kind of crazy. We're downloading code from the internet, written by unknown individuals, that we haven't read, and we execute it with full permissions on our laptops and servers, where we keep our most important data. When you think of it like that, it's actually a miracle that the system works at all, and that it's continued to work for so long.

[00:09:25] Another reason why people don't read code is that npm doesn't make it easy. If you open up the website and go to this Explore tab here, you can't even see the code that you're going to be downloading when you're selecting a package. So developers have a pretty hard time making good decisions and actually reading the code. They often have to resort to clicking on the GitHub link and going to GitHub to read the code. But attackers can publish different code to npm and GitHub. Often the code is different in the associated GitHub repository, and attackers know this and will take advantage of it. npm does not guarantee that the code on GitHub matches the code on npm.

[00:10:07] So no one is looking at the code. You might think that's okay. Linus Torvalds has this quote about how, given enough eyeballs, all bugs are shallow, and to some extent this is true. But if everyone is relying on someone else to read the code, then who is finding the malware? Maybe this is why, on average, a malicious package is available for 209 days before it's publicly reported. This is from a 2020 research paper, and I personally find this number very shocking. Another paper in 2021 found similar results, including that 20% of malware persists in package managers for over 400 days and has more than 1,000 downloads.

[00:10:57] Finally, the last reason why this is happening now is that popular tools give a false sense of security. It's very common to use vulnerability scanning tools to tell you whether open source is safe or not, but scanning for known vulnerabilities is not enough. The entire security industry is obsessed with scanning for known vulnerabilities, an approach which is too reactive to stop an active supply chain attack. Vulnerabilities can take weeks or months to be discovered, and in today's culture of fast development, a malicious dependency can be updated, merged and running in production in days or even hours. That really isn't enough time for a vulnerability report to be created and make its way into the vulnerability scanning tools that a lot of teams use.

[00:11:42] What it comes down to is that supply chain attacks and vulnerabilities are very different, and they need different solutions. Vulnerabilities are accidentally introduced by open source maintainers, and it's sometimes okay to ship them to production if they're low impact. Supply chain attacks, on the other hand, are intentionally introduced by an attacker, and it's never okay to ship such code to production. It's never okay to ship malware to production. You must catch it before you install it or depend on it in your application. Vulnerability scanners will not catch the next supply chain attack.

Attack vectors: hijacked packages, typosquatting and dependency confusion

[00:12:18] Now that we understand why supply chain attacks are happening now, let's dig in a little bit to how a supply chain attack actually works. There are two types of things I want to talk about: attack vectors and attack tactics. Vectors are how the attacker tricks you, and tactics are what the code actually does when it runs.

[00:12:43] Let's start with attack vectors, how the attacker tricks you. The first and most common attack technique is hijacked packages. These are often the source of a lot of the headlines that you see in the news around supply chain attacks and open source packages that have been compromised. This is the same thing that happened in the example we started this talk with, ua-parser-js. Packages get hijacked for a number of reasons. It could be that the maintainer chose a weak password; that's what happened in the case of ua-parser-js. It could be that the maintainer gives access to a malicious actor by mistake. Maintainers can become malicious themselves; that actually happened in January. Maintainers can also use their packages to protest. And maintainers could also just get malware on their laptops. All of this is exacerbated by the fact that npm doesn't enforce 2FA, though this is starting to improve somewhat in recent times.

[00:13:45] The next tactic is typosquatting. Typosquatting is a pretty nefarious trick. If you look at these two packages here, one of them is real and one of them is fake, but you might be hard-pressed to guess which is real and which is fake. I'll just tell you: the first one is real, the second one's fake. If you were to make the mistake of installing the fake package, you would be greeted with a nice supply chain attack. What you have here is the contents of the fake package, and as you'll note, the line here says that a script is going to run automatically. If we open up that file to see what's inside, we're greeted with a completely obfuscated file. Even without being able to decipher what this code does, it's very clear this is not something that you should be running on your machine or in production. This is not going to do anything that you want it to do.

[00:14:42] Let's talk about the next tactic, dependency confusion. Dependency confusion is closely related to typosquatting, but instead of relying on the user making a mistake about the specific dependency they install, this attack works when a company publishes packages to its own internal private npm registry and uses a name that hasn't been registered on the public registry. An attacker can come along and register a package with the same name on the public registry, and then later some internal tools may get confused and use the public version of the package instead of the internal version. That's why it's called the dependency confusion attack.

[00:15:27] There have been plenty of examples of this recently, with many companies affected. Just looking through the recently deleted npm packages, we found a bunch of likely dependency confusion attacks. Most of these had malicious code in them, and all of these packages have names which appear to conflict with likely internal package names. You can see here that all kinds of organizations were affected, including really large companies and the federal government, and there are even more on this page. So this is quite a widespread problem. It also isn't helped by the fact that npm recently had an issue where they leaked data on these private package names, which made attackers' jobs even easier, because they could just pull a list.

Attack tactics: install scripts, stealing data and destroying data

[00:16:18] Now let's talk about tactics. What does the attack code do? Once this code's running on your machine, what's it actually going to do? The most common thing by far is install scripts. Most malware is in an install script. If you're an attacker, why wouldn't you want your code to run automatically the moment that the user installs the package? A 2022 paper found that almost 94% of malicious packages had at least one install script. Unfortunately, install scripts have legitimate uses, so it's not an easy solution to just disable them. We've already seen many examples of this, but this single line here is enough to cause malware to execute automatically.

[00:17:02] Now let's talk about the second tactic, stealing data. This is very common in a lot of the malware that we see on npm. If we take that last example again and dive into the scripts it's attempting to run, this is a very, very frequent example of a piece of malware that you'll see on npm. You'll see it's making an HTTP request, and the data that it's sending from your system is going to this domain here. The data that's actually being sent is process.env, which is your environment variables. That means your tokens, your keys, whatever environment variables are in the environment, are going to get exfiltrated by this script.

[00:17:46] There are different techniques that attackers like to use. Sometimes an HTTP request can get blocked by a firewall, so there's also a DNS technique. This uses a DNS lookup to exfiltrate the data: it puts it into the subdomain of the URL, and again sends the environment variables off to the attacker. The attacker can of course use this data however they like. They can post it online, they can use it to break into your systems, or whatever they feel like.

[00:18:14] Finally, the last tactic is to delete or ransom your data. Some examples of this are the recent protestware that we've seen, peacenotwar and node-ipc. We'll take a look at that example here. This is a piece of malware written by some activists who wanted to make a statement about the war in Ukraine, so they included this code in their package. If you take a close look at it, it's a little bit obfuscated, but the key line is down here, where it's going to iterate through all the files on your machine and write garbage data over them, effectively deleting all your files. Not a friendly piece of software, and not something that you want to be running on any machine that's yours.

Choose better dependencies

[00:19:03] So how can you protect your app? This is where we talk about potential solutions to these problems. The first thing you can do is choose better dependencies. If you ship code to production, you are responsible for it. We really need a mindset shift as developers. Node, or whatever process is running your code, doesn't care whether that code was written by you as the developer or whether it came from open source. This mindset of "it's not my problem because it's open source" needs to change, because ultimately it's all going to run in the same program. Node doesn't care who wrote the code. It's all part of your application once it's bundled up and running in production.

[00:19:52] If you look at the actual open source licenses that are part of all the open source packages you consume, the most popular license, the MIT license, literally says this: open source is provided "as is," without warranty of any kind, and in no event shall the author be liable for any claim, damages or liability. This is actually the truth. You're ultimately responsible for the open source code that you've taken and that you run on your production servers. So we need to do a better job picking dependencies.

[00:20:22] How do we do this? Most of us aren't going to look at the actual code, so we look at heuristics, signals that give us an indication as to whether an open source package is quality and trustworthy. We look at things like: first of all, does it get the job done? Does it have an open source license, so we can use it? Does it have good docs? Does it have a lot of downloads and GitHub stars? Does it have recent commits? Does it have types? Does it have tests? This is good stuff to look at, but it doesn't really dig deep into the package, and it's not going to give you an indication of whether a package has malware in it or not. Most of the time this is going to work, but in those rare cases where a package that you've been depending on is compromised, it might check all these boxes and still be affected. So we really need to dig a little deeper.

[00:21:13] I'm going to show you an example of a tool. This is the tool that we built at Socket that can actually dig into the contents of a package and tell you what it does. There are probably other tools out there, but this is a good example of what you can do with really good static analysis. Here we have a package, bufferutil, and you can see that it's going to run code automatically on installation, and it's going to run some native code. We call that out right there on the package page. This package actually turns out to be totally benign.

[00:21:42] But what about something sketchy? What about a package that's doing something kind of weird? What would that look like? Here's a package that's quite popular. It puts a little overlay on the page; it's a React-like overlay component, so it's just a web component. But if you dig into its dependencies, you'll find that it's actually doing quite a lot of interesting things. You'll see here that it has code that runs automatically. It has telemetry, which means it's sending tracking data back to the maintainer, so it's tracking who installs the package. And it does a bunch of other things, like run shell commands and access the network. So this is worth digging into a little further, and if you click in here, you can learn a little bit more about what this package is actually doing.

[00:22:27] Now let's take a look at an actual piece of malware. If you were to look it up on Socket, you'd see that we've detected that it's going to run code on installation and that it accesses the network. If you click into the alerts there, you'll get jumped straight to the line of code where it does the given behavior. You'll see here that the package is accessing environment variables on this line, and here it's just sending them off to some server on the internet. A typical data theft attack. So make sure to take a close look at the dependencies that you use.

Update at the right cadence and audit with automation

[00:23:07] The next tip is to update dependencies at the right cadence. A lot of us are using bots like Dependabot to stay on the latest version of code, and that's usually a good security practice. However, the quicker you update your dependencies, the fewer eyeballs have had a chance to actually look at the code, so there's really a trade-off here. How quickly should you update? This is a really tough question. If you update too slowly, you're exposed to known vulnerabilities, and that's not good. If you update too quickly, though, you're exposed to supply chain attacks, because now you're running code that no eyeballs have looked at, that no one has seen yet. There's no good answer here; there are really just trade-offs all around. But this is something to at least think about as a team, and come up with a policy for what you want to do.

[00:24:05] The third thing you can do is use automation to audit every dependency. How closely should you actually audit a dependency? Again, there's a trade-off here. You can do a full audit. A full audit means reading every single line of code of every dependency in your project. If you do this, it's thorough, it's the best-in-class thing you can do, but it's a lot of work. It's also slow and time-consuming, and therefore expensive, because it takes a lot of time to properly audit that much code.

[00:24:37] On the other hand, a lot of teams are doing nothing, and if you do that, you're vulnerable to supply chain attacks. You're just running code that you haven't audited or looked at in any way, and hoping that it's good. This is risky, and it can be expensive in a different way: in terms of PR cost to the company, or in terms of breaches.

[00:24:56] The happy medium here is to lean on automation. What we recommend is using static analysis to audit every dependency and detect indicators of packages doing suspicious things, such as using privileged APIs like the file system or the network, or containing obfuscated code. If you detect packages doing these things, then you can manually audit just those packages, just the most suspicious ones. That way you can spend security team resources on the highest-impact tasks instead of doing it all or nothing. And if you do this and put the security information directly in pull requests, so that developers can see it, then you empower developers to solve security issues before they're deployed into production, and you can find these attacks before they affect your users.

Using Socket, and a mindset shift

[00:25:51] Our final recommendation, and I know I'm biased here, is to consider using Socket to do this. You can use the tool, as we showed earlier, to research packages. This is a free tool, free to the community. Anyone can use it. You can look up packages, and all this data is open and accessible. Feel free to do this when you're considering which packages to use, or when you're auditing an application. We also have the ability to monitor pull requests for bad dependencies. This gives you an active way to monitor for bad dependencies before they're merged in. This is what it looks like: it's a bot that will come to your pull request and leave a comment telling you what issues are present in this dependency, and give the developer the information they need to make an informed decision about this dependency. If you're interested in this, you can install it for free at socket.dev.

[00:26:51] I want to end on a positive note here and just say: I want you for supply chain security. I think developers, security practitioners and really entire teams need to change the way they think about dependencies in order to solve the software supply chain security crisis that we're in right now. We need a mindset shift around dependencies. We need to think of them as part of our apps, because they are part of our apps. And we need to really understand the threats that are out there, and what steps we can take as teams to protect ourselves from this emerging threat.

[00:27:28] With that, I'll end. Thank you for having me here, I appreciate it. My contact info is here on the slide if you're interested in reaching out. And also, just a shout-out that we're hiring developers at Socket. Thanks.