Are we Forever Doomed to Software Supply Chain Security?

Hi, everyone, and thank you for joining my talk, Are We Forever Doomed by Software Supply Chain Risks? If you joined this talk, then it means you care about software and you care about software security, which is great. Most of all, you probably are curious, like I am, how does open-source software supply chain risks impact all of us? You, me, everyone here.

So I wanted to begin this with interesting stories, such as, let's take a moment to reflect on this picture, what you're seeing on your screen right now. And what comes to your mind? Is it a futuristic outlook of the world, like the robots? Are they uprising and becoming a key part of our lives? Perhaps it's other things, like how much does the robot actually learn and upload all of that to the cloud? You know, where is it stored? Is it stored safely? What happens when someone can do something malicious with that data? Most of all, what happens if someone hacks in and interacts with my child? This is my kid, if so to say. And what happens when that interaction takes place, when someone is able to compromise that? So, yeah, that's where we are getting started, you know. These are some of the things that are keeping me up at night.

Today I would like to share with you some real-world stories of how developers play a fundamental role in recent and growing security incidents. And also, you know, why should you care about security, software security and software supply chain risks? And also, leave you off to think about who you actually trust.

So, in case you had a doubt, we're seeing more and more open source software being developed. Year after year, right, open source software repositories are growing with more and more open source software and that is more software footprint. The thing is the application that we build are ever-growing in their dependency of open source software, not just the fact that we're using open source software, but the applications themselves are using more and more of that as well. So, more of us, basically, software engineers, are now accustomed to the way of how people work in open source, like opening issues and becoming maintainers of open source. More and more of us are maintainers of open source software and that growth of open source software doesn't come without any risks, right? This is why we're all here, because we care about it.

We are continuously witnessing the growth of open source software security vulnerabilities in these ecosystems, like NPM and Java and others and this could be anything from CVE-based vulnerability reports that when you install a software you get that installation that said, you know, there are 1,000 vulnerabilities there. What happens when that takes place? Also the incidents themselves of malicious packages and different targeted attacks on developers because us as developers, we rely on open source software packages and so we are targeted as well and we'll get to that right now.

Let's rewind back, first of all, in time to get an early glimpse of how one developer perceived the risks of open source software. So in 1984, the Turing Award-winning Ken Thompson wrote a short essay titled, Reflections on Trusting Trust, in which he describes how he added a backdoor to the Unix login C program and then he continued and added a backdoor to the C compiler and then he added a backdoor further on with his chains of attacks into the compiler that compiles the C program, that compiles the compiler. Right, so in his paper here about Reflections on Trusting Trust, he actually explains how software can be taught to learn specific traits and pass them on. So, realistically, it is very, very hard to find the traces of things like Trojan horses and backdoors unless you have actually written everything from scratch. And I mean everything such as the compiler, the linker, the CPU, the display that you're watching something, the keyboard, everything. It is very hard to trust an ecosystem. So as we learned by Thompson's Trojan horse story, this is dating back to 1984, developers have been targeted as a vehicle to distribute malicious backdoors and other sorts of malware for a very long time now.

Let's explore some of these incidents. I'm sure you've probably heard of some of them. In 2018, the JavaScript ecosystem witnessed its first high impact, spearheaded, targeted attack on maintainers and developers alike where they are working in open source in the ecosystem themselves and actually have been the attack target, the vehicle as well to actually distribute malicious JavaScript code to all of us using those software. And so the attack itself targeted also developers using open source, specifically a Bitcoin wallet application. And this was the well-known Event Stream incident. Event Stream existed on the NPM registry since 2011, for a very long time now, as you could see. It's practically didn't receive any new releases in the last two years. But gained millions of downloads per week.

Out of the blue someone chimes in and say, I wanna help. They get into the project, they help, they open a pull request as you normally do in open-source software. One of those pull requests later has actually introduced the dependency. At that point in time, this was a trusted individual. So they received a different kind of like access to the repository, and publishing new packages, and new versions of that package, and so on. And later on when they added that dependency, now that dependency exists in a different state, and they could just add that backdoor to that new version that they released. And now new versions of Event Stream that use that dependency pull that, the new version in, and they get that dependency with the backdoor that now the original maintainer of Event Stream did not know about. They did not know that this is happening and they can't control it. Because this is how software package managers work in specific ecosystems like NPM, like others. So this actually resulted in two versions of the copy Bitcoin wallet application, including the malware, three months. For three months until we found this, this is the Event Stream one.

Now, an academic research paper published in 2019 investigated this properties of language-based software ecosystems, and it found that more than a half, 61% of open source packages on NPM could be considered abandoned because they did not receive any release in the last 12 months. That's exactly what happened with the Event Stream story. And as if we didn't learn anything from this incident, one year later, a similar action comes into life, Electronative Notify, and similar incident happening. What happening in short, this one point in time, this existing Electronative Notify package, which is not malicious, but it's added as a dependency to a different project, this EasyDEX GUI thing. Later on, the developer that owns Electronative Notify releases a new version that now includes a malware, a backdoor to it. What happens, the Electronative Notify project is now embedded into a different app, this EasyDEX things, which now pulls a new version of Electronative Notify with the malicious version and releases new versions. Again, sounds similar? Exactly what happened before. How much thought are we actually giving to the security of our own development infrastructure? For example, resources like Cloud instances and your continuous integration tooling. In January 2021, a security researcher broke into Microsoft Visual Studio code, GitHub repository. Essentially, this is providing him any capabilities like read and write access to modify the code, of the very well beloved IDE VS code that many developers love. But due to command injection that this researcher had been able to exploit, it was able to give him an attack vector to basically create code injection. Now, by opening a pull request, all he had to do, open a new code pull request, this researcher can now execute code into VS code's own CI ecosystems without being authenticated or authorized. All of this really just provides him the ability to create, to submit, or if you would like to say, spawn remote reverse shells on the CI servers running code. And from that, being able to basically push and write access to the repository itself. Now, fortunately for us, all of us here in this ecosystem, this researcher responsibly reported the flaw to Microsoft before, hopefully anyone else could have gained access to it. But you have to ask, right? How much do we know about the current state of open source security? What does it actually entail? So in an effort to explore the security awareness of Python and JavaScript open source communities, a group of researchers investigated how maintainers work in the open source community and how they release fixes relating to security vulnerabilities. One of the research questions was how quickly do maintainers mitigate a newly published security vulnerability? Once it's published, once there's a CDE out, the research found that it takes almost 100 days on average for both JavaScript and Python maintainers to start mitigating a vulnerability. You have to ask yourself if this is fast enough. And this is a bit different between the different communities because for Python, it has been remediated much sooner versus JavaScript, which took its time from 2018 until now to basically be a bit more mature and aware about the security risks. As a case study exactly of this, we can refer to Marked. It's a library, Marked's own security vulnerabilities from several years back. Now, what happened is Marked is a Markdown library. It's a parser for, you know, JavaScript and Node.js and it's very popular, getting millions of downloads a week. One day a security researcher just pops up, open a code pull request to fix a vulnerability. So this says, all you have to do is merge it, basically, as you can see here. But, as it is with open source software, maintainers are really trying to do their best, but they're not contracted or obligated legally or something to support you. So, me as a maintainer, you, everyone else, we're really trying to do our best in time of what we can do. But this vulnerability stayed out there for a year, knowingly, exact what's happening with this, and this security vulnerability itself being public for more than a year. When we're all so much very dependent on open source software, we can't ignore the question of, you know, where do we put our trust? What are the mitigations and controls that we have to cope with the risks involved? In 2017, a security researcher working with the Node.js Foundation, conducted a research, which he wanted to explore and assess, right, the state of weak NPM credentials on the NPM registry. His work really revealed devastating truths about developers and their lack of security hygiene.

What I mean by that is, this security researcher was able to gain published access to 14% of NPM ecosystem modules. This is a lot. Some of these modules were downloaded millions of times, tens of millions of times a week. And they are still an essential key part of this thriving JavaScript ecosystem, which I am also a part of. So this is very concerning for me, as well.

The thing is, the problem was rooted with insecure passwords chosen by well-known maintainer accounts, such as literally the word password for one of the accounts running millions of downloads for one of their packages. So if it can happen to them, it can happen to other packages, as well. If our code packages can reach millions of developers, shouldn't we have more protections in place as citizens of open source communities? Shouldn't we do better for our account hygiene? So you would imagine so.

But in NPM, the largest registry of open-source software, even though two-factor authentication was enabled since 2017, only 7.1% of NPM package maintainers have enabled 2FA. You'd think that we have done better so far, but a year later, by 2020, another interesting insight that we get into what has been happening is that it has only grown by 2%. Only 9% of developer account on NPM by 2020 have enabled 2FA. We should do better.

As cybersecurity expert Bruce Schneier says, and had said and written in his book, 'Secrets and Lies,' humans often represent the weakest link in the chain. Given enough eyeballs, all bags are shallow. As a term coined, as Linux is low, backed by Eric Raymond in his work, the Cathedral and the Bazaar by 1999. He explored the differences of how software development takes different shifts between open source and enterprises. And one of the things was that if everyone are looking at it, you know, we'll find the problems or a lot of people looking at this code. Really, are we going to find it? Because in January, 2021, it was discovered that sudo, the common utility installed on many Linux distributions had a security vulnerability. Specifically, it allowed any unprivileged users to gain root access simply by the default sudo configuration. Now, this was so daunting as a vulnerability because it was hiding in plain sight. This is from the, you know, the disclosure itself. For a whole 10, whole years, this was just showing up there.

We can't deny being a part of this open source ecosystems as consumers, contributors, or maybe even as maintainers for some of us. We all play some parts in a world where 90% of our application code is open source software components. We reached a point where, you know, we take open source for granted, right? Open source registries are open in their nature. Anyone can go in and publish their new packages, even malicious ones, you know, we will be surprised, maybe it doesn't get caught. We have been accustomed to this, you know, opening and issuing a project source code or asking for help, asking for a feature, you know, asking someone to get our bag fixed hopefully. But we import those open source into our projects and do we understand what's going on at that point in time? Because open source registries are open in their nature, but they're not gonna be there all the time. What happens when maintainers remove their packages and dependencies? And this is a story that exactly happened and for a package that was very pivotal in the ecosystems, but caused a widespread breakage of continuous integration systems all around. So at the very least, it taught us that enterprises and organizations that were prompted this attack were not understanding how to deal with open source software and registries themselves did not know either.

They were missing how even this could be possible and wouldn't be possible. What kind of malicious activities and assets can we track back to open source software? Time after time, we're finding more malicious packages hitting the NPM ecosystem, typo spawning mistakes and others. But they're not only a sole problem of the NPM, the JavaScript ecosystem, because PyPI had also seen thousands of packages brute forcingly published in bulk to basically add malicious packages.

To further show us how attackers can actually harness open source to their advantage, Alex Berchan published his research in February 2021 about how he exploited design flaws in package managers and human errors and registries basically to infiltrate into corporations like Apple, Microsoft, and others. And so all of this is happening in open source.

And I would like to leave you off with some questions. Are we going to have less or more software in the future? Are we going to use more open source software in the future? Who do you trust and what can you do about it? Thank you for coming to my talk. My name is Iran Tal, I am a developer advocate at Snyk where we build a security platform to help developers build securely with open source software. Thank you all for joining my talk. Have a nice conference.

That was a wonderful session. I am super glad that Liran brought this topic today because security is something that is very important for web development. And yeah, thanks for that.

So let's take a look at the results of the pool. And as we can see here, it seems that people hear a lot more about Trojan source than any of the other options that you left there. What's your comments on that, Liran?

Yeah, that makes a lot of sense because I guess a lot of Trojan source has been in the news recently. That's more recent than the others, but they are different types of attacks. And that's why I gave them that kind of shows you got, what are you seeing? Which is a lot of the media stuff. But for example, an event stream, which, you know, if you didn't know about before, then you should probably know now because of like, I was chatting about it on my slides just in this talk. It's one of the most sophisticated attacks that we've seen on like the supply chain ecosystem for NPM, event stream is. And recently that was 2018, but now we've had recently about a month back, the UA parser agent and COA, and other packages that were probably hijacked and added, they had like malware and cryptocurrency slash ransom kind of malware in them. So that happened pretty recently, but Trojan source, interestingly, that's made it into the news. CodeCuff. Have you heard about CodeCuff before, because that's kind of like a Docker related region of the problem. They haven't. Would you like to talk about them?

Yeah, I'll talk in brief because again, like these are all kinds of different things like Trojan sources from like an academic research paper that caused a whole lot of stuff happening. And it is, Trojan source is not actually an incident, it's more like an academic paper that describes how those sorts of Trojans can, like any bad codes can be injected into open source projects, kind of like using invisible characters and stuff like that. Like by the original unique code characters that changed the way that the text looks. But it still is like the compiler doesn't have any issues with this, because the code is okay. And interestingly, that was like very easily, or I would say very quickly, at least, kind of like addressed where people were both I think, GitHub, Snyk, like a lot of organizations were kind of like acting, responding quickly to fix this and highlight those kinds of issues. So that is very particular case. CodeCov, on the other hand, is we've seen it happening with like the bash uploader security incidents, which stole tokens and stuff like that. So a lot of people using the NPM package for it or the GitHub action for it to like do code coverage, which this is very, sitting this conference, like testing a conference, but the origin of that incident, like why did it happen is simply due to the Docker image that was built for the CodeCov tool chain was actually leaking, had some layers inside, actually leaked private keys that allowed someone when they looked at it and like, you know, know where to change files, how to change them. It went on for like four months until this got discovered. So like super serious issue as well. And it was addressed, you know, very well by the CodeCov team with a lot of transparency and incident response on this. So like, you know, kudos to them to handle this, but it just shows like, you know, security incidents and supply-chain security is just all around. So that's kind of like. It just shows how security is important. And we have, we don't, we can't just look into one part of the testing because security is part of the software development in general, right? So we have some questions.

And the first one is from Jim Nuro. Is a solution better vulnerability scanning software? I don't know if I understood exactly the question. Did you get it? Is a solution better vulnerability scanning software? Okay so there are a few. I will not discuss the whole sort of like the spectrum of like a security testing software. There's a bunch, but mostly you probably wanna have the SCA because we use a lot of open source. SCA means software composition analysis. It's basically looking at your requirements that decks, you know, Ruby gems files, all of the like packages and all of those package managers related files. Checking out which has vulnerabilities in them depending on the versions and then tracking that towards a vulnerability database. Now that's important to have because we rely a lot on open source and you can see a lot of like these incidents related to open source dependencies. So what is better, right? It's kind of like, you know, it's very much like, what works really well for you and that's like super important. But I would say there are two things that would make a software better. I think for me, when you do the vulnerabilities, one is it has to have like developer related workflows in mind. So not just to show you the problem but actually help you fix them and like, you know, help developers like actually remediate the issues. And the other one is a very, very good vulnerability database because if you have something that just works but it doesn't really cover a lot of like the vulnerabilities or doesn't cover them on time or it's like has a lot of false positives, all of those things that kind of like add noise and in the end you'll just end up getting frustrated and not to use this. So these are two traits and features that I'd look into a software that does that.

Awesome. There's another question here, which is how do you envision the future yourself? Do you envision your own utopia for package security? It's an interesting one. Yeah, but could you repeat it's a vision of what I didn't really catch that one. How do you envision the future yourself? Do you envision your own utopia for package security? Oh, I got it. Okay. So I'll tell you what the ecosystem looks at right now from being involved in a lot of like working groups and security forums and things like that. There's a lot of work right now being done in the open SSF to this is the open source security foundation recently by the way kind of like a relatively recently kind of created. And the idea there is that's kind of like prevailing is first of all, a lot of best practices. Like first of all, we need to get developers, engineers ops security, everyone kind of like knowing what to do best for the best practices, how to do them. So this is why I'm doing these talks, like helping developers like writing blog posts where practices to all of those kinds of things. So people could like upskill themselves. Like this is like half of the problem. The other it's the other is like tooling and there is definitely like some work around SIG store and other projects which are basically if you like Google's SIG store, you'll find it really quick. The idea is how do we right now a lot of those issues or because we cannot trust that the package wasn't tampered with. So like if I publish the package on NPM and you downloaded it, you do not know if this new version was really signed like an integrity check that was really me producing it or someone else that's had a takeover of my account like they know that username and password and they did it but they don't have access to my private key or something like that. So like they couldn't give you that kind of like a feeling of safety. So SIG store and other tooling around this is really more about the authenticity, the integrity everything around the package that makes it hopefully like will make our lives a lot safer but I'm pretty sure we're gonna see more threats once we solve that one, right? Like it's a rolling ball. Yeah, yeah. I want to ask a question before we go with the next one which is like I have in a previous experience implemented in a JavaScript code base the Dependabot from GitHub where it automatically sends you pull requests fixing vulnerability issues and things like that. Do you have any comments on doing that and how often, because in the beginning it was funny because it was just too many pull requests and we have to limit how many we wanted to review every week so that we could work on other stuff as well. And at some point, it became like a more of a number that we could manage, but I wanted to hear your opinion on that. Yeah, and I think it's a good example. Like if NPM Auditor dependent but works for you, I mean, that's good. That means that maybe you're happy with this and that's okay. I'll tell you what are the addressing specifically your concern? Like what are the more developer oriented tools are and that kind of what we're doing at Snyk. And so one of those first issues that you just said, like you are concerned because it's flooding you with maybe hundreds or tens of full requests. So on Snyk, what you could do is you could say, Hey, you know what, I want to limit my upgrade requests of packages to like, say five. I don't know whatever numbers make sense for your team, but let's say it's five. So Snyk knows that it will not open more upgrade full requests if there are already five open, of course like the security one will always open up because I want to keep you safe and protected. You should like upgrade as soon as you can.

But all the general chatter around, like the full request chatter about like opening more and more and more just to stay up to date. We always limit that to an extent. And that helps your team kind of like have that control of like how to do things. So it sounds like a small feature, but it's like just it's exactly that developer first kind of mindsets tooling that Snyk has, which is, you know, free and like open source with CLI and tooling. So like, that's really addresses developers in terms of like how they should like, you know, work in a very kind of safe and comfortable environment.

The other... Completely says, yeah. Sure, yeah. The other aspect of it is for example, we were looking at some security incidents in the past before. And one of the things that we learned is, which also like relates to what you said, with a lot of like the, you know, flooding you with those baggage updates. And that is sometimes like updating very, very quickly is actually bad, right? Because if this is now in like version three, five, two, whatever, is like the newest version you, and someone tells you like upgrades is dependable or something else, updated this and then like two weeks later, you found out you upgraded, you know, too fast. And now this is a malicious version, like UA parser COV, et cetera. Then upgrading really, really fast, actually puts you in a position of risk. So we've actually done this even like, I think two years ago already, like 2020, even maybe 2020, 2019, where we analyze some of like the prior security incidents that happened, and we figured out what is an optimal timeline where we will introduce a new package version, which is not too soon, but also like not too late. And I think that's kind of like 20 or 21 days. And so if there's like a new package, we will not tell you about it until 20 or 21 days. And afterwards you cannot, we will. So like that's kind of like, you know, have been, there's like some safety around it. That's super nice. That's super.

And last question, how can we better promote supply chain security practices? Best question. You know, it's a lot. Like, you know, help me raise the awareness, you know, talk about it with your friends, with your colleagues, you know, do the best practices. There is a ton of learning on the sneak blog. If you kind of like Google like best practices for NPM or Noders Java, you'll find a ton of the stuff that we've been writing, like cheat sheets and PDFs and everything from like tooling that we're building. And the idea is really just to help you. And, you know, I'm a developer myself. I write all of these tools, actually wrote an ES linked plugin to detect and prevent you from having Trojan source attacks, which, you know, a lot of people here said that they knew of, so like, why not use it? We're doing a lot in a lot of like activism to like really promote awareness of supply chain security, open source security in general. And you know, the more that we talk about it, the more that we use the tooling, the more that we take responsibility for what we use and the due diligence that we do is, you know, one step further to actually like, you know, promote the whole notion for everyone else and advance this story. Yeah, a final question from myself, which resources do you recommend for reading about like security testing? There's quite a few.

I mean, OS is pretty good in terms of, like they have a cheat sheet series, which I highly recommend. If you like Google OS cheat sheets, you'll get to create a landing page with a ton of those. There's the OSASVS and which is like, and the Proactive Controls which are, Proactive Controls is very developer-oriented. I would say the only kind of like caveat bit with OS is that it's still very, sort of security-minded. So like it's kind of geared towards security people, not towards developers in a sense, like the documentation, the way that stuff are built, the terminology. So it could be very overwhelming when you read this as a developer and you hear, like you read on the screen, like CVEs, CVSS Core, like a lot of stuff that you may not know of and like it will deter you away.