Accounts of remarkable real-world hacks are as thrilling as works of fiction - the surgical precision of Stuxnet to quietly corrupt machinery only in Iran’s uranium refineries, or NotPetya breaking out of its geofence to infect the entire world.

Wired recently published an account of the biggest supply-chain hack in recent history - SolarWinds distributed their Orion software to 10s of thousands of customers, and it secretly contained a backdoor for hackers to infiltrate the host. This article largely tells the story of the people who cracked the case, and the drama unfolds as the number of compromised companies expands to include governments, big tech companies, and cloud service providers.

It includes the revelation that the government had traced an intrusion to SolarWinds, months before it was acknowledged among the security community, which irked other agencies. The attackers got spooked and tried to cover their tracks, except that one old copy of their software was left behind (on a virtual image snapshot no less). It’s a good read, check it out!

Callers from the NSA and CISA were suddenly livid, according to a person on the line—because for the first time, they were learning that Justice had detected the hackers months earlier. The FBI guy “phrased it like it was no big deal,” the attendee recalls. - Wired

But as an engineer, I’m more enthralled by the technical details of how the attackers injected their malicious code into SolarWind’s binaries. (There’s a paragraph detailing this in the Wired article, but if you’d like to go deep on the actual code, check out CrowdStrike’s analysis). Security researchers were initially bewildered by the root cause, as the binaries appeared to be digitally signed and delivered by SolarWinds, with no evidence of tampering.

So how did a major software player get hacked to distribute malware to all of their customers?

Timeline of attack events from SolarWinds

This attack was carried out by compromising SolarWind’s build servers, to inject a backdoor into the code of a monitoring product called Orion. Orion happens to be a popular tool, it is (was) used by 10,000s of organizations, including global companies and government agencies. To pull off this hack while remaining undetected, the malicious code had to be incredibly smart!

The attackers placed an agent on the servers that compiled SolarWind’s software, that would check what processes were running, once per second. The code was looking for instances of MsBuild.exe, which is Microsoft’s compiler. Whenever the compiler was running, the agent would determine if Orion software was being built by inspecting the build process parameters stored in memory, and if not it would do nothing. It even determined which version of Orion code was being built, to ensure Orion’s code would be compatibility with the malicious code containing a backdoor.

Then the code would replace a legitimate source code file with one that included the backdoor, before that file was compiled into the binary, and after it was compiled, switch back the original code file.

The backdoor code would disable compiler warnings so as not to alert developers that something was amiss. Strings in the code were pre-obfuscated to resemble pre-compiled output.

The malicious code would even log encrypted entries to its own log file, with sequence numbers indicating how far execution had progressed, for attackers to debug their own code.

(Can you imagine, trying to debug your malicious code, while running on your target’s compromised servers, undetected? “Let’s just use printfs” one of your colleagues suggests??)

Did you try using some print statements?

This is all incredibly sophisticated! It requires advanced systems and compilers knowledge.

Officials have identified SVR (Russian civilian intelligence agency) as the attackers, and it’s likely they stole SolarWinds’ source code a year prior to carrying out their exploit of Orion. We’ve seen more than a few high-profile source code thefts in the past year, and SolarWinds is a reminder of how damaging that can be. Who can predict the effects of Okta’s breach that occurred just a few months ago?

One part of a security course I took at Stanford has stuck with me - the professor continuously emphasized that unless you’ve built the compiler, it’s impossible to say that your code is “secure”. This is just as true twenty years later.