Ebury backdoors SSH servers in hosting providers, giving the malware extraordinary reach.
Infrastructure used to maintain and distribute the Linux operating system kernel was infected for two years, starting in 2009, by sophisticated malware that managed to get a hold of one of the developers’ most closely guarded resources: the /etc/shadow files that stored encrypted password data for more than 550 system users, researchers said Tuesday.
The unknown attackers behind the compromise infected at least four servers inside kernel.org, the Internet domain underpinning the sprawling Linux development and distribution network, the researchers from security firm ESET said. After obtaining the cryptographic hashes for 551 user accounts on the network, the attackers were able to convert half into plaintext passwords, likely through password-cracking techniques and the use of an advanced credential-stealing feature built into the malware. From there, the attackers used the servers to send spam and carry out other nefarious activities. The four servers were likely infected and disinfected at different times, with the last two being remediated at some point in 2011.FURTHER READING
Who rooted kernel.org servers two years ago, how did it happen, and why?
An infection of kernel.org came to light in 2011, when kernel maintainers revealed that 448 accounts had been compromised after attackers had somehow managed to gain unfettered, or “root,” system access to servers connected to the domain. Maintainers reneged on a promise to provide an autopsy of the hack, a decision that has limited the public’s understanding of the incident.
Besides revealing the number of compromised user accounts, representatives of the Linux Kernel Organization provided no details other than saying that the infection:
- Occurred no later than August 12, 2011, and wasn’t detected for another 17 days
- Installed an off-the-shelf rootkit known as Phalanx on multiple servers and personal devices belonging to a senior Linux developer
- Modified the files that both servers and end user devices inside the network used to connect through OpenSSH, an implementation of the SSH protocol for securing remote connections.
FURTHER READING
10,000 Linux servers hit by malware serving tsunami of spam and exploits
In 2014, ESET researchers said the 2011 attack likely infected kernel.org servers with a second piece of malware they called Ebury. The malware, the firm said, came in the form of a malicious code library that, when installed, created a backdoor in OpenSSH that provided the attackers with a remote root shell on infected hosts with no valid password required. In a little less than 22 months, starting in August 2011, Ebury spread to 25,000 servers. Besides the four belonging to the Linux Kernel Organization, the infection also touched one or more servers inside hosting facilities and an unnamed domain registrar and web hosting provider.
A 47-page report summarizing Ebury’s 15-year history said that the infection hitting the kernel.org network began in 2009, two years earlier than the domain was previously thought to have been compromised. The report said that since 2009, the OpenSSH-dwelling malware has infected more than 400,000 servers, all running Linux except for about 400 FreeBSD servers, a dozen OpenBSD and SunOS servers, and at least one Mac.
Researcher Marc-Etienne M. Léveillé wrote:
In our 2014 paper, we mentioned that there was evidence that kernel.org, hosting the source code of the Linux kernel, had been a victim of Ebury. Data now at our disposal reveals additional details about the incident. Ebury had been installed on at least four servers belonging to the Linux Foundation between 2009 and 2011. It seems these servers acted as mail servers, name servers, mirrors, and source code repositories at the time of the compromise. We cannot tell for sure when Ebury was removed from each of the servers, but since it was discovered in 2011 it is likely that two of the servers were compromised for as long as two years, one for one year and the other for six months.
The perpetrator also had copies of the /etc/shadow files, which overall contained 551 unique username and hashed password pairs. The cleartext passwords for 275 of those users (50%) are in possession of the attackers. We believe that the cleartext passwords were obtained by using the installed Ebury credential stealer, and by brute force.
The researcher said in an email that the Ebury and Phalanx infections appear to be separate compromises by two unrelated threat groups. Representatives of the Linux Kernel Organization didn’t respond to emails asking if they were aware of the ESET report or if its claims were accurate. There is no indication that either infection resulted in tampering with the Linux kernel source code.
ARS VIDEO
How The Callisto Protocol’s Team Designed Its Terrifying, Immersive Audio
Constant churn for 15 years (and counting)
Tuesday’s report said that in the 10 years since ESET last detailed Ebury, company researchers have continued to uncover new information that illustrates the extraordinary reach and power of the sprawling, profit-generating botnet it assembles. Léveillé explained:
While 400,000 is a massive number, it’s important to mention that this is the number of compromises over the course of almost 15 years. Not all of those machines were compromised at the same time. There is a constant churn of new servers being compromised while others are being cleaned up or decommissioned. The data at our disposal doesn’t indicate when the attackers lost access to the systems, so it’s difficult to know the size of the botnet at any specific point in time. However, each time we were able to measure the number of servers currently compromised with Ebury, whether that was 10 years ago or in the past few years, we ended up with more or less 40,000 IP addresses. In 2023, that number ballooned to 110,000, mostly due to the compromise of a large hosting provider, as described below.
While this might seem less than some of the massive malware campaigns targeting Windows, let’s remind ourselves that almost all compromised systems are servers, not end user devices. Servers help run the Internet by hosting web pages, acting as authoritative name servers, performing financial transactions, etc.
When ESET researchers first encountered Ebury, they thought its sole means of spreading was through the compromise of SSH credentials. By backdooring OpenSSH files on both servers and client devices, the compromise of one server would reveal the SSH credentials for accessing one or more new servers. The process would repeat in an almost exponential fashion. Since then, ESET has identified five new methods for propagating. They are:Advertisement
- Credential stuffing
- Spreading from hypervisors or containers to all subsystems
- Spreading from a hosting provider to all connected servers
- Exploitation of vulnerabilities, including one known as Dirty COW and what was once a zero-day in the Control Web Panel for managing servers inside host facilities
- Using their access to one server to perform adversary-in-the-middle attacks to steal SSH credentials from other servers in the same network.
One of the most notable propagation methods is Ebury using its presence inside data centers to perform AitM attacks at scale on servers deemed to be high value. Elements involved are:
- Identifing a compromised machine with Ebury installed within the same subnet as the targeted system
- Confirming network segmentation by examining the ARP cache after pinging the target
- Installing necessary tools such as arpspoof
- Executing arpspoof to intercept network traffic and establish iptables rules to redirect network traffic to an operator-controlled system designed to capture SSH credentials.
More details are available in the section headed “Performing AitM Attacks.”
Infections at scale
A key to the proliferation of the malware has been its success in spreading in virtually any type of environment, including universities, small and large enterprises, Internet service providers, cryptocurrency exchanges, Tor exit nodes, shared hosting providers, and dedicated server providers. In 2019, for instance, Ebury infected the “infrastructure of a large and popular US-based domain registrar and web hosting provider.” The feat gave the attackers access to both the source code and credentials available to roughly 2,500 physical and 60,000 virtual servers hosted inside the unnamed organization.
Most, if not all, of the commandeered ISP servers were shared among multiple users for hosting websites, with a total of more than 1.5 million accounts. Since the compromised infrastructure ran mostly the same configurations and OS and application versions, the attackers were able to deploy a variety of newly added malware tools at a massive scale.
Last year, ESET researchers tracked a large number of compromised servers located in a single US-based autonomous system (AS) that was used by a limited number of dedicated and virtual server rental brands. The researchers suspected that a single server-hosting organization using the AS had been compromised and used to spread Ebury at large on any server running inside the organization.
To test their theory, the researchers rented a virtual server from the hosting provider and used it as a “honeypot” to see if it would be infected. The researchers did nothing with the server other than logging into it over SSH just once. Within a week, Ebury was installed on their device. From then on, the attackers connected daily to the server to check for any new SSH credentials and provide malware updates. In all, 70,000 servers from the unnamed hosting provider were compromised last year with Ebury.
One key area of evolution involves new ways to generate revenue from compromised servers. The methods include payment card information theft, stealing of cryptocurrency, traffic redirection, the sending of spam, and credential stealing. Some of the new tools added are:
- A set of Apache modules used for different purposes such as proxying raw traffic, exfiltrating sensitive information from HTTP requests, and redirecting visitors to an attacker-controlled website
- A kernel module that modifies HTTP traffic to perform redirection
- Tools to hide and allow malicious traffic through the firewall
- Scripts used to carry out AitM within hosting providers’ data centers, aiming to compromise specific targets.
ESET included the following graphic depicting some of the various malware for generating revenue, along with some of the resulting consequences for people who interact with compromised servers:
The ESET report provides IP addresses, file hashes, and other technical indicators that administrators can use to determine if they have been targeted by Ebury. It also notes the benefit of protecting SSH servers with multi-factor authentication (MFA) so that a compromise of the password isn’t sufficient to take control of the server it protects. Unfortunately, MFA faces some disadvantages in the SSH server context.
“It relies on third-party PAM modules such as the Google Authenticator PAM module and needs manual configuration,” Léveillé explained in the report. “It is not the default security setting of any popular Linux distribution. As a result, MFA is rarely deployed for the SSH service.”
Given Ebury’s continuous evolution over 15 years and its multiple ways of propagating, it’s not hard to see how it was able to infect kernel.org and the many other organizations it has claimed. Given that the malware continues to spread and shows no signs of stopping, it’s imperative that organizations of all sizes remain vigilant.
“Ebury poses a serious threat and a challenge to the Linux security community,” Léveillé concluded. “There is no simple fix that would make Ebury ineffective, but a handful of mitigations can be applied to minimize its spread and impact. One thing to realize is that it doesn’t happen only to organizations or individuals that care less about security. A lot of very tech-savvy individuals and large organizations are among the list of victims.”
