Most high-profile intrusions of the past decade haven’t been found due to intrusion detection sensors or complicated forensic tools. Instead, it was the malfunctioning code in the rootkits that made them crash the system with weird error messages.
Most of you already know that the rootkit is the basic set of tools an intruder uses to keep control over a compromised system. Common features of a rootkit include:
- Remote access
- Ability to intercept data
- Hiding modifications on the filesystem
- Hiding processes
- Hiding ‘magic’ users
- Hiding remote access ports/connections.
There are two types of rootkits, based on the way they interfere with user actions: userland and kernel land rootkits. The difference between them comes from the methods they use to hijack the operating system functions.
Rootkits work by subverting the normal operations of the system, either by modifying files on the system with backdoor versions (userland rootkits) or hijacking the operating system function/handler pointers in memory to alter the normal behavior (kernel land rootkits) in order to keep SUPER USER rights.
In order to be able to keep control over a compromised system, a userland rootkit has to modify or alter a binary file in some way, either by replacing the file, modifying parts of the file on disk or modifying parts of the process memory space. Such modifications are, however, easily spotted by a trained admin eye because userland rootkits provide a poor means of disguising the presence of the attacker and his tools, a poor means of hiding the remote connection, not many options to survive reboot, etc.
On the other hand, kernel land rootkits need to interfere with binaries only during the booting phase, to make sure they are loaded during the system initialization procedure. Also, they provide an incredible means of hiding an attacker’s presence inside a computer system, by manipulating the most low-level routines of the operating system. Also, they make remote access really easy for the attacker, with little to no warnings to the administrator. They are very hard to detect by rootkit detection tools. However, the complexity of code may result in unexpected behavior.
So how do anti-rootkit tools work? They are founded on the principle that the rootkit needs to make certain changes to an operating system in order to work. User land rootkits will alter files on disk, timestamps, file sizes, the directory structure, etc. Kernel land rootkits alter system calls and functions, most of them focusing on the syscall table. Any type of rootkit will add files to the file system; maybe start new processes or new remote connections. For instance, the fact that rootkits keep the same file structure for their own files makes them easily spotted by traditional anti rootkit tools such chkrootkit or rkhunter because these tools use the directory and file structure of rootkits to generate signatures of malware presence (Figures 1-5).
They parse certain system directories looking for files known to be the part of malware, so when such a suspected file is found, the user is alerted.
However, as you can easily see, newer or slightly modified versions of rootkits go undetected by traditional signature scanning methods. Therefore, there is a need for a periodically updated database with the latest rootkit signatures and even then, there is no guarantee that a rootkit did not evade the signature scan.
The new technology has been developed in order to avoid relying on signatures or known-to-be-sane system fingerprints to scan for. Instead, it uses live system information it gathers to be able to heuristically determine signs of rootkit activity. Furthermore, we are able to uniquely pinpoint the rootkit code, thus being able to analyze it and determine its exact functionalities, and also extract a unique signature specific to the typology of the rootkit that can be used to speed up further scans, and also, to heuristically determine new or slightly modified versions of the rootkit.
So, how do newer rootkits get detected using traditional means? Well, they don’t. Instead, rootkits make visible changes to the operating system, sometimes resulting in system crashes and weird error messages, mostly due to insufficiently tested code. For example, Phalanx tries to disguise itself as Xnest to access /dev/mem without checking if there is Xnest on the system. An admin would be irritated seeing a message such as “Program Xnest tried to access /dev/mem between 0->8000000” when he doesn’t even have Xnest running. And other kernel rootkits have their own distinctive flaws.
So basically, rootkits are found because they hijack predictable places, mostly aiming at the syscall table, Interrupt Descriptor Table or hijacking filesystem operations. Most modern anti-rootkit detection tools check all these places for inconsistencies with the values previously gathered in their database making visible such a modification. The major flaw of this approach is that if the system is already compromised, or if the attacker has access to the stored fingerprint, or fingerprint update mechanism, they can easily go unnoticed.
Another problem of a rootkit is that they usually tend to employ as many functionalities as possible. This leads to complex code which is a very dangerous bet, especially in Kernel land as it increases the number of possible points of failure. Also, most rootkits don’t get a consistent grasp of the specifics of the system they run on.
Another major problem of a rootkit is the remote access, because an alert admin could easily spot unknown traffic going to suspicious ports, especially if it’s encrypted or encapsulated. Eavesdropping the connection can provide important information about the intruder.
But as we have realized, current Unix anti-rootkit tools provide little to no accuracy in detection of rootkits, the impossibility to clean the system from a rootkit infection or the ability to analyze the malware. It looks like anti-rootkit tools have to step the notch up a bit and raise the bar a little higher for rootkit programmers, so that they come up with better and newer detection mechanisms instead of relying on known fingerprints or signatures, as these can easily be evaded or forged.
AB Consultancy Software SRL
Is a newly merged computer security company located in Bucharest, Romania whose main area of activity is penetration testing and forensics examination. Our experts have over 20 years of international experience in the field of computer security research, both offensive and defensive security, ranging from malware and antimalware research, software audit, exploit development or cryptology. Our customers range from government, military and financial industries, both based in Romania and abroad.