SAR

The first command I would like to discuss is 'sar'. Sar stands for System Activity Registration and is standard installed on all Linux/Unix systems. If sar isn't installed on your system you can install it using your package manager, in CentOS it will be 'yum install sysstat'. Sar will show you the use of resources of a system, adding parameters to sar will show you the resources of the selected 'hardware'. With sar you can also view the use of resources from the past, default it will show you the use of resources of the current day (starting from midnight).

By default sar holds data from the past 8 days. During office hours the data files of sar will be filled every 20 minutes with data and outside the office hours every hour. These statistic counters for sar are filled bij a cron into a logfile. By default these logfiles are named '/var/adm/sa/saXX' on Unix and '/var/log/sa/saXX' on Linux', replace 'XX' with the day of the month. Now the basics are known let us go through a few parameters of sar:

Kind of hardware: processor(s)
Command to execute: sar -u
Explanation: When a process is working through it's own code the used CPU time is totalized in %usr (nice value is default or lower then default) or in %nice (nice value is higher then default, means lower priority). If a process is issueing a system call the used CPU time is totalized in %sys. If you add the %usr, %nice and %sys together you get the total busy time of the processor(s).

Kind of hardware: disk(s)
Command to execute: sar -dp
Explanation: This command gives you information about the disks with human readable dev names. It will show you the I/O request, write and read speed etc. Look at the manpage of sar for the -d option to show all explanations.

NETSTAT

Network delays are difficult to analyse, normally this is something for specialists, but with netstat you can get some information from several protocols since boot time. The information shown will not give you information about network delays it can also be the case that the receiving side has a processor that is running at 100% or it is swapping memory, but it gives you an indication about what is going on over the network. Let us go through a few usefull parameters:

Command to execute: netstat -i
Explanation: Displays the results of all network packages send and received of all interfaces since boot time.

Command to execute: netstat -s
Explanation: Displays a statistics summary of each protocol since boot time.

Command to execute: netstat -n | grep 8022 | grep ESTABLISHED | wc –l
Explanation: For example if you have a SFTP server running on port 8022, this displays the current open connections on this port.

LSOF

If you found a TCP or UDP port in the output of your netstat command that looks weird to you or you want to find out which process is using this port you can use the command lsof. This will display the process that is using that port. Also lsof can do a lot more things. Look into the manpage of lsof for further options.

For example. You saw with 'netstat -a' that there is a IPP port in use on your system. You can find out which port is using this process with the command: lsof -i @<hostname>:631

PSACCT / ACCTON / ACCT

If you would like to monitor performance from a process point of view you need to get into 'psacct'. I am not going to discuss it here, but I will give some links to a website where you can get some more information:

http://www.cyberciti.biz/faq/linux-unix-bsd-varaccountpacct-or-varlogaccountpacct-file/
http://www.cyberciti.biz/tips/howto-log-user-activity-using-process-accounting.html

VMSTAT / IOSTAT

The command vmstat gives you information about processes, memory, paging, block IO, traps, and cpu activity. The command iostat reports Central Processing Unit (CPU) statistics and input/output statistics for devices, partitions and network filesystems (NFS). For these commands I also refer you to this website: http://www.cyberciti.biz/tips/top-linux-monitoring-tools.html