resource monitoring
(Written by Paul Cobbaut, https://github.com/paulcobbaut/, with contributions by: Alex M. Schapelle, https://github.com/zero-pytagoras/)
Monitoring is the process of obtaining information about the utilization
of memory, cpu, bandwidth and storage. You should start monitoring your
system as soon as possible, to be able to create a baseline. Make sure
that you get to know your system! This baseline is important because it
allows you to see a steady or sudden growth in resource utilization
and likewise steady (or sudden) decline in resource availability. It
will allow you to plan for scaling up or scaling out.
Let us look at some tools that go beyond ps fax,
df -h, free -om and
du -sh.
four basic resources
The four basic resources to monitor are:
- cpu
- network
- ram memory
- storage
top
To start monitoring, you can use top. This tool will
monitor ram memory, cpu and swap. Top will automatically refresh. Inside
top you can use many commands, like k to kill processes, or t and
m to toggle displaying task and memory information, or the number 1
to have one line per cpu, or one summary line for all cpu\'s.
top - 12:23:16 up 2 days, 4:01, 2 users, load average: 0.00, 0.00, 0.00
Tasks: 61 total, 1 running, 60 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.3% us, 0.5% sy, 0.0% ni, 98.9% id, 0.2% wa, 0.0% hi, 0.0% si
Mem: 255972k total, 240952k used, 15020k free, 59024k buffers
Swap: 524280k total, 144k used, 524136k free, 112356k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1 root 16 0 2816 560 480 S 0.0 0.2 0:00.91 init
2 root 34 19 0 0 0 S 0.0 0.0 0:00.01 ksoftirqd/0
3 root 5 -10 0 0 0 S 0.0 0.0 0:00.57 events/0
4 root 5 -10 0 0 0 S 0.0 0.0 0:00.00 khelper
5 root 15 -10 0 0 0 S 0.0 0.0 0:00.00 kacpid
16 root 5 -10 0 0 0 S 0.0 0.0 0:00.08 kblockd/0
26 root 15 0 0 0 0 S 0.0 0.0 0:02.86 pdflush
...
You can customize top to display the columns of your choice, or to display only the processes that you find interesting.
[student@linux ~]$ top p 3456 p 8732 p 9654
free
The free command is common on Linux to monitor free
memory. You can use free to display information every x seconds, but the
output is not ideal.
[student@linux gen]$ free -om -s 10
total used free shared buffers cached
Mem: 249 222 27 0 50 109
Swap: 511 0 511
total used free shared buffers cached
Mem: 249 222 27 0 50 109
Swap: 511 0 511
[student@linux gen]$
watch
It might be more interesting to combine free with the
watch program. This program can run commands with a
delay, and can highlight changes (with the -d switch).
[student@linux ~]$ watch -d -n 3 free -om
...
Every 3.0s: free -om Sat Jan 27 12:13:03 2007
total used free shared buffers cached
Mem: 249 230 19 0 56 109
Swap: 511 0 511
vmstat
To monitor CPU, disk and memory statistics in one line there is
vmstat. The screenshot below shows vmstat running every
two seconds 100 times (or until the Ctrl-C). Below the r, you see the
number of processes waiting for the CPU, sleeping processes go below b.
Swap usage (swpd) stayed constant at 144 kilobytes, free memory dropped
from 16.7MB to 12.9MB. See man vmstat for the rest.
[student@linux ~]$ vmstat 2 100
procs ----------memory--------- --swap-- ---io--- --system-- ---cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
0 0 144 16708 58212 111612 0 0 3 4 75 62 0 1 99 0
0 0 144 16708 58212 111612 0 0 0 0 976 22 0 0 100 0
0 0 144 16708 58212 111612 0 0 0 0 958 14 0 1 99 0
1 0 144 16528 58212 111612 0 0 0 18 1432 7417 1 32 66 0
1 0 144 16468 58212 111612 0 0 0 0 2910 20048 4 95 1 0
1 0 144 16408 58212 111612 0 0 0 0 3210 19509 4 97 0 0
1 0 144 15568 58816 111612 0 0 300 1632 2423 10189 2 62 0 36
0 1 144 13648 60324 111612 0 0 754 0 1910 2843 1 27 0 72
0 0 144 12928 60948 111612 0 0 312 418 1346 1258 0 14 57 29
0 0 144 12928 60948 111612 0 0 0 0 977 19 0 0 100 0
0 0 144 12988 60948 111612 0 0 0 0 977 15 0 0 100 0
0 0 144 12988 60948 111612 0 0 0 0 978 18 0 0 100 0
[student@linux ~]$
iostat
The iostat tool can display disk and cpu statistics. The
-d switch below makes iostat only display disk information (500 times
every two seconds). The first block displays statistics since the last
reboot.
[student@linux ~]$ iostat -d 2 500
Linux 2.6.9-34.EL (RHELv8u3.localdomain) 01/27/2007
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
hdc 0.00 0.01 0.00 1080 0
sda 0.52 5.07 7.78 941798 1445148
sda1 0.00 0.01 0.00 968 4
sda2 1.13 5.06 7.78 939862 1445144
dm-0 1.13 5.05 7.77 939034 1444856
dm-1 0.00 0.00 0.00 360 288
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
hdc 0.00 0.00 0.00 0 0
sda 0.00 0.00 0.00 0 0
sda1 0.00 0.00 0.00 0 0
sda2 0.00 0.00 0.00 0 0
dm-0 0.00 0.00 0.00 0 0
dm-1 0.00 0.00 0.00 0 0
...
[student@linux ~]$
You can have more statistics using iostat -d -x, or display only cpu
statistics with iostat -c.
[student@linux ~]$ iostat -c 5 500
Linux 2.6.9-34.EL (RHELv8u3.localdomain) 01/27/2007
avg-cpu: %user %nice %sys %iowait %idle
0.31 0.02 0.52 0.23 98.92
avg-cpu: %user %nice %sys %iowait %idle
0.62 0.00 52.16 47.23 0.00
avg-cpu: %user %nice %sys %iowait %idle
2.92 0.00 36.95 60.13 0.00
avg-cpu: %user %nice %sys %iowait %idle
0.63 0.00 36.63 62.32 0.42
avg-cpu: %user %nice %sys %iowait %idle
0.00 0.00 0.20 0.20 99.59
[student@linux ~]$
mpstat
On multi-processor machines, mpstat can display
statistics for all, or for a selected cpu.
student@linux:~$ mpstat -P ALL
Linux 2.6.20-3-generic (laika) 02/09/2007
CPU %user %nice %sys %iowait %irq %soft %steal %idle intr/s
all 1.77 0.03 1.37 1.03 0.02 0.39 0.00 95.40 1304.91
0 1.73 0.02 1.47 1.93 0.04 0.77 0.00 94.04 1304.91
1 1.81 0.03 1.27 0.13 0.00 0.00 0.00 96.76 0.00
student@linux:~$
sadc and sar
The sadc tool writes system utilization data to
/var/log/sa/sa??, where ?? is replaced with the current
day of the month. By default, cron runs the sal script
every 10 minutes, the sal script runs sadc for one second. Just before
midnight every day, cron runs the sa2 script, which in
turn invokes sar. The sar tool will read the daily data
generated by sadc and put it in /var/log/sa/sar??. These sar reports
contain a lot of statistics.
You can also use sar to display a portion of the statistics that were gathered. Like this example for cpu statistics.
[student@linux sa]$ sar -u | head
Linux 2.6.9-34.EL (RHELv8u3.localdomain) 01/27/2007
12:00:01 AM CPU %user %nice %system %iowait %idle
12:10:01 AM all 0.48 0.01 0.60 0.04 98.87
12:20:01 AM all 0.49 0.01 0.60 0.06 98.84
12:30:01 AM all 0.49 0.01 0.64 0.25 98.62
12:40:02 AM all 0.44 0.01 0.62 0.07 98.86
12:50:01 AM all 0.42 0.01 0.60 0.10 98.87
01:00:01 AM all 0.47 0.01 0.65 0.08 98.80
01:10:01 AM all 0.45 0.01 0.68 0.08 98.78
[student@linux sa]$
There are other useful sar options, like sar -I PROC to display
interrupt activity per interrupt and per CPU, or sar -r
for memory related statistics. Check the manual page of sar for more.
ntop
The ntop tool is not present in default Red Hat
installs. Once run, it will generate a very extensive analysis of
network traffic in html on http://localhost:3000 .
iftop
The iftop tool will display bandwidth by socket
statistics for a specific network device. Not available on default Red
Hat servers.
1.91Mb 3.81Mb 5.72Mb 7.63Mb 9.54Mb
--------------|-------------|--------------|-------------|--------|----
laika.local => barry 4.94Kb 6.65Kb 69.9Kb
<= 7.41Kb 16.4Kb 766Kb
laika.local => ik-in-f19.google.com 0b 1.58Kb 14.4Kb
<= 0b 292b 41.0Kb
laika.local => ik-in-f99.google.com 0b 83b 4.01Kb
<= 0b 83b 39.8Kb
laika.local => ug-in-f189.google.com 0b 42b 664b
<= 0b 42b 406b
laika.local => 10.0.0.138 0b 0b 149b
<= 0b 0b 256b
laika.local => 224.0.0.251 0b 0b 86b
<= 0b 0b 0b
laika.local => ik-in-f83.google.com 0b 0b 39b
<= 0b 0b 21b
iptraf
Use iptraf for a colourful display of ip traffic over the network
cards.
[root@linux ~]# iptraf
[root@linux ~]# iptraf -i eth0
nload
nload displays current network traffic in the command line. Use the
arrow keys to walk through devices.
Device wlan0 [192.168.1.35] (2/2):
==============================================================================================
Incoming:
||
.. ## || .. Curr: 13.20 kBit/s
## ## ##.. ## Avg: 656.33 kBit/s
##.. ##||#### ## Min: 0.00 Bit/s
#### ########## ##.. Max: 4.44 MBit/s
.#### ..########## ..#### Ttl: 895.44 MByte
Outgoing:
Curr: 11.84 kBit/s
Avg: 105.90 kBit/s
Min: 0.00 Bit/s
Max: 518.48 kBit/s
.. ............ .. Ttl: 672.49 MByte
nmon
Another popular and all round tool is nmon.
htop
You can use htop instead of top.
