Verbose Data

Data is reported in this form when either --verbose is used OR if there is at least one type of data requested that doesn't have a brief form such as any detail data or ionodes, processes or slabs. Specifying some of the lustre output options with --lustopts such as B, D and M will also force verbose format.

Buddy (Memory Fragmentation) Data, collectl -sb

# MEMORY FRAGMENTATION SUMMARY (4K pages)
#     1Pg    2Pgs    4Pgs    8Pgs   16Pgs   32Pgs   64Pgs  128Pgs  256Pgs  512Pgs 1024Pgs
This table shows the total number of memory fragments by pagesize in increasing powers of 2 for all the memory types.

CPU, collectl -sc

# CPU SUMMARY (INTR, CTXSW & PROC /sec)
# User  Nice   Sys  Wait   IRQ  Soft Steal Guest NiceG  Idle  CPUs  Intr  Ctxsw  Proc  RunQ   Run   Avg1  Avg5 Avg15 RunT BlkT
These are the percentage of time the system in running is one of the modes, noting that these are averaged across all CPUs. While User and Sys modes are self-eplanitory, the others may not be:

User Time spent in User mode, not including time spend in "nice" mode.
Nice Time spent in Nice mode, that is lower priority as adjusted by the nice command and have the "N" status flag set when examined with "ps".
Sys This is time spent in "pure" system time.
Wait Also known as "iowait", this is the time the CPU was idle during an outstanding disk I/O request. This is not considered to be part of the total or system times reported in brief mode.
Irq Time spent processing interrupts and also considered to be part of the summary system time reported in "brief" mode.
Soft Time spent processing soft interrupts and also considered to be part of the summary system time reported in "brief" mode.
Steal Time spent in other operating systems when running in a virtualized environment
Guest Time spent running a virtual CPU for guest operating systems under the control of the Linux kernel, new since 2.6.24
NiceG Time spent running a niced guest (virtual CPU for guest operating systems under the control of the Linux kernel), new since 2.6.33

This next set of fields apply to processes

ProcProcess creations/sec.
RunqNumber of processes in the run queue.
RunNumber of processes in the run state.
Avg1, Avg5, Avg15Load average over the last 1,5 and 15 minutes.
RunTTotal number of process in the run state, not counting collectl itself
BlkTTotal number of process blocked, waiting on I/O

Disks, collectl -sd

If you specify filtering with --dskfilt, the disks that match the pattern(s) will either be included or excluded from the the summary data. However, the data will still be collected so if recorded to a file can later be viewed.

# DISK SUMMARY (/sec)
#KBRead RMerged  Reads SizeKB   KBWrit WMerged Writes SizeKB
KBReadKB read/sec
RMergedRead requests merged per second when being dequeued.
ReadsNumber of reads/sec
SizeKBAverage read size in KB
KBWriteKB written/sec
WMerged Write requests merged per second when being dequeued.
WritesNumber of writes/sec
SizeKBAverage write size in KB

Inodes/Filesystem, collectl -si

# INODE SUMMARY
#    Dentries      File Handles    Inodes
# Number  Unused   Alloc  MaxPct   Number
   40585   39442     576    0.17    38348
DCache
Dentries NumberNumber of entries in directory cache
Dentried UnusedNumber of unused entries in directory cache
Handles AllocNumber of allocated file handles
handles % MaxPercentage of maximum available file handles
Inodes NumberNumber of inodes in use

NOTE - as of this writing I'm baffled by the dentry unused field. No matter how many files and/or directories I create, this number goes up! Sholdn't it go down?

Infiniband, collectl -sx

# INFINIBAND SUMMARY (/sec)
#  KBIn   PktIn  SizeIn   KBOut  PktOut SizeOut  Errors
KBInKB received/sec.
PktInPackets received/sec.
SizeInAverage incoming packet size in KB
KBOutKB transmitted/sec.
PktOutPackets transmitted/sec.
SizeOutAverage outgoing packet size in KB
ErrsCount of current errors. Since these are typically infrequent, it is felt that reporting them as a rate would result in either not seeing them OR round-off hiding their values.

Lustre

Lustre Client, collectl -sl

There are several formats here controlled by the --lustopts switch. There is also detail data for these available as well. Specifying -sL results in data broken out by the file system and --lustopts O further breaks it out by OST. Also note the average read/write sizes are only reported when --lustopts is not specified.

# LUSTRE CLIENT SUMMARY
# KBRead  Reads SizeKB  KBWrite Writes SizeKB
KBReadKB/sec delivered to the client.
ReadsReads/sec delivered to the client, not necessarily from the lustre storage servers.
SizeKBAverage read size in KB
KBWriteKB Writes/sec delievered to the storage servers.
WritesWrites/sec delievered to the storage servers.
SizeKBAverage write size in KB
# LUSTRE CLIENT SUMMARY: METADATA
# KBRead  Reads KBWrite Writes  Open Close GAttr SAttr  Seek Fsynk DrtHit DrtMis
KBReadKB/sec delivered to the client.
ReadsReads/sec delivered to the client, not necessarily from the lustre storage servers.
KBWriteKB Writes/sec delievered to the storage servers.
WritesWrites/sec delievered to the storage servers.
OpenFile opens/sec
CloseFile closes/sec
GAttrgetattrs/sec
Seekseeks/sec
Fsyncfsyncs/sec
DrtHitdirty hits/sec
DrtMisdirty misses/sec
# LUSTRE CLIENT SUMMARY: READAHEAD
# KBRead  Reads KBWrite Writes  Pend  Hits Misses NotCon MisWin FalGrb