Client1: (groupid=0, jobs=1): err= 0: pid=16109: Sat Jun 24 12:07:54 2017 write: IOPS=88, BW=623KiB/s (638kB/s)(30.4MiB/50032msec) slat (nsec): min=500, max=145500, avg=8318.00, stdev=4781.50 clat (usec): min=170, max=78367, avg=4019.02, stdev=8293.31 lat (usec): min=174, max=78375, avg=4027.34, stdev=8291.79 clat percentiles (usec): | 1.00th=[ 302], 5.00th=[ 326], 10.00th=[ 343], 20.00th=[ 363], | 30.00th=[ 392], 40.00th=[ 404], 50.00th=[ 416], 60.00th=[ 445], | 70.00th=[ 816], 80.00th=[ 6718], 90.00th=, 95.00th=, | 99.00th=, 99.50th=, 99.90th=, 99.95th=, | 99.99th= bw ( KiB/s): min= 532, max= 686, per=0.10%, avg=622.87, stdev=24.82, samples= 100 iops : min= 76, max= 98, avg=88.98, stdev= 3.54, samples= 100 lat (usec) : 250=0.04%, 500=64.11%, 750=4.81%, 1000=2.79% lat (msec) : 2=4.16%, 4=1.84%, 10=4.90%, 20=11.33%, 50=5.37% lat (msec) : 100=0.65% cpu : usr=0.27%, sys=0.18%, ctx=12072, majf=0, minf=21 IO depths : 1=85.0%, 2=13.1%, 4=1.8%, 8=0.1%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=0,4450,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=8
- The string before the colon shows the I/O direction the statistics are for. IOPS is the average I/Os performed per second. BW is the average bandwidth rate shown as: value in power of 2 format (value in power of 10 format). The last two values show: (total I/O performed in power of 2 format / runtime of that thread).
- Submission latency (min being the minimum, max being the maximum, avg being the average, stdev being the standard deviation). This is the time it took to submit the I/O. For sync I/O this row is not displayed as the slat is really the completion latency (since queue/complete is one operation there). This value can be in nanoseconds, microseconds or milliseconds — fio will choose the most appropriate base and print that (in the example above nanoseconds was the best scale). Note: in --minimal mode latencies are always expressed in microseconds.
- Completion latency. Same names as slat, this denotes the time from submission to completion of the I/O pieces. For sync I/O, clat will usually be equal (or very close) to 0, as the time from submit to complete is basically just CPU time (I/O has already been done, see slat explanation).
- Total latency. Same names as slat and clat, this denotes the time from when fio created the I/O unit to completion of the I/O operation.1. fio - Flexible I/O tester rev. 3.12 — fio 3.12 documentation
- Bandwidth statistics based on samples. Same names as the xlat stats, but also includes the number of samples taken (samples) and an approximate percentage of total aggregate bandwidth this thread received in its group (per). This last value is only really useful if the threads in this group are on the same disk, since they are then competing for disk access.