Skip to content

fio tips

Jump to bottom
Vincent Fu edited this page Sep 19, 2023 · 2 revisions

Fio has a formidable set of options and it is easy to be overwhelmed reading through the voluminous documentation. Here is a collection of tips that might be useful to fio users.

  1. Fio documentation is available at https://fio.readthedocs.io/en/latest/. The man page is of course available but users may find the online documentation more aesthetically pleasing and easier to browse as references to various sections are hyperlinked. Read the Docs even provides a pdf version to download.

online documentation

  1. By default a given build of fio will always generate offsets for random I/O in a predictable sequence. This is because fio uses default seeds for its random number generators in order for results to be reproducible from run to run. In the two runs below the off= values list the read request offsets which are identical for the two runs:
user@ubuntu:~$ fio --name=test --debug=io --ioengine=null --number_ios=4 --size=1M --rw=randread | grep queue
io       127725 queue: io_u 0x563ac4536f40: off=0xf000,len=0x1000,ddir=0,file=test.0.0
io       127725 queue: io_u 0x563ac4536f40: off=0xbd000,len=0x1000,ddir=0,file=test.0.0
io       127725 queue: io_u 0x563ac4536f40: off=0xd7000,len=0x1000,ddir=0,file=test.0.0
io       127725 queue: io_u 0x563ac4536f40: off=0x78000,len=0x1000,ddir=0,file=test.0.0
user@ubuntu:~$ fio --name=test --debug=io --ioengine=null --number_ios=4 --size=1M --rw=randread | grep queue
io       127801 queue: io_u 0x558e2e33cf40: off=0xf000,len=0x1000,ddir=0,file=test.0.0
io       127801 queue: io_u 0x558e2e33cf40: off=0xbd000,len=0x1000,ddir=0,file=test.0.0
io       127801 queue: io_u 0x558e2e33cf40: off=0xd7000,len=0x1000,ddir=0,file=test.0.0
io       127801 queue: io_u 0x558e2e33cf40: off=0x78000,len=0x1000,ddir=0,file=test.0.0

This can be changed with the randrepeat option. If randrepeat is set to false, fio will generate new random seeds for every invocation. The offsets from the two runs below with randrepeat=0 differ from the ones above and from each other:

user@ubuntu:~$ fio --name=test --debug=io --ioengine=null --number_ios=4 --size=1M --rw=randread --randrepeat=0 | grep queue
io       127953 queue: io_u 0x55b7bf65bfc0: off=0x82000,len=0x1000,ddir=0,file=test.0.0
io       127953 queue: io_u 0x55b7bf65bfc0: off=0x34000,len=0x1000,ddir=0,file=test.0.0
io       127953 queue: io_u 0x55b7bf65bfc0: off=0xe000,len=0x1000,ddir=0,file=test.0.0
io       127953 queue: io_u 0x55b7bf65bfc0: off=0x2f000,len=0x1000,ddir=0,file=test.0.0
user@ubuntu:~$ fio --name=test --debug=io --ioengine=null --number_ios=4 --size=1M --rw=randread --randrepeat=0 | grep queue
io       128029 queue: io_u 0x565082228fc0: off=0xb1000,len=0x1000,ddir=0,file=test.0.0
io       128029 queue: io_u 0x565082228fc0: off=0x8d000,len=0x1000,ddir=0,file=test.0.0
io       128029 queue: io_u 0x565082228fc0: off=0x88000,len=0x1000,ddir=0,file=test.0.0
io       128029 queue: io_u 0x565082228fc0: off=0xf5000,len=0x1000,ddir=0,file=test.0.0
  1. By default fio will not touch the same offset more than once. If we direct fio to issue four random 4 KiB read commands on a 16 KiB file, fio will always cover the entire file by default (although not necessarily in the same order since we have randrepeat=0). The offsets identified by the off= values in the output below cover the entire file in both of the runs.
user@ubuntu:~$ fio --name=test --debug=io --ioengine=null --number_ios=4 --size=16K --rw=randread --randrepeat=0 | grep queue
io       130176 queue: io_u 0x560a47021fc0: off=0x1000,len=0x1000,ddir=0,file=test.0.0
io       130176 queue: io_u 0x560a47021fc0: off=0x2000,len=0x1000,ddir=0,file=test.0.0
io       130176 queue: io_u 0x560a47021fc0: off=0x3000,len=0x1000,ddir=0,file=test.0.0
io       130176 queue: io_u 0x560a47021fc0: off=0x0,len=0x1000,ddir=0,file=test.0.0
user@ubuntu:~$ fio --name=test --debug=io --ioengine=null --number_ios=4 --size=16K --rw=randread --randrepeat=0 | grep queue
io       130252 queue: io_u 0x5575cf215fc0: off=0x2000,len=0x1000,ddir=0,file=test.0.0
io       130252 queue: io_u 0x5575cf215fc0: off=0x3000,len=0x1000,ddir=0,file=test.0.0
io       130252 queue: io_u 0x5575cf215fc0: off=0x0,len=0x1000,ddir=0,file=test.0.0
io       130252 queue: io_u 0x5575cf215fc0: off=0x1000,len=0x1000,ddir=0,file=test.0.0

In order to relax this constraint, use the norandommap option. This instructs fio to not keep track of offsets accessed and as a result some offsets will be read more than once and others may be completely untouched as in the following two examples:

user@ubuntu:~$ fio --name=test --debug=io --ioengine=null --number_ios=4 --size=16K --rw=randread --randrepeat=0 --norandommap | grep queue
io       130467 queue: io_u 0x5626665fa0c0: off=0x3000,len=0x1000,ddir=0,file=test.0.0
io       130467 queue: io_u 0x5626665fa0c0: off=0x1000,len=0x1000,ddir=0,file=test.0.0
io       130467 queue: io_u 0x5626665fa0c0: off=0x3000,len=0x1000,ddir=0,file=test.0.0
io       130467 queue: io_u 0x5626665fa0c0: off=0x3000,len=0x1000,ddir=0,file=test.0.0
user@ubuntu:~$ fio --name=test --debug=io --ioengine=null --number_ios=4 --size=16K --rw=randread --randrepeat=0 --norandommap | grep queue
io       130543 queue: io_u 0x5579dff1f0c0: off=0x2000,len=0x1000,ddir=0,file=test.0.0
io       130543 queue: io_u 0x5579dff1f0c0: off=0x0,len=0x1000,ddir=0,file=test.0.0
io       130543 queue: io_u 0x5579dff1f0c0: off=0x2000,len=0x1000,ddir=0,file=test.0.0
io       130543 queue: io_u 0x5579dff1f0c0: off=0x3000,len=0x1000,ddir=0,file=test.0.0

In the first example, offset 0x3000 is read three times, offset 0x1000 is read once and the two remaining offsets are untouched. In the second example, offsets 0x0 and 0x3000 are each read once, offset 0x2000 is read twice and offset 0x1000 is not read at all.

  1. The output listings above take advantage of fio's debug logging feature. For troubleshooting or to confirm that fio is doing what you intend, the debug output is invaluable. Specify the type of debug output desired by invoking fio with the debug option set to a comma-separated list of one or more of the following types:
blktrace parse
compress process
diskutil profile
file random
helperthread rate
io steadystate
job time
mem verify
mutex zbd
net

  1. If you don't want to wait until the end of a long running fio job to see the interim results, send your fio process the USR1 signal and it will print the current summary output at the time of the signal and then continue running. For an example see Example 3 in the steady state blog post.

  2. Fio's example job files are useful for better understanding how to use various fio features. Would you like fio to read a file backwards? There is an example for that:

# Demonstrates how to read backwards in a file.

[backwards-read]
bs=4k
# seek -8k back for every IO
rw=read:-8k
filename=128m
size=128m

This uses a feature of the rw option to specify a delta other than zero for consecutive sequential offsets.

Are you wondering why the filecreate ioengine doesn't work the way you expect? Take a look at its example job file:

filecreate-ioengine.fio 
# Example filecreate job
#
# create_on_open is needed so that the open happens during the run and not the
# setup.
#
# openfiles needs to be set so that you do not exceed the maximum allowed open
# files.
#
# filesize needs to be set to a non zero value so fio will actually run, but the
# IO will not really be done and the write latency numbers will only reflect the
# open times.
[global]
create_on_open=1
nrfiles=31250
ioengine=filecreate
fallocate=none
filesize=4k
openfiles=1

[t0]
[t1]
[t2]
[t3]
[t4]
[t5]
[t6]
[t7]
[t8]
[t9]
[t10]
[t11]
[t12]
[t13]
[t14]
[t15]

It turns out that options like size, io_size, and number_ios are largely irrelevant for the filecreate ioengine. The ioengine's queue function does not actually queue I/O and reported bandwidth numbers are not meaningful. Instead, latencies are recorded when fio opens (creates) each file for I/O, the latency report summarizes the latency of file creation, and the nrfiles option is used to specify how many operations to carry out. The filedelete and filestat ioengines operate in a similar fashion.

  1. Fio has a status-interval option that directs it to produce cumulative (from job start) summary output periodically with the specified time interval between output sets. Although the bandwidth and IOPS values are cumulative from the start of the job, the output values can nevertheless be used to produce live "instantaneous" bandwidth and IOPS measurements.

The output below is from a three-second job where status-interval output was requested every second:

Example job and output 
root@ubuntu:~# fio --name=test --runtime=3s --time_based --ioengine=null --size=1T --status-interval=1s
test: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=null, iodepth=1
fio-3.30-91-gc08f95
Starting 1 process

test: (groupid=0, jobs=1): err= 0: pid=143816: Wed Aug 24 08:14:29 2022
  read: IOPS=3589k, BW=13.7GiB/s (14.7GB/s)(12.3GiB/898msec)
    clat (nsec): min=16, max=30985, avg=21.52, stdev=102.87
     lat (nsec): min=55, max=31030, avg=69.49, stdev=167.18
    clat percentiles (nsec):
     |  1.00th=[   17],  5.00th=[   17], 10.00th=[   18], 20.00th=[   19],
     | 30.00th=[   19], 40.00th=[   20], 50.00th=[   20], 60.00th=[   21],
     | 70.00th=[   22], 80.00th=[   23], 90.00th=[   25], 95.00th=[   25],
     | 99.00th=[   27], 99.50th=[   38], 99.90th=[   44], 99.95th=[   48],
     | 99.99th=[   61]
   bw (  MiB/s): min=13730, max=13730, per=97.96%, avg=13730.35, stdev= 0.00, samples=1
   iops        : min=3514974, max=3514974, avg=3514974.00, stdev= 0.00, samples=1
  lat (nsec)   : 20=39.28%, 50=60.69%, 100=0.03%, 250=0.01%, 500=0.01%
  lat (nsec)   : 1000=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
  cpu          : usr=100.00%, sys=0.00%, ctx=4, majf=0, minf=7
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=3223023,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=13.7GiB/s (14.7GB/s), 13.7GiB/s-13.7GiB/s (14.7GB/s-14.7GB/s), io=12.3GiB (13.2GB), run=898-898msec
Jobs: 1 (f=1)
test: (groupid=0, jobs=1): err= 0: pid=143816: Wed Aug 24 08:14:30 2022
  read: IOPS=3619k, BW=13.8GiB/s (14.8GB/s)(26.2GiB/1898msec)
    clat (nsec): min=16, max=30985, avg=21.17, stdev=97.58
     lat (nsec): min=55, max=31030, avg=68.65, stdev=166.06
    clat percentiles (nsec):
     |  1.00th=[   17],  5.00th=[   17], 10.00th=[   18], 20.00th=[   19],
     | 30.00th=[   19], 40.00th=[   19], 50.00th=[   20], 60.00th=[   20],
     | 70.00th=[   21], 80.00th=[   22], 90.00th=[   23], 95.00th=[   25],
     | 99.00th=[   33], 99.50th=[   39], 99.90th=[   43], 99.95th=[   46],
     | 99.99th=[   77]
   bw (  MiB/s): min=13730, max=14370, per=100.00%, avg=14149.27, stdev=363.00, samples=3
   iops        : min=3514974, max=3678956, avg=3622213.33, stdev=92924.48, samples=3
  lat (nsec)   : 20=42.90%, 50=57.07%, 100=0.02%, 250=0.01%, 500=0.01%
  lat (nsec)   : 750=0.01%, 1000=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
  cpu          : usr=99.95%, sys=0.00%, ctx=10, majf=0, minf=8
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=6868643,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=13.8GiB/s (14.8GB/s), 13.8GiB/s-13.8GiB/s (14.8GB/s-14.8GB/s), io=26.2GiB (28.1GB), run=1898-1898msec
Jobs: 1 (f=1)
test: (groupid=0, jobs=1): err= 0: pid=143816: Wed Aug 24 08:14:31 2022
  read: IOPS=3643k, BW=13.9GiB/s (14.9GB/s)(40.3GiB/2898msec)
    clat (nsec): min=16, max=30985, avg=21.03, stdev=97.92
     lat (nsec): min=55, max=31030, avg=68.15, stdev=164.24
    clat percentiles (nsec):
     |  1.00th=[   17],  5.00th=[   17], 10.00th=[   18], 20.00th=[   18],
     | 30.00th=[   19], 40.00th=[   19], 50.00th=[   20], 60.00th=[   20],
     | 70.00th=[   21], 80.00th=[   22], 90.00th=[   23], 95.00th=[   24],
     | 99.00th=[   30], 99.50th=[   37], 99.90th=[   43], 99.95th=[   45],
     | 99.99th=[   80]
   bw (  MiB/s): min=13730, max=14433, per=99.92%, avg=14217.61, stdev=284.73, samples=5
   iops        : min=3514974, max=3694986, avg=3639708.80, stdev=72889.56, samples=5
  lat (nsec)   : 20=44.30%, 50=55.67%, 100=0.02%, 250=0.01%, 500=0.01%
  lat (nsec)   : 750=0.01%, 1000=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
  cpu          : usr=99.79%, sys=0.00%, ctx=12, majf=0, minf=8
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=10556962,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=13.9GiB/s (14.9GB/s), 13.9GiB/s-13.9GiB/s (14.9GB/s-14.9GB/s), io=40.3GiB (43.2GB), run=2898-2898msec
Jobs: 1 (f=1): [R(1)][-.-%][r=14.1GiB/s][r=3692k IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=143816: Wed Aug 24 08:14:31 2022
  read: IOPS=3643k, BW=13.9GiB/s (14.9GB/s)(41.7GiB/3001msec)
    clat (nsec): min=16, max=30985, avg=21.02, stdev=97.76
     lat (nsec): min=55, max=31030, avg=68.13, stdev=164.18
    clat percentiles (nsec):
     |  1.00th=[   17],  5.00th=[   17], 10.00th=[   18], 20.00th=[   18],
     | 30.00th=[   19], 40.00th=[   19], 50.00th=[   20], 60.00th=[   20],
     | 70.00th=[   21], 80.00th=[   22], 90.00th=[   23], 95.00th=[   24],
     | 99.00th=[   30], 99.50th=[   37], 99.90th=[   43], 99.95th=[   45],
     | 99.99th=[   87]
   bw (  MiB/s): min=13730, max=14433, per=99.91%, avg=14217.61, stdev=284.73, samples=5
   iops        : min=3514974, max=3694986, avg=3639708.80, stdev=72889.56, samples=5
  lat (nsec)   : 20=44.41%, 50=55.57%, 100=0.02%, 250=0.01%, 500=0.01%
  lat (nsec)   : 750=0.01%, 1000=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
  cpu          : usr=99.87%, sys=0.00%, ctx=12, majf=0, minf=9
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=10932587,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=13.9GiB/s (14.9GB/s), 13.9GiB/s-13.9GiB/s (14.9GB/s-14.9GB/s), io=41.7GiB (44.8GB), run=3001-3001msec

The first three output panels are the result of status-interval and the last panel is the usual final summary output. We can use the total data read and the total number of requests issued from the cumulative output panels to calculate periodic bandwidth and IOPS values.

The first output panel occurred 898ms into the run at which point 12.3GiB had been read and 3223023 read requests had been issued. The next output panel appeared 1898ms into the run at which point 26.2GiB had been read and 6868643 requests had been issued. Thus, by differencing the cumulative values we can determine that read bandwidth in this one-second period was 13.9GiB/s and that the IOPS rate was 3646K.

In practice the calculations could be done with a script and possibly using the terse or JSON output formats. Note that producing the status-interval output involves locking and issuing some malloc calls. So depending on the workload there may be a measurable impact on performance metrics. Fio also of course has write_bw_log and write_iops_log for periodic bandwidth and IOPS measurements.

Clone this wiki locally