shuf

The shuf command helps you randomize the input lines. And there are features to limit the number of output lines, repeat lines and even generate random positive integers.

Randomize input lines

By default, shuf will randomize the order of input lines. Here's an example:

$ cat purchases.txt
coffee
tea
washing powder
coffee
toothpaste
tea
soap
tea

$ shuf purchases.txt
tea
coffee
tea
toothpaste
soap
coffee
washing powder
tea

You can use the --random-source=FILE option to provide your own source for randomness. With this option, the output will be the same across multiple runs. See Sources of random data for more details.

shuf doesn't accept multiple input files. Use cat for such cases.

Limit output lines

Use the -n option to limit the number of lines you want in the output. If the value is greater than the number of lines in the input, it would be similar to not using the -n option.

$ printf 'apple\nbanana\ncherry\nfig\nmango' | shuf -n2
mango
cherry

As seen in the example above, shuf will add a newline character even if it is not present for the last input line.

Repeated lines

The -r option helps if you want to allow input lines to be repeated. This option is usually paired with -n to limit the number of lines in the output.

$ cat fruits.txt
banana
papaya
mango

$ shuf -n3 -r fruits.txt
banana
mango
banana

$ shuf -n5 -r fruits.txt
papaya
banana
mango
papaya
papaya

If a limit using -n is not specified, shuf -r will produce output lines indefinitely.

Specify input lines as arguments

You can use the -e option to specify multiple input lines as arguments to the command.

# quote the arguments as necessary
$ shuf -e hi there 'hello world' good
hello world
good
hi
there

$ shuf -n1 -e brown green blue
blue

$ shuf -n4 -r -e brown green blue
blue
green
brown
blue

The shell will autocomplete unquoted glob patterns (provided there are files that match the given expression). You can thus easily construct a solution to get a random selection of files matching the given glob pattern.

$ echo *.csv
marks.csv mixed_fields.csv report_1.csv report_2.csv scores.csv

$ shuf -n2 -e *.csv
scores.csv
marks.csv

Generate random numbers

The -i option will help generate random positive integers. The argument has to be a range, with - as the separator between the two numbers.

$ shuf -i 5-8
5
8
7
6

$ shuf -n3 -i 100-200
170
112
148

$ shuf -n5 -r -i 0-1
1
0
0
1
1

2^64 - 1 is the maximum allowed integer when I tested it on my machine.

$ shuf -i 18446744073709551612-18446744073709551615
18446744073709551615
18446744073709551614
18446744073709551612
18446744073709551613

$ shuf -i 18446744073709551612-18446744073709551616
shuf: invalid input range: ‘18446744073709551616’:
Value too large for defined data type

# seq can help in such cases
# but remember that shuf needs to read the entire input
$ seq 100000000000000000000000000000 100000000000000000000000000105 | shuf -n2
100000000000000000000000000039
100000000000000000000000000018

seq can also help when you need negative and floating-point numbers.

$ seq -10 -8 | shuf
-9
-10
-8

$ seq -f'%.4f' 100 0.25 3000 | shuf -n3
1627.7500
1303.5000
2466.2500

See unix.stackexchange: generate random strings if numbers aren't enough for you.

Specifying output file

The -o option can be used to specify the output file to be used for saving the results. This is more useful for in-place editing, since you can simply use shell redirection to save the output in a different file.

$ cat book_list.txt
Cradle
Mage Errant
Mother of Learning
Super Powereds
The Umbral Storm
The Weirkey Chronicles

$ shuf book_list.txt -o book_list.txt
$ cat book_list.txt
Super Powereds
Cradle
Mage Errant
The Weirkey Chronicles
Mother of Learning
The Umbral Storm

NUL separator

Use the -z option if you want to use NUL character as the line separator. In this scenario, shuf will ensure to add a final NUL character even if not present in the input.

$ printf 'apple\0banana\0cherry\0fig\0mango' | shuf -z -n3 | cat -v
banana^@mango^@cherry^@

Exercises

The exercises directory has all the files used in this section.

1) What's wrong with the given command?

$ shuf --random-source=greeting.txt fruits.txt books.txt
shuf: extra operand ‘books.txt’
Try 'shuf --help' for more information.

# expected output
##### add your solution here
banana
Cradle:::Mage Errant::The Weirkey Chronicles
Mother of Learning::Eight:::::Dear Spellbook:Ascendant
papaya
Mark of the Fool:Super Powereds:::Ends of Magic
mango

2) What do the -r and -n options do? Why are they often used together?

3) What does the following command do?

$ shuf -e apple banana cherry fig mango

4) Which option would you use to generate random numbers? Given an example.

5) How would you generate 5 random numbers between 0.125 and 0.789 with a step value of 0.023?

# output shown below is a sample, might differ for you
##### add your solution here
0.378
0.631
0.447
0.746
0.723