xargs
xargs is a command on Unix and most Unix-like operating systems used to build and execute command lines from standard input. Commands such as grep
and awk
can accept the standard input as a parameter, or argument by using a pipe. However, others such as cp
and echo
disregard the standard input stream and rely solely on the arguments found after the command. Additionally, under the Linux kernel before version 2.6.23, and under many other Unix-like systems, arbitrarily long lists of parameters cannot be passed to a command,[1] so xargs
breaks the list of arguments into sublists small enough to be acceptable.
Contents
About
For example, shell commands such as:
rm /path/*
or
rm `find /path -type f`
may fail with an error message of "Argument list too long" (meaning that the exec system call's limit on the length of a command line was exceeded) if there are too many files in /path
.
However, the version below (functionally equivalent to rm `find /path -type f`
) will not fail:
find /path -type f -print | xargs rm
In the above example, the find
utility feeds the input of xargs
with a long list of file names. xargs
then splits this list into sublists and calls rm
once for every sublist.
The previous example is more efficient than this functionally equivalent version which calls rm
once for every single file:
find /path -type f -exec rm '{}' \;
Note, however, that with modern versions of find
, the following variant does the same thing as the xargs
version:
find /path -type f -exec rm '{}' +
xargs often covers the same functionality as the backquote (`) feature of many shells, but is more flexible and often also safer, especially if there are blanks or special characters in the input. It is a good companion for commands that output long lists of files such as find
, locate
and grep
, but only if you use -0
, since xargs
without -0
deals badly with file names containing ', " and space. GNU Parallel is a similar tool that offers better compatibility with find, locate and grep if file names may contain ', " and space (newline still requires -0
).
Shell trick
The xargs command only offers limited options if one wants the listed arguments to be inserted at some position other than the end of the command-line. While the -I {} option can help, it changes several aspects of the behavior of find at once.
$ mkdir ~/backups
$ find /path -type f -name '*~' -print0 | xargs -0 -I % cp -a % ~/backups
Another way to achieve a similar effect is to use bash
as the launched command, and deal with the complexity in that shell, for example:
$ mkdir ~/backups
$ find /path -type f -name '*~' -print0 | xargs -0 bash -c 'for filename; do cp -a "$filename" ~/backups; done
' bash
The word bash
at the end of the line is interpreted by bash -c
as special parameter $0. If the word bash
weren't present, the name of the first matched file would be assigned to $0
and the file wouldn't be copied to ~/backups
. Any word can be used instead of bash
, but since $0
usually expands to the name of the shell or shell script being executed, bash
is a good choice.
Separator problem
Many Unix utilities are line-oriented. These may work with xargs
as long as the lines do not contain ', " or space. Some of the Unix utilities can use NUL as record separator (e.g. Perl (requires -0
and \0
instead of \n
), locate
(requires using -0
), find
(requires using -print0
), grep
(requires -z
or -Z
), sort
(requires using -z
)). Using -0
for xargs deals with the problem, but many Unix utilities cannot use NUL as separator (e.g. head
, tail
, ls
, echo
, sed
, tar -v
, wc
, which
).
But often people forget this and assume xargs
is also line-oriented, which is not the case (per default xargs
separates on newlines and blanks within lines, substrings with blanks must be single- or double-quoted).
The separator problem is illustrated here:
touch important_file
touch 'not important_file'
find . -name not\* | tail -1 | xargs rm
mkdir -p '12" records'
find \! -name . -type d | tail -1 | xargs rmdir
Running the above will cause important_file
to be removed but will remove neither the directory called 12" records
, nor the file called not important_file
.
The proper fix is to use find -print0
, but tail (and other tools) do not support NUL-terminated strings:
touch important_file
touch 'not important_file'
find . -name not\* -print0 | xargs -0 rm
mkdir -p '12" records'
find \! -name . -type d -print0 | xargs -0 rmdir
When using the syntax find -print0, entries are separated by a null character instead of an end-of-line. This is equivalent to the more verbose command:
find . -name not\* | tr \\n \\0 | xargs -0 rm
or shorter, by switching xargs to line-oriented mode with the -d
(delimiter) option:
find . -name not\* | xargs -d '\n' rm
but in general using the -0
option should be preferred, since newlines in filenames are still a problem.
GNU Parallel is an alternative to xargs
that is designed to have the same options, but be line-oriented. Thus, using GNU Parallel instead, the above would work as expected.[2]
For Unix environments where xargs
does not support the -0
option (e.g. Solaris, AIX), the following can not be used as it does not deal with ' and " (GNU Parallel would work on Solaris, though):
find . -name not\* | sed 's/ /\\ /g' | xargs rm
For Solaris, do not use these examples to fix file perms as they do not deal correctly with names such as 12" records (GNU Parallel instead of xargs would work, though):
find . -type d -print | sed -e 's/^/"/' -e 's/$/"/' | xargs chmod 755
find . -type f -print | sed -e 's/^/"/' -e 's/$/"/' | xargs chmod 644
Operating on a subset of arguments at a time
One might be dealing with commands that can only accept one or maybe two arguments at a time. For example the diff
command operates on two files at a time. The -n flag to xargs specifies how many arguments at a time to supply to the given command. The command will be invoked repeatedly until all input is exhausted. Note that on the last invocation one might get fewer than the desired number of arguments if there is insufficient input. Use xargs to break up the input into two arguments per line:
$ echo {0..9} | xargs -n 2
0 1
2 3
4 5
6 7
8 9
In addition to running based on a specified number of arguments at time, one can also invoke a command for each line of input at a time with -L 1. One can use an arbitrary number of lines at a time, but one is most common. Here is how one might diff every git commit against its parent.[3]
$ git log --format="%H %P" | xargs -L 1 git diff
Encoding problem
The argument separator processing of xargs is not the only problem with using the xargs program in its default mode. Most Unix tools which are often used to manipulate filenames (for example sed, basename, sort and so on) are text processing tools. However, Unix path names are not really text. Consider a path name /aaa/bbb/ccc. The /aaa directory and its bbb subdirectory can in general be created by different users with different environments. That means these users could have a different locale setup, and that means that aaa and bbb do not even necessarily have to have the same character encoding. For example, aaa could be in UTF-8 and bbb in Shift JIS. As a result, an absolute path name in a Unix system may not be correctly processable as text under a single character encoding. Tools which rely on their input being text may fail on such strings.
One workaround for this problem is to run such tools in the C locale, which essentially processes the bytes of the input as-is. However, this will change the behavior of the tools in ways the user may not expect (for example, some of the user's expectations about case-folding behavior may not be met).
References
- ↑ Lua error in package.lua at line 80: module 'strict' not found.
- ↑ Differences Between xargs and GNU Parallel. GNU.org. Accessed February 2012.
- ↑ Lua error in package.lua at line 80: module 'strict' not found.
External links
- The Single UNIX® Specification, Issue 7 from The Open Group : construct argument lists and invoke utility – Commands & Utilities Reference,
Manual pages
- GNU Findutils reference –
- FreeBSD General Commands Manual : construct argument list(s) and execute utility –
- NetBSD General Commands Manual : construct argument list(s) and execute utility –
- OpenBSD General Commands Manual : construct argument list(s) and execute utility –
- Solaris 10 User Commands Reference Manual : construct argument lists and invoke utility –