Forgotten Unix commands: nice and renice

Today we are going to shed some light onto the way processes can be (re-)prioritised when it comes to scheduling using the nice and renice commands.

Typically (but not always), priority values range from -20 (run with top priority) to +19/20 (run when nothing else runs), let’s have a look at an excerpt of a processlist with “ps axl” on a CentOS server:

ps axl
F   UID   PID  PPID PRI  NI    VSZ   RSS WCHAN  STAT TTY        TIME COMMAND
4     0     1     0  20   0   4116   876 poll_s Ss   ?          0:00 init
1     0     2     1  20   0      0     0 kthrea S    ?          0:00 [kthreadd/1049]
1     0     3     2  20   0      0     0 worker S    ?          0:00 [khelper/1049]
5     0   134     1  16  -4  10420   612 poll_s S<s  ?          0:00 /sbin/udevd -d
1     0   560     1  20   0  63596  1216 poll_s Ss   ?          0:00 /usr/sbin/sshd
1     0   740     1  20   0 281748 10376 poll_s Ss   ?          0:59 /usr/sbin/httpd
1   497   756   749  25   5  64836  1056 hrtime SN   ?          0:54 /usr/sbin/zabbix_agentd
1     0   760     1  20   0 116668  1212 hrtime Ss   ?          0:03 crond
5    48  7776   740  20   0 282272  7352 inet_c S    ?          0:00 /usr/sbin/httpd
5    48  8022   740  20   0 282128  6508 inet_c S    ?          0:00 /usr/sbin/httpd
...

Except for udevd, everything is running at priority 20, without any sort of nicing. Unprivileged users can only lower the priority of their processes (so as to not interfere with underlying OS stability), the superuser can also increase the piority of a process, though. Let’s have a look at how this is done for processes that have already started:

# renice -n -4 -p 7776; ps axl
7776: old priority -4, new priority -4
F   UID   PID  PPID PRI  NI    VSZ   RSS WCHAN  STAT TTY        TIME COMMAND
...
5    48  7776   740  16  -4 282656  7580 inet_c S<   ?          0:00 /usr/sbin/httpd

As we can see, the PRI and NI columns have changed for this single process. The arguments here are -n and -p, -n takes an integer value to modify the priority by the value given, and -p takes the process ID as argument.

The nice command works in a similar fashion, it takes an integer for its -n parameter value, followed by the command as such, e.g.:

nice -n 2 ps axl results in:

F   UID   PID  PPID PRI  NI    VSZ   RSS WCHAN  STAT TTY        TIME COMMAND
...
4     0 14503 12698  22   2 105464   892 -      RN+  pts/0      0:00 ps axl
...

whereas nice -n -2 ps axl yields:

F   UID   PID  PPID PRI  NI    VSZ   RSS WCHAN  STAT TTY        TIME COMMAND
...
4     0 14533 12698  18  -2 105468   892 -      R<+  pts/0      0:00 ps axl
...

So…that’s all very NICE, yes, but what does it actually mean? Modifying the priority of processes can be useful when processes need more power than the CPU can actually deliver. In such cases, (re)nicing processes can help to stabilise a system, to avoid (too much) contention for resources, and to keep vital functions up and functioning.

A typical example may be compressing a large file, where it is not important how long that process actually takes, but it should not have any (or at least only very little) effect on everything else on the system. In such cases, one might run the compressing process with nice -n 19 gzip … thus giving it a lower priority. Here, however, it is also important to mention that Linux often features a program called ionice, which is specifically intended for scheduling I/O (as opposed to primarily scheduling CPU).

There is, however, no general rule as to “how much” more priority a process gets based on the differences in integer values.

nice and renice have lost importance over the years with CPU power ever increasing, but nevertheless they might come in handy at times, and we hope this quick intro will be useful to you when it comes to administrating your virtual private server or your dedicated server.

 

Forgotten Unix commands: xargs

In today’s post we will cover a bit about xargs - also very useful to perform rather complex (and repetitive) operations on the Linux shell.

xargs is particularly useful when it comes to piping (“|“) and chaining commands and their arguments together.

xargs reads items from the standard input or from pipes, delimited by blanks or newlines, and then executes the command one or more times with any initial arguments followed by items read from standard input. Blank lines on the standard input are ignored1.

Let’s do some examples:

# echo Where is my tux? | xargs
Where is my tux?

That was easy, especially since echo is the default command anyway.

Now something more useful maybe:

# find /var/log -name 'secure-*' -type f -print | xargs /bin/ls -las
4 -rw------- 1 root root  593 Dec 28 20:02 /var/log/secure-20131229
8 -rw------- 1 root root 6734 Jan  4 14:38 /var/log/secure-20140105
4 -rw------- 1 root root 3793 Jan 10 15:33 /var/log/secure-20140112
4 -rw------- 1 root root 1182 Jan 16 08:40 /var/log/secure-20140119

You do not really need the -print here, and you can do it with find alone as well:

# find /var/log -name 'secure-*' -type f -ls
11903950    4 -rw-------   1 root     root          593 Dec 28 20:02 /var/log/secure-20131229
11903981    8 -rw-------   1 root     root         6734 Jan  4 14:38 /var/log/secure-20140105
11903996    4 -rw-------   1 root     root         3793 Jan 10 15:33 /var/log/secure-20140112
11904057    4 -rw-------   1 root     root         1182 Jan 16 08:40 /var/log/secure-20140119

or:

# find /var/log -name 'secure-*' -type f -exec ls -las {} \;
4 -rw------- 1 root root 593 Dec 28 20:02 /var/log/secure-20131229
8 -rw------- 1 root root 6734 Jan  4 14:38 /var/log/secure-20140105
4 -rw------- 1 root root 3793 Jan 10 15:33 /var/log/secure-20140112
4 -rw------- 1 root root 1182 Jan 16 08:40 /var/log/secure-20140119

But maybe you forgot that syntax? It never hurts to have several approaches at hand!

Some more useful examples, the next one is to clean up things a bit. Say you have a couple of files in several directories:

# ls -las dir1; ls -las dir2
total 8
4 drwxr-xr-x 2 root root 4096 Jan 24 09:45 .
4 drwxr-xr-x 5 root root 4096 Jan 24 09:42 ..
0 -rw-r--r-- 1 root root    0 Jan 24 09:42 1
0 -rw-r--r-- 1 root root    0 Jan 24 09:42 2
total 8
4 drwxr-xr-x 2 root root 4096 Jan 24 09:45 .
4 drwxr-xr-x 5 root root 4096 Jan 24 09:42 ..
0 -rw-r--r-- 1 root root    0 Jan 24 09:42 3
0 -rw-r--r-- 1 root root    0 Jan 24 09:42 4

For some reason, you want them all to be moved to one single, new directory, let’s call it dir3:

# ls -las dir3
total 8
4 drwxr-xr-x 2 root root 4096 Jan 24 09:45 .
4 drwxr-xr-x 5 root root 4096 Jan 24 09:42 ..

You can now embark on a mv orgy, or you can do it a bit faster:

# find ./ -type f -print0 | xargs -0 -I {} mv {} dir3
# ls -las dir*
dir1:
total 8
4 drwxr-xr-x 2 root root 4096 Jan 24 09:46 .
4 drwxr-xr-x 5 root root 4096 Jan 24 09:42 ..

dir2:
total 8
4 drwxr-xr-x 2 root root 4096 Jan 24 09:46 .
4 drwxr-xr-x 5 root root 4096 Jan 24 09:42 ..

dir3:
total 8
4 drwxr-xr-x 2 root root 4096 Jan 24 09:46 .
4 drwxr-xr-x 5 root root 4096 Jan 24 09:42 ..
0 -rw-r--r-- 1 root root    0 Jan 24 09:42 1
0 -rw-r--r-- 1 root root    0 Jan 24 09:42 2
0 -rw-r--r-- 1 root root    0 Jan 24 09:42 3
0 -rw-r--r-- 1 root root    0 Jan 24 09:42 4

Nice, isn’t it? {} acts as placeholder for the input files and argument list, the -0 and -Ioptions are to handle special characters in filenames (which is also why we used -print0in the find command) and to replace a specified string occurrence with one from the standard input.

Now, let’s cp all these files into a single directory (e.g. useful to make a quick “backup” of files to an external drive, etc.):

# find /tmp/xargs/ -type f -print0 | xargs -0 -r -I file cp -v -p file --target-directory=/tmp/xargs/dir3
`/tmp/xargs/dir2/3' -> `/tmp/xargs/dir3/3'
`/tmp/xargs/dir2/4' -> `/tmp/xargs/dir3/4'
`/tmp/xargs/dir1/1' -> `/tmp/xargs/dir3/1'
`/tmp/xargs/dir1/2' -> `/tmp/xargs/dir3/2'
# ls -las dir3
total 8
4 drwxr-xr-x 2 root root 4096 Jan 24 09:58 .
4 drwxr-xr-x 5 root root 4096 Jan 24 09:42 ..
0 -rw-r--r-- 1 root root    0 Jan 24 09:42 1
0 -rw-r--r-- 1 root root    0 Jan 24 09:42 2
0 -rw-r--r-- 1 root root    0 Jan 24 09:42 3
0 -rw-r--r-- 1 root root    0 Jan 24 09:42 4

Looks good, all files copied.

Last, but not least, instead very nifty indeed, is quickly creating an archive of files (and/or directories):

# find /tmp/xargs -type f | xargs tar rvf archive.tar
tar: Removing leading `/' from member names
/tmp/xargs/dir2/3
/tmp/xargs/dir2/4
/tmp/xargs/dir1/1
/tmp/xargs/dir1/2
# ls -las
total 32
 4 drwxr-xr-x  5 root root  4096 Jan 24 10:01 .
 4 drwxrwxrwt. 4 root root  4096 Jan 24 09:42 ..
12 -rw-r--r--  1 root root 10240 Jan 24 10:01 archive.tar
 4 drwxr-xr-x  2 root root  4096 Jan 24 09:52 dir1
 4 drwxr-xr-x  2 root root  4096 Jan 24 09:52 dir2
 4 drwxr-xr-x  2 root root  4096 Jan 24 10:01 dir3
# tar -tf archive.tar
tmp/xargs/dir2/3
tmp/xargs/dir2/4
tmp/xargs/dir1/1
tmp/xargs/dir1/2

As we can see, all files are in the archive we created!

xargs isn’t something one can learn on the fly, especially the more complex operations that it can handle – but the latter fact makes it a very valuable tool for system administrators of both virtual private servers and dedicated servers.

Now it is time to wish you good luck and a lot of fun exploring the world of xargs!

1 from the man page of GNU’s version of xargs