11 Replies Latest reply on Jul 21, 2008 8:09 PM by user4994457

arg list too long, on MV command with directory holding 30.000+ files

807567 Apr 17, 2008 8:04 AM

I'm trying to move all files, in a given directory, to another directory. Special thing about is, is that this directory holds a lot of files, in this case somewhat more than 30.000 files. The command runs for a while, then states "arg list too long", and stops. No file at all is being copied !

1. Re: arg list too long, on MV command with directory holding 30.000+ files

807567 Apr 17, 2008 12:13 PM (in response to 807567)

You can solve this problem using less number of files to move at a single try. The script below makes it for you.

1. create a script with the contents below the ##### line
2. edit DSTDIR and SRCDIR variables to match your source and destination directories
3. save the script and chmod +x it.
4. execute it giving the number of files to copy in a single step as the first argument for example: ./script.sh 500 (will move 500 files from source to destdir).
5. execute the script several times until all files relocate to the new directory.

####################################################

#!/bin/sh

DSTDIR=/home/mef/nisbackup/test/
SRCDIR=/home/mef/nisbackup/src/

if [ $1 ]
then

SALUT=`/usr/bin/ls -a -1 |tail -$1 | xargs`

for FILE in $SALUT
do
/usr/bin/mv $SRCDIR/$FILE $DSTDIR
done

else
echo 'usage: ./$0 chunk_size'
exit 1;
fi
Like Show 0 Likes(0)

Actions
2. Re: arg list too long, on MV command with directory holding 30.000+ files

807567 Apr 17, 2008 2:28 PM (in response to 807567)
You can try something like this:

tar -cf - * | (cd ../dst; tar -xf -)
For this example, cd into your source directory. the command written in the sub-shell assumes your destination directory is ../dst. Modify this to fit your environment.
Like Show 0 Likes(0)

Actions
3. Re: arg list too long, on MV command with directory holding 30.000+ files

807567 Apr 18, 2008 8:25 AM (in response to 807567)

Thanks,
I'll have a look at both proposed solutions

Any chance that Solaris may solve this bug ?
Like Show 0 Likes(0)

Actions
4. Re: arg list too long, on MV command with directory holding 30.000+ files

807567 Apr 18, 2008 8:32 AM (in response to 807567)

This is not a bug, but a feature rather.

Every UNIX shell has it's argument list limit.
Like Show 0 Likes(0)

Actions
5. Re: arg list too long, on MV command with directory holding 30.000+ files

807567 Apr 18, 2008 9:43 PM (in response to 807567)

99480 wrote:
I'm trying to move all files, in a given directory, to another directory. Special thing about is, is that this directory holds a lot of files, in this case somewhat more than 30.000 files. The command runs for a while, then states "arg list too long", and stops. No file at all is being copied !
Are there no "old school" system administrators left? :-)

Assume that the source directory is /export/home/ and the new directory is /local/home
# cd /export
# find home -print | cpio -pdmv /local/
# pwd
/export
# rm -r home

Cheers,

Edited by: John.Kotches on Apr 18, 2008 4:42 PM
Like Show 0 Likes(0)

Actions
6. Re: arg list too long, on MV command with directory holding 30.000+ files

800381 Apr 18, 2008 9:44 PM (in response to 807567)
Assuming no subdirectories, you can also use the "find" command:

find . -exec mv {} /dest/dir \;
And it might even work for subdirectories, but I'm not going to guarantee it.
Like Show 0 Likes(0)

Actions
7. Re: arg list too long, on MV command with directory holding 30.000+ files

807567 Apr 21, 2008 2:38 PM (in response to 800381)
AndrewHenle wrote:
Assuming no subdirectories, you can also use the "find" command:

find . -exec mv {} /dest/dir \;
And it might even work for subdirectories, but I'm not going to guarantee it.
When you use find -exec you fork a shell for each file which is quite CPU intensive. Not much different than my find in practical use.

Cheers,
Like Show 0 Likes(0)

Actions
8. Re: arg list too long, on MV command with directory holding 30.000+ files

800381 Apr 21, 2008 10:12 PM (in response to 807567)
John.Kotches wrote:
AndrewHenle wrote:
Assuming no subdirectories, you can also use the "find" command:

find . -exec mv {} /dest/dir \;
And it might even work for subdirectories, but I'm not going to guarantee it.
When you use find -exec you fork a shell for each file which is quite CPU intensive. Not much different than my find in practical use.

Cheers,
I think there are three practical differences.

First, by using cpio to do a copy and then having to come back and delete the original files, if the source and target directories are located on the same file system there has to be enough empty space to hold an extra copy of the entire directory. And if the source and target directory do happen to be on the same file system, a simple rename() is all that is necessary to move the file, which has minimal CPU utilization as the actual contents of the file never move on the physical disk. An actual copy of the file data would be unnecessary in this case. So, yes, while a fork() and rename() does have some overhead, it's less than a single child process recreating each file and copying its data over to the new copy. And if the source and target directory are on separate disks, the CPU cost of the file creation and data copy dwarfs the impact of a fork() for each file anyway.

Second, by accomplishing the task of moving the files with one command, there's no opportunity to mistakenly type "rm -f -r *" in the wrong location.

Third, your method will work for directory trees. I'm not so sure about mine.
Like Show 0 Likes(0)

Actions
9. Re: arg list too long, on MV command with directory holding 30.000+ files

807567 Apr 28, 2008 2:41 PM (in response to 800381)

AndrewHenle wrote:
I think there are three practical differences.

First, by using cpio to do a copy and then having to come back and delete the original files, if the source and target directories are located on the same file system there has to be enough empty space to hold an extra copy of the entire directory. And if the source and target directory do happen to be on the same file system, a simple rename() is all that is necessary to move the file, which has minimal CPU utilization as the actual contents of the file never move on the physical disk. An actual copy of the file data would be unnecessary in this case. So, yes, while a fork() and rename() does have some overhead, it's less than a single child process recreating each file and copying its data over to the new copy. And if the source and target directory are on separate disks, the CPU cost of the file creation and data copy dwarfs the impact of a fork() for each file anyway.
This depends on the number of files.... If we're talking a few files yes. 30K files? Well that's a little different.

Second, by accomplishing the task of moving the files with one command, there's no opportunity to mistakenly type "rm -f -r *" in the wrong location.
Which is why I suggested the pwd first ;-) Having two copies of the data also acts as a safety net.

Third, your method will work for directory trees. I'm not so sure about mine.
I never use mv for bulk moves including subtrees. I use either cpio or cp -pr whichever suits the task better. Given the size of modern disk hardware unless it's a "stupid big" move I don't worry about a temporary second copy.
Like Show 0 Likes(0)

Actions
10. Re: arg list too long, on MV command with directory holding 30.000+ files

807567 Jul 15, 2008 2:46 PM (in response to 807567)

I need to have a look at the latest updates, I'm still hitting the same bug. I'm also trying to look how many files are in the directory, but the "ls" command fails as well. I don't consider this a bug, I think it is a severe bug ... a cp or mv command not working, I consider pretty nasty. Now having issues with directories containing 120.000 files in them. If the system commands cannot handle that amount of files, they should not allow them to be created therein. Of course, the bug would become more obvious then. Anyway, I'm considering a compression tool to work around this bug.
Like Show 0 Likes(0)

Actions
11. Re: arg list too long, on MV command with directory holding 30.000+ files

user4994457 Jul 21, 2008 8:09 PM (in response to 807567)
99480 wrote:
I need to have a look at the latest updates, I'm still hitting the same bug. I'm also trying to look how many files are in the directory, but the "ls" command fails as well.
How is 'ls' being executed, and what error are you getting? You may want to use 'ls -f', otherwise ls will try to sort the filenames, and that's not a good idea when you have multi-thousands of files.

I don't consider this a bug, I think it is a severe bug
It's not a bug. There are things the system is not supposed to be able to do. This is one of them.

... a cp or mv command not working, I consider pretty nasty.

Now having issues with directories containing 120.000 files in them. If the system commands cannot handle that amount of files, they should not allow them to be created therein. Of course, the bug would become more obvious then. Anyway, I'm considering a compression tool to work around this bug.
There is a difference between the limits of files in a directory and the argument limits to a process. They are not related.

Argument list size failures are documented for exec() functions.

# man execve [...] The exec functions will fail if: E2BIG The number of bytes in the new process's argument list is greater than the system-imposed limit of ARG_MAX bytes. The argument list limit is sum of the size of the argument list plus the size of the environment's exported shell variables.
# grep ARG_MAX /usr/include/limits.h
* ARG_MAX is calculated as follows:
#define ARGMAX32 1048320 /* max length of args to exec 32-bit program */
#define ARGMAX64 2096640 /* max length of args to exec 64-bit program */
#define ARG_MAX ARGMAX64 /* max length of arguments to exec */
#define ARG_MAX ARGMAX32 /* max length of arguments to exec */
#define POSIXARG_MAX 4096

--
Darren
Like Show 0 Likes(0)

Actions

Go to original post