Comments on: Recursive MD5 Sum Script

By: Jonathan

Jonathan — Sat, 28 Jul 2012 23:52:41 +0000

I’ve had a need for verifying integrity of backups/mirrors which contain a large number of files and ended up writing a command-line program called MassHash. It’s written in Python. A GTK+ Launcher is also available. You may want to check it out…

http://code.google.com/p/masshash/

By: enars

enars — Sat, 28 Jul 2012 09:09:58 +0000

Another solution would be the md5deep utility ->

http://md5deep.sourceforge.net/

under terminal (ubuntu 10.10)

$ md5deep

$ sudo apt-get install md5deep

By: Ben in Seattle

Ben in Seattle — Sun, 01 Aug 2010 04:19:13 +0000

Uhh… duh…. I neglected to remove the “/” or “$1″ from my tweak of David E’s version. Not a big deal, since I think my later version is better, but I feel silly now.

By: Ben in Seattle

Ben in Seattle — Sun, 01 Aug 2010 04:11:23 +0000

I love seeing the evolution of this script in the comments as it gets tighter and better. It really shows you how Unix is a toolbox: even with only basic knowledge of the shell you can get your work done, but as you become an expert you findsharp, precise tools and the knowledge to handle them. David E's version looks well-nigh perfect, but it has a couple extraneous parts that can be cut out. First is the slash in "./" which can just be typed ".". The second is the "$1" which I think was a typo or a remnant of a shell script. So, my tweak on his version would be: find ./ -type f -exec md5sum $1 {} \; > checksums.md5 Or, if you're using GNU/Linux (which you probably are if you're using the Linux kernel), you can omit the period since GNU find defaults to searching the current working directory. Also, instead of using "\;", which runs a separate md5sum process for every file, GNU find has the nifty plus syntax, "+", which passes all the filenames found as command line arguments to one instance of md5sum. That will make it run faster and also has the benefit that plus is not a shell special character so it doesn't have to be escaped by a backslash. So, my final recommendation (though I'm sure somebody will come along and improve upon this), is: find -type f -exec md5sum {} + > checksums.md5 Of course, to check the sums, it's still the same md5sum -c checksums.md5

By: spargonaut

spargonaut — Sun, 10 Jan 2010 02:00:33 +0000

this is great.
thanks to everyone for the input.

I’m trying to accomplish something similar ( in addition to sharpening my shell chops ).
After creating an md5sum for each file, I want it to start back at the beginning of the file and search for duplicates, and write the possible duplicates to another file.

this thread gives me a great head start.

-js

By: David E.

David E. — Fri, 27 Nov 2009 15:23:33 +0000

I like the comment about using the following command: “find * ! -type d -print0 | xargs -0 md5sum”

You could accomplish all that by using the find command as follows: “find ./ -type f -exec md5sum $1 {} \;”

The important thing to note about the script in this post is that when parsing files by filename, its usually best to avoid using the “for” command; as it uses a space as the default field separator… This means that if it runs across a filename with a space in it, which is common if you are scripting against a file server, it will run into a lot of problems.

If you have an aversion against using find, it would be more advisable to do the following: “ls | while read FILE; do md5sum “$FILE”; done”

By: TheLinuxBlog.com

TheLinuxBlog.com — Thu, 12 Nov 2009 14:45:55 +0000

@APROTIM nothing is wrong with md5sum

/* for finding files within a directory, but it won’t go into the directories. It would basically do the same thing as “md5sum $directory’/'$x;” the $x is there in case I wanted to extend it to multiple levels.

By: aprotim

aprotim — Wed, 11 Nov 2009 21:13:01 +0000

What’s wrong with:

md5sum $directory/*

It will print errors to stderr for non-plain files found, but stdout will have the hashes, as required…

By: der Dennis

der Dennis — Fri, 09 Oct 2009 07:03:54 +0000

Another solution would be the md5deep utility ->

http://md5deep.sourceforge.net/

By: log69

log69 — Fri, 30 Jan 2009 14:04:46 +0000

or a better one ;)

find ! -type d -print0 | xargs -0 md5sum | md5sum