I have a method that creates a MessageDigest (a hash) from a file, and I need to do this to a lot of files (>= 100,000). How big should I make the buffer used to read from the files to maximize performance?
Most everyone is familiar with the basic code (which I'll repeat here just in case):
MessageDigest md = MessageDigest.getInstance( "SHA" ); FileInputStream ios = new FileInputStream( "myfile.bmp" ); byte buffer = new byte[4 * 1024]; // what should this value be? int read = 0; while( ( read = ios.read( buffer ) ) > 0 ) md.update( buffer, 0, read ); ios.close(); md.digest();
What is the ideal size of the buffer to maximize throughput? I know this is system dependent, and I'm pretty sure its OS, FileSystem, and HDD dependent, and there maybe other hardware/software in the mix.
(I should point out that I'm somewhat new to Java, so this may just be some Java API call I don't know about.)
Edit: I do not know ahead of time the kinds of systems this will be used on, so I can't assume a whole lot. (I'm using Java for that reason.)
Edit: The code above is missing things like try..catch to make the post smaller