Java – Partially load large text file with different encodings

encodingjavalarge-filesloadtext

I am writing a Java text component, and is trying to partially load some large text file in the middle (for speed reasons).

My question is if the text is in some multi-bytes encoding format, like UTF8, Big5, GBK, etc. How can I align the bytes so that I can correctly decode the text?

Best Solution

I can't speak for the other formats but utf8 shouldn't be too hard.

Just look at the first byte of the chunk you grabbed and figure out from there:

Taken from wikipedia:

00000000-01111111   00-7F   0-127   US-ASCII (single byte)
10000000-10111111   80-BF   128-191 2'nd, 3rd, or 4'th byte of a multi-byte sequence
11000000-11000001   C0-C1   192-193 start of a 2-byte sequence, but code point <= 127
11000010-11011111   C2-DF   194-223 Start of 2-byte sequence
11100000-11101111   E0-EF   224-239 Start of 3-byte sequence
11110000-11110100   F0-F4   240-244 Start of 4-byte sequence

If the byte is in the 2'nd or 3'rd group then you know you missed part of a character. If it's in the 1'st,4'th,5'th,6'th group then you know you are on the start of a character. Proceed accordingly from there.

Related Solutions

Git – Managing large binary files with Git

I discovered git-annex recently which I find awesome. It was designed for managing large files efficiently. I use it for my photo/music (etc.) collections. The development of git-annex is very active. The content of the files can be removed from the Git repository, only the tree hierarchy is tracked by Git (through symlinks). However, to get the content of the file, a second step is necessary after pulling/pushing, e.g.:

$ git annex add mybigfile
$ git commit -m'add mybigfile'
$ git push myremote
$ git annex copy --to myremote mybigfile ## This command copies the actual content to myremote
$ git annex drop mybigfile ## Remove content from local repo
...
$ git annex get mybigfile ## Retrieve the content
## or to specify the remote from which to get:
$ git annex copy --from myremote mybigfile

There are many commands available, and there is a great documentation on the website. A package is available on Debian.

Java – How to check if a file exists in Java

Using java.io.File:

File f = new File(filePathString);
if(f.exists() && !f.isDirectory()) { 
    // do something
}

Best Solution

Related Solutions

Git – Managing large binary files with Git

Java – How to check if a file exists in Java

Related Question