Read all text from a file
Java 11 added the readString() method to read small files as a String
, preserving line terminators:
String content = Files.readString(path, StandardCharsets.US_ASCII);
For versions between Java 7 and 11, here's a compact, robust idiom, wrapped up in a utility method:
static String readFile(String path, Charset encoding)
throws IOException
{
byte[] encoded = Files.readAllBytes(Paths.get(path));
return new String(encoded, encoding);
}
Read lines of text from a file
Java 7 added a convenience method to read a file as lines of text, represented as a List<String>
. This approach is "lossy" because the line separators are stripped from the end of each line.
List<String> lines = Files.readAllLines(Paths.get(path), encoding);
Java 8 added the Files.lines()
method to produce a Stream<String>
. Again, this method is lossy because line separators are stripped. If an IOException
is encountered while reading the file, it is wrapped in an UncheckedIOException
, since Stream
doesn't accept lambdas that throw checked exceptions.
try (Stream<String> lines = Files.lines(path, encoding)) {
lines.forEach(System.out::println);
}
This Stream
does need a close()
call; this is poorly documented on the API, and I suspect many people don't even notice Stream
has a close()
method. Be sure to use an ARM-block as shown.
If you are working with a source other than a file, you can use the lines()
method in BufferedReader
instead.
Memory utilization
The first method, that preserves line breaks, can temporarily require memory several times the size of the file, because for a short time the raw file contents (a byte array), and the decoded characters (each of which is 16 bits even if encoded as 8 bits in the file) reside in memory at once. It is safest to apply to files that you know to be small relative to the available memory.
The second method, reading lines, is usually more memory efficient, because the input byte buffer for decoding doesn't need to contain the entire file. However, it's still not suitable for files that are very large relative to available memory.
For reading large files, you need a different design for your program, one that reads a chunk of text from a stream, processes it, and then moves on to the next, reusing the same fixed-sized memory block. Here, "large" depends on the computer specs. Nowadays, this threshold might be many gigabytes of RAM. The third method, using a Stream<String>
is one way to do this, if your input "records" happen to be individual lines. (Using the readLine()
method of BufferedReader
is the procedural equivalent to this approach.)
Character encoding
One thing that is missing from the sample in the original post is the character encoding. There are some special cases where the platform default is what you want, but they are rare, and you should be able justify your choice.
The StandardCharsets
class defines some constants for the encodings required of all Java runtimes:
String content = readFile("test.txt", StandardCharsets.UTF_8);
The platform default is available from the Charset
class itself:
String content = readFile("test.txt", Charset.defaultCharset());
Note: This answer largely replaces my Java 6 version. The utility of Java 7 safely simplifies the code, and the old answer, which used a mapped byte buffer, prevented the file that was read from being deleted until the mapped buffer was garbage collected. You can view the old version via the "edited" link on this answer.
Contrary to the answers here, you DON'T need to worry about encoding if the bytes don't need to be interpreted!
Like you mentioned, your goal is, simply, to "get what bytes the string has been stored in".
(And, of course, to be able to re-construct the string from the bytes.)
For those goals, I honestly do not understand why people keep telling you that you need the encodings. You certainly do NOT need to worry about encodings for this.
Just do this instead:
static byte[] GetBytes(string str)
{
byte[] bytes = new byte[str.Length * sizeof(char)];
System.Buffer.BlockCopy(str.ToCharArray(), 0, bytes, 0, bytes.Length);
return bytes;
}
// Do NOT use on arbitrary bytes; only use on GetBytes's output on the SAME system
static string GetString(byte[] bytes)
{
char[] chars = new char[bytes.Length / sizeof(char)];
System.Buffer.BlockCopy(bytes, 0, chars, 0, bytes.Length);
return new string(chars);
}
As long as your program (or other programs) don't try to interpret the bytes somehow, which you obviously didn't mention you intend to do, then there is nothing wrong with this approach! Worrying about encodings just makes your life more complicated for no real reason.
Additional benefit to this approach: It doesn't matter if the string contains invalid characters, because you can still get the data and reconstruct the original string anyway!
It will be encoded and decoded just the same, because you are just looking at the bytes.
If you used a specific encoding, though, it would've given you trouble with encoding/decoding invalid characters.
Best Solution
EDIT: In the answer below I've referred to the intern pool as being AppDomain-specific; I'm pretty sure that's what I've observed before, but the MSDN docs for String.Intern suggest that there's a single intern pool for the whole process, making this even more important.
Original answer
(I was going to add this as a comment, but I think it's an important enough point to need an extra answer...)
As others have explained, string interning occurs for all string literals, but not on "dynamically created" strings (e.g. those read from a database or file, or built using
StringBuilder
orString.Format
.)However, I wouldn't suggest calling
String.Intern
to get round the latter point: it will populate the intern pool for the lifetime of yourAppDomain
. Instead, use a pool which is local to just your usage. Here's an example of such a pool:You'd then just use something like:
(Note that the pool isn't thread-safe; normal usage wouldn't need it to be.)
This way you can throw away your pool as soon as you no longer need it, rather than having a potentially large number of strings in memory forever. You could also make it smarter, implementing an LRU cache or something if you really wanted to.
EDIT: Just to clarify why this is better than using
String.Intern
... suppose you read a bunch of strings from a database or log file, process them, and then move onto another task. If you callString.Intern
on those strings, they will never be garbage collected as long as yourAppDomain
is alive - and possibly not even then. If you load several different log files, you'll gradually accumulate strings in your intern pool until you either finish or run out of memory. Instead, I'm suggesting a pattern like this:Here you get the benefit of multiple strings in the same file only existing once in memory (or at least, only getting past gen0 once) but you don't pollute a "global" resource (the intern pool).