C++ – Is endian conversion required for wchar_t data

c++data-transferendiannesswchar-t

In C/C++, if a multi-byte wide character (wchar_t) value is transmitted from a big-endian system to a little-endian system (or vice-versa), will it come out the same value on the other side? Or will the bytes need to be swapped?

Best Solution

Yes you will need to swap them.
The bytes will be retrieved from the transport in the same order they were put in. Just at the other end the ordering of these bytes has a different meaning. So you need to convert them to the correct endian-ness (is that a word?).

The tried and true method is to convert to network byte order before transport. Then convert back to host specific byte order (from network byte order) on receipt.

A set of function to help with endian conversion:

ntohs   Convert a 16-bit quantity from network byte order to host byte order
ntohl   Convert a 32-bit quantity from network byte order to host byte order
htons   Convert a 16-bit quantity from host byte order to network byte order
htonl   Convert a 32-bit quantity from host byte order to network byte order

Just to add another note of caution.
Different systems use different size for wchar_t so do not assume sizeof(wchar_t) == 2.

Additionally each host may use a different representational format for wchar_t.
To help deal with this most systems convert the text to a known format for transport (UTF-8 or UTF-16 are good choices). The convert the text back to the host specific format at the other end.

You could look at IBM's icu this has all this functionality.

Related Solutions

C++ – Detecting endianness programmatically in a C++ program

I don't like the method based on type punning - it will often be warned against by compiler. That's exactly what unions are for !

bool is_big_endian(void)
{
    union {
        uint32_t i;
        char c[4];
    } bint = {0x01020304};

    return bint.c[0] == 1; 
}

The principle is equivalent to the type case as suggested by others, but this is clearer - and according to C99, is guaranteed to be correct. gcc prefers this compared to the direct pointer cast.

This is also much better than fixing the endianness at compile time - for OS which support multi-architecture (fat binary on Mac os x for example), this will work for both ppc/i386, whereas it is very easy to mess things up otherwise.

Sqlite – Improve INSERT-per-second performance of SQLite

Several tips:

Put inserts/updates in a transaction.
For older versions of SQLite - Consider a less paranoid journal mode (pragma journal_mode). There is NORMAL, and then there is OFF, which can significantly increase insert speed if you're not too worried about the database possibly getting corrupted if the OS crashes. If your application crashes the data should be fine. Note that in newer versions, the OFF/MEMORY settings are not safe for application level crashes.
Playing with page sizes makes a difference as well (PRAGMA page_size). Having larger page sizes can make reads and writes go a bit faster as larger pages are held in memory. Note that more memory will be used for your database.
If you have indices, consider calling CREATE INDEX after doing all your inserts. This is significantly faster than creating the index and then doing your inserts.
You have to be quite careful if you have concurrent access to SQLite, as the whole database is locked when writes are done, and although multiple readers are possible, writes will be locked out. This has been improved somewhat with the addition of a WAL in newer SQLite versions.
Take advantage of saving space...smaller databases go faster. For instance, if you have key value pairs, try making the key an INTEGER PRIMARY KEY if possible, which will replace the implied unique row number column in the table.
If you are using multiple threads, you can try using the shared page cache, which will allow loaded pages to be shared between threads, which can avoid expensive I/O calls.
Don't use !feof(file)!

I've also asked similar questions here and here.

Best Solution

Related Solutions

C++ – Detecting endianness programmatically in a C++ program

Sqlite – Improve INSERT-per-second performance of SQLite

Related Question