Windows - UnicodeDecodeError: ‘charmap’ codec can’t decode byte X in position Y: character maps to

decodefile-iopython-3.xunicodewindows

I'm trying to get a Python 3 program to do some manipulations with a text file filled with information. However, when trying to read the file I get the following error:

 Traceback (most recent call last):  
     File "SCRIPT LOCATION", line NUMBER, in <module>  
     `text = file.read()`  
     File "C:\Python31\lib\encodings\cp1252.py", line 23, in decode  
     `return codecs.charmap_decode(input,self.errors,decoding_table)[0]`  
     UnicodeDecodeError: 'charmap' codec can't decode byte 0x90 in position 2907500: character maps to `<undefined>`

Best Solution

The file in question is not using the CP1252 encoding. It's using another encoding. Which one you have to figure out yourself. Common ones are Latin-1 and UTF-8. Since 0x90 doesn't actually mean anything in Latin-1, UTF-8 (where 0x90 is a continuation byte) is more likely.

You specify the encoding when you open the file:

file = open(filename, encoding="utf8")

Related Solutions

Python - UnicodeEncodeError: ‘ascii’ codec can’t encode character u’\xa0′ in position 20: ordinal not in range(128)

You need to read the Python Unicode HOWTO. This error is the very first example.

Basically, stop using str to convert from unicode to encoded text / bytes.

Instead, properly use .encode() to encode the string:

p.agent_info = u' '.join((agent_contact, agent_telno)).encode('utf-8').strip()

or work entirely in unicode.

Python JSON to CSV - bad encoding, UnicodeDecodeError: ‘charmap’ codec can’t decode byte

You need to know cyrylic encoding of which file are you going to open. For example that is enough in python3:

with open(args.input_file, 'r', encoding="cp866") as input_file:
        data = input_file.read()
        structure = json.loads(data)

In python3 data variable is automatically utf-8. In python2 there might be problem with feeding input to json.

Also try to print out in python interpreter line and see if symbols are right. Without input file is hard to tell if everything is right. Also are you sure that it is python, not excel related problem? Did you tried to open in notepad++ or similar encodings respecting editors?

Most important thing working with encodings is cheking that input and output is right. I would suggest to look here.

Best Solution

Related Solutions

Python - UnicodeEncodeError: ‘ascii’ codec can’t encode character u’\xa0′ in position 20: ordinal not in range(128)

Python JSON to CSV - bad encoding, UnicodeDecodeError: ‘charmap’ codec can’t decode byte

Related Question