A Mysterious Crawler Bug Actually Made Me Give Up Python!? The Reason Behind It Is Heartbreaking~

Behind every successful programmer, there may stand countless bald men—providing them with various development tools & code libraries, of course including... all kinds of mysterious bugs...

The Beginning of the Mystery

Recently, while working on a crawler project with Python, I ran into a very strange problem, and it doesn't even trigger every time—truly baffling...

The error message is as follows:

UnicodeEncodeError: 'latin-1' codec can't encode character '\u2026' in position 512: ordinal not in range(256)

I took the error message to a search engine and looked it up. The Chinese community said it was because there was Chinese data in the request body or headers,

and the solution was: first encode it as UTF-8, then decode it with latin-1 encoding.

But there was no Chinese in my request data!

Thus began a hair-loss-inducing journey of hunting down the bug

Looking at the Code

(Long and tedious—don't bother, suggest skipping to what follows)

First, let's look at the format of the data I submitted~

This is related to authentication (too long, so only a snippet)

"spider9": {
  "Authorization": "Basic Z2VjZW50ZXJfYWR",
  "Blade-Auth": "bearer eyJhbGciOiJIUzI1NiIsI",
  "cookie": "oauth=eyJhY2Nlc3NfdG"
}

Here's the header-related code

headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:88.0) Gecko/20100101 Firefox/88.0',
    'Accept': 'application/json, text/plain, */*',
    'Accept-Language': 'zh-CN,zh;q=0.8,zh-TW;q=0.7,zh-HK;q=0.5,en-US;q=0.3,en;q=0.2',
    'Accept-Encoding': 'gzip, deflate', 'Content-Type': 'application/json;charset=utf-8',
    'Authorization': config['spider9']['Authorization'],
    'Blade-Auth': config['spider9']['Blade-Auth'].encode('utf-8').decode('latin1'),
    'Content-Length': '60', 'Connection': 'keep-alive',
    'Cookie': config['spider9']['cookie'],
    'Pragma': 'no-cache', 'Cache-Control': 'no-cache',
}

Here's the requests request code:

response = requests.post(
    url, headers=headers, verify=False,
    json={
        'cityCode': '1234',
        'createTimeFrom': None,
        'createTimeTo': None
    }
)

Just looking at this code, there shouldn't be any problem at all. In fact, my other crawlers are written the same way and have been running stably for over a year. It's just this newly written crawler that doesn't work... Sometimes code problems are just this mysterious...

As mentioned earlier, I found netizens suggesting the encode-then-decode method, so I tried adding it to the three fields Authorization, Blade-Auth, and Cookie in the headers:

'Authorization': config['spider9']['Authorization'].encode('utf-8').decode('latin1')

That way it no longer threw this UnicodeEncodeError, but the backend service directly reported an error saying I wasn't logged in...

So what kind of problem is this then?

I continued searching on Stack Overflow. A Russian netizen said they also ran into this problem, and there was a reply below suggesting trying to set an environment variable. OK, so I followed along and tried:

export PYTHONUTF8=1

Then I printed the system encoding and locale in Python:

import sys
import locale

print(sys.getfilesystemencoding())
print(locale.getpreferredencoding())

Output

utf-8
UTF-8

Oh wow~ Tried running it again... still didn't work. I'm speechless. So it wasn't this problem at all.

Alright, I surrender. I don't want to keep grinding away at it. Where I fall, there I lie down

So what was the problem? To this day it remains an unsolved mystery...

Aftermath

So tired. Switched to writing crawlers in C#, giving up Python...

What happened to “Life is short, I use Python”? How did it become such a hassle TAT...

(PS: I previously built a crawler platform that can schedule crawler programs implemented in different languages, providing a unified configuration center and a unified data persistence interface~ So it doesn't make much difference which language each crawler is written in)

I'm writing this article as a record, hoping that someday in the future this problem can be solved~ (A beautiful wish)

References

• https://stackoverflow.com/questions/64769797/latin-1-codec-cant-encode-character-u2019
• https://www.cnblogs.com/xtmp/p/12693862.html