Removing Invisible Unicode Characters from Strings in C#

Background

Recently I found that in data pulled down from a data collection system, the JSON in some fields was inexplicably truncated, causing JSON parsing to fail during subsequent data analysis.

Something like this

{"title": "你好

Or like this, with an extra double quote or something

{"title":""你好"}

Because the database is Oracle, I initially thought this old relic Oracle was the problem. After some fussing around, I pulled out the SQL statement for every write, and the JSON format inside all looked fine.

This is way too weird. It looks fine, so why was the JSON randomly truncated?

Finally I tried putting the entire SQL into Rider's query console and executing it, then reading this JSON back from the database, and surprisingly found it had become this:

{"title":"？你好"}

Ah, seeing this big question mark, I immediately knew that this “你好” was more than just these two characters—it definitely contained invisible Unicode characters.

Then I copied this JSON out, opened it in hex mode, and sure enough saw a \u0020 character in front of “你好”…

Unicode Code Charts

• 0000-007F: C0 Control Characters and Basic Latin (C0 Control and Basic Latin)
• 0080-00FF: C1 Control Characters and Latin-1 Supplement (C1 Control and Latin 1 Supplement)
• 0100-017F: Latin Extended-A (Latin Extended-A)
• 0180-024F: Latin Extended-B (Latin Extended-B)
• 0250-02AF: IPA Extensions (IPA Extensions)
• 02B0-02FF: Spacing Modifier Letters (Spacing Modifiers)
• ……

Here I’m also attaching part of the Unicode table

You can see that \u0020 above is in the third row, first column—it’s an invisible character, hiding in front of the title

It was because of this Unicode character that Oracle couldn’t parse it correctly, which caused things to get messed up when inserting data

So the case is solved: the system’s front-end users somehow got a Unicode character in there when typing…

The fix is for me to do another round of filtering when collecting on my side…

Didn’t expect filtering Unicode in C# to be such a hassle—there’s so little material…

In the end I still put it together by referring to Java materials. = =...

Code

The code is as follows

I wrote an extension method to filter

public static class StringExt { 
    // 控制字符
    private static readonly Regex ControlCharRegex = new Regex(@"[\p{C}]", RegexOptions.Compiled);

    /// <summary>
    /// 移除控制字符
    /// </summary>
    public static string RemoveControlChars(this string text) {
        return ControlCharRegex.Replace(text, string.Empty);
    }
}

When you need to use it, do it like this

var outStr = "带有Unicode的字符串".RemoveControlChars();

Done.

References

• Unicode Code Chart and Some Invisible Character Filtering Approaches - https://www.cnblogs.com/fan-yuan/p/8176886.html
• https://stackoverflow.com/questions/6198986/how-can-i-replace-non-printable-unicode-characters-in-java