Notes on C# Crawler Development

Preface

I've been busy since 2023. With the Spring Festival approaching, there are even more miscellaneous chores, but having put off writing articles for so long always feels uncomfortable, so I'm stealing some time from my busy schedule to jot down some notes on recently writing crawlers in C#.

Crawlers are generally written in Python—the ecosystem is rich, dynamic languages are fast to develop with, and debugging is convenient too

But

I have to say but—dynamic languages have their limitations too. As a veteran crawler mentor, I've worked with almost every language. This task isn't complicated, so using my favorite C# is a piece of cake~

Getting Started

When I was working on the OneCat project before, the initial data collection module was built with C#. I also integrated Chloe as the ORM and used Nancy for HTTP APIs. Combined with C#'s powerful concurrency capabilities, the results were pretty good.

This time I need to crawl some wallpapers—a very simple scenario—so I reused some utility classes from the previous OneCat project and made some improvements.

HttpHelper

For network requests I directly use HttpClient from the .NET Core standard library. This library requires using a singleton. In ASP.NET Core you'd typically use dependency injection, but for this simple crawler a Console app is enough.

After crawling the HTML, you still need to parse it. In Python you'd typically use BeautifulSoup; in C# you can use AngleSharp, which is also quite nice~

For convenience, I wrapped another utility class that integrates HttpClient and AngleSharp.

public static class HttpHelper {
    public const string UserAgent =
        "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36";

    public static HttpClientHandler Handler { get; }

    public static HttpClient Client { get; }

    static HttpHelper() {
        Handler = new HttpClientHandler();
        Client = new HttpClient(Handler);
        Client.DefaultRequestHeaders.Add("User-Agent", UserAgent);
    }

    public static async Task<IHtmlDocument> GetHtmlDocument(string url) {
        var html = await Client.GetStringAsync(url);
        // todo 这个用法有内存泄漏问题，得优化一下
        return new HtmlParser().ParseDocument(html);
    }

    public static async Task<IHtmlDocument> GetHtmlDocument(string url, string charset) {
        var res = await Client.GetAsync(url);
        var resBytes = await res.Content.ReadAsByteArrayAsync();
        var resStr = Encoding.GetEncoding(charset).GetString(resBytes);
        // todo 这个用法有内存泄漏问题，得优化一下
        return new HtmlParser().ParseDocument(resStr);
    }
}

There are two todos in this code. The memory leak isn't a big deal in a simple crawler, so I'll optimize it later if there's a large-scale need~

Working with HTML

Most crawlers get data from web pages

If the page is rendered on the backend, with no JS dynamically loading data, you can basically get any data you want with CSS selectors + regular expressions.

With the wrapping above, requesting a page + parsing HTML takes just one line of code

IHtmlDocument data = await HttpHelper.GetHtmlDocument(url);

After getting the IHtmlDocument object, pass a CSS selector to QuerySelector and you can get all kinds of elements.

For example, like this, extracting the addresses of all links under <li> elements

var data = await HttpHelper.GetHtmlDocument(url);
foreach (var item in data.QuerySelectorAll(".pagew li")) {
    var link = item.QuerySelector("a");
    var href = link?.GetAttribute("href");
    if (href != null) await CrawlItem(href);
}

Or combined with regular expressions

var data = await HttpHelper.GetHtmlDocument(url);
var page = data.QuerySelector(".pageinfo");
Console.WriteLine("拿到分页信息：{0}", page?.TextContent);
var match = Regex.Match(page?.TextContent ?? "", @"共\s(\d+)页(\d+)条");
var pageCount = int.Parse(match.Groups[1].Value);
for (int i = 1; i <= pageCount; i++) {
    await CrawlPage(i);
}

Regular expressions are extremely useful—a must-have for crawlers~

Here's another useful thing I'd recommend: Runoob's online regex tester. After getting a string, first write a matching regex in the tester, then put it in the program—much more efficient~

URL: https://c.runoob.com/front-end/854/

JSON Handling

A well-worn topic

JSON is very common in web development. Whether for API interaction or saving data locally, it's a great format

The built-in System.Text.Json in .NET Core is pretty good—no need to manually install dependencies. If you don't have special requirements, just use this

The scenario here is saving collected data to JSON, i.e. serialization. Just go all-in with the following config code and it can handle most scenarios

var jsonOption = new JsonSerializerOptions {
    WriteIndented = true,
    Encoder = JavaScriptEncoder.UnsafeRelaxedJsonEscaping
};

Writing to a file

await File.WriteAllTextAsync("path", JsonSerializer.Serialize(data, jsonOption));

Further Reading

• A Few Common Issues and Solutions When Using System.Text.Json in .NET 6
• ASP.NET Core Development Notes: Nested Recursion Problem When Returning JSON Objects from APIs

Downloading Files

The simplest way is to get the Response directly with HttpClient, then CopyToAsync into a file stream

This approach is fine for downloading a few small files, but multithreaded downloads, resume on disconnect, retry on failure, and similar methods you'd have to implement yourself—pretty tedious.

So this time I just used the third-party library Downloader. This library looks pretty powerful with lots of features. I won't translate them—see the project homepage for details

Project URL: https://github.com/bezzad/Downloader

Similarly, I also wrapped the download functionality into HttpHelper

Add this part of the code

public static IDownloadService Downloader { get; }

public static DownloadConfiguration DownloadConf => new DownloadConfiguration {
    BufferBlockSize = 10240, // 通常，主机最大支持8000字节，默认值为8000。
    ChunkCount = 8, // 要下载的文件分片数量，默认值为1
    // MaximumBytesPerSecond = 1024 * 50, // 下载速度限制，默认值为零或无限制
    MaxTryAgainOnFailover = 5, // 失败的最大次数
    ParallelDownload = true, // 下载文件是否为并行的。默认值为false
    Timeout = 1000, // 每个 stream reader  的超时（毫秒），默认值是1000
    RequestConfiguration = {
        Accept = "*/*",
        AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate,
        CookieContainer = new CookieContainer(), // Add your cookies
        Headers = new WebHeaderCollection(), // Add your custom headers
        KeepAlive = true,
        ProtocolVersion = HttpVersion.Version11, // Default value is HTTP 1.1
        UseDefaultCredentials = false,
        UserAgent = UserAgent
    }
};

static HttpHelper() {
    // ...
    Downloader = new DownloadService(DownloadConf);
}

Usage is still one line of code

await HttpHelper.Downloader.DownloadFileTaskAsync(url, filepath);

This time I didn't wrap a download method directly, but instead made the IDownloadService object a property, because you often want to add some "buffs" when downloading

For example, listening to download progress—see the code below

HttpHelper.Downloader.DownloadStarted += DownloadStarted;
HttpHelper.Downloader.DownloadFileCompleted += DownloadFileCompleted;
HttpHelper.Downloader.DownloadProgressChanged += DownloadProgressChanged;
HttpHelper.Downloader.ChunkDownloadProgressChanged += ChunkDownloadProgressChanged;

This library provides four events, namely:

• Download started
• Download completed
• Download progress changed
• Chunk download progress changed

Progress Bar

With these events, you can implement a download progress bar display. The progress bar introduced next is also the one used in the official examples of the Downloader library

Project URL: https://github.com/Mpdreamz/shellprogressbar

First, forget the examples on the official site—those examples aren't very useful in practice.

Tick Mode

This progress bar has two modes. One is its own Tick method: first define the total number of tasks, and each execution means one task is completed, for example:

using var bar = new ProgressBar(10, "正在下载所有图片", BarOptions);

The code above defines 10 tasks. Each call to bar.Tick() means one task is completed; after executing it 10 times the whole thing is done~

IProgress<T> Mode

This IProgress<T> is a type from the C# standard library, used for handling progress bars.

The ProgressBar object can be converted to an IProgress<T> object using the AsProgress<T> method, then call the Report method of IProgress<T> to report progress.

This is well suited for nonlinear tasks like download progress, where the completed progress differs with each update

Downloader's download progress update event uses percentages, so this IProgress<T> mode is a good fit.

Nested Progress Bars

This crawler project is for collecting wallpapers. The wallpapers are organized by albums, and one album may have multiple images

To handle this scenario, you can use one progress bar to show overall progress, indicating which album is currently being downloaded

Then nest a child progress bar indicating that the nth image of the current album is being downloaded

And then after that, nest yet another grandchild progress bar showing the download progress of the specific image (percentage)

This uses the Spawn method of ProgressBar, which generates a ChildProgressBar object. Then you just update the child progress bar object's value.

Just look at the code

var list = // 加载图集列表
using var bar = new ProgressBar(list.Count, "正在下载所有图片", BarOptions);

foreach (var item in list) {
    bar.Message = $"图集：{item.Name}";
    bar.Tick();

    foreach (var imgUrl in item.Images) {
        using (var childBar = bar.Spawn(item.ImageCount,$"图片：{imgUrl}",ChildBarOptions)) {
            childBar.Tick();
            // 具体的下载代码
        }
    }
}

This way the main progress bar shows which album is being downloaded, and the child progress bar shows which image is being downloaded.

Then in the actual download code, use Downloader's event listeners and Spawn another new progress bar to show the download progress of a single image.

The code is as follows:

private async Task Download(IProgressBar bar, string url, string filepath) {
    var percentageBar = bar.Spawn(100, $"正在下载：{Path.GetFileName(url)}", PercentageBarOptions);

    HttpHelper.Downloader.DownloadStarted += DownloadStarted;
    HttpHelper.Downloader.DownloadFileCompleted += DownloadFileCompleted;
    HttpHelper.Downloader.DownloadProgressChanged += DownloadProgressChanged;

    await HttpHelper.Downloader.DownloadFileTaskAsync(url, filepath);

    void DownloadStarted(object? sender, DownloadStartedEventArgs e) {
        Trace.WriteLine(
            $"图片, FileName:{Path.GetFileName(e.FileName)}, TotalBytesToReceive:{e.TotalBytesToReceive}");
    }

    void DownloadFileCompleted(object? sender, AsyncCompletedEventArgs e) {
        Trace.WriteLine($"下载完成, filepath:{filepath}");
        percentageBar.Dispose();
    }

    void DownloadProgressChanged(object? sender, DownloadProgressChangedEventArgs e) {
        percentageBar.AsProgress<double>().Report(e.ProgressPercentage);
    }
}

Note that all ProgressBar objects need to be disposed after use, so here we Dispose them inside the DownloadFileCompleted event.

The above uses a using statement directly for automatic disposal.

Progress Bar Configuration

This thing's customization features are pretty decent.

You can configure colors, display characters, display position, and so on.

var barOptions = new ProgressBarOptions {
    ForegroundColor = ConsoleColor.Yellow,
    BackgroundColor = ConsoleColor.DarkYellow,
    ForegroundColorError = ConsoleColor.Red,
    ForegroundColorDone = ConsoleColor.Green,
    BackgroundCharacter = '\u2593',
    ProgressBarOnBottom = true,
    EnableTaskBarProgress = RuntimeInformation.IsOSPlatform(OSPlatform.Windows),
    DisplayTimeInRealTime = false,
    ShowEstimatedDuration = false
};

The EnableTaskBarProgress option can also update the progress on the Windows taskbar at the same time.

For specific configuration options, you can look at the source code directly; the comments inside are very detailed.

If a child progress bar spawned with Spawn has no options configured, it will inherit the parent level's configuration.

Summary

Using C# to write crawlers is still comfortable, at least much better than Java.

Building console apps and packaging them as an exe is also convenient for distribution.

Besides the third-party libraries mentioned in this article, there are other useful approaches for developing console apps with C#.

For example, these two:

• https://github.com/gui-cs/Terminal.Gui
• https://github.com/Tyrrrz/CliFx

If you want a GUI and need it to be cross-platform, then WinForms, WPF, and the like are out of consideration.

Microsoft's MAUI seems a bit of a minefield, and it has no official Linux support, so that's a pass too.

A more mature option would be Avalonia.

For something lightweight, you can try: https://github.com/picoe/Eto

Also, I recommend a tool called RoslynPad, which seems to be imitating LinqPad.

You can quickly execute C# snippets like writing Python scripts, and it also supports importing NuGet packages. It's very convenient for writing crawlers or simple code experiments.

Most importantly, it's open source and free! LinqPad is just too expensive. RoslynPad is getting better with each update; I feel it can slowly catch up to LinqPad~

PS: I still haven't made the New Year WeChat official account red envelope cover; I'll try to get it done tonight~