C# Crawler: CatSpider# Development Notes

Introduction

(PS: I used the # symbol here in place of the word Sharp)

CatSpider is the data collection module in my graduation project. Originally, crawler-type applications would of course be developed with Python, but when using request_html for parsing, Python's dynamic typing really disgusted me, and I felt this library wasn't very mature either, html5lib wasn't easy to use, and I didn't feel like digging deeper. I'd seen some experts building crawlers on the .net core platform, so I decided to give it a try too. I didn't expect the results to be so good, especially together with LinqPad—write a code snippet then Dump it directly for data display, super convenient.

After the code tested fine, I wrote it straight into the project, used a lightweight ORM to write to the database, felt great. However, scraping some websites is more troublesome due to anti-crawling mechanisms, and in this regard it's not as good as Python, because Python has lots of wheels—I can just find what others have made and integrate it. After all, crawling isn't the main focus of this project, so I can't waste too much time and energy. So how do you integrate C# and Python modules together? emmm of course it's RPC, though adding a Flask in the Python crawler for calling would also work, but data exchange performance would take a big hit.

Got a bit off topic, continuing to document the .net core crawler~

Network Requests

If it were Python, then I think the requests library is the only best choice. In NetCore, there's now a very useful library HttpClient. Unlike requests, this one is official, made very easy to use, and all calls are asynchronous.

The official recommendation is to use the singleton pattern, so I made an HttpHelper static class to use~

public static class HttpHelper
{
    private static readonly HttpClientHandler handler;
    private static readonly HttpClient client;

    public static HttpClientHandler Handler { get => handler; }
    public static HttpClient Client { get => client; }

    static HttpHelper()
    {
        handler = new HttpClientHandler();
        client = new HttpClient(handler);
        client.DefaultRequestHeaders.Add("User-Agent", "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36");
    }
}

Using it is very simple:

var data = await HttpHelper.Client.GetStringAsync("http://example.com");

Parsing HTML Documents

Parsing is equally convenient. With the AngleSharp library, you can quickly locate page elements like CSS selectors, much better than Python's commonly used BeautifulSoup.

For convenience, I wrapped a method to quickly parse documents:

public static async Task<IHtmlDocument> GetHtmlDocument(string url)
{
    var html = await client.GetStringAsync(url);
    return new HtmlParser().ParseDocument(html);
}

public static async Task<IHtmlDocument> GetHtmlDocument(string url, string charset)
{
    var res = await client.GetAsync(url);
    var resBytes = await res.Content.ReadAsByteArrayAsync();
    var resStr = Encoding.GetEncoding(charset).GetString(resBytes);
    return new HtmlParser().ParseDocument(resStr);
}

The second method has a charset parameter to specify the document encoding. Some older domestic websites don't use utf-8 and don't indicate the encoding either, so Chinese characters downloaded this way are all garbled and need to be handled.

Starting Data Collection

Here are some simple examples, crawling a list:

Using linq, the world feels so beautiful, hahaha

public static async Task<List<ListArticle>> CrawlHotList()
{
    var dom = await HttpHelper.GetHtmlDocument("https://www.com/");
    var links = dom.QuerySelectorAll(".hot-list ul li a");
    var hotList = links.Select((elem, result) => 
    new ListArticle{
		Title = elem.TextContent,
        Link = elem.GetAttribute("href"),
        Source = "来源"});
    return new List<ListArticle>(hotList);
}

Crawling article content, the code is especially simple:

public static async Task<Article> CrawlArticle(string url)
{
    var dom = await HttpHelper.GetHtmlDocument(url);
    var data = new Article
    {
        Title = dom.QuerySelector("#cb_post_title_url").TextContent,
        Source = "来源",
        Content = dom.QuerySelector(".postBody").TextContent,
        Link = url,
        PublishTime = DateTime.Parse(dom.QuerySelector("#post-date").TextContent),
        AddTime = DateTime.Now,
        Author = dom.QuerySelector(".postDesc a").TextContent
    };
    return data;
}

And what about when you encounter large amounts of data? That's when you need to bring in parallel tasks. C#'s high-performance advantage over Python shows here. Here's the code:

public static async Task<List<CnBlogListArticle>> CrawlList2(int page = 10)
{
    var http = HttpHelper.Client;
    var parser = new HtmlParser();
    var data = await Task.WhenAny(
        Enumerable.Range(1, page)
        .Select(async page =>
        {
        	string pageData = await http.GetStringAsync($"https://www.cnblogs.com/sitehome/p/{page}");
            IHtmlDocument doc = await parser.ParseDocumentAsync(pageData);
            return doc.QuerySelectorAll(".post_item").Select(tag => new CnBlogListArticle
            {
                Title = tag.QuerySelector(".titlelnk").TextContent,                     Page = page,
                UserName = tag.QuerySelector(".post_item_foot .lightblue").TextContent,
                PublishTime = DateTime.Parse(Regex.Match(tag.QuerySelector(".post_item_foot").ChildNodes[2].TextContent, @"(\d{4}\-\d{2}\-\d{2}\s\d{2}:\d{2})", RegexOptions.None).Value),
                CommentCount = int.Parse(tag.QuerySelector(".post_item_foot .article_comment").TextContent.Trim()[3..^1]),
                ViewCount = int.Parse(tag.QuerySelector(".post_item_foot .article_view").TextContent[3..^1]),
                BriefContent = tag.QuerySelector(".post_item_summary").TextContent.Trim(),
            });
})).ConfigureAwait(true);
    return new List<CnBlogListArticle>(await data);
}

You can also use IEnumerable's AsParallel() method to parallelize LINQ. Won't expand on that.

Data Persistence

Data persistence is doable. The .net core platform has many useful ORMs, such as Microsoft's official EF Core, SqlSugar, Dapper, and so on, but EF Core feels relatively heavy, and when I was doing this, I hadn't yet learned how to use it standalone.

Then I found a lightweight ORM made by a Chinese developer, Chloe. Looking at the docs it seemed simple to use, so I gave it a try. Model code:

[Table("ListArticles")]
public class ListArticle
{
    [Column("Id", IsPrimaryKey = true)]
    [AutoIncrement]
    public int Id { get; set; }
    public string Title { get; set; }
    public string Source { get; set; }
    public string Link { get; set; }
}

This ORM requires adding attributes on the model class to define the primary key, table name, etc. EF Core doesn't need this—it follows conventions completely. In this regard it's not as convenient as EF Core.

And it can't auto-generate tables, so I had to create tables manually. Thumbs down.

Next, the usual operations, create a DBContext. Most ORMs are similar:

public class SQLiteConnectionFactory : IDbConnectionFactory
{
    /// <summary>
    /// 数据库连接字符串，如下
    /// Data Source=dapperTest.db
    /// </summary>
    string _connString = null;
    public SQLiteConnectionFactory(string connString)
    {
        this._connString = connString;
    }
    public IDbConnection CreateConnection()
    {
        // 得先安装Sqlite的驱动
        // Microsoft.Data.Sqlite
        // System.Data.Sqlite
        SQLiteConnection conn = new SQLiteConnection(_connString);
        return conn;
    }
}

Oh right, you need to configure the connection first:

public static class SQLiteContextFactory
{
    public static SQLiteContext GetContext()
    {
        string connString = "Data Source=CatSpider.db";
        return new SQLiteContext(new SQLiteConnectionFactory(connString));
    }
}

Using it is very simple:

var context = SQLiteContextFactory.GetContext();
obj = context.Insert(obj);

For more operations, go look at the docs. This article won't expand on that.

Providing an HTTP API

Basic functionality is implemented. Previously, considering interoperability with other languages or modules, I thought we could interact via HTTP APIs (though now I don't think it's the best solution)

This is simple, just find a lightweight server framework. I found one called Nancy, which sounds like a person's name, but it turned out to be a web framework.

Using it is very simple, just start it:

private string host = "http://localhost";
private int port = 50010;
private NancyHost nancy;

public Program()
{
    var uri = new Uri($"{host}:{port}/");
    nancy = new NancyHost(uri);
}
public void Start()
{
    nancy.Start();
    logger.Debug($"nancy server started at {host}:{port}");
    Console.ReadKey();
    nancy.Stop();
}
static void Main(string[] args)
{
    new Program().Start();
}

Defining APIs

This framework has a Module concept, pretty much like a Controller. Defining them is simple. I'll put up the test code; business code won't be posted for now:

public class MainModule : NancyModule
{
    public MainModule()
    {
        Get("/", _ => "hello");
        Get("404", _ => HttpStatusCode.NotFound);
        Get("test", _ =>
            {
                var response = (Response)JsonConvert.SerializeObject(new int[] { 1, 2, 3 });
                response.ContentType = "application/json";
                return response;
            });
        Get("test2", _ => JsonConvert.SerializeObject(new int[] { 1, 2, 3 }));
    }
}

Logging

Finally, a word about logging. I used nlog, a lightweight logging engine.

First, configure NLog.config and set it to copy automatically to the output folder on build:

<?xml version="1.0" encoding="utf-8" ?>
<nlog xmlns="http://www.nlog-project.org/schemas/NLog.xsd"
      xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
      autoReload="true"
      throwConfigExceptions="true">
  <targets>
    <!-- 配置说明：https://github.com/NLog/NLog/wiki/ColoredConsole-target -->
    <target name="f1" xsi:type="File" fileName="CatSpiderLog.txt"/>
    <target name="n1" xsi:type="Network" address="tcp://localhost:4001"/>
    <target name="c1" xsi:type="Console" encoding="utf-8"
            error="true"
            detectConsoleAvailable="true" />
    <target name="c2" xsi:type="ColoredConsole" encoding="utf-8"
          useDefaultRowHighlightingRules="true"
          errorStream="true"
          enableAnsiOutput="true"
          detectConsoleAvailable="true"
          DetectOutputRedirected="true">
    </target>
  </targets>

  <rules>
    <logger name="*" maxLevel="Debug" writeTo="c2" />
    <logger name="*" maxLevel="Debug" writeTo="f1" />
    <!--<logger name="*" minLevel="Info" writeTo="f1" />-->
  </rules>
</nlog>

The official recommendation is to use one logger instance per class:

private static Logger logger = LogManager.GetCurrentClassLogger();

Usage:

logger.Debug($"列表：{obj}");

Very convenient.

That's about it. I'll continue writing about other things when I have time~

Welcome to Connect

For questions, please leave a message on the WeChat official account backend. I'll reply to every message~

• WeChat Official Account: 画星星高手
• Coding livestream: https://live.bilibili.com/11883038
• Zhihu: https://www.zhihu.com/people/dealiaxy
• Column: https://zhuanlan.zhihu.com/deali
• Jianshu: https://www.jianshu.com/u/965b95853b9f