StarBlog Development Notes (17): Automatically Download External Images in Articles

Series Articles

• Building a Blog Project with .NetCore: StarBlog - (1) Why Write Your Own Blog?
• Building a Blog Project with .NetCore: StarBlog - (2) Environment Setup and Creating the Project
• Building a Blog Project with .NetCore: StarBlog - (3) Model Design
• Building a Blog Project with .NetCore: StarBlog - (4) Batch Import of Markdown Blogs
• Building a Blog Project with .NetCore: StarBlog - (5) Starting to Build the Web Project
• Building a Blog Project with .NetCore: StarBlog - (6) Page Development: Blog Post List
• Building a Blog Project with .NetCore: StarBlog - (7) Page Development: Post Detail Page
• Building a Blog Project with .NetCore: StarBlog - (8) Displaying the Category Hierarchy
• Building a Blog Project with .NetCore: StarBlog - (9) Batch Import of Images
• Building a Blog Project with .NetCore: StarBlog - (10) Image Waterfall Layout
• Building a Blog Project with .NetCore: StarBlog - (11) Implementing Visit Statistics
• Building a Blog Project with .NetCore: StarBlog - (12) Dynamic Compilation of Razor Pages
• Building a Blog Project with .NetCore: StarBlog - (13) Adding a Friend Links Feature
• Building a Blog Project with .NetCore: StarBlog - (14) Implementing Theme Switching
• Building a Blog Project with .NetCore: StarBlog - (15) Generating Random-Size Images
• Building a Blog Project with .NetCore: StarBlog - (16) Some New Features (Monitoring/Statistics/Configuration/Initialization)
• Building a Blog Project with .NetCore: StarBlog - (17) Automatically Download External Images in Articles

Preface

It's been a long time since I last updated the blog. After publishing an article about StarBlog development at the end of last month, I wasted a lot of time on miscellaneous things like offline training, a poetry contest, and so on. For a while I was so busy with all that that I had no time to write code...

PS: I shared this poem at the poetry contest: Reading Bai Juyi's "Twelve Chapters on Birds and Insects"

Then recently I bought Yang Zhongke's newly published AspNetCore Technology Internals, and I found it fascinating. I spent more than a week roughly going through the book's content. I've long wanted to learn DDD (Domain-Driven Design), but never found good introductory material. This book by the expert is quite good and easy to understand. Even so, DDD is still a relatively complex approach that needs to be mastered through continuous practice.

Although I've done so many things recently, work has also been busy. There's a project that needs to go live before September. I originally planned to practice DDD, but as I wrote I found I still couldn't quite get a handle on it, so I had to first use my previous DjangoStarter framework, and later gradually refactor my StarBlog blog with DDD thinking~

Oh, another reason I haven't updated the blog for so long is that during use I've become very dissatisfied with the current admin backend (developed with Vue2+ElementUI). The user experience is extremely poor, so I've also been thinking about which technology to use to refactor the admin frontend. Currently there are a few options:

• blazor (develop the frontend with C#, very cool)
• react (among the others, my favorite frontend tech stack)
• still vue, but rewrite the existing architecture (less work)

I haven't made up my mind yet. Until the refactor is done, I can only hold my nose and use the existing admin backend, and I most likely won't add new features to the current frontend project either.

Back to the Topic

OK, back to the content of this article. While using the blog, sometimes I copy some markdown snippets from other websites, or copy markdown content from my blogs on other platforms (Cnblogs, Juejin, etc.). The copied markdown may contain some images. If left unprocessed, this can cause problems, such as remote images failing to display in StarBlog due to hotlink protection, or images being lost when the original site operator shuts down. For data, it's more reassuring to keep it firmly in your own hands.

So I built this feature: download the remote images in markdown articles, and replace the links in the markdown.

The principle is simple: scan the markdown, extract the image links and download them, while replacing the image links with addresses on StarBlog. Below I'll introduce step by step how to implement this in code.

Downloading Images

First is the image download feature. In C#, you can use the HttpClient standard library to access the network.

The simplest usage looks like this:

var client = new HttpClient();
await client.GetAsync("图片地址");

However, the official documentation does not recommend this usage. Best practice is to maintain only one HttpClient object in a program.

In AspNetCore, we can use dependency injection of IHttpClientFactory to manage HttpClient objects.

Register the service in Program.cs

builder.Services.AddHttpClient();

Inject IHttpClientFactory where needed. For example, in this project, we create a new CommonService.cs to hold the file-download code. Considering this feature may also be used elsewhere later, we make it generic and not coupled with PostService.

The code is as follows:

public class CommonService {
  private readonly ILogger<CommonService> _logger;
  private readonly IHttpClientFactory _httpClientFactory;

  public CommonService(ILogger<CommonService> logger, IHttpClientFactory httpClientFactory) {
    _logger = logger;
    _httpClientFactory = httpClientFactory;
  }
  
  public async Task<string?> DownloadFileAsync(string url, string savePath) {
    var httpClient = _httpClientFactory.CreateClient();
    try {
      var resp = await httpClient.GetAsync(url, HttpCompletionOption.ResponseHeadersRead);

      // 生成随机文件名
      var fileName = GuidUtils.GuidTo16String() + Path.GetExtension(url);
      var filePath = Path.Combine(savePath, WebUtility.UrlEncode(fileName));
      await using var fs = new FileStream(filePath, FileMode.OpenOrCreate, FileAccess.Write);
      await resp.Content.CopyToAsync(fs);

      return fileName;
    }
    catch (Exception ex) {
      _logger.LogError("下载文件出错，信息：{Error}", ex);
      return null;
    }
  }
}

Let's analyze some of the code:

• Line 13 uses HttpClient's GetAsync method to download data, adding an HttpCompletionOption.ResponseHeadersRead parameter, so we don't have to wait until all the data is loaded into memory before doing stream reading and similar operations; instead, we can proceed to the next step as soon as the response headers are returned. This avoids OutOfMemoryException caused by the file being too large, which is very important for a file-download program!
• Line 16 uses a wrapped Guid utility to generate a 16-character GUID. Using Guid.NewGuid().ToString() directly also works; that's 32 characters.
• Lines 18-19 write the HTTP response content to the file stream

Done. The file download code is fairly simple. For error-prone places involving IO operations, you need to handle the details well to ensure program stability.

PS: Don't forget to register the service!

builder.Services.AddSingleton<CommonService>();

Processing Markdown

After the image download feature is done, we continue with the markdown processing part.

Regarding processing Markdown in C#, I've already explored it several times before, so I can say I'm quite familiar with it by now hhh~

Attached are previous articles about Markdown processing:

• Parsing Markdown documents in C# and replacing image links
• Building a Blog Project with .NetCore: StarBlog - (4) Batch Import of Markdown Blogs

Still using the Markdig library (seems there's no other choice for processing markdown on .NetCore anyway)

Add code in PostService.cs

/// <summary>
/// Markdown中外部图片下载
/// <para>如果Markdown中包含外部图片URL，则下载到本地且进行URL替换</para>
/// </summary>
private async Task<string> MdExternalUrlDownloadAsync(Post post) {
  if (post.Content == null) return string.Empty;

  // 得先初始化目录
  InitPostMediaDir(post);

  var document = Markdown.Parse(post.Content);
  foreach (var node in document.AsEnumerable()) {
    if (node is not ParagraphBlock {Inline: { }} paragraphBlock) continue;
    foreach (var inline in paragraphBlock.Inline) {
      if (inline is not LinkInline {IsImage: true} linkInline) continue;

      var imgUrl = linkInline.Url;
      
      // 跳过空链接
      if (imgUrl == null) continue;
      // 跳过本站地址的图片
      if (imgUrl.StartsWith(Host)) continue;

      // 下载图片
      _logger.LogDebug("文章：{Title}，下载图片：{Url}", post.Title, imgUrl);
      var savePath = Path.Combine(_environment.WebRootPath, "media", "blog", post.Id!);
      var fileName = await _commonService.DownloadFileAsync(imgUrl, savePath);
      linkInline.Url = fileName;
    }
  }

  await using var writer = new StringWriter();
  var render = new NormalizeRenderer(writer);
  render.Render(document);
  return writer.ToString();
}

Code explanation:

• Line 9's directory initialization simply checks whether this article has a corresponding directory and creates it if not. It's simple so I won't paste the code. You can see the complete code in the GitHub project
• Starting from line 12, two nested loops traverse the markdown document tree to find the image links
• Line 22 checks whether the image is external or internal; internal images don't need to be downloaded

This completes downloading external images in the markdown and replacing the links~

Modifying Article Save Logic

Next, let's modify the article save logic.

Still in this PostService.cs, saving and adding articles share one method: InsertOrUpdateAsync

Here's the code directly

public async Task<Post> InsertOrUpdateAsync(Post post) {
  // 是新文章的话，先保存到数据库
  if (await _postRepo.Where(a => a.Id == post.Id).CountAsync() == 0) {
    post = await _postRepo.InsertAsync(post);
  }

  // 检查文章中的外部图片，下载并进行替换
  post.Content = await MdExternalUrlDownloadAsync(post);
  // 修改文章时，将markdown中的图片地址替换成相对路径再保存
  post.Content = MdImageLinkConvert(post, false);

  // 处理完内容再更新一次
  await _postRepo.UpdateAsync(post);
  return post;
}

Code notes:

• For a new article, it is saved once first as a draft.
• First download the external images, then replace them with local image links*(Regarding image link replacement, you can refer to article 4 of this series; there is a link above)*
• After completing these steps, save again. Note that the article is still in draft status at this point; you need to use another method to set the article's IsPublish property to true. However, this isn't closely related to this article, so I won't include the code here for now. This process will be covered in detail later in the articles on RESTFul API development.

That's it~

References

• Official documentation: https://docs.microsoft.com/en-us/dotnet/api/system.net.http.httpclient?view=net-6.0
• C#: A detailed guide to using HttpClient: https://blog.csdn.net/u010476739/article/details/119782562
• A summary of using HttpClient in C#: https://zhuanlan.zhihu.com/p/89106847