StarBlog Blog Development Notes (4): Bulk Import of Markdown Blogs

Series of Articles

• Developing a Blog Project with .NetCore: StarBlog - (1) Why Build Your Own Blog?
• Developing a Blog Project with .NetCore: StarBlog - (2) Environment Setup and Project Creation
• Developing a Blog Project with .NetCore: StarBlog - (3) Model Design
• Developing a Blog Project with .NetCore: StarBlog - (4) Batch Importing Markdown Blogs
• ...

Introduction

Last week, we discussed the model design for the blog. Now that the model is ready, it's time to start importing data.

The goal is to import all the markdown files within a folder, using the directory structure as the article categories, the file names as the article titles, and the file creation and update dates as the publication dates.

The general approach is to first traverse the directory using .Net's standard library, process the article content with a third-party markdown parsing library, and then write it to the database via ORM.

PS: Tomorrow is Labor Day. Happy May Day to all the working-class laborers!

Related Technologies

• File I/O related APIs
• Regular expressions
• ORM: FreeSQL
• Markdown parsing library: Markdig

Getting Started with Code

We'll begin by discussing the most critical parts: parsing Markdown content, extracting images, and handling headings.

To process Markdown content, I searched for relevant libraries and discovered that for .NET Core, the only available option is the Markdig library. Since it's still in development and lacks complete documentation, I had to piece things together by reading the GitHub repository's minimal explanations and experimenting with examples. Unfortunately, there aren't many good alternatives, and I'm too lazy (or maybe just bad at coding) to reinvent the wheel, so I'll just make do with it.

Markdig official website: https://github.com/xoofx/markdig

Create a new class in the StarBlog.Migrate project called PostProcessor. We'll implement the Markdown file processing logic in this class.

The complete code for PostProcessor.cs is here: https://github.com/Deali-Axy/StarBlog/blob/master/StarBlog.Migrate/PostProcessor.cs

Constructor:

private readonly Post _post;
private readonly string _importPath;
private readonly string _assetsPath;

public PostProcessor(string importPath, string assetsPath, Post post) {
    _post = post;
    _assetsPath = assetsPath;
    _importPath = importPath;
}

Here:

• Post: The article model designed in our previous section
• importPath: The path to the Markdown folder to be imported
• assetsPath: The storage path for resource files, used to store images from the Markdown files. In this project, the path is set to StarBlog.Web/wwwroot/media/blog

Article Summary Extraction

For article summary extraction, I implemented a simple approach: converting the Markdown content into plain text, then truncating it to the first N characters to form the summary. The code is as follows:

public string GetSummary(int length) {
    return _post.Content == null
        ? string.Empty
        : Markdown.ToPlainText(_post.Content).Limit(length);
}

Article Status and Title Handling

Previously, when writing blogs using local markdown files, out of personal habit, I would add status prefixes to the file names. For example, an unfinished article would look like this:

(Unfinished) StarBlog Development Notes(4): Batch Markdown Blog Import

Or for completed but unpublished articles, I would add (Unpublished).

Once published, I would remove the prefix. Therefore, during import, I need to use a regular expression to extract this prefix so that the blog article title imported into the database no longer carries the prefix.

The code is as follows:

public (string, string) InflateStatusTitle() {
    const string pattern = @"^（(.+)）(.+)$";
    var status = _post.Status ?? "Published";
    var title = _post.Title;
	if (string.IsNullOrEmpty(title)) return (status, $"Untitled Article {_post.CreationTime.ToLongDateString()}");
    var result = Regex.Match(title, pattern);
    if (!result.Success) return (status, title);

    status = result.Groups[1].Value;
    title = result.Groups[2].Value;

    _post.Status = status;
    _post.Title = title;

    if (!new[] { "Published", "Released" }.Contains(_post.Status)) {
        _post.IsPublish = false;
    }

    return (status, title);
}

The logic is straightforward: first check if the title is empty (which is unlikely for a file name, but done for thoroughness). Then, use a regular expression for matching. If a match is found, extract the status; if not, default to "Published".

Image Extraction & Replacement

The most complex part of Markdown content processing is this section, so I previously wrote a separate article to explain it in detail. Therefore, I won't repeat too much here. For more information, you can refer to my earlier article: Parsing Markdown Documents in C# and Implementing Image Link Replacement

Now, returning to our blog project, the code for this part is as follows:

public string MarkdownParse() {
    if (_post.Content == null) {
        return string.Empty;
    }

    var document = Markdown.Parse(_post.Content);

    foreach (var node in document.AsEnumerable()) {
        if (node is not ParagraphBlock { Inline: { } } paragraphBlock) continue;
        foreach (var inline in paragraphBlock.Inline) {
            if (inline is not LinkInline { IsImage: true } linkInline) continue;

            if (linkInline.Url == null) continue;
            if (linkInline.Url.StartsWith("http")) continue;

            // Path handling
            var imgPath = Path.Combine(_importPath, _post.Path, linkInline.Url);
            var imgFilename = Path.GetFileName(linkInline.Url);
            var destDir = Path.Combine(_assetsPath, _post.Id);
            if (!Directory.Exists(destDir)) Directory.CreateDirectory(destDir);
            var destPath = Path.Combine(destDir, imgFilename);
            if (File.Exists(destPath)) {
                // Handle duplicate image names
                var imgId = GuidUtils.GuidTo16String();
                imgFilename = $"{Path.GetFileNameWithoutExtension(imgFilename)}-{imgId}.{Path.GetExtension(imgFilename)}";
                destPath = Path.Combine(destDir, imgFilename);
            }

            // Replace image link
            linkInline.Url = imgFilename;
            // Copy image
            File.Copy(imgPath, destPath);

            Console.WriteLine($"Copying {imgPath} to {destPath}");
        }
    }


    using var writer = new StringWriter();
    var render = new NormalizeRenderer(writer);
    render.Render(document);
    return writer.ToString();
}

The implementation steps are roughly as follows:

• Use the Markdown parsing functionality from the Markdig library.
• Extract all image links.
• Then, based on the importPath directory passed in the constructor earlier, construct the full path of the image.
• Next, copy the images into the assetsPath directory.
• Finally, replace the image paths in the Markdown with the newly generated image filenames.

Summary

Currently, this solution handles most images in Markdown files without issues, but there is still a problem!

Images with filenames containing spaces cannot be recognized!

This issue is a limitation of the Markdig library, I suppose. I tried reading through Markdig's source code to see if I could fix it, but unfortunately, I couldn't understand it well enough. So for now, there's no perfect workaround—I'll just have to submit an issue to the official repository. This library is actively maintained, so there's hope that the developers will address and fix this problem.

Traversing Directories

We've covered the key parts earlier; now let's talk about the relatively simple task of traversing directory files. If you're already proficient with file I/O, feel free to skip this section~

I implemented this using a recursive method, referencing an official Microsoft blog post: https://docs.microsoft.com/zh-cn/dotnet/csharp/programming-guide/file-system/how-to-iterate-through-a-directory-tree

The key code is as follows; the complete code can be found here: https://github.com/Deali-Axy/StarBlog/blob/master/StarBlog.Migrate/Program.cs

void WalkDirectoryTree(DirectoryInfo root) {
    Console.WriteLine($"Scanning folder: {root.FullName}");

    FileInfo[]? files = null;
    DirectoryInfo[]? subDirs = null;

    try {
        files = root.GetFiles("*.md");
    }
    catch (UnauthorizedAccessException e) {
        Console.WriteLine(e.Message);
    }
    catch (DirectoryNotFoundException e) {
        Console.WriteLine(e.Message);
    }

    if (files != null) {
        foreach (var fi in files) {
            Console.WriteLine(fi.FullName);
            // Code for processing articles, omitted
        }
    }

    subDirs = root.GetDirectories();

    foreach (var dirInfo in subDirs) {
        if (exclusionDirs.Contains(dirInfo.Name)) {
            continue;
        }

        if (dirInfo.Name.EndsWith(".assets")) {
            continue;
        }

        WalkDirectoryTree(dirInfo);
    }
}

This method is called "pre-order traversal," meaning it processes the files in the directory first, then processes the subdirectories within it.

The recursive method is simple to write, but it has a drawback: if the directory structure is deeply nested, it might cause a stack overflow. You could consider switching to a traversal based on the Stack<T> pattern. However, since a blog's directory hierarchy shouldn't be too deep, I opted for the simpler approach~

Writing to the Database

The ORM used in this project is FreeSQL. ORM operations will be covered in more detail in subsequent website development articles, so I'll skip them here. The code for writing article data to the database is quite straightforward—you can view it directly at: https://github.com/Deali-Axy/StarBlog/blob/master/StarBlog.Migrate/Program.cs

Conclusion

OK, that covers everything about batch blog import. Once you handle a few tricky parts, it's not difficult anymore. Having the article data ready makes it much easier to start developing the blog website~

That's about it for now. See you in the next post~

Also, all project code has been uploaded to GitHub. Everyone is welcome to Star/Fork!

• Blog backend + frontend project address: https://github.com/Deali-Axy/StarBlog
• Admin backend frontend project address: https://github.com/Deali-Axy/StarBlog-Admin