Using Python to Parse Images in Markdown Documents and Save Them Locally~

Background

A while ago Jianshu seemed to be going under, which made me a bit panicked, since most of my blogs are hosted on Jianshu. Although Jianshu provides a package export feature, it can only export text—images still live on Jianshu's servers. Plus I've always wanted to rebuild a personal blog, so this article came about.

Approach

First I need to parse the markdown documents, then get all the images in them, and save the images into directories organized by md file.

Parsing markdown documents

Here I used the misaka module, which is said to have the best performance among Python markdown parsers. But its documentation is really sparse—too little content, and it's written unclearly. Its basic function seems to be parsing markdown documents into HTML documents, but it doesn't seem to have methods for directly operating on markdown elements.

No problem, I can parse the HTML the way I usually write crawlers, so I can get the images in a roundabout way~ I'll use BeautifulSoup here

Downloading images

Very simple, just requests, nothing much to say.

Implementation

Traversing files

First I need to traverse all md documents in the folder:

def get_files_list(dir):
    """
    获取一个目录下所有文件列表，包括子目录
    :param dir:
    :return:
    """
    files_list = []
    for root, dirs, files in os.walk(dir, topdown=False):
        for file in files:
            files_list.append(os.path.join(root, file))

    return files_list

Parsing md documents and getting all images

First convert markdown to HTML with misaka, then extract all img tags.

def get_pics_list(md_content):
    """
    获取一个markdown文档里的所有图片链接
    :param md_content:
    :return:
    """
    md_render = misaka.Markdown(misaka.HtmlRenderer())
    html = md_render(md_content)
    soup = BeautifulSoup(html, features='html.parser')
    pics_list = []
    for img in soup.find_all('img'):
        pics_list.append(img.get('src'))

    return pics_list

Downloading images

def download_pics(url, file):
    img_data = requests.get(url).content
    filename = os.path.basename(file)
    dirname = os.path.dirname(file)
    targer_dir = os.path.join(dirname, f'{filename}.assets')
    if not os.path.exists(targer_dir):
        os.mkdir(targer_dir)

    with open(os.path.join(targer_dir, f'{uuid.uuid4().hex}.jpg'), 'w+') as f:
        f.buffer.write(img_data)

Complete code

The complete code for this project has been uploaded to GitHub, at the following address: https://github.com/Deali-Axy/Markdown-Image-Parser

Running

pip install -r requirements.txt
python spider.py

Welcome to connect with me

• Coding livestream: https://live.bilibili.com/11883038
• WeChat official account: DealiAxy
• Zhihu: https://www.zhihu.com/people/dealiaxy
• Blog: https://blog.deali.cn
• Jianshu: https://www.jianshu.com/u/965b95853b9f