Run Crawlers on Your Phone! A First Look at a Flutter Crawler Framework~

Recently I needed a lot of data to train models for my graduation project. Many sites also have anti-crawling mechanisms, so I have to crawl for a while then pause, which makes it extremely slow. Running a crawler on my computer 24 hours a day would use too much electricity, so I thought of using my phone~

Then I thought of Flutter—old Android and Apple phones can all be put to use for data collection.

ok, then I started building. After a day of tinkering I put together a rough framework. Finally I can conveniently add crawler tasks. It integrates basic features like config, logging, and data persistence, then wraps network requests and webpage parsing. It works reasonably well.

Except, as everyone knows, Dart is a single-threaded language. Although it provides an isolate mechanism, it can't share memory, so using it feels like having your hands tied. And Flutter even disabled reflection! The Google team is really something. After an afternoon of struggling I got so pissed off by this piece of junk that I simply abandoned the project.

If I want to implement a multi-threaded crawler, it looks like I'd have to write some native code anyway, in which case I might as well go fully native...

(ps: Why use this junk Flutter for crawling when perfectly good Android native is sitting there?? I must have gone nuts. So pissed off.)

App Screenshots

System Architecture

Drew a few diagrams

The SpiderTask base class. All crawler classes derive from this class. SpiderTask itself maintains a TaskConfig task configuration object and a logging object. As shown below:

As for data persistence, I used another class with the singleton pattern, initialized when the app starts.

Workflow

1. Register crawler tasks in the task list on the home page
2. After launching the app, manage each crawler task on the main page
3. Select a task and start it
4. Entering the detail page automatically binds to the crawler task's logging object so you can see log output

A Simple Example

Here's a simple example. When run, it crawls news from the CN Radio website.

import 'package:flutter/src/widgets/framework.dart';
import 'package:flutter_spider_fx/framework/index.dart';

class CNRadioNewsSpider extends SpiderTask {
  var url = 'http://news.cnr.cn/';

  CNRadioNewsSpider(BuildContext context) : super(context, 'CNRadioNewsSpider');

  @override
  start() async {
    super.start();
    var dom = await CatHttp.getDocument(url, encoding: 'gb2312');
    var links = dom.querySelectorAll('.contentPanel .lh30 a');
    links.forEach((link) {
      logging.debug(link.attributes['href']);
      logging.info(link.text);
    });
  }
}

Framework Code

For more details, you can check out the project code on GitHub~

Project URL: https://github.com/Deali-Axy/flutter_spider_fx

One last thing: junk Flutter. Don't blindly worship Google's tech. This thing is still a trap right now. Fine for writing a simple GUI to do CRUD, but don't even think about advanced stuff.

Welcome to Chat

For questions, please leave a message on the WeChat official account backend. I'll reply to every message~

• WeChat official account: 画星星高手
• Coding livestream: https://live.bilibili.com/11883038
• Zhihu: https://www.zhihu.com/people/dealiaxy
• Column: https://zhuanlan.zhihu.com/deali
• Jianshu: https://www.jianshu.com/u/965b95853b9f