Selenium Crawler Practice: AJAX Request Capture and Browser Exit

Preface

I've been working on the company's internal systems lately and I'm exhausted. I need to export data from another internal system into a database for analysis. There's a lot of data collection work, and I have no way to get that system's APIs directly. It's so hard that I can only scrape it. But cookies expire frequently. To avoid being asked to re-inject Cookie every time a login expires, I wrote a mobile web page to control Selenium in the background for automatic login and to intercept tokens and cookies.

AJAX Request Capture Approach

Searching for materials was really painful, but luckily the time wasn't wasted. I eventually solved the problem……

According to the materials I found, there are several ways to capture data from AJAX requests in Selenium

• Use a local proxy: browsermob-proxy (the method used in this article)
• Use Selenium's JS execution to inject an ajax hook and run it, then start a local server to receive the intercepted ajax data (see the third reference)
• Use the third-party library selenium-wire, an open-source GitHub project that can directly intercept response_code and body; the principle is probably also a proxy
• Enable Selenium performance capture and make changes in the performance logs to intercept response_body (see the first reference for details)

Using a Local Proxy

This article uses the Browsermob-Proxy proxy server. It's written in Java and has a Python wrapper package for convenient interaction……

Download it first: https://github.com/lightbody/browsermob-proxy/releases

Install the Python package:

pip install browsermob-proxy

Using it in code. Here I've excerpted part of the project's code—just take a look. The complete code can be found in the official docs or the references~

There are a few pitfalls to watch out for, which I've marked in the code

# 创建代理服务器
self.server = Server(
    # Windows就是bat，如果Linux就是另一个不带后缀名的
    r'path\bin\browsermob-proxy.bat',
    # 这里可以自定义端口
    options={'port': 9090}
)
# 这里启动服务器，等会机会要关掉，不然下次用就端口占用冲突了
self.server.start()
# 注意这里一定要 trustAllServers 不然等会selenium会报 error_tunnel 错误
self.proxy = self.server.create_proxy(params={'trustAllServers': 'true'})

# 设置selenium的代理
options = ChromeOptions()
options.add_argument('--ignore-certificate-errors')
options.add_argument(f'--proxy-server={self.proxy.proxy}')
self.driver = webdriver.Chrome(options=options)

Using the proxy for packet capture. This project of mine needs to extract the token and cookie from the AJAX request headers. The key excerpt is as follows:

self.proxy.new_har('抓包名称 自己起一个', options={'captureHeaders': True, 'captureContent': True})

# 找到需要点击的元素
elem_query = self.driver.find_element_by_css_selector(elem_css_selector)
elem_query.click()

# 点击按钮后等待 并把数据取出来
time.sleep(5)
result = self.proxy.har

data = {}

for entry in result['log']['entries']:
    url = entry['request']['url']
    # 根据URL找到数据接口
    if 'xxx/query' in url:
        _response = entry['response']
        _content = _response['content']['text']

        for item in entry['request']['headers']:
            # 提取出header里面的 token
            if item['name'] == 'Authorization':
                data['authorization'] = item['value']
            # 提取出header里面的 cookie
            if item['name'] == 'Cookie':
                data['cookie'] = item['value']
        break

print(data)

The code above is likewise not complete, but it fully expresses the specific packet capture process. Those who need it can code according to their actual requirements. As long as you can capture the data, everything else is easy~

Exiting the Browser and Proxy Server

Not much to write here, but there's a small pitfall too, so I'll fill some space~

As you can also see from the code above, I wrote a class to operate Selenium. After the program finishes, you definitely need to shut down the proxy and the server, otherwise Selenium will leave a chromedriver.exe process occupying resources in the background. Over time, system memory fills up.

I added code to close the proxy server and the browser in the class's __del__ method, as follows:

def __del__(self):
    print('SeleniumFxxkUnicom has been deleted.')
    self.proxy.close()
    self.server.stop()
    for win in self.driver.window_handles:
        self.driver.switch_to.window(win)
        self.driver.close()
    os.system('taskkill /im chromedriver.exe /F')

Note this loop of driver.close(). You can't normally execute driver.quit() in __del__. In theory quit is the best exit method, but it has to import all sorts of messy modules, which caused it to fail in this __del__, so I had to take a roundabout approach: first close all tabs, then kill the process with a system command…… It's a bit hardcoded. That's how it is—I'm tired.

References

• Capturing AJAX requests from web pages with Selenium: https://www.cnblogs.com/fish-101/p/13170021.html
• Selenium crawler - getting browser Network requests and responses: http://www.spiderpy.cn/blog/detail/47/
• How to process and save Ajax data in real time with Hooks
• How to easily intercept Ajax data in Selenium
• Closing the browser with driver.close() and driver.quit() in the Selenium framework: https://blog.csdn.net/yangfengjueqi/article/details/84338167

Welcome to Discuss

The Program Design Lab focuses on exploring popular new internet technologies and team agile development practices. Reply linux, flutter, c#, netcore, android, kotlin, java, python, etc. in the backend of the WeChat official account 「程序设计实验室」 to get related technical articles and materials. If you have any questions, you can also leave a message in the official account backend~

• Blog Garden: https://www.cnblogs.com/deali/
• Coding livestream: https://live.bilibili.com/11883038
• Zhihu: https://www.zhihu.com/people/dealiaxy