python requests爬网页加速

143 阅读 0 评论 95 点赞

我是靠谱客的博主过时月饼，这篇文章主要介绍python requests爬网页加速，现在分享给大家，希望可以做个参考。

import requests

req = requests.get(url)

一般用上述写法，访问网页久了会变卡，那么可以用下面方法替代

session = requests.Session()
req = session.get(url)

也就是说，requests 让 Session对象能够保持连接：

The Session object allows you to persist certain parameters across requests. It also persists cookies across all requests made from the Session instance, and will use urllib3’s connection pooling. So if you’re making several requests to the same host, the underlying TCP connection will be reused, which can result in a significant performance increase (see HTTP persistent connection).