利用Python发送 http 请求
迟业 人气:0前言:
假如有一个文件,里面有 10 万个 url,需要对每个 url
发送 http
请求,并打印请求结果的状态码,如何编写代码尽可能快的完成这些任务呢?
Python 并发编程有很多方法,多线程的标准库 threading
,concurrency
,协程 asyncio
,当然还有 grequests
这种异步库,每一个都可以实现上述需求,下面一一用代码实现一下,本文的代码可以直接运行,给你以后的并发编程作为参考:
1.队列+多线程
定义一个大小为 400 的队列,然后开启 200 个线程,每个线程都是不断的从队列中获取 url 并访问。
主线程读取文件中的 url 放入队列中,然后等待队列中所有的元素都被接收和处理完毕。
代码如下:
from threading import Thread import sys from queue import Queue import requests concurrent = 200 def doWork(): while True: url = q.get() status, url = getStatus(url) doSomethingWithResult(status, url) q.task_done() def getStatus(ourl): try: res = requests.get(ourl) return res.status_code, ourl except: return "error", ourl def doSomethingWithResult(status, url): print(status, url) q = Queue(concurrent * 2) for i in range(concurrent): t = Thread(target=doWork) t.daemon = True t.start() try: for url in open("urllist.txt"): q.put(url.strip()) q.join() except KeyboardInterrupt: sys.exit(1)
运行结果如下:
有没有 get 到新技能?
2.线程池
如果使用线程池,推荐使用更高级的 concurrent.futures 库:
import concurrent.futures import requests out = [] CONNECTIONS = 100 TIMEOUT = 5 urls = [] with open("urllist.txt") as reader: for url in reader: urls.append(url.strip()) def load_url(url, timeout): ans = requests.get(url, timeout=timeout) return ans.status_code with concurrent.futures.ThreadPoolExecutor(max_workers=CONNECTIONS) as executor: future_to_url = (executor.submit(load_url, url, TIMEOUT) for url in urls) for future in concurrent.futures.as_completed(future_to_url): try: data = future.result() except Exception as exc: data = str(type(exc)) finally: out.append(data) print(data)
3.协程 + aiohttp
协程也是并发非常常用的工具了,
import asyncio from aiohttp import ClientSession, ClientConnectorError async def fetch_html(url: str, session: ClientSession, **kwargs) -> tuple: try: resp = await session.request(method="GET", url=url, **kwargs) except ClientConnectorError: return (url, 404) return (url, resp.status) async def make_requests(urls: set, **kwargs) -> None: async with ClientSession() as session: tasks = [] for url in urls: tasks.append( fetch_html(url=url, session=session, **kwargs) ) results = await asyncio.gather(*tasks) for result in results: print(f'{result[1]} - {str(result[0])}') if __name__ == "__main__": import sys assert sys.version_info >= (3, 7), "Script requires Python 3.7+." with open("urllist.txt") as infile: urls = set(map(str.strip, infile)) asyncio.run(make_requests(urls=urls))
4.grequests[1]
这是个第三方库,目前有 3.8K 个星,就是 Requests + Gevent[2],
让异步 http 请求变得更加简单。Gevent
的本质还是协程。
使用前:
pip install grequests
使用起来那是相当的简单:
import grequests urls = [] with open("urllist.txt") as reader: for url in reader: urls.append(url.strip()) rs = (grequests.get(u) for u in urls) for result in grequests.map(rs): print(result.status_code, result.url)
注意 :grequests.map(rs) 是并发执行的。
运行结果如下:
也可以加入异常处理:
>>> def exception_handler(request, exception): ... print("Request failed") >>> reqs = [ ... grequests.get('http://httpbin.org/delay/1', timeout=0.001), ... grequests.get('http://fakedomain/'), ... grequests.get('http://httpbin.org/status/500')] >>> grequests.map(reqs, exception_handler=exception_handler) Request failed Request failed [None, None, <Response [500]>]
最后的话:
今天分享了并发 http 请求的几种实现方式,有人说异步(协程)性能比多线程好,其实要分场景看的,没有一种方法适用所有的场景,笔者就曾做过一个实验,也是请求 url,当并发数量超过 500 时,协程明显变慢。所以,不能说哪个一定比哪个好,需要划分情况。
加载全部内容