安装fake-useragent模块
~~~
pip install fake-useragent
~~~
在middlewares.py文件中导入模块
~~~
from fake_useragent import UserAgent
~~~
定义UAMiddleware中间件
~~~
class UAMiddleware(object):
def __init__(self, crawler):
super(UAMiddleware, self).__init__()
self.ua = UserAgent()
@classmethod
def from_crawler(cls, crawler):
return cls(crawler)
def process_request(self, request, spider):
request.headers.setdefault('User-Agent', self.ua.random)
~~~
在settings.py配置文件DOWNLOADER_MIDDLEWARES中添加该中间件
