Using goagent agent in Scrapy

Scrapy crawl with goagent agent I say you goagent list address: http://127.0.0.1:8087 and you create a scrapy project named: myscrapy. and you pwd is myscrapy

# file: myscrapy/settings.py
...

USER_AGENT = 'http://127.0.0.1:8087'
DOWNLOADER_MIDDLEWARES = {
 'myscrapy.middlewares.MyProxyMiddleware': 100,
 'scrapy.contrib.downloadermiddleware.httpproxy.HttpProxyMiddleware': 110,
 'scrapy.contrib.downloadermiddleware.useragent.UserAgentMiddleware': None,
}
...

# file: myscrapy/middlewares.py
from myscrapy.settings import USER_AGENT

class MyProxyMiddleware(object):
 def process_request(self, request, spider):
 request.meta['proxy'] = USER_AGENT

网站的维护离不开大家的支持鼓励，捐赠让我更有动力走的更远&& 也可以关注我的微信公众号,发布更多的干货