昊天SEO

Using goagent agent in Scrapy

Scrapy crawl with goagent agent I say you goagent list address: http://127.0.0.1:8087 and you create a scrapy project named: myscrapy. and you pwd is myscrapy

 

# file: myscrapy/settings.py
...

USER_AGENT = 'http://127.0.0.1:8087'
DOWNLOADER_MIDDLEWARES = {
 'myscrapy.middlewares.MyProxyMiddleware': 100,
 'scrapy.contrib.downloadermiddleware.httpproxy.HttpProxyMiddleware': 110,
 'scrapy.contrib.downloadermiddleware.useragent.UserAgentMiddleware': None,
}
...

# file: myscrapy/middlewares.py
from myscrapy.settings import USER_AGENT

class MyProxyMiddleware(object):
 def process_request(self, request, spider):
 request.meta['proxy'] = USER_AGENT

本文地址官网    新乡网站建设,seo

分享到:更多 ()
a