在爬取网站内容的时候，最常遇到的问题是：网站对IP有限制，会有防抓取功能，最好的办法就是IP轮换抓取（加代理）
下面来说一下Scrapy如何配置代理，进行抓取
1.在Scrapy工程下新建“middlewares.py”
# Importing base64 library because we'll need it ONLY in case if the proxy we are going to use requires authenticationimport base64 # Start your middleware classclass ProxyMiddleware(object): # overwrite process request def process_request(self, request, spider): # Set the location of the proxy request.meta['proxy&<a style="color:transparent">来@源gao*daima.com搞@代#码网</a>#39;] = "http://YOUR_PROXY_IP:PORT" # Use the following lines if your proxy requires authentication proxy_user_pass = "USERNAME:PASSWORD" # setup basic authentication for the proxy encoded_user_pass = base64.encodestring(proxy_user_pass) request.headers['Proxy-Authorization'] = 'Basic ' + encoded_user_pass
2.在项目配置文件里(./pythontab/settings.py)添加
DOWNLOADER_MIDDLEWARES = { 'scrapy.contrib.downloadermiddleware.httpproxy.HttpProxyMiddleware': 110, 'pythontab.middlewares.ProxyMiddleware': 100,}
搞代码网（gaodaima.com）提供的所有资源部分来自互联网，如果有侵犯您的版权或其他权益，请说明详细缘由并提供版权或权益证明然后发送到邮箱[email protected]，我们会在看到邮件的第一时间内为您处理，或直接联系QQ：872152909。本网站采用BY-NC-SA协议进行授权
转载请注明原文链接：python爬虫之Scrapy 使用代理配置
转载请注明原文链接：python爬虫之Scrapy 使用代理配置
