前言
本文的文字及图片来源于网络,仅供学习、交流使用,不具有任何商业用途,如有问题请及时联系我们以作处理。
最近发现了一个不错的数据网站，叫“淘数据”。里面的数据都是淘宝的商家数据，包含了店名、类目、标价、成交均价、销量、销售金额等
这个网站还是一位同学和我说才知道的，既然这样，那就开始爬
项目目标
爬取淘宝假发专业数据，假发是我当时随便选的，再想选别的，要收费了
可能是命运的安排吧，知道程序员需要什么
受害者地址
https://www.taosj.com/industry/index.html<span>#</span><span>/data/hotitems/?cid=50023283&brand=&type=&pcid=</span>
www#gaodaima.com来源gao!%daima.com搞$代*!码$网搞代码
环境
Python3.6
pycharm
爬虫代码
导入所需工具
<span>import</span><span> requests </span><span>import</span> csv
分析网页，先F12打开开发者工具，复制你所需要的数据，找数据所在的标签
找到所需要的URL和headers中的参数
url = <span>"</span><span>https://www.taosj.com/data/industry/hotitems/list?cid=50023283&brand=&type=ALL&date=1596211200000&pageNo=1&pageSize=10&orderType=desc&orderField=</span><span>"</span><span>.format(page)
headers </span>=<span> {
</span><span>"</span><span>Host</span><span>"</span>:<span>"</span><span>www.taosj.com</span><span>"</span><span>,
</span><span>"</span><span>Referer</span><span>"</span>:<span>"</span><span>https://www.taosj.com/industry/index.html</span><span>"</span><span>,
</span><span>"</span><span>User-Agent</span><span>"</span>:<span>"</span><span>Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.138 Safari/537.36</span><span>"</span><span>,
}
response </span>= requests.get(url=url, headers=<span>headers)
html_data </span>= response.json()
在json数据中提取相关数据
lis = html_data[<span>"</span><span>data</span><span>"</span>][<span>"</span><span>list</span><span>"</span><span>]
</span><span>for</span> li <span>in</span><span> lis:
tb_url </span>= <span>"</span><span>https://detail.tmall.com/item.htm?id={}</span><span>"</span>.format(li[<span>"</span><span>id</span><span>"</span><span>])
dit </span>=<span> {
</span><span>"</span><span>标题</span><span>"</span>: li[<span>"</span><span>title</span><span>"</span><span>],
</span><span>"</span><span>店铺名称</span><span>"</span>: li[<span>"</span><span>shop</span><span>"</span><span>],
</span><span>"</span><span>类目</span><span>"</span>: li[<span>"</span><span>nextCatName</span><span>"</span><span>],
</span><span>"</span><span>标价</span><span>"</span>: li[<span>"</span><span>oriPrice</span><span>"</span><span>],
</span><span>"</span><span>成交均价</span><span>"</span>: li[<span>"</span><span>price</span><span>"</span><span>],
</span><span>"</span><span>销售量</span><span>"</span>: li[<span>"</span><span>offer30</span><span>"</span><span>],
</span><span>"</span><span>销售金额</span><span>"</span>: li[<span>"</span><span>price30</span><span>"</span><span>],
</span><span>"</span><span>淘宝地址</span><span>"</span><span>: tb_url,
}</span>
保存数据
f = open(<span>"</span><span>淘数据.csv</span><span>"</span>, mode=<span>"</span><span>a</span><span>"</span>, encoding=<span>"</span><span>utf-8-sig</span><span>"</span>, newline=<span>""</span><span>) csv_writer </span>= csv.DictWriter(f, fieldnames=[<span>"</span><span>标题</span><span>"</span>, <span>"</span><span>品牌</span><span>"</span>, <span>"</span><span>店铺名称</span><span>"</span>, <span>"</span><span>类目</span><span>"</span>, <span>"</span><span>标价</span><span>"</span>, <span>"</span><span>成交均价</span><span>"</span>, <span>"</span><span>销售量</span><span>"</span>, <span>"</span><span>销售金额</span><span>"</span>, <span>"</span><span>淘宝地址</span><span>"</span><span>]) csv_writer.writeheader() </span><span>print</span>(dit)
效果图
完整代码
<span>import</span><span> requests
</span><span>import</span><span> csv
f </span>= open(<span>"</span><span>淘数据.csv</span><span>"</span>, mode=<span>"</span><span>a</span><span>"</span>, encoding=<span>"</span><span>utf-8-sig</span><span>"</span>, newline=<span>""</span><span>)
csv_writer </span>= csv.DictWriter(f, fieldnames=[<span>"</span><span>标题</span><span>"</span>, <span>"</span><span>品牌</span><span>"</span>, <span>"</span><span>店铺名称</span><span>"</span>, <span>"</span><span>类目</span><span>"</span>, <span>"</span><span>标价</span><span>"</span>, <span>"</span><span>成交均价</span><span>"</span>, <span>"</span><span>销售量</span><span>"</span>, <span>"</span><span>销售金额</span><span>"</span>, <span>"</span><span>淘宝地址</span><span>"</span><span>])
csv_writer.writeheader()
</span><span>for</span> page <span>in</span> range(1, 51<span>):
url </span>= <span>"</span><span>https://www.taosj.com/data/industry/hotitems/list?cid=50023282&brand=&type=ALL&date=1596211200000&pageNo={}&pageSize=10&orderType=desc&orderField=amount&searchKey=</span><span>"</span><span>.format(page)
</span><span>"""</span><span>
复制开发者工具中的requests headers 中的参数 记得加cookie
</span><span>"""</span><span>
headers </span>=<span> {
</span><span>"</span><span>Host</span><span>"</span>: <span>"</span><span>www.taosj.com</span><span>"</span><span>,
</span><span>"</span><span>Referer</span><span>"</span>: <span>"</span><span>https://www.taosj.com/industry/index.html</span><span>"</span><span>,
</span><span>"</span><span>User-Agent</span><span>"</span>: <span>"</span><span>Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.138 Safari/537.36</span><span>"</span><span>,
}
response </span>= requests.get(url=url, headers=<span>headers)
html_data </span>=<span> response.json()
lis </span>= html_data[<span>"</span><span>data</span><span>"</span>][<span>"</span><span>list</span><span>"</span><span>]
</span><span>for</span> li <span>in</span><span> lis:
tb_url </span>= <span>"</span><span>https://detail.tmall.com/item.htm?id={}</span><span>"</span>.format(li[<span>"</span><span>id</span><span>"</span><span>])
dit </span>=<span> {
</span><span>"</span><span>标题</span><span>"</span>: li[<span>"</span><span>title</span><span>"</span><span>],
</span><span>"</span><span>品牌</span><span>"</span>: li[<span>"</span><span>brand</span><span>"</span><span>],
</span><span>"</span><span>店铺名称</span><span>"</span>: li[<span>"</span><span>shop</span><span>"</span><span>],
</span><span>"</span><span>类目</span><span>"</span>: li[<span>"</span><span>nextCatName</span><span>"</span><span>],
</span><span>"</span><span>标价</span><span>"</span>: li[<span>"</span><span>oriPrice</span><span>"</span><span>],
</span><span>"</span><span>成交均价</span><span>"</span>: li[<span>"</span><span>price</span><span>"</span><span>],
</span><span>"</span><span>销售量</span><span>"</span>: li[<span>"</span><span>offer30</span><span>"</span><span>],
</span><span>"</span><span>销售金额</span><span>"</span>: li[<span>"</span><span>price30</span><span>"</span><span>],
</span><span>"</span><span>淘宝地址</span><span>"</span><span>: tb_url,
}
csv_writer.writerow(dit)
</span><span>print</span>(dit)
搞代码网（gaodaima.com）提供的所有资源部分来自互联网，如果有侵犯您的版权或其他权益，请说明详细缘由并提供版权或权益证明然后发送到邮箱[email protected]，我们会在看到邮件的第一时间内为您处理，或直接联系QQ：872152909。本网站采用BY-NC-SA协议进行授权
转载请注明原文链接：Python爬取淘数据平台商品数据，发现假发的市场原来那么火
转载请注明原文链接：Python爬取淘数据平台商品数据，发现假发的市场原来那么火
