Python实战快速上手BeautifulSoup库爬取专栏标题和地址

文章目录[隐藏]

解析标签
解析属性

解析标签

from bs4 import BeautifulSoup
import requests
url='https://blog.gaodaima.com/weixin_42403632/category_11076268.html'
headers={'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:93.0) Gecko/20100101 Firefox/93.0'}
html=requests.get(url,headers=headers).text
s=BeautifulSoup(html,'html.parser')
title =s.select('h2')
for i in title:
    print(i.text)

第一行代码：导入BeautifulSoup库

第二行代码：导入requests

第三、四、五行代码：获取url的html

第六行代码：激活BeautifulSoup库 ‘html.parser’设置解析器为HTML解析器

第七行代码：选取所有<h2>标签

解析属性

BeautifulSoup库支持根据特定属性解析网页元素

根据class值解析

from bs4 import BeautifulSoup
import requests
url='https://blog.gaodaima.com/weixin_42403632/category_11076268.html'
headers={'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:93.0) Gecko/20100101 Firefox/93.0'}
html=requests.get(url,headers=headers).text
s=BeautifulSoup(html,'html.parser')
title =s.select('.column_article_title')
for i in title:
    print(i.text)

根据ID解析

from bs4 import BeautifulSoup
html='''<div class="crop-img-before">
         <img src="" alt="<a href="http://www.gaodaima.com">搞代码</a>" id="cropImg">
      </div>
        <div id='title'>
        测试成功
        </div>
      <div class="crop-zoom">
         <a href="javascript:;" rel="external nofollow"  rel="external nofollow"  rel="external nofollow"  rel="external nofollow"  class="bt-reduce">-</a><a href="javascript:;" rel="external nofollow"  rel="external nofollow"  rel="external nofollow"  rel="external nofollow"  class="bt-add">+</a>
      </div>
      <div class="crop-img-after">
         <div  class="final-img"></div>
      </div>'''
s=BeautifulSoup(html,'html.parser')
title =s.select('#title')
for i in title:
    print(i.text)

多层筛选

from bs4 import BeautifulSoup
html='''<div class="crop-img-before">
         <img src="" alt="<a href="http://www.gaodaima.com">搞代码</a>" id="cropImg">
      </div>
        <div id='title'>
        456456465
        <h1>测试成功</h1>
        </div>
      <div class="crop-zoom">
         <a href="javascript:;" rel="external nofollow"  rel="external nofollow"  rel="external nofollow"  rel="external nofollow"  class="bt-reduce">-</a><a href="javascript:;" rel="external nofollow"  rel="external nofollow"  rel="external nofollow"  rel="external nofollow"  class="bt-add">+</a>
      </div>
      <div class="crop-img-after">
         <div  class="final-img"></div>
      </div>'''
s=BeautifulSoup(html,'html.parser')
title =s.select('#title')
for i in title:
    print(i.text)
title =s.select('#title h1')
for i in title:
    print(i.text)

提取a标签中的网址

title =s.select('a')
for i in title:
    print(i['href'])

实战-获取博客专栏标题+网址

from bs4 import BeautifulSoup
<strong style="color:transparent">本文来源gao@daima#com搞(%代@#码网@</strong>import requests
import re
url='https://blog.gaodaima.com/weixin_42403632/category_11298953.html'
headers={'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:93.0) Gecko/20100101 Firefox/93.0'}
html=requests.get(url,headers=headers).text
s=BeautifulSoup(html,'html.parser')
title =s.select('.column_article_list li a')
for i in title:
    print((re.findall('原创.*?\n(.*?)\n',i.text))[0].lstrip())
    print(i['href'])

搞代码网（gaodaima.com）提供的所有资源部分来自互联网，如果有侵犯您的版权或其他权益，请说明详细缘由并提供版权或权益证明然后发送到邮箱[email protected]‍，我们会在看到邮件的第一时间内为您处理，或直接联系QQ：872152909。本网站采用BY-NC-SA协议进行授权
转载请注明原文链接：Python实战快速上手BeautifulSoup库爬取专栏标题和地址