回复内容：
context_re = r’
(.*?)
‘你准备的这个正则表达式啊，truncated！断在了
这里，所以只能爬第一段。爬取新浪军事论坛需要做三件事：
一、
上CSDN汪海老师的专栏，http://blog.gaodaima.com/column/details/why-bug.html，学习一个。
二、
按F12看一下前端。
三、
<code class="language-python"><span class="kn">from</span> <span class="nn">bs4</span> <span class="kn">import</span> <span class="n">BeautifulSoup</span><span class="kn">import</span> <span class="nn">requests</span><span class="n">response</span> <span class="o">=</span> <span class="n">requests</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s">"http://club.mil.news.sina.com.cn/thread-666013-1-1.html?retcode=0"</span><span class="p">)</span> <span class="c">#硬点网址</span><span class="n">response</span><span class="o">.</span><span class="n">encoding</span> <span class="o">=</span> <span class="s">'gb18030'</span> <span class="c">#中文编码</span><span class="n">soup</span> <span class="o">=</span> <span class="n">BeautifulSoup</span><span class="p">(</span><span class="n">response</span><span class="o">.</span><span class="n">text</span><span class="p">,</span> <span class="s">'html.parser'</span><span class="p">)</span> <span class="c">#构建BeautifulSoup对象</span><span class="n">divs</span> <span class="o">=</span> <span class="n">soup</span><span class="p">(</span><span class="s">'div'</span><span class="p">,</span> <span class="s">'mainbox'</span><span class="p">)</span> <span class="c">#每个楼层</span><span class="k">for</span> <span class="n">div</span> <span class="ow">in</span> <span class="n">divs</span><span class="p">:</span> <span class="n">comments</span> <span class="o">=</span> <span class="n">div</span><span class="o">.</span><span class="n">find_all</span><span class="p">(</span><span class="s">'div'</span><span class="p">,</span><span class="s">'cont f14'</span><span class="p">)<strong>本文来源gaodai#ma#com搞@@代~&码*网2</strong></span> <span class="c">#每个楼层的正文</span> <span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="s">'Sina_Military_Club.txt'</span><span class="p">,</span><span class="s">'a'</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span> <span class="n">f</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="s">'</span><span class="se">\n</span><span class="s">'</span><span class="o">+</span><span class="nb">str</span><span class="p">(</span><span class="n">comments</span><span class="p">)</span><span class="o">+</span><span class="s">'</span><span class="se">\n</span><span class="s">'</span><span class="p">)</span></code>刚好几个小时前就在写一个爬取网站会员（公司）资料的小程序
具体的编程问题就不回答了，跟用什么语言写代码无关，关键是你要分析好这个页面的html代码结构，写出合适的正则表达式来进行匹配，如果想简化的话，可以进行分次匹配（比如先得到里面的第一个里面的内容就是原帖的地址，然后再进一步处理）
大数据分析就不会了，还请赐教。<code class="language-python"><span class="kn">import</span> <span class="nn">requests</span><span class="kn">from</span> <span class="nn">bs4</span> <span class="kn">import</span> <span class="n">BeautifulSoup</span><span class="n">r</span> <span class="o">=</span> <span class="n">requests</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s">"http://club.mil.news.sina.com.cn/thread-666013-1-1.html"</span><span class="p">)</span><span class="n">r</span><span class="o">.</span><span class="n">encoding</span> <span class="o">=</span> <span class="n">r</span><span class="o">.</span><span class="n">apparent_encoding</span><span class="n">soup</span> <span class="o">=</span> <span class="n">BeautifulSoup</span><span class="p">(</span><span class="n">r</span><span class="o">.</span><span class="n">text</span><span class="p">)</span><span class="n">result</span> <span class="o">=</span> <span class="n">soup</span><span class="o">.</span><span class="n">find</span><span class="p">(</span><span class="n">attrs</span><span class="o">=</span><span class="p">{</span><span class="s">"class"</span><span class="p">:</span> <span class="s">"cont f14"</span><span class="p">})</span><span class="k">print</span> <span class="n">result</span><span class="o">.</span><span class="n">text</span></code>用beautifulSoup吧,正则太多了看着都头疼.先用了BeautifulSoup爬取数据
<code class="language-python"><span class="c"># -*- coding:utf-8 -*-</span><span class="kn">import</span> <span class="nn">re</span><span class="o">,</span> <span class="nn">requests</span><span class="kn">from</span> <span class="nn">bs4</span> <span class="kn">import</span> <span class="n">BeautifulSoup</span><span class="kn">import</span> <span class="nn">sys</span><span class="nb">reload</span><span class="p">(</span><span class="n">sys</span><span class="p">)</span><span class="n">sys</span><span class="o">.</span><span class="n">setdefaultencoding</span><span class="p">(</span><span class="s">'utf-8'</span><span class="p">)</span><span class="n">url</span> <span class="o">=</span> <span class="s">"http://club.mil.news.sina.com.cn/viewthread.php?tid=666013&extra=page%3D1&page=1"</span><span class="n">req</span> <span class="o">=</span> <span class="n">requests</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">url</span><span class="p">)</span><span class="n">req</span><span class="o">.</span><span class="n">encoding</span> <span class="o">=</span> <span class="n">req</span><span class="o">.</span><span class="n">apparent_encoding</span><span class="n">html</span> <span class="o">=</span> <span class="n">req</span><span class="o">.</span><span class="n">text</span><span class="n">soup</span> <span class="o">=</span> <span class="n">BeautifulSoup</span><span class="p">(</span><span class="n">html</span><span class="p">)</span><span class="nb">file</span> <span class="o">=</span> <span class="nb">open</span><span class="p">(</span><span class="s">'sina_club.txt'</span><span class="p">,</span> <span class="s">'w'</span><span class="p">)</span><span class="n">x</span> <span class="o">=</span> <span class="mi">1</span><span class="k">for</span> <span class="n">tag</span> <span class="ow">in</span> <span class="n">soup</span><span class="o">.</span><span class="n">find_all</span><span class="p">(</span><span class="s">'div'</span><span class="p">,</span> <span class="n">attrs</span> <span class="o">=</span> <span class="p">{</span><span class="s">'class'</span><span class="p">:</span> <span class="s">"cont f14"</span><span class="p">}):</span> <span class="n">word</span> <span class="o">=</span> <span class="n">tag</span><span class="o">.</span><span class="n">get_text</span><span class="p">()</span> <span class="n">line1</span> <span class="o">=</span> <span class="s">"---------------评论"</span> <span class="o">+</span> <span class="nb">str</span><span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="o">+</span> <span class="s">"---------------------"</span> <span class="o">+</span> <span class="s">"</span><span class="se">\n</span><span class="s">"</span> <span class="n">line2</span> <span class="o">=</span> <span class="n">word</span> <span class="o">+</span> <span class="s">"</span><span class="se">\n</span><span class="s">"</span> <span class="n">line</span> <span class="o">=</span> <span class="n">line1</span> <span class="o">+</span> <span class="n">line2</span> <span class="n">x</span> <span class="o">+=</span> <span class="mi">1</span> <span class="nb">file</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="n">line</span><span class="p">)</span><span class="nb">file</span><span class="o">.</span><span class="n">close</span><span class="p">()</span></code>哎，扒就扒吧，发了paper能不能告诉我刊号页数让我看一下？我们自己都没做大数据分析……建议用一下正则测试工具 你需要pyquery，可以使用jquery一样的语法。你值得拥有。
https://pythonhosted.org/pyquery/搞代码网（gaodaima.com）提供的所有资源部分来自互联网，如果有侵犯您的版权或其他权益，请说明详细缘由并提供版权或权益证明然后发送到邮箱[email protected]，我们会在看到邮件的第一时间内为您处理，或直接联系QQ：872152909。本网站采用BY-NC-SA协议进行授权
转载请注明原文链接：如何编写 Python 程序爬取新浪军事论坛？
