php爬小说技巧_若何用python爬取收集小说

文章目录 [+]

<!DOCTYPE html><head><title>三寸人间最新章节_三寸人间全文阅读 - 顶点小说网</title><meta http-equiv="Content-Type" content="text/html; charset=utf8" /><meta name="keywords" content="三寸人间,三寸人间最新章节,三寸人间全文阅读"/><meta name="description" content="三寸人间最新章节及全文阅读，三寸人间无弹窗广告免费阅读尽在顶点小说网。 " /><meta http-equiv="Cache-Control" content="no-siteapp" /><meta http-equiv="Cache-Control" content="no-transform" /><meta http-equiv="mobile-agent" content="format=html5; url=https://m.23us.so/wapbook/43.html" /><meta http-equiv="mobile-agent" content="format=xhtml; url=https://m.23us.so/wapbook/43.html" /><link rel="stylesheet" href="/23us/css/style.css" type="text/css"/><script language="javascript" type="text/javascript" src="/23us/js/xiaoshuo.js?1"></script></head><body><div id="a_head"><ul><li><a href="/">首页</a></li><li><a href="/list/1_1.html">玄幻邪术</a></li><li><a href="/list/2_1.html">武侠修真</a></li><li><a href="/list/3_1.html">都邑言情</a></li><li><a href="/list/4_1.html">历史军事</a></li><li><a href="/list/5_1.html">网游竞技</a></li><li><a href="/list/6_1.html">科幻小说</a></li><li><a href="/list/7_1.html">胆怯灵异</a></li><li><a href="/list/8_1.html">女生小说</a></li><li><a href="/list/9_1.html">其他小说</a></li><li><a href="/full.html">全本小说</a></li><script type="text/javascript">show_search_top();</script></ul></div><div id="a_main"><div class="bdtop"></div><div class="bdsub"><dl><dt><p class="fr"><a rel="nofollow" href="Javascript:void(0);" onclick="javascript:Ajax.Request('/addbookcase/43.php',{onComplete:function(){if('-1'==this.response){alert('先登录再收藏！ ');}else{alert('加入书架成功！ ');}}});" target="_blank">加入书架</a> | <a rel="nofollow" href="Javascript:void(0);" onclick="javascript:Ajax.Request('/recommend/43/',{onComplete:function(){alert('推举本书成功！ ');}});" target="_blank">推举本书</a> | <a href="https://www.23us.so/xiaoshuo/43.html">返回书页</a></p><a href="https://www.23us.so/">顶点小说网</a>-> <a href="/list/2_1.html">武侠仙侠</a> -><a href="https://www.23us.so/xiaoshuo/43.html">三寸人间</a></dt><dd><h1>三寸人间最新章节</h1></dd><div class="adlist"><script>show_list();</script></div><dd><h3>作者：耳根</h3><br><h3>更新韶光：2019-10-02 14:48:20</h3></dd><dd><dd style="margin-left: 20px;margin-top: 10px;">推举阅读：<a href="https://www.23us.so/xiaoshuo/3574.html"><strong>飞剑问道</strong></a> <a href="https://www.23us.so/xiaoshuo/13332.html"><stongr>元尊</strong></a> <a href="https://www.23us.so/xiaoshuo/43.html"><strong>三寸人间</strong></a> <a href="https://www.23us.so/xiaoshuo/9495.html"><strong>不朽凡人</strong></a> <a href="https://www.23us.so/xiaoshuo/14239.html"><strong>永恒国度</strong></a> <a href="https://www.23us.so/xiaoshuo/9579.html"><strong>凡人修仙传仙界篇</strong></a> <a href="https://www.23us.so/xiaoshuo/4874.html"><strong>一品道门</strong></a> <a href="https://www.23us.so/xiaoshuo/2527.html"><strong>非凡洪荒</strong></a> <a href="https://www.23us.so/xiaoshuo/13577.html"><strong>最强反派系统</strong></a> <a href="https://www.23us.so/xiaoshuo/955.html"><strong>逍遥派</strong></a> </dd><table cellspacing="1" cellpadding="0" bgcolor="#E4E4E4" id="at"><tr><td class="L"><a href="https://www.23us.so/files/article/html/0/43/3615670.html">写在连载前</a></td><td class="L"><a href="https://www.23us.so/files/article/html/0/43/3615671.html">第一章我要减肥！ </a></td><td class="L"><a href="https://www.23us.so/files/article/html/0/43/11243828.html">第二章王宝乐，你干了什么！ </a></td><td class="L"><a href="https://www.23us.so/files/article/html/0/43/11249082.html">第三章好同学，统统有我！ </a></td><td class="L"><a href="https://www.23us.so/files/article/html/0/43/11249807.html">第四章飘渺道院</a></td></tr><tr><td class="L"><a href="https://www.23us.so/files/article/html/0/43/11250120.html">第五章特招学子</a></td><td class="L"><a href="https://www.23us.so/files/article/html/0/43/11252789.html">第六章麻烦大了</a></td><td class="L"><a href="https://www.23us.so/files/article/html/0/43/11253517.html">第七章全民矿工</a></td><td class="L"><a href="https://www.23us.so/files/article/html/0/43/11256673.html">第八章才智与反击！ </a></td><td class="L"><a href="https://www.23us.so/files/article/html/0/43/11257447.html">第九章太虚噬气诀！ </a></td></tr><tr><td class="L"><a href="https://www.23us.so/files/article/html/0/43/11260698.html">第十章无敌战武系</a></td><td class="L"><a href="https://www.23us.so/files/article/html/0/43/11261589.html">第十一章老师，带我一个</a></td><td class="L"><a href="https://www.23us.so/files/article/html/0/43/11264649.html">第十二章打破</a></td><td class="L"><a href="https://www.23us.so/files/article/html/0/43/11265419.html">第十三章化清丹</a></td><td class="L"><a href="https://www.23us.so/files/article/html/0/43/11269342.html">第十四章法兵系的上风</a></td></tr><tr><td class="L"><a href="https://www.23us.so/files/article/html/0/43/11270039.html">第十五章这便是抢钱！ </a></td><td class="L"><a href="https://www.23us.so/files/article/html/0/43/11273120.html">第十六章上品灵石</a></td><td class="L"><a href="https://www.23us.so/files/article/html/0/43/11273825.html">第十七章专门陵暴我们战武系？</a></td><td class="L"><a href="https://www.23us.so/files/article/html/0/43/11276835.html">第十八章封身境</a></td><td class="L"><a href="https://www.23us.so/files/article/html/0/43/11277527.html">第十九章太虚擒拿术</a></td></tr><tr><td class="L"><a href="https://www.23us.so/files/article/html/0/43/11280524.html">第二十章搏击俱乐部</a></td><td class="L"><a href="https://www.23us.so/files/article/html/0/43/11281242.html">第二十一章叫爸爸！ </a></td><td class="L"><a href="https://www.23us.so/files/article/html/0/43/11284172.html">第二十二章碎了……</a></td><td class="L"><a href="https://www.23us.so/files/article/html/0/43/11284907.html">第二十三章无耻胖兔</a></td><td class="L"><a href="https://www.23us.so/files/article/html/0/43/11287888.html">第二十四章我反面你打</a></td></tr><tr><td class="L"><a href="https://www.23us.so/files/article/html/0/43/11288570.html">第二十五章太虚擒拿术大成</a></td><td class="L"><a href="https://www.23us.so/files/article/html/0/43/11291595.html">第二十六章鄙人谢海洋</a></td><td class="L"><a href="https://www.23us.so/files/article/html/0/43/11292307.html">第二十七章去世神丹！ </a></td><td class="L"><a href="https://www.23us.so/files/article/html/0/43/11295421.html">第二十八章封身大圆满</a></td><td class="L"><a href="https://www.23us.so/files/article/html/0/43/11296102.html">第二十九章绝不犹豫！ </a></td></tr><tr><td class="L"><a href="https://www.23us.so/files/article/html/0/43/11301568.html">第三十章申请考察！ </a></td><td class="L">

是不是看的有些头疼？而且最主要的是每个章节列表的网址没有规律，虽然只是末了六七位数字不一样，但是我找半天末了的数字的规律也没有找到，网上有一张一张爬的，而且实现的也很明白，但是我最初的思路是在目录页获取所有的章节地址，然后一章一章的去下载，开始没有思路，但是后来想了一个本办法，便是下面的代码：

import requestsfrom bs4 import BeautifulSoupheaders = { 'user-agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36'}ls = []cl=[]def getHTMLText(url): try: r = requests.get(url, timeout=30) r.raise_for_status() r.encoding = 'utf-8' return r.text except: return ""def get_maninfo(url): try: res=requests.get(url,headers=headers) res.raise_for_status() res.encoding = res.apparent_encoding soup=BeautifulSoup(res.text,'html.parser') ceshi=soup.find_all('a') for i in ceshi: try: href = i.attrs['href'] ls.append(href) except: continue print(len(ls)) except: return ""def getText(soup): a = str(soup.find_all('dd',{'id':'contents'})) a=a.replace('[<dd id="contents">','') return a.replace('<br/>','')def getHead(soup): a = str(soup.find_all('h1')).replace('<h1>','') return a.replace('</h1>','')def putText(text,h1): dingdian = open("D:\pythonapk\ceshi\三寸人间.txt","a",encoding = "utf-8") dingdian.write(h1 + '\n') dingdian.write(text+'\n\n\n\n') dingdian.closedef man(): url="https://www.23us.so/files/article/html/0/43/index.html" get_maninfo(url) for x in range(28, len(ls)): cl.append(ls[x]) for url in cl: r = getHTMLText(url) soup = BeautifulSoup(r, "html.parser") putText(getText(soup), getHead(soup))man()

结果比较成功，当然速率比较慢

php爬小说技巧_若何用python爬取收集小说 php爬小说技巧_若何用python爬取收集小说 SEO优化