江苏龙网

  1. 首页 > 生活 > >

怎样使用爬虫获取新加坡PSI信息

瓶颈主要在硬盘IO上，30线程爬网页 vs 单线程写入.csv文件。改成并发写入数据库应该会快很多。有时刚开始就会出现报递归Error，可是我代码中没用递归啊。Traceback (most recent call last): File "C:\\Python 3.5\\lib\\multiprocessing\\queues.py", line 241, in _feed obj = ForkingPickler.dumps(obj) File "C:\\Python 3.5\\lib\\multiprocessing\\reduction.py", line 50, in dumps cls(buf, protocol).dump(obj)RecursionError: maximum recursion depth exceeded翻了源码发现是Queue在传递数据时序列化引起的，在Google了一番找到解决办法，把递归深度设置大一点，比如一百万：sys.setrecursionlimit(1000000)#!/usr/bin/env python# -*- coding: utf-8 -*-# @Author: loveNightimport osimport sysimport csvimport timeimport requestsimport threadingfrom multiprocessing import Queuefrom datetime import datetime, timedeltafrom bs4 import BeautifulSoup as BSfrom multiprocessing.dummy import Poolsys.setrecursionlimit(1000000) # 递归深度，默认只有900os.chdir(sys.path)url_pattern = r"http://www.nea.gov.sg/anti-pollution-radiation-protection/air-pollution-control/psi/historical-psi-readings/year/{0}/month/{1}/day/{2}"# 表头table_header = headers = { "Accept-Encoding": "gzip,deflate,sdch", "User-Agent": "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2125.122 Safari/537.36 SE 2.X MetaSr 1.0", "Host": "www.nea.gov.sg",}session = requests.Session()session.headers = headersdelay = 0 # 网络请求之间的间隔QUIT = "Quit"queue = Queue()# 要下载的日期dt = datetime(2014, 4, 1)dt_now = datetime.now()todo = while dt \u0026lt;= dt_now: todo.append(dt) dt += timedelta(days=1)# 打开网页def getPage(url): if delay: time.sleep(delay) return session.get(url).text# 写入文件def save(filename): start = time.time() with open(filename, "w", newline="") as output: writer = csv.writer(output) writer.writerow(table_header) while True: lines = queue.get() if isinstance(lines, str) and lines == QUIT: break else: print("拿到数据，正在写入", datetime.now()) writer.writerows(lines) print("写入完成！用时 %s " % (time.time() - start))# 解析指定日期的页面def resolvePage(dt): year = dt.year month = dt.month day = dt.day html = getPage(url_pattern.format(year, month, day)) soup = BS(html, "lxml") # 需要安装第三方库lxml，也可以使用自带的html.parser table = soup.find(name="table", class_="text_psinormal") if table: trs = table.find_all("tr") trs = trs # 去除表头 lines = for tr in trs: datas = + lines.append(datas) queue.put(lines) # 传入整张表# 开始下载filename = "data.csv"t = threading.Thread(target=save, args=(filename,))t.daemon = Truet.start()pool = Pool(30)pool.map(resolvePage, todo)pool.close()pool.join()queue.put(QUIT)用时127秒。然后用Excel打开脚本同目录下的data.csv，按Year, Month, Day, Time排序就行了，一共1万4千多行。


推荐阅读


上一篇：手机■200元就能网购苹果手机？常州11岁男孩被骗16300元

下一篇：运维想转python程序员，没有项目经验咋办