Fulongのblog

python爬虫

2018/03/27 Python 共 3134 字，约 9 分钟

有个爬虫软件:metaseeker

使用urllib爬取网页

使用urllib的requert爬取网页

import urllib.request
f=urllib.request.urlopen("http://www.baidu.com")
data=f.read();
# print(data);

fhandle=open("D:/Code/pycharmProject/1.html","wb");
fhandle.write(data);
fhandle.close();

urllib.request.urlcleanup();

print(f.info())
print(f.getcode()) # the state code
# 200
print(f.geturl()) # the url

当输入特殊字符串如汉字等不符合URL标准的字符时，需要编码

# 编码
t1=urllib.request.quote("http://www.sina.com.cn")
print(t1)
# 解码
t2=urllib.request.unquote(t1);
print(t2)

# http://www.baidu.com
# http%3A//www.sina.com.cn
# http://www.sina.com.cn

模拟浏览器

403 forbideen错误，禁止访问，需要让爬虫模拟成浏览器，设置Uer-Agent信息(可以在浏览器中获取)，两种解决方式

  1. 使用build_opener()修改报头
    import urllib.request
url="https://blog.csdn.net/fanyun_01/article/details/79406368"
urlagent="Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36"
headers=("User-Agent",urlagent)
opener=urllib.request.build_opener()
opener.addheaders=[headers];
data=opener.open(url).read();
print(data)
  2. 使用add_header()添加报头
    import urllib.request
url="https://blog.csdn.net/fanyun_01/article/details/79406368"
urlagent="Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36"
req=urllib.request.Request(url);
req.add_header('User-Agent',urlagent)
data=urllib.request.urlopen(req).read();
print(data)

超时设置

import urllib.request
url="http://www.baidu.com"
for i in range(1,100):
    try:
        f=urllib.request.urlopen(url,timeout=1)
        # 设置超时限制为1秒
        data=f.read();
        print(len(data))
    except Exception as e:
        print("出现异常-->"+str(e))

HTTP协议请求实战

get

实现爬虫自动地查询关键词为hello可通过url+”/s?wd=”+”hello”实现(get方法)

import urllib.request
url="http://www.baidu.com/s?wd="
keywd="hello"
url=url+keywd
req=urllib.request.Request(url)
data=urllib.request.urlopen(req).read()
fhandle=open("D:\Code\pycharmProject/1.html","wb");
fhandle.write(data);
fhandle.close();

对特殊字符编码

import urllib.request
url="http://www.baidu.com/s?wd="
keywd="gakki's husband "
url=url+urllib.request.quote(keywd);
print(url)
req=urllib.request.Request(url)
data=urllib.request.urlopen(req).read()
fhandle=open("D:\Code\pycharmProject/1.html","wb");
fhandle.write(data);
fhandle.close();

post

实验网址:http://www.iqianyue.com/mypost

里面有个表单

首先对数据进行处理，使用urllib.parse.urlencode方法，然后add_header()模拟浏览器爬取。。。

import urllib.request
import urllib.parse
url="http://www.iqianyue.com/mypost/"
print(url)
postdata=urllib.parse.urlencode({"name":"mafulong","pass":"wahhh"}).encode("utf-8")

req=urllib.request.Request(url,postdata)
urlagent="Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36"
req.add_header('User-Agent',urlagent)
data=urllib.request.urlopen(req).read()
print(data)
fhandle=open("D:\Code\pycharmProject/1.html","wb");
fhandle.write(data);
fhandle.close();

代理服务器的设置

免费代理IP网址整理：http://www.xicidaili.com/ 找验证时间最短的，时间长的可能会失效

install_opener对全局的urllib.request有效

def use_proxy(proxy_addr,url):
    import urllib.request
    proxy=urllib.request.ProxyHandler({'http':proxy_addr})
    opener=urllib.request.build_opener(proxy,urllib.request.HTTPHandler)
    urllib.request.install_opener(opener)
    data=urllib.request.urlopen(url).read().decode('utf-8')
    return data;

proxy_addr="122.114.31.177"
data=use_proxy(proxy_addr,"http://www.baidu.com");
print(len(data));
print(data)

Search

    Table of Contents

    Popular Posts