working with large web resources in python
Saving the file to disk.1
import urllib2
import shutil
req = urllib2.urlopen(url)
with open(filename, 'wb') as f:
shutil.copyfileobj(req, f)
Reading GZIP compressed CSV files:2
import csv
import gzip
with gzip.open(filename) as f:
reader = csv.reader(f, quoting=csv.QUOTE_NONE)
header = csv.next()
for row in reader:
entry = dict(zip(header, row))
# ...