wget

How to download multiple URLs using wget using a single command?

by

From man wget: 2 Invoking By default, Wget is very simple to invoke. The basic syntax is: wget [option]… [URL]… So, just use multiple URLs: wget URL1 URL2 Or using the links from comments: $ cat list.txt http://www.vodafone.de/privat/tarife/red-smartphone-tarife.html http://www.verizonwireless.com/smartphones-2.shtml http://www.att.com/shop/wireless/devices/smartphones.html and your command line: wget -E -H -k -K -p -e robots=off -P /Downloads/ -i … Read more

How can I use Python’s Requests to fake a browser visit a.k.a and generate User Agent? [duplicate]

by

Provide a User-Agent header: import requests url=”http://www.ichangtou.com/#company:data_000008.html” headers = {‘User-Agent’: ‘Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36’} response = requests.get(url, headers=headers) print(response.content) FYI, here is a list of User-Agent strings for different browsers: List of all Browsers As a side note, there is a pretty useful third-party package called … Read more

How can I set a proxy for Wget?

by

For all users of the system via the /etc/wgetrc or for the user only with the ~/.wgetrc file: use_proxy=yes http_proxy=127.0.0.1:8080 https_proxy=127.0.0.1:8080 or via -e options placed after the URL: wget … -e use_proxy=yes -e http_proxy=127.0.0.1:8080 …

Downloading Xcode with wget or curl

by

For Chrome, Install cookies.txt Chrome extension Login to Apple Developer site and get the url for downloading Run cookies.txt extension and download cookies.txt file From the cookies.txt download directory, load cookies into wget and start resumable download. For example, to download Xcode_7.dmg, you would run: wget –load-cookies=cookies.txt -c http://adcdownload.apple.com/Developer_Tools/Xcode_7/Xcode_7.dmg