Build a URL using Requests module Python

requests is basically a convenient wrapper around urllib (and 2,3 and other related libraries). You can import urljoin(), quote() from requests.compat, but this is essentially the same as using them directly from urllib and urlparse modules: >>> from requests.compat import urljoin, quote_plus >>> url = “http://some-address.com/api/” >>> term = ‘This is a test’ >>> urljoin(url, …

Read more

Is there a way to globally override requests’ timeout setting?

The simplest way is to “shim” the session’s request function: import requests import functools s = requests.Session() s.request = functools.partial(s.request, timeout=3) # now all get, post, head etc requests should timeout after 3 seconds # following will fail s.get(‘https://httpbin.org/delay/6’) # we can still pass higher timeout when needed # following will succeed s.get(‘https://httpbin.org/delay/6’, timeout=7)

How can I use Python’s Requests to fake a browser visit a.k.a and generate User Agent? [duplicate]

Provide a User-Agent header: import requests url=”http://www.ichangtou.com/#company:data_000008.html” headers = {‘User-Agent’: ‘Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36’} response = requests.get(url, headers=headers) print(response.content) FYI, here is a list of User-Agent strings for different browsers: List of all Browsers As a side note, there is a pretty useful third-party package called …

Read more

How to fix “403 Forbidden” errors when calling APIs using Python requests?

It seems the page rejects GET requests that do not identify a User-Agent. I visited the page with a browser (Chrome) and copied the User-Agent header of the GET request (look in the Network tab of the developer tools): import requests url=”http://worldagnetwork.com/” headers = {‘User-Agent’: ‘Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5) AppleWebKit/537.36 (KHTML, like …

Read more

how to use github api token in python for requesting

Here’s some code that might help you out. Examples: Example 1 (auth): username=”user” token = ‘token’ login = requests.get(‘https://api.github.com/search/repositories?q=github+api’, auth=(username,token)) Example 2 (headers): headers = {‘Authorization’: ‘token ‘ + token} login = requests.get(‘https://api.github.com/user’, headers=headers) print(login.json()) Example 3 (delete repo): user=”username” repo = ‘some_repo’ # Delete this repo headers = {‘Authorization’: ‘token ‘ + token} login …

Read more