There are two options. The first (and better) one is using the Fetch as Google option in Webmaster Tools that Mike Flynn commented about. Here are detailed instructions: Go to: https://www.google.com/webmasters/tools/ and log in If you haven’t already, add and verify the site with the “Add a Site” button Click on the site name for … Read more
The user-agent should be specified as a field in the header. Here is a list of HTTP header fields, and you’d probably be interested in request-specific fields, which includes User-Agent. If you’re using requests v2.13 and newer The simplest way to do what you want is to create a dictionary and specify your headers directly, … Read more
There is legal precedent for this. Field v. Google Inc., 412 F. Supp. 2d 1106, (U.S. Dist. Ct. Nevada 2006). Google won a summary judgement based on several factors, most notably that the author did not utilize a robots.txt file in the metatags on his website, which would have prevented Google from crawling and caching … Read more