Searching in Google with Python

I assume you are using this library by Mario Vilas because of the stop=20 argument which appears in his code. It seems like this library is not able to return anything but the URLs, making it horribly undeveloped. As such, what you want to do is not possible with the library you are currently using.

I would suggest you instead use abenassi/Google-Search-API. Then you can simply do:

from google import google
num_page = 3
search_results = google.search("This is my query", num_page)
for result in search_results:
    print(result.description)

You can also use a third-party service like SerpApi which is a Google search engine results. It solves the issues of having to rent proxies and parsing the HTML results. JSON output is particularly rich.

It's easy to integrate with Python:

from serpapi import GoogleSearch

params = {
    "q" : "Coffee",
    "location" : "Austin, Texas, United States",
    "hl" : "en",
    "gl" : "us",
    "google_domain" : "google.com",
    "api_key" : "demo",
}

query = GoogleSearch(params)
dictionary_results = query.get_dict()

GitHub: https://github.com/serpapi/google-search-results-python

Not exactly what I was looking for, but I found myself a nice solution for now (I might edit this if I will able to make this better). I combined searching in Google like I did (returning only URL) and the Beautiful Soup package for parsing HTML pages:

from googlesearch import search
import urllib
from bs4 import BeautifulSoup

def google_scrape(url):
    thepage = urllib.urlopen(url)
    soup = BeautifulSoup(thepage, "html.parser")
    return soup.title.text

i = 1
query = 'search this'
for url in search(query, stop=10):
    a = google_scrape(url)
    print str(i) + ". " + a
    print url
    print " "
    i += 1

This gives me a list of the title of pages and the link.

And another great solutions:

from googlesearch import search
import requests

for url in search(ip, stop=10):
            r = requests.get(url)
            title = everything_between(r.text, '<title>', '</title>')

Most of them I tried using, but didn't work out for me or gave errors like search module not found despite importing packages. Or I did work out with selenium web driver and it works great if used with Firefox or chrome or Phantom web browser, but still I felt it was a bit slow in terms of execution time, as it queried browser first and then returned search result.

So I thought of using google api and it works amazingly quick and returns results accurately.

Before I share the code here are few quick tips to follow:-

1. Register on Google Api to get a Google Api key (free version)
2. Now search for Google Custom Search and set up your free account to get a custom search id
3. Now add this package(google-api-python-client) in your python project (can be done by writing !pip install google-api-python-client )

That is it and all you have to do now is run this code:-

from googleapiclient.discovery import build

my_api_key = "your API KEY TYPE HERE"
my_cse_id = "YOUR CUSTOM SEARCH ENGINE ID TYPE HERE"

def google_search(search_term, api_key, cse_id, **kwargs):
      service = build("customsearch", "v1", developerKey=api_key)
      res = service.cse().list(q=search_term, cx=cse_id, **kwargs).execute()
      return res['items']

results= google_search("YOUR SEARCH QUERY HERE",my_api_key,my_cse_id,num=10) 

for result in results:
      print(result["link"])