How to Spy on Competitors with Python & Data Studio (Advanced SEO)

How to Spy on Competitors with Python & Data Studio
‘ );

h3_html = ‘



cta = ‘‘+cat_head_params.cta_text.textual content+’
atext = ‘


scdetails = scheader.getElementsByClassName( ‘scdetails’ );
sappendHtml( scdetails[0], h3_html );
sappendHtml( scdetails[0], atext );
sappendHtml( scdetails[0], cta );
// emblem
sappendHtml( scheader, “” );
sc_logo = scheader.getElementsByClassName( ‘sc-logo’ );
logo_html = ‘‘;
sappendHtml( sc_logo[0], logo_html );

sappendHtml( scheader, ‘


‘ );

if(“undefined”!=typeof __gaTracker)
} // endif cat_head_params.sponsor_logo

Whether you’re new to search engine optimization or a seasoned professional, investigating random drops in key phrase rankings is simply a part of the commerce.

There are many strategies to investigating why your focused key phrases could have dropped, however utilizing Python and Data Studio collectively is a recreation changer!

A fast disclaimer: I didn’t create this script myself. A fellow search engine optimization and developer, Evan from Architek, collaborated with me to clear up a specific want I had. Evan was the brains behind creating the script.

I at all times wished to have the option to view a big set of Google outcomes at scale, with out having to conduct these searches manually one-by-one.

Evan talked about that Python often is the ultimate resolution for fixing my search engine optimization investigation wants.

Why Python Is Relevant to search engine optimization

Python is an extremely highly effective programing language that may do absolutely anything. One of the extra widespread makes use of for Python is automating every day monotonous duties.

One of the best issues about Python is that there are a number of other ways to accomplish the identical job. However, this additionally provides a brand new stage of problem.

Most of the pattern Python scripts on the market generally is a bit outdated, so that you’ll end up doing quite a lot of trial and error.

Python has many purposes for analyzing search engine optimization information. The secret is to have the best thought. If you’ve an thought to automate a job, likelihood is that there’s a script that may be created for it.

To keep up to date with the most recent use circumstances for Python in search engine optimization, Hamlet Batista has printed some superior articles right here at Search Engine Journal.

What This Specific Script Does & Doesn’t Do

Most key phrase rating instruments report a mean rating for a key phrase over a specified timeframe. This Python script runs a single crawl on the time you run it out of your IP deal with. This script shouldn’t be meant to monitor key phrase rankings.

The goal of this script was to clear up a problem I used to be having with investigating sudden drops of rankings throughout my consumer and their competitors.

Most key phrase rating instruments will let you know which pages your area is rating for, however not your competitor’s highest rating pages per key phrase.

So Why’s That Important?

In this situation, we aren’t tacking web page efficiency for the long-term. We’re merely making an attempt to get fast information.

This script permits us to shortly determine tendencies throughout the natural panorama and see which pages are performing finest.

What You’ll Need to Get Started

If you’re new to Python, I like to recommend trying out the official Python tutorial or Automate the Boring Stuff.

For this tutorial I’m utilizing PyCharm CE, however you need to use Sublime Text or no matter your most well-liked growth setting is.

This script is written in Python three and could also be a bit superior for people new to this programming language.

If you haven’t discovered an interpreter or arrange your first digital setting but, this information may help you get began.

Once you’re arrange with a brand new digital setting, you’ll want the next libraries:

Now that you simply’re all arrange, let’s dive into some analysis collectively.

1. Make a List of Keywords to Investigate

We’re going to be utilizing some pattern information to examine some key phrases we’re pretending to monitor.

Let’s fake that you simply checked out your key phrase monitoring software program and observed that the next key phrases dropped greater than 5 positions:

  • search engine optimization Tips
  • Local search engine optimization Advice
  • Learn search engine optimization
  • Search Engine Optimization Articles
  • search engine optimization Blog
  • search engine optimization Basics

Disclaimer: *Searching for too many key phrases could lead to your IP getting quickly banned. Pinging Google for this many outcomes directly could seem spammy and can drain their sources. Use with warning and moderation.

The very first thing we are going to do is place these key phrases in a easy textual content file. The key phrases needs to be separated with a line break, as proven within the screenshot beneath.

Example of keywords to search with Python

2. Run the Python Ranking Investigation Script

The TL;DR of this script is that it does three primary features:

  • Locates and opens your searches.txt file.
  • Uses these key phrases and searches the primary web page of Google for every outcome.
  • Creates a brand new CSV file and prints the outcomes (Keyword, URLs, and web page titles).

We’ve made this script simple to run, so you may simply copy and paste all the script and run it .

from urllib.parse import urlencode, urlparse, parse_qs

from lxml.html import fromstring
from requests import get
import csv

Next, you’ll have the option to enter the primary perform of this script in a single copy/paste motion. This a part of the script units the precise steps taken, however won’t execute the command till the third step.

def scrape_run():
    with open('searches.txt') as searches:
        for search in searches:
           userQuery = search
           uncooked = get("" + userQuery).textual content
           web page = fromstring(uncooked)
           hyperlinks = web page.cssselect('.r a')
           csvfile = 'information.csv'
           for row in hyperlinks:
               raw_url = row.get('href')
               title = row.text_content()
               if raw_url.startswith("/url?"):
                   url = parse_qs(urlparse(raw_url).question)['q']
                   csvRow = [userQuery, url[0], title]
                   with open(csvfile, 'a') as information:
                       author =

Now you’re prepared to run the command. The ultimate step is to copy/paste the command beneath and click on the return key.


That’s it!

three. Use Data Studio to Analyze the Results

By working this command, you might discover new CSV file has been created known as information.csv. These are your uncooked outcomes, which we’ll want for the ultimate step.

My company has created a helpful Data Studio template for analyzing your outcomes. In order to use this free report, you’ll want to paste your outcomes into Google Sheets.

The web page within the hyperlink above has in-depth directions on how to arrange your Data Studio report.

How to Analyze Our Results

Now that you’ve your new Data Studio report applied, it’s time to make sense of all this information.

What we’re on the lookout for are patterns. Yes, you could find patterns within the uncooked information, however this Data Studio template has a helpful function that permits us to shortly determine which pages are rating essentially the most steadily for our focused key phrases.

This is useful, as a result of it permits us to see which opponents are performing effectively, and which particular pages are performing effectively.

Data Studio template for analyzing python web scraping data

As you may see within the Data Studio screenshot above, Moz and Ahrefs are the highest two opponents rating for our searched key phrases.

However, that doesn’t actually assist us work out precisely what they’re doing to rank for these key phrases.

That’s the place the second chart is useful. This shows every rating web page and what number of instances they happen for all of our search queries. We’re shortly in a position to determine the highest three performing pages for our key phrases.

Need to filter down to a web page or key phrase particular stage?

We’ve included filters on the prime of the Data Studio template to simplify this.

data studio filters

Once you’ve made a listing of the best-performing pages, you may conduct additional on-page and off-page evaluation to work out why these pages are performing so effectively.

Getting Stuck?

If you’re getting caught, attain out to the inventor of this script for ideas or customized programming options.

What’s Your Idea?

Hopefully, this has sparked some inventive concepts on how you need to use Python to assist automate your search engine optimization processes.

More Resources:

Image Credits

All screenshots taken by writer, May 2019

Source hyperlink search engine optimization

Be the first to comment

Leave a Reply

Your email address will not be published.