DEV Community: Anthony Bloomer

Monitoring Scrapy using the Python Agent API

Anthony Bloomer — Sun, 19 Jul 2020 17:07:48 +0000

Recently I worked on a project to learn more about the Scrapy framework and to demonstrate how a customer can monitor a framework such as Scrapy using the Python Agent API.

Basic Instrumentation using Agent Background Tasks

A trivial example of how a customer can monitor their Scrapy application is to use the background task decorator with the New Relic Python agent API. Consider the following Spider that scrapes content from Quotes to Scrape:

import scrapy


class QuotesSpider(scrapy.Spider):
    name = "quotes"
    start_urls = [
        'http://quotes.toscrape.com/page/1/',
    ]

    def parse(self, response):
        for quote in response.css('div.quote'):
            yield {
                'text': quote.css('span.text::text').get(),
                'author': quote.css('small.author::text').get(),
                'tags': quote.css('div.tags a.tag::text').getall(),
            }

        for a in response.css('li.next a'):
            yield response.follow(a, callback=self.parse)

To add basic instrumentation using the New Relic Python agent, we just need to add three additional lines of code!

import newrelic.agent
newrelic.agent.initialize('newrelic.ini')

import scrapy


class QuotesSpider(scrapy.Spider):
    name = "quotes"
    start_urls = [
        'http://quotes.toscrape.com/page/1/',
    ]

    @newrelic.agent.background_task()
    def parse(self, response):
        for quote in response.css('div.quote'):
            yield {
                'text': quote.css('span.text::text').get(),
                'author': quote.css('small.author::text').get(),
                'tags': quote.css('div.tags a.tag::text').getall(),
            }

        for a in response.css('li.next a'):
            yield response.follow(a, callback=self.parse)

In the example above, the initialize method is used to initialize the agent with the specified newrelic.ini configuration file. The @newrelic.agent.background_task() decorator is used to instrument the parse function as a background task. This transaction is then displayed as a non-web transactions in the APM UI and separated from web transactions.

Advanced Instrumentation using Scrapy Extensions

To go one step further with instrumenting Scrapy applications is to use Scrapy Extensions. The extensions framework built into Scrapy provides a mechanism for inserting your own custom functionality into Scrapy. Extensions are just regular classes that are instantiated at Scrapy startup, when extensions are initialized.

I worked on an extension that collects some statistics and records a New Relic custom event that can be queried using New Relic Insights. Scrapy uses signals to notify when certain events occur. You can catch some of those signals in your Scrapy application using a custom extension to perform tasks or extend Scrapy to add functionality not provided out of the box.

In my custom New Relic extension, I gather some basic statistics when the Spider is opened, closed, scraped, etc. In the closed method, I send the gathered data using the record_custom_event API method.

You can find the custom extension below:

import newrelic.agent

import logging
import datetime

from scrapy import signals
from scrapy.exceptions import NotConfigured

logger = logging.getLogger(__name__)


class NewRelic(object):

    def __init__(self):
        self.event_stats = {}

    @classmethod
    def from_crawler(cls, crawler):
        if not crawler.settings.getbool('MYEXT_ENABLED'):
            raise NotConfigured

        o = cls()
        crawler.signals.connect(o.spider_opened, signal=signals.spider_opened)
        crawler.signals.connect(o.spider_closed, signal=signals.spider_closed)
        crawler.signals.connect(o.item_scraped, signal=signals.item_scraped)
        crawler.signals.connect(o.item_dropped, signal=signals.item_dropped)
        crawler.signals.connect(o.response_received, signal=signals.response_received)
        return o

    def set_value(self, key, value):
        self.event_stats[key] = value

    def spider_opened(self, spider):
        self.set_value('start_time', datetime.datetime.utcnow())

    def spider_closed(self, spider, reason):
        self.set_value('finish_time', datetime.datetime.utcnow())
        application = newrelic.agent.application()
        self.event_stats.update({'spider': spider.name})
        newrelic.agent.record_custom_event("ScrapyEvent", self.event_stats, application)

    def inc_value(self, key, count=1, start=0, spider=None):
        d = self.event_stats
        d[key] = d.setdefault(key, start) + count

    def item_scraped(self, item, spider):
        self.inc_value('item_scraped_count', spider=spider)

    def response_received(self, spider):
        self.inc_value('response_received_count', spider=spider)

    def item_dropped(self, item, spider, exception):
        reason = exception.__class__.__name__
        self.inc_value('item_dropped_count', spider=spider)
        self.inc_value('item_dropped_reasons_count/%s' % reason, spider=spider)

The above example includes only a few of the signal events available. For a full list of signals go to: https://docs.scrapy.org/en/latest/topics/signals.html

Testing the Project

To try this project, follow these steps:

Clone the repo: git clone https://github.com/AnthonyBloomer/nrscrapy.git
Install the requirements. Run pip install -r requirements.txt
Update newrelic.ini with your license key or export your license key as an environment variable.
Run cd tutorial
Run scrapy crawl quotes

Please note that the information presented here is done so as is without warranty or support. Scrapy is an unsupported framework, therefore New Relic Technical Support cannot offer assistance with this.

Building a Fitness Insights Dashboard

Anthony Bloomer — Sun, 19 Jul 2020 16:55:15 +0000

Introduction

As a motivator to maintain a healthy diet and to get into a routine of daily workouts, I have started to track my intake and exercise habits using MyFitnessPal and other mobile applications. Pulling data from those sources and creating custom events in New Relic Insights allows me to keep track of my progress and provides a way for me to query my data using NRQL and to create visualizations.

Data Sources

The two main applications I use for tracking is MyFitnessPal and MapMyWalk. MyFitnessPal allows you to track your diet and exercise to determine optimal caloric intake and nutrients for your overall goal. MapMyWalk allows you to track cardiovascular activities such as running, walking, and even stationary bike exercises.

Although MyFitnessPal does have an API, it is private-access only. To get around this, I found a Python library called python-myfitnesspal which allows you to scrape your data from MyFitnessPal. See the example below:

import myfitnesspal

client = myfitnesspal.Client('my_username')
day = client.get_date(2013, 3, 2)
day
# >> <03/02/13 {'sodium': 3326, 'carbohydrates': 369, 'calories': 2001, 'fat': 22, 'sugar': 103, 'protein': 110}>

For a daily summary of your nutrition information, you can use a Day object's totals property:

day.totals
# >> {'calories': 2001,
#     'carbohydrates': 369,
#     'fat': 22,
#     'protein': 110,
#     'sodium': 3326,
#     'sugar': 103}

If you want to see the amount of water you've recorded:

day.water
# >> 1

Perfect. These are exactly the data points I am interested in. So, once a day, I call the API to get those values and record a custom event using the Python Agent API:

def send_intake_data():
    client = myfitnesspal.Client(USERNAME, PASSWORD)
    day = client.get_date(datetime.datetime.now())
    totals = day.totals
    totals.update({'water': day.water})
    newrelic.agent.record_custom_event('Meal', totals, application)

Similarly to MyFitnessPal, MapMyWalk does have an API but it is private access only. To address this issue, I wrote a simple script using Selenium which logs into the MapMyWalk website and then goes to a JSON resource which holds my workout data. Once every day, I call this script to get workout data for that day and then record a custom event in Insights:

def login():
    logger.info("Logging into MapMyWalk")
    driver = webdriver.Chrome(options=chrome_options)
    driver.get(LOGIN_URL)
    driver.find_element_by_id('email').send_keys(USERNAME)
    driver.find_element_by_id('password').send_keys(PASSWORD)
    driver.find_element_by_id('password').send_keys(Keys.ENTER)
    return driver

def get_workout_data():
    d = login()
    time.sleep(3)
    today = datetime.date.today()
    d.get(WORKOUTS % (today.month, today.year))
    time.sleep(3)
    pre = d.find_element_by_tag_name("pre").text
    data = json.loads(pre)
    logger.info("JSON response: %s" % data)
    wd = data['workout_data']['workouts']
    d.close()
    d.quit()
    today = str(today)
    if today not in wd:
        logger.info("No workouts found for today.")
        return None
    else:
        return wd[today]

def send_workout_data():
    workout_data = get_workout_data()
    if workout_data is not None:
        for w in workout_data:
            logger.info(w)
            newrelic.agent.record_custom_event('Workout', w, application)

You can find the complete source code on my GitHub. The documentation covers project setup and deployment instructions.

Insights Dashboard

Installation

This project uses python-myfitnesspal to programmatically interact with and analyze the information I have entered into MyFitnessPal. To authenticate, you'll need to export your username and password as environment variables:

export FITNESS_PAL_USERNAME='YOUR_USERNAME'
export FITNESS_PAL_PASSWORD='YOUR_PASSWORD'

I'm using Selenium and the ChromeDriver to automate data retrieval from the MapMyWalk website. You will need to have Google Chrome and ChromeDriver installed. If you are using macOS, you can install ChromeDriver using Homebrew:

brew cask install chromedriver

To send metrics to New Relic Insights, you will also need to export your application name and license key:

export NEW_RELIC_LICENSE_KEY='YOUR_LICENSE_KEY'
export NEW_RELIC_APP_NAME='YOUR_APP_NAME'

Then, create a new Virtual Environment and install the project requirements:

virtualenv env
source env/bin/activate
pip install -r requirements.txt

Finally, launch the worker:

python fitnesspal.py

The program uses schedule to send metrics once a day at 23:30pm.

Deploying to Heroku

Since I'm using Selenium to automate data retrieval from MapMyWalk, you'll need to install the following buildpacks:

You will also need to configure the environment variables outlined in the installation above, for example:

heroku config:set NEW_RELIC_LICENSE_KEY=YOUR_LICENSE_KEY

Querying Insights Data

This tool creates two custom events, Meal and Workout, which you can query using NRQL in Insights.

Examples

Get your average calorie intake since 1 week ago:

SELECT average(calories) FROM Meal SINCE 1 week AGO TIMESERIES 1 day

Get the total number of steps walked since 1 week ago:

SELECT sum(steps) FROM Workout SINCE 1 week AGO TIMESERIES 1 day

Get the total amount of water consumed:

SELECT sum(water) FROM Meal SINCE 1 week AGO TIMESERIES 1 day

Lambda Function to send your Github Traffic Statistics to New Relic Insights

Anthony Bloomer — Sun, 19 Jul 2020 16:51:52 +0000

For repositories that you have push access to within Github, the Github traffic API provides access to the information provided in the graphs section of the repositories UI.

This information includes referral sources, referral paths, views, and clones. The API returns data over the past 14 day period. As the API only returns data over the past 14 day period, this solution aims to give users access to historical Traffic data for their Github repositories within Insights.

This Lambda function calls each endpoint every 15 days and creates custom events that you can then query and create visualisations within New Relic Insights.

See the New Relic One Dashboard below:

How it works

The function uses Github's token based API authentication to gain access to the user's repository data. For each of the users repostories, the script makes a call to each Traffic API endpoint and records a custom event using the Event API if there has been any Traffic data reported.

def lambda_handler(event, context):
  base_url = "https://api.github.com"
  user = os.getenv("GITHUB_USERNAME")
  print("Getting repositories for %s" % user)
  req = api_req("%s/users/%s/repos" % (base_url, user))

  ur = [(r["name"]) for r in req]
  print("Found %s repositories" % len(ur))

  print(" ")

  for repo_name in ur:

    print("Getting referral data for %s" % repo_name)
    referrals = api_req(
      "%s/repos/%s/%s/traffic/popular/referrers" % (base_url, user, repo_name)
    )
    if len(referrals) > 0:
      for ref in referrals:
        referred = {
          "eventType": "Referral",
          "repo_name": repo_name,
          "referrer": ref["referrer"],
          "count": ref["count"],
          "uniques": ref["uniques"],
        }
        print(referred)
        record_custom_event(referred)

        print("Getting top referral path data for %s" % repo_name)

        paths = api_req(
          "%s/repos/%s/%s/traffic/popular/paths" % (base_url, user, repo_name)
        )
        if len(paths) > 0:
          for ref in paths:
            paths = {
              "eventType": "ReferralPath",
              "repo_name": repo_name,
              "path": ref["path"],
              "title": ref["title"],
              "count": ref["count"],
              "uniques": ref["uniques"],
            }
            print(paths)
            record_custom_event(paths)

            print("Getting views for %s" % repo_name)
            req = api_req("%s/repos/%s/%s/traffic/views" % (base_url, user, repo_name))
            if req["count"] > 0 or req["uniques"] > 0:
              viewed = {
                "eventType": "View",
                "repo_name": repo_name,
                "count": req["count"],
                "uniques": req["uniques"],
              }
              print(viewed)
              record_custom_event(viewed)

              print("Getting clones for %s" % repo_name)
              req = api_req("%s/repos/%s/%s/traffic/clones" % (base_url, user, repo_name))
              if req["count"] > 0 or req["uniques"] > 0:
                cloned = {
                  "eventType": "Clone",
                  "repo_name": repo_name,
                  "count": req["count"],
                  "uniques": req["uniques"],
                }
                record_custom_event(cloned)
                print(" ")

You can view the complete source code on my Github.

Setup

Clone the repository: git clone https://github.com/AnthonyBloomer/github-traffic-insights.git

Update the following environment variables in the template.yml file:

GITHUB_TOKEN - Create a personal access token using your Personal access tokens settings page.
GITHUB_USERNAME - Your GitHub username.
NEW_RELIC_ACCOUNT_ID - Your New Relic Account ID
NEW_RELIC_INSERT_KEY - Your New Relic Insert Key

Deploy using the AWS CLI:

aws cloudformation deploy --template template.yml --stack-name github

Sample Dashboard

The project also includes the sample dashboard JSON shown above which you can import into your New Relic account using the Dashboards API.

Periodically share a New Relic Dashboard using the GMail API

Anthony Bloomer — Sun, 19 Jul 2020 16:45:03 +0000

Overview

A common request I have received from customers is the ability to create performance reports to share with other members of their team or to their clients. While New Relic does provide email weekly performance reports, it is often the case that customers wish to email custom reports with the performance and business metrics they care about most.

With the New Relic NerdGraph API, it is now possible to retrieve the link for the current snapshot of your New Relic Dashboard! This opens up a lot of programmable capabilities for our customers. One use-case would be to programmatically generate a snapshot of a performance dashboard that could be mailed every week.

Given this new functionality, I worked on a project to periodically share a New Relic Dashboard using the GMail API.

How it works

As a starting point, I developed a small Python client that abstracts the API call made to the NerdGraph API. The client retrieves the latest snapshot URL for a given dashboard and then downloads that file:

def exporter(
    guid: str,
    file_type: str = "PDF",
    output_directory: str = os.getcwd(),
    filename: str = int(time.time()),
    width: int = 2000,
    height: int = 2000,
):
    if not os.getenv("NEW_RELIC_PERSONAL_API_KEY"):
        sys.exit("New Relic API key and region environment variables required.")
    headers = {
        "Content-Type": "application/json",
        "API-Key": os.getenv("NEW_RELIC_PERSONAL_API_KEY"),
    }
    endpoint_url = (
        "https://api.eu.newrelic.com/graphql"
        if os.getenv("NEW_RELIC_REGION") == "EU"
        else "https://api.newrelic.com/graphql"
    )
    if not guid or not guid.strip():
        sys.exit("Dashboard GUID required.")
    query = '''
    mutation {
        dashboardCreateSnapshotUrl(guid: "%s")
    }
    ''' % guid
    req = requests.post(url=endpoint_url, json={"query": query}, headers=headers)

    if not req.ok:
        req.raise_for_status()
    resp = req.json()
    url = resp["data"]["dashboardCreateSnapshotUrl"]
    if url is None:
        sys.exit("Dashboard not found.")
    url = url.split("?", 1)[0] + "?format=%s&width=%s&height=%s" % (
        file_type,
        width,
        height,
    )
    resp = requests.get(url=url)
    content_type = resp.headers["content-type"]
    extension = mimetypes.guess_extension(content_type)
    of = "%s/%s%s" % (output_directory, filename, extension)
    with open(of, "wb") as f:
        f.write(resp.content)
        f.close()
    return of

Next, I wrote some code to interact with the GMail API. Google provides a client library for Python that simplifies a lot of the process involved with things like API authorization and sending emails.

The process is to first authorize the application to allow the API to perform actions on behalf of the user. Once a token has been generated, the GMail API handles token refreshes so initial authorization only needs to be done once:

def get_service():
    scopes = "https://mail.google.com/"
    credentials = None
    if os.path.exists("token.pickle"):
        with open("token.pickle", "rb") as token:
            credentials = pickle.load(token)
    if not credentials or not credentials.valid:
        if credentials and credentials.expired and credentials.refresh_token:
            credentials.refresh(Request())
        else:
            flow = InstalledAppFlow.from_client_secrets_file("credentials.json", scopes)
            credentials = flow.run_local_server(port=0)
        with open("token.pickle", "wb") as token:
            pickle.dump(credentials, token)

    service = build("gmail", "v1", credentials=credentials)
    return service

Once the user has authorized the application, it is then easy to perform actions on their behalf:

def send_email(session, user_id, message):
    k = session.users().messages().send(userId=user_id, body=message).execute()
    print("Message Id: %s" % k["id"])
    return k

The final step of this project was to write a script to periodically download the latest dashboard image and send it is an email:

def job():
    print("Exporting New Relic dashboard...")
    dashboard = exporter(
        guid=config["dashboard"]["guid"],
        file_type=config["dashboard"]["file_type"],
        width=int(config["dashboard"]["width"]),
        height=int(config["dashboard"]["height"]),
    )
    print("Creating message...")
    message = create_message_with_attachment(
        sender=config["email"]["sender"],
        to=config["email"]["to"],
        subject=config["email"]["subject"],
        message_text=config["email"]["text"],
        file=dashboard,
    )
    print("Authenticating with Google...")
    service = get_service()
    print("Sending email...")
    send_email(message=message, user_id="me", session=service)
    print("Done!")

if __name__ == "__main__":
    # Run the initial job to authenticate with Google.
    job()
    schedule.every().friday.do(job)

Check the installation instructions below if you'd like to test this project out! :)

Installation

Clone the Github repository and set up a virtual environment.

git clone https://github.com/AnthonyBloomer/dashboard-email-scheduler.git
cd dashboard-email-scheduler
virtualenv env
source env/bin/activate

Install the project requirements.

pip install -r requirements.txt

Export your Personal API Key as an environment variable.

export NEW_RELIC_PERSONAL_API_KEY = "YOUR_API_KEY"

If you have an EU based account, you will also need to export the NEW_RELIC_REGION environment variable:

export NEW_RELIC_REGION = "EU"

Edit config.py with the New Relic Dashboard and Email settings.

config = {
    "email": {
        "sender": "Email sender.",
        "to": "Email receiver.",
        "subject": "Email subject",
        "text": "Email message text",
    },
    "dashboard": {
        "guid": "The New Relic Dashboard GUID",
        "file_type": "PDF",
        "width": "2000",
        "height": "2000",
    },
}

Go to the Google Developer Console and create a new project. Once the API is enabled, download the credentials.json file to the root directory of this project.

Run python scheduler.py. This will run an initial job so you can authenticate the application. By default, the job will run every Friday. Refer to the Schedule project documentation to configure the scheduler to your liking.

If you've made it this far, thanks for reading! :D I hope you find this project useful.