DEV Community: Jesse Orshan

3 Examples of DevOps Remediation with Datadog and WayScript

Jesse Orshan — Wed, 16 Jun 2021 20:19:58 +0000

As a scaling software company, we experience periods of unpredictable platform usage spikes which can come from all over the world. This volatility fosters complexity around instance scaling and other core dev infrastructure. If our instances don’t scale properly, our users experience a less performant product. If our servers are too large, it is a waste of dollars.

Hooking into Datadog gave our company newfound and much-needed insights into our AWS infrastructure. Moreover, the alerting functionality built into Datadog’s event monitoring unlocked the ability for us to monitor, investigate, and ultimately resolve issues with our infrastructure. However, although Datadog alerted us to issues, we still had to manually manage our AWS infra. Our goal was to build a platform that could remediate infra challenges in an automated way by leveraging Datadog alerts.

...

Examples of Datadog Remediation with WayScript

1) AWS EC2 Instance Management —

I eluded to this use case in the intro paragraph. Mainly, with Datadog alerting and WayScript, users can set up alerting for high or low levels of usage of a particular instance. To do this, users quickly set up a Datadog trigger on WayScript.

Once you Datadog trigger is setup, you can run Shell Scripts, Python, JavaScript, Java, or SQL queries directly from WayScript. In this case, we use the boto3 python library to automate a variety of different EC2 remediation functions (full example tutorial and ReadMe).

For example, if we get an alert that our instance has high traffic load, we can automate running a script to add another EC2 instance to our system with this type of code:

import boto3

def turn_instance_on( instance_id ):
    ec2 = build_client()
    current_state = check_instance_state( instance_id )
    if current_state == 'not running':
        try:
            response = ec2.start_instances(InstanceIds=[instance_id], DryRun=False)
            new_state = response.get('StartInstances')[0].get('CurrentState').get('Name')
            return 'Success'
        except ClientError as response:
            return response
    else:
        return 'Instance Already Running'

The same type of logic can be run to turn instances off with low traffic.

Full Example Here.

...

2) Rollback to a previous deployment with CircleCI after an admin confirmation via text message.

When building a remediation tool, we found ourselves needing automated tasks mixed with ‘Human-in-the-loop’ interactions. Mainly, we wanted to design a program that would roll our production server back to the previous version based on a Datadog alert. Moreover, once the alert hit our backend we wanted to generate a text message approval by our lead backend development team. Once approved by a text message response, the rollback kicks off via CircleCI.

So how does this work? First, we set up an event alert on Datadog that is linked to our Rollbar incident reporting for deployments. If this incident is marked as a bad_deploy, our trigger fires. Next, a python script interprets the event and determines if a rollback is necessary:

event = variables['Event']
title = event.get('Title')
try:
    if 'bad_deploy' in title:
        status = 'bad'
        variables['status'] = status
    else:
        status = 'good'
except:
    variables['status'] = 'good'

If a rollback is necessary, we use the Twilio API to send a text message to our backend dev team. If/when a dev response with ‘approve’, CircleCI is set to rollback to the previous working version of our production system.

Full Example Here.

...

3) Terminate a deadlocked DB query and log the issues.

As we continue to scale, there have been instances where we have experienced unanticipated database issues such as deadlocks from long running queries. This type of event can cause significant degradation of performance for our user base. Therefore, we wanted to build a process for logging the deadlocked process, but then ultimately killing the query in an automated way (we determined this is better than user wide degradation).

In order to do this we set up Datadog alerts for High CPU usage (degraded status) or High memory usage on our database. When this alert hits WayScript, it kicks of a couple of processes.

Initially, we use Python & SQL to grab all currently running queries on our db (RDS on AWS). The first process builds a Pandas Dataframe of the running queries information, stores this in a file, and then emails the file to our backend dev team. The second process looks for queries that have exceeded an expected time threshold. For these queries, they are passed to a third process which kills them based on their RDS ID.

Example of Pulling the Running Queries:

import boto3
from botocore.exceptions import ClientError
def build_client():
    ec2 = boto3.client(
    'rds',
    region_name = 'us-east-2',
    aws_access_key_id=context['key_id'], #stored in .secrets
    aws_secret_access_key=context['key_secret'] # stored in .secrets
)
    return rds
rds = build_client()
response = rds.execute_statement(
    continueAfterTimeout=False,
    database='database-1',
    includeResultMetadata=False,
    resourceArn='aws:rds:us-east-<DB_ID>',
    schema='string',
    secretArn='string',
    sql='string',
    transactionId='string'
)

Full Example Here.

...

Our goal was to build a platform that could remediate infra challenges in an automated way by leveraging Datadog alerts.

WayScript let’s users trigger processes from Datadog monitors and events to automatically address infrastructure needs. WayScript is a virtual development environment that runs scripts in Shell, Python, or JavaScript and connects with external services like EC2, CircleCI, and SQL Server.

5 Things Low-Code Is Missing

Jesse Orshan — Thu, 28 Jan 2021 19:48:59 +0000

Currently, the ‘Low-Code’/’No-Code’ (LC/NC) space is a ‘hot’ market with a growing number of high-profile venture capital investments. Startups are receiving funding to forge various visions of ‘the future of software development’. These platforms have a spectrum of target users ranging from the non-technical ‘citizen developer’ to more technical developers. They also have varying use cases that range from interface/GUI front-end building to backend event processing with logic.

Low-Code has the power to materially improve the lives of developers looking to rapidly build production tooling. However, many ‘visual programming’ platforms are architected in ways that limit usability by developers.

A disclaimer, I am a co-founder @ WayScript — a development hub for internal tooling. We have been building a platform in the developer tooling space (which happens to have some low-code overlap which I’ll get to later). I am a backend software developer and come from a background in application development. As we continue to learn and build, we’ve studied myriad features/decisions in the spectrum of low-code products coming to market.

So here’s 5 missing components from current platforms that would be game-changing for developers.

1) A Cloud Based Dev Environment that feels like my Localhost Environment.

The Need:

As a software developer, my workflow is building software in a local dev environment which is subsequently stored in a cloud based git repository such as GitHub. When I am writing scripts, they are often built in a virtual environment which contains code, a complex file directory structure, and third-party dependencies. The trouble is that in order to turn these local command-line scripts into production tools, I need to do a lot more work for basic production (provisioning servers, deploying source code, configuration files, etc.). I also have to do much, much more work for robustness (CI/CD, unit testing, etc.). A lot of this infrastructure work ends up taking as much or more time than the core application. What I want is a cloud based development environment that feels like my local host environment, but seamlessly converts my script directories into stable production software.

The Focus of Current Platforms and What’s Missing:

At their core, many LC/NC platforms are ‘JSON parsing’ utilities. Mainly, the platforms enable users to pull JSON data (usually from third party apis), and parse particular key/value pairs to pass to another API or interface.

sample_json_data = {
 'name'  : 'Jesse Orshan',
 'email' : 'jesse[at]wayscript.com',
 ...
}

As an example, users are empowered to pass sample_json_data.email from an event trigger to another API without having to build the underlying API queries.

I want to mention that of course JSON parsing services are useful and can be used to build compelling automations / interfaces. However, core product architecture built around JSON Parsing limits the types of applications that can be enhanced by low-code.

A Solution:

A low-code platform architecture built as a virtual environment file-system which can read, write, and delete arbitrary file structures. When ready for production, any project can be executed through the core platform engine on a dedicated cloud service. JSON data from an event trigger can be passed into the script as an importable variable (e.g. runs when a new customer is added to Salesforce and passes the data). Moreover, this virtual file system can support various programming languages and dependencies out of the box (python, java, c#, …). The virtual environment is cloud based and serves as a uniform runtime environment across varying devices and operating systems. Even when working on my local desktop, this environment runs in the cloud but feels indistinguishable from my local machine. Finally, this virtual environment can interact with third party file structures such as GitHub, S3, Dropbox, etc. The system is seamless because a configuration file for production execution on a cloud server is automatically constructed by the platform (and editable if more customization is required).

2) Prioritizing Coding Ease, not Dragging and Dropping.

The Need:

A platform that offers the ease of event-triggering and software hosting/running for any code repository (servers, tasks, etc.).

The Focus of Current Platforms and What’s Missing:

When most people think low-code, they think visual programming (a.k.a. end-user development). Although visual programming is an example, in reality just about everything in software development is ‘low-code’ — the abstraction of lower-level code through a higher-level interface. For example, the Python programming language can be interpreted as a low-code abstraction of the C programming language. Secondly, a popular API such as Twilio is a low-code improvement over building a text-messaging interface from scratch. In other words, an API is just as much a low-code product as a visual programming interface. The important point here is just that there should be a distinction between low-code and visual programming.

This distinction fosters two questions. First, is visual programming the right paradigm for production grade scripting and tooling? Secondly, what are the most valuable features of these platforms?

For the first question, the answer is that visual programming can be a useful aid in software development, but it can also be a hindrance when it’s forced for its own sake. An example of a useful visual interface might be setting up a cron job to execute a script or interacting with a simple API. Often, the use cases that benefit from visual programming are components of software that require a lot of specialized code to deliver simple and common functionality. However, by forcing an entire platform to use visual programming, you end up stuck in places where it becomes a hindrance. Often, code is the simplest and fastest way to express what you want to do. For example, building boolean logic such as if/else statements with visual programming quickly become unruly to setup and manage. As a developer, I’d rather just write my logic in python.

So what are the most valuable features of visual programming platforms? First, the builtin event based triggering mechanisms are a great benefit (e.g. run whenever a new customer is added to my database or CRM). This is useful because setting up this type of trigger system requires another coding project (hosting a server for incoming webhooks) and distracts me from the core problem I am trying to solve. Secondly, the ability of these platforms to seamlessly run the constructed automations in the cloud without any dev ops process is beneficial.

A Solution:

A virtual environment platform with integrated event triggering from third party APIs (at the least a generalizable HTTP endpoint) that can push the request response data into scripts at execution of the application. Secondly, baked in APIs that handle oauth/credentialing, easy parameter inputs, and proper querying can save time. These APIs also benefit from visual interfaces (sending a custom email, slack message, etc.).

3) File Dependency Management.

The Need:

The service should be ‘smart’ about understanding dependencies and treat this as a first class problem to solve. For example, if an internal tool relies on a file in a particular directory location, a particular db schema, or another constant then at a minimum the platform should recognize breaking changes and alert stakeholders immediately with an explanation of what changed. The best experience would allow users to easily build unit-tests with minimal added work.

The Focus of Current Platforms and What’s Missing:

When working with teams building internal scripts and workflow automations, we often find that shared files are a potential pressure point / weakness in production stability. For example, we worked with a company that was running workflow automations on a Google Sheet which was simultaneously being edited by employees. If a sheet, column, or particular cell was accidentally edited, the automations would break without any indication (at least not immediately). Tracking down the root cause here when debugging is hard, time consuming, and doesn’t prevent this issue in the future. In another case, a company had a few internal tools that expected a database schema to be in a certain format and the tools all crashed when the db was upgraded. The reality is that both file dependencies and real world data is messy, and many platforms expect users to manage this mess.

A Solution:

The platform’s auto-generated configuration file for each program should include a mapping of all files and folders utilized within the program. Any references to these files should be stored in the metadata. In other words, the application ‘understands’ file dependencies and the nature of the dependency (directory location, internal data structures, typing, etc.). With this architecture, one can envision a low-code mapping that shows all file interdependencies within a system of scripts and tools that are running.

4) Versioning & Repo Integration.

The Need:

A platform paradigm that enables users to seamlessly run code, scripts, and servers in a cloud virtual environment without dev ops or setup (it feels like your local environment). Versioning and collaboration are much easier to apply to a file architecture than to a proprietary program saved in a third party services database. In other words, the user owns the source code instead of the service.

The Focus of Current Platforms and What’s Missing:

Many End-User Programming LC/NC platforms are architected so that the program content lives in the company database (Zapier stores your ‘zaps’, etc.). This paradigm is antithetical to the git/repository paradigm utilized by most developers when building tooling. The benefits of a git paradigm like GitHub are version control, shareability, portability, and collaboration (open source repos as an example). For building tooling, a paradigm that empowers developers to store and collaborate on a program by treating all elements as files in a directory is crucial. This returns to the notion that the low code program at its core should work as an editable file within a system where the purpose of the platform is to aid users in configuration (whether through visual programming or great documentation). This paradigm unlocks the ability for more sophisticated tooling and can leverage many of the tools software developers already use such as GitHub for source control. As an added benefit, this same architecture unlocks a more modular development experience where more fundamental pieces can be manipulated (in other words, users can edit / understand the lowest level code if they need to).

In my opinion, the platform GitBook does this well for building documentation. Mainly, the visual interface GitBook provides makes it super easy to build dynamic documentation and the underlying content is stored in each users GitHub account as a repository of markdown files (see our own docs here and the underlying file structure). By utilizing GitHub, GitBook is able to leverage the platform for conflict management and facilitates better low-level capabilities for devs.

A Solution:

A LC platform that aids/automatically constructs a configuration file in the source code to run arbitrary code on a cloud service (scripts, servers/endpoints, etc.). All the files related to the script are owned and stored by the user instead of the services database. All the service stores is references to where these files are stored (GitHub, S3, Dropbox, etc.)

5) Vendor Flexibility

The Need:

Forced vendor lock in creates a true barrier for developers. What happens if a company goes out of business or spikes their pricing? What happens if I suddenly need a feature or more resources that the service is unable to provide? In these instances, it is a problem if I am locked in.

The Focus of Current Platforms and What’s Missing:

When building software tools, many tend to become mission critical over time. As a developer, a huge value proposition of building my tooling from scratch is that I have complete autonomy. I can decide which frameworks to use, where to host, etc.

Today, most visual programming platforms inherently lock users into their system with their paradigm and treat this ‘stickiness’ as a business benefit. For example, using AirTable is highly convenient in many cases but also creates some uneasiness as a choice when building tooling. Mainly, I do not have the ability to access the underlying relational database if I needed/wanted to.

A Solution:

A platform that treats each component of the service it offers (such as event triggering, hosting, credentials, etc.) as a file inside a repository. With this paradigm, I can write all my code however I want and could always replace a configuration file with another service (swap a Heroku ProcFile for an AWS config file as an example). Yes, sometimes this work is tedious BUT I need to have the option. Secondly, this architecture aligns incentives for the service to offer best in class service of their components or those pieces will be swapped for a better service.

Conclusion

Low-code has the power to materially improve the lives of developers looking to rapidly build production tooling. However, ‘visual programming’ platforms which leverage low-code are often architected in ways that limit use by developers. A service that solves these problems would be a major value prop to all developers.

What do you think? Do you have other major problems with these platforms as a developer?

At WayScript, we are building a platform for developers looking to quickly build internal tooling. In a future post, I will write a follow-up as to how our platform is architected to address the issues raised in this article.

Automate SQL Queries in the cloud

Jesse Orshan — Thu, 30 Jan 2020 19:30:06 +0000

You can use the WayScript beta (free) to automate running SQL queries on a hourly, data, or weekly basis in the cloud. You can also output the results and send them via Slack, email, text message, etc.

How to Schedule a Python Script to Run Daily

Jesse Orshan — Mon, 27 Jan 2020 21:38:44 +0000

When I started programming, I remember feeling frustrated that I didn't have a simple way to take my python script and get it to automatically run everyday. With the WayScript beta (free tool), we created a super rapid way for you to get your script running in the cloud without any DevOps setup.

Sample Template - https://wayscript.com/shared/D9ToJA2M

Tutorial Video -

Build Personalized Discord Bots with Python and WayScript

Jesse Orshan — Wed, 02 Oct 2019 22:10:08 +0000

In this video we'll look at how we can build custom commands onto a WayScript discord bot using python programming, discord, and WayScript.com

Python Code:

import re

keywords = inputs['keywords']
result = inputs['result']
discord_message = inputs['Discord Message Text']

command = re.compile(r'\![a-z]*')
matches = command.findall(discord_message)
print(matches[0])
found_keyword = matches[0]
print(keywords)
position = keywords.index(found_keyword)
action = result[position]

outputs['action'] = action

Run a Python Script with a Google Assistant Voice Command

Jesse Orshan — Mon, 09 Sep 2019 21:36:21 +0000

The "WayScript Trigger" Google Assistant Action

In this tutorial, you will learn how to use WayScript to run your Python script in the cloud using a Google Assistant voice command. (Check out the Google Assistant Bot template to try it out!)

This is done using the WayScript Trigger action, which you will first need to enable on your Google account.

After enabling the action, you will then need to link it to your WayScript account using the Google Assistant app.

The Google Assistant Action

The next step is to build a WayScript program with a Google Assistant Trigger. To do so, first go to your WayScript dashboard and select "Create a new Script".

Give your program a name, like "Run Python with Google Assistant".

Once your program is created, you'll start by adding a Google Assistant Trigger.

Click the "+" icon next to "Add Trigger(s)" and select the Google Assistant Trigger.

You should now have the Google Assistant Trigger in the "Triggers" section of your program.

Set up the Google Assistant Trigger

Now that you have added the trigger to your program, let's get it set up! Start by clicking on the trigger to open it in the settings panel.

You should see something like this:

Set "Activate Trigger" to "On".

If you haven't yet linked your Google Assistant account, select "Add an Account..." from the "Select an Account" dropdown.

This will show a dialog with the option to "Enable Action and Link Account" on the Google Assistant site.

Now, notice that the trigger shows the text "Hey Google, ask WayScript Trigger to run 'Run Python with Google Assistant'."

That's kind of a mouthful, so you'll set the "Program Alias" field to create a phrase that's easier to say and easier for your Google Assistant to understand.

In this case, you'll simply set the Program Alias field to "Python".

Now you have a much nicer phrase!

Add a Python Script

With the Google Assistant Trigger in place, it's time to add a Python Script. Start by dragging the Python module into your program.

Then, click on the Python module and paste your script into the code box.

That's it! You're done!

Run your Python script from any of your Google Assistant-enabled devices by saying "Hey Google, ask WayScript Trigger to run 'Python'."

Of course, you don't have to stop at Python. You can run any WayScript program - including an evergrowing list of modules - using the Google Assistant Trigger.

Run any Python Script with an Alexa Voice Command

Jesse Orshan — Fri, 06 Sep 2019 17:51:20 +0000

https://wayscript.com/view_template/Amazon%20Alexa%20Bot

Standup a Web App on the Cloud in 1 Minute

Jesse Orshan — Thu, 29 Aug 2019 22:05:17 +0000

Using WayScript, you are able to turn your scripts and tools into live web applications rapidly. Anything you build is automatically running in the cloud. You can run javascript (and install libraries with npm), python (can pip install libraries), and more.

Building a Google Calendar Slack Bot - 5 minute tutorial

Jesse Orshan — Tue, 30 Jul 2019 17:46:58 +0000

I am one of the developers of WayScript - a visual programming language for developers

Web Scraping with Python Tutorial

Jesse Orshan — Wed, 24 Jul 2019 17:23:13 +0000

https://youtu.be/JU2GjEEj0TY

In this tutorial, I use BeautifulSoup and Requests in Python to pull the market cap of a stock (AAPL) from Yahoo Finance. Next, we set our program to run everyday and to send us an sms text message with the results.

Documentation Overhaul - Our Startups Sprint

Jesse Orshan — Fri, 19 Jul 2019 16:32:10 +0000

Our team (4 developers) decided it was time to overhaul our docs. Therefore, we all stopped what we were doing and spent the week sprinting on building much better docs for WayScript (drag and drop programming language).

Here are the results: https://docs.wayscript.com/

As a startup that is constantly shipping, changing, and altering our software.. we are striving to make sure we prioritize always having the most up to date docs we can.

Building a Slack SQL "inventory" Bot

Jesse Orshan — Fri, 21 Jun 2019 16:04:45 +0000

I have been recording tutorial videos on youtube. In this video, I build a Slack Bot that relays SQL database data back to users posting on a Slack channel using WayScript.

https://youtu.be/VIcTgs5Uw4Q