DEV Community: dillan teagle

stack overflow is down

dillan teagle — Wed, 08 Jul 2020 15:41:26 +0000

Really? disk space?

New gatsby github readme component

dillan teagle — Wed, 03 Jun 2020 13:36:58 +0000

This is a new component for gatsby that renders a preview of a github readme that is very similiar to dev.to. There are a few additions like the language bar but ultimately i followed the same design princibles.

So this is not an ordinary remark or source plugin. Its simply a react component package. The github data is fetched on the gatsby build. So all you have to worry about is how you want to render your readme, in jsx, remark, or mdx?

This npm package is new and still undergoing testing to please feel obligated to leave issues as questions, problems, or suggestions.

github is down

dillan teagle — Tue, 21 Apr 2020 16:05:09 +0000

I said it...its true. for how long?

New Gatsby theme powered by Jupyter!

dillan teagle — Sat, 08 Feb 2020 15:32:05 +0000

Introducing Gatsby Theme Binder

teaglebuilt / gatsby-theme-binder

🌔 Gatsby theme for interactive remark code blocks powered by jupyter kernels for code execution.

Gatsby Theme Binder

Power your gatsby site with kernels from jupyter Powered by mybinder.org

This project was initially inspired by Min RK's Thebelab package. Then finding Juniper. The goal is take this functionality wrapped into a gatsby package that allows for customization and accesibility.

How it works

The theme will connect to a launched docker image of your chosen github repository as a jupyter notebook. An active connection will persist to the given kernel that is active in your notebook. You can choose which kernel to connect to in your gatsby config. You can use this theme just for a source connection with server sent events, or you can also import the interactive text editor and execute / render results from the kernel itself as a remark component. There are further instructions below.

Note, this is my first gatsby theme and I am not sure if this is…

View on GitHub

Many of us know the great benefits of using gatsby. In 2020, writing your content from markdown is very popular and has many different benefits in terms of SEO, delivery, and performance. None the less, able to integrate react components into your static content written in markdown. We all know this.

Well, we also have seen examples of Gatsby used in the context if SSR. It is a little tricky in Gatsby without the capabilities of using Suspense and Lazy loading. Although, with server sent events, we can empower a gatsby static website with an active kernel from Jupyter Notebooks.

What is Jupyter

Jupyter Notebooks are commonly used for data science and bring the capability to build notebooks of code with active execution for discovery purposes.

What is Binder

MyBinder is a website that launches a docker image of your repository with the dependencies as a jupyter notebook. For example, I can launch a repository with python and its project dependencies, and open a jupyter notebook with a python2 or 3 kernel and the project dependencies installed from requirements.txt...

Where are you going with all of this?

In Gatsby Theme Binder we can connect to mybinder from server sent events and power our gatsby website with the given kernel your using. Therefore, you can render a remark "markdown" react component that is a text editor, capable of executing code and rendering results just like a shell and cell from jupyter notebooks.

How to use it

A base folder is created with two child folders...


/whatever_you_name_it
    /code
    /posts

/code will hold the code files that you want to render inside your text editor on page load.

Instructions are noted in the readme of the options needed in your gatsby config. One important note is that you will need to import the renderAST component into your page template or layout component and pass in the htmlAST from graphql. This will enable rendering the component in markdown.

To use the component, simply add it to your markdown post with the name of the file and the language


<codeblock filename="test" lang="python"></codeblock>

Reference of Technologies used

You may of heard of jupyterlab. It is a python package that has interface that far exceeds original notebooks with extensible features for customization. Jupyterlab can be extended using NPM packages to their public api.

Below are two NPM packages that I am using to integrate jupyter notebooks into my gatsby site.

jupyterlab / jupyterlab

JupyterLab computational environment.

JupyterLab

An extensible environment for interactive and reproducible computing, based on the Jupyter Notebook and Architecture. Currently ready for users.

JupyterLab is the next-generation user interface for Project Jupyter offering all the familiar building blocks of the classic Jupyter Notebook (notebook terminal, text editor, file browser, rich outputs, etc.) in a flexible and powerful user interface JupyterLab will eventually replace the classic Jupyter Notebook.

JupyterLab can be extended using npm packages that use our public APIs. The prebuilt extensions can be distributed via PyPI conda, and other package managers. The source extensions can be installed directly from npm (search for jupyterlab-extension) but require additional build step. You can also find JupyterLab extensions exploring GitHub topic jupyterlab-extension. To learn more about extensions, see the user documentation.

The current JupyterLab releases are suitable for general…

View on GitHub

JupyterLab - OutputArea

An interface for interacting with Jupyter, a large project split into packages relative to particular use cases and given functionality.

Jupyterlab - RenderMime

An interface for interacting with Jupyter, a large project split into packages relative to particular use cases and given functionality

So these two packages above is how I retrieve a given running kernel and interact with it directly through Javascripts EventSource() class.

The next challenge is how to render the results generated from the kernel. Originally I have used PrismJS and syntax highlighting to display code in my blog. For the purpose of creating interactive code blocks, I dediced to use CodeMirror.

CodeMirror

An interface for interacting with Jupyter, a large project split into packages relative to particular use cases and given functionality.

codemirror / CodeMirror

In-browser code editor

CodeMirror

CodeMirror is a versatile text editor implemented in JavaScript for the browser. It is specialized for editing code, and comes with over 100 language modes and various addons that implement more advanced editing functionality. Every language comes with fully-featured code and syntax highlighting to help with reading and editing complex code.

A rich programming API and a CSS theming system are available for customizing CodeMirror to fit your application, and extending it with new functionality.

You can find more information (and the manual) on the project page. For questions and discussion, use the discussion forum.

See CONTRIBUTING.md for contributing guidelines.

The CodeMirror community aims to be welcoming to everybody. We use the Contributor Covenant (1.1) as our code of conduct.

Installation

Either get the zip file with the latest version, or make sure you have Node installed and run:

npm install codemirror

NOTE: This is…

View on GitHub

Inspiration

This project was initially inspired by Min RK's
Thebelab package. Then finding Juniper. The goal is take this functionality wrapped into a gatsby package that allows for customization and accesibility.

with & without lambda in python

dillan teagle — Fri, 29 Nov 2019 23:02:39 +0000

A quick understanding of using functional methods like map, filter, & sorted with and without lambda functions. A great example of how lambda functions work in python.

use the filter function to remove items from a list

without lambda


def filterFunc(x):
    if x % 2 == 0:
        return False
    return True


nums = (1, 8, 4, 5, 12, 26, 381, 58, 47)
odds = list(filter(filterFunc, nums))

[1, 5, 381, 47]

with lambda

mylist = [1, 2, 3, 4, 5]

odds = list(filter((lambda x: x % 2 != 0), mylist))

[1, 5, 381, 47]

remove uppercase from string

without lambda

def filterFunc2(x):
    if x == x.upper():
        return False
    return True

chars  = "aBcDeFghoiJk"

lower = list(filter(filterFunc2, chars))

['a', 'c', 'e', 'g', 'h', 'o', 'i', 'k']

with lambda

lower = list(filter((lambda x: x == x.upper()), chars))

['a', 'c', 'e', 'g', 'h', 'o', 'i', 'k']

return new list with index squared

without lambda

def squareFunc(x):
    return x ** 2

ints = [1, 2, 3, 4, 5]
squared = list(map(squareFunc, ints))

[1, 4, 9, 16, 25]

with lambda

squared = list(map(lambda x: x ** 2, ints))

[1, 4, 9, 16, 25]

use sorted and map to change numbers to grades


def toGrade(x):
    if x  >= 90:
        return "A"
    elif x >= 80:
        return "B"
    elif x >= 70:
        return "C"
    elif x >= 60:
        return "D"
    else:
        return "F"

grades = (81, 89, 94, 78, 61, 99, 74, 90)

order = sorted(grades)
final = list(map(toGrade, order))

['D', 'C', 'C', 'B', 'B', 'A', 'A', 'A']

Challenge:

Is using a lambda function a good choice here? If so, how will you do it?

Continuous delivery test automation pipeline - Part One

dillan teagle — Sun, 17 Nov 2019 19:27:16 +0000

Hey my name is Dillan, I decided to start documenting some of my work. You can find this post at my website. This particular post is the first of three seperate articles.

Website

Source Code

Part One

Lets list out exactly what were going to cover

[] Build test infrastructure with docker and SeleniumGrid
[] Build python test suite to run against each container
[] Configure CentOs server and register gitlab-runner
[] On push run gitlab ci to build up containers and execute tests,
[] Build up infrastructure and run tests on Jenkins triggered from gitlab

Why are we using SeleniumGrid & Docker?

Each container represents a separate browser and operating system. The idea is to have all automated tests centralized in one container. This container will run and execute tests directly against the Hub. The hub is the control center to all the other nodes. When the hub container is triggered, it will run the given tests across each node/container.

This technology can help us consolidate time, compare behavior across multiple browsers & operating systems, and provide a higher level of code coverage for frontend changes. So now how are we going to use it?

Dockerfile

Used to build image for our test container

FROM python:3.6

ADD . /uitests

WORKDIR /uitests

RUN pip install -r requirements.txt

RUN chmod +x wait-for-it.sh

docker-compose.yml

Used to pull down image for each browser and build containers for hub & nodes

We define a network in which we also include the image of our tests from the Dockerfile

version: '2'
services:

  selenium_hub:
    container_name: selenium_hub
    image: selenium/hub
    networks:
      cao_ui_prodchecks: {}
    ports:
      - "4444"

  firefoxnode:
    container_name: firefox_node
    image: selenium/node-firefox
    networks:
      cao_ui_prodchecks: {}
    environment:
      - HUB_PORT_4444_TCP_ADDR=selenium_hub
      - HUB_PORT_4444_TCP_PORT=4444
    depends_on:
      - selenium_hub
    ports:
      - 5554

  chromenode:
    container_name: chrome_node
    image: selenium/node-chrome
    networks:
      cao_ui_prodchecks: {}
    environment:
      - HUB_PORT_4444_TCP_ADDR=selenium_hub
      - HUB_PORT_4444_TCP_PORT=4444
    depends_on:
      - selenium_hub
    ports:
      - 5555

  robottests:
    container_name: robottests
    command: /bin/sleep infinity
    networks:
      cao_ui_prodchecks: {}
    depends_on:
      - selenium_hub
    build: .
    volumes:
      - ./reports:/cao_ui_tests/reports

networks:
  cao_ui_prodchecks:
    driver: bridge

The Dockerfile will copy over all of the folders of the Repository. Click Repository to view the project configuration. I decided to use RobotFramework for a quick example.

Commands

# Build robot tests
docker build -t robottests:1 .

# Launch services 
docker-compose up -d

# execute tests
docker-compose exec robottests scripts/wait_for_it.sh -t 15 selenium_hub:4444 -- robot -A run_tests.robot

You should see all services up and running!

Creating network "cao_ui_prodchecks_cao_ui_prodchecks" with driver "bridge"
Creating selenium_hub ... done
Creating robottests ... done
Creating firefox_node ... done
Creating chrome_node ... done

So let's take a look at the services we have up and running.

The Selenium Hub - controls delivery of test execution to each node
Firefox Node - test suite executes against firefox browser
Chrome Node - test suite executes against chrome browser
Test Container - dockerized automated test suites

Let's execute the test container. Note the command is listed in the dropdown list above ^.

docker exec robottests scripts/wait_for_it.sh -t 15 selenium_hub:4444 -- robot -A run_tests.robot
wait_for_it.sh: waiting 15 seconds for selenium_hub:4444
wait_for_it.sh: selenium_hub:4444 is available after 0 seconds
==============================================================================
UI Prod Tests                                                                 
==============================================================================
UI Prod Tests.Login Page                                                      
==============================================================================
Authentication | PASS |
------------------------------------------------------------------------------
UI Prod Tests.Login Page | PASS |
1 critical test, 1 passed, 0 failed
1 test total, 1 passed, 0 failed
==============================================================================
UI Prod Tests | PASS |
1 critical test, 1 passed, 0 failed
1 test total, 1 passed, 0 failed
==============================================================================
Debug: /cao_ui_tests/reports/debug.logenv
Output: /cao_ui_tests/reports/output.xml
Log: /cao_ui_tests/reports/log.html
Report: /cao_ui_tests/reports/report.html

If you look close you will notice something that I have not mentioned. Before the test suite begins, a shell script executes, with the output

wait_for_it.sh: waiting 15 seconds for selenium_hub:4444
wait_for_it.sh: selenium_hub:4444 is available after 0 seconds

This is a shell script that is vital to execution and is located in the scripts directory of the Repository.

Before executing the Selenium tests, it's necessary to make sure that the test runner can create a Selenium Grid Session with a given webdriver. It means that Selenium Grid or Selenium Chrome Node should be up and ready.

wait-for-it.sh script provides you with an opportunity to wait for the service to be up but not ready

So lets go through the checklist at the beginning of this post and see where we are.

[X] ~~Build test infrastructure with docker and SeleniumGrid~~
[X] ~~Build python test suite to run against each container~~
[] Configure CentOs server and register gitlab-runner
[] On push run gitlab ci to build up containers and execute tests,
[] Build up infrastructure and run tests on Jenkins triggered from gitlab

In the next post, we will discuss configuring a centOs server to run our CI tests that will build up our infrastructure and run our test suites and code changes in our automated tests. Jenkins will pull from the repository so it is important to make sure we are not adding breaking changes into our codebase.

A look at itertools chain method

dillan teagle — Sat, 26 Oct 2019 20:06:03 +0000

The python itertools module is a collection of tools for handling iterators. I want to make an honest effort to break the habit of repetitive standard patterns of basic python functionality. For example...

We have a list of tuples.

a = [(1, 2, 3), (4, 5, 6)]

In order to iterate over the list and the tuples, we have a nested for loop.

for x in a:
  for y in x:
    print(y)

 # output 

1
2
3
4
5
6

I have decided to challenge myself to start using all that itertools has to offer in my every day solutions. Today lets take a look at the chain method!

This first approach will only return the two tuples as as index 0 and 1. Instead of nesting another loop, lets apply the chain method.

from itertools import chain

a = [(1, 2, 3), (4, 5, 6)]

for _ in chain(a, b):
  print(_)

 # output 
 (1, 2, 3)
 (4, 5, 6)

now we have access to iterate over the tuples without nesting another iteration with a for loop.

for _ in chain.from_iterable(a):
  print(_)

# output

1
2
3
4
5
6

I think the chain method gives the flexibility to use this as a default choice for list iteration.

Let Python worry about tweeting for Wordpress

dillan teagle — Fri, 18 Oct 2019 18:12:43 +0000

Lets use Python to automatically tweet recent posts from wordpress

For discovery purposes, I chose to build a blog with headless wordpress, React, GraphQL/Apollo. I would like to find a way to automate these posts across all platforms as much as possible.

I think we could come up with a solution to get posts from the wordpress API and and send a simple tweet about the new article. After thinking about this, I have realized that we will need to store the ID of the post that is selected to prevent retweets of the same post.

I will walk you through the first messy version of automating this task and maybe we can clean it up a bit along the way.

First things first, we have Twython as a dependency to interact with twitter's API.

Now lets create a class that interact with Wordpress. For now, we will grab all posts and store in a json file.

Wordpress.py

class WPGateway:

    def __init__(self):
        self.url = os.environ.get("WP_POSTS")

    def get_posts(self):
        response = requests.get(self.url, verify=False)
        wp_content = response.json()
        return wp_content

    def get_image(self, img_data):
        url = [x["href"] for x in img_data] 
        r = requests.get(url[0])
        if r.ok:
            data = json.loads(r.content.decode('UTF-8'))
            image_url = data['guid']['rendered']
            return image_url

Tweeter.py

import json
import csv
import logging
import requests
import urllib3
urllib3.disable_warnings()

logger = logging.getLogger(__name__)

wp = WPGateway()



if __name__ == "__main__":
    logging.basicConfig(
        format="%(levelname)s: %(message)s",
        level=logging.INFO, stream=sys.stdout
    )
    with open("posts.json", "w") as write_file:
        data = wp.get_posts()
        json.dump(data, write_file, indent=4)

So this is our main script which calls get_posts() which will populate a json file with the given response from the rest api.

The schema of a wordpress response is consistent by default unless there have been custom modifications to the api with custom post types, taxonomies or so on. So parsing this Json should be similiar to your experience if not the same.

[
    {
        "id": 15,
        "date": "2019-10-13T14:48:17",
        "date_gmt": "2019-10-13T14:48:17",
        "guid": {
            "rendered": "https://{domain}.com/?p=15"
        },
        "modified": "2019-10-13T17:13:35",
        "modified_gmt": "2019-10-13T17:13:35",
        "slug": "{title}",
        "status": "publish",
        "type": "post",
        "link": "https://{domain}.com/{title}/",
        "title": {
            "rendered": "{title}"
        },
        "content": {
            "rendered": "\n<p>Main content</p>,
            "protected": false
        },
        "excerpt": {
            "rendered": "<p>Excerpt Content.</p>\n",
            "protected": false
        },
        "author": 2,
        "featured_media": 16,
        "comment_status": "open",
        "ping_status": "open",
        "sticky": false,
        "template": "",
        "format": "standard",
        "meta": [],
        "categories": [
            2
        ],
        "tags": [
            16
        ],
        "acf": [],
        "_links": {
            "self": [
                {
                    "href": "https://{domain}.com/wp-json/wp/v2/posts/{id}"
                }
            ],
            "collection": [
                {
                    "href": "https://{domain}.com/wp-json/wp/v2/posts"
                }
            ],
            "about": [
                {
                    "href": "https://{domain}.com/wp-json/wp/v2/types/post"
                }
            ],
            "author": [
                {
                    "embeddable": true,
                    "href": "https://{domain}.com/wp-json/wp/v2/users/{amount}"
                }
            ],
            "replies": [
                {
                    "embeddable": true,
                    "href": "https://{domain}.com/wp-json/wp/v2/comments?post={id}"
                }
            ],
            "version-history": [
                {
                    "count": 38,
                    "href": "https://{domain}.com/wp-json/wp/v2/posts/{id}/revisions"
                }
            ],
            "predecessor-version": [
                {
                    "id": 60,
                    "href": "https://{domain}.com/wp-json/wp/v2/posts/{id}/revisions/60"
                }
            ],
            "wp:featuredmedia": [
                {
                    "embeddable": true,
                    "href": "https://{domain}.com/wp-json/wp/v2/media/16"
                }
            ],
            "wp:attachment": [
                {
                    "href": "https://{domain}.com/wp-json/wp/v2/media?parent={id}"
                }
            ],
            "wp:term": [
                {
                    "taxonomy": "category",
                    "embeddable": true,
                    "href": "https://{domain}.com/wp-json/wp/v2/categories?post={id}"
                },
                {
                    "taxonomy": "post_tag",
                    "embeddable": true,
                    "href": "https://{domain}.com/wp-json/wp/v2/tags?post={id}"
                }
            ],
            "curies": [
                {
                    "name": "wp",
                    "href": "https://api.w.org/{rel}",
                    "templated": true
                }
            ]
        },
        "_embedded": {
            "author": [
                {
                    "id": 2,
                    "name": "Your name",
                    "url": "",
                    "description": "",
                    "link": "https://{domain}.com/author/admin/",
                    "slug": "admin",
                    "avatar_urls": {
                        "24": "https://secure.gravatar.com/avatar/57323407ed9d034148c1489a370970cd?s=24&d=mm&r=g",
                        "48": "https://secure.gravatar.com/avatar/57323407ed9d034148c1489a370970cd?s=48&d=mm&r=g",
                        "96": "https://secure.gravatar.com/avatar/57323407ed9d034148c1489a370970cd?s=96&d=mm&r=g"
                    },
                    "acf": [],
                    "_links": {
                        "self": [
                            {
                                "href": "https://{domain}.com/wp-json/wp/v2/users/2"
                            }
                        ],
                        "collection": [
                            {
                                "href": "https://{domain}.com/wp-json/wp/v2/users"
                            }
                        ]
                    }
                }
            ],
            "wp:featuredmedia": [
                {
                    "id": 16,
                    "date": "2019-10-13T14:46:52",
                    "slug": "slug",
                    "type": "attachment",
                    "link": "https://{domain}.com/title",
                    "title": {
                        "rendered": "title"
                    },
                    "author": 2,
                    "acf": [],
                    "caption": {
                        "rendered": "<p>title</p>\n"
                    },
                    "alt_text": "slack with django",
                    "media_type": "image",
                    "mime_type": "image/png",
                    "media_details": {
                        "width": 512,
                        "height": 256,
                        "file": "2019/10/postimage.png",
                        "sizes": {
                            "thumbnail": {
                                "file": "postimage-150x150.png",
                                "width": 150,
                                "height": 150,
                                "mime_type": "image/png",
                                "source_url": "https://{domain}.com/wp-content/uploads/2019/10/slackdjango-150x150.png"
                            },
                            "medium": {
                                "file": "postimage-300x150.png",
                                "width": 300,
                                "height": 150,
                                "mime_type": "image/png",
                                "source_url": "https://{domain}.com/wp-content/uploads/2019/10/slackdjango-300x150.png"
                            },
                            "full": {
                                "file": "postimage.png",
                                "width": 512,
                                "height": 256,
                                "mime_type": "image/png",
                                "source_url": "https://teaglebuilt.com/wp-content/uploads/2019/10/slackdjango.png"
                            }
                        },
                        "image_meta": {
                            "aperture": "0",
                            "credit": "",
                            "camera": "",
                            "caption": "",
                            "created_timestamp": "0",
                            "copyright": "",
                            "focal_length": "0",
                            "iso": "0",
                            "shutter_speed": "0",
                            "title": "",
                            "orientation": "0",
                            "keywords": []
                        }
                    },
                    "source_url": "https://{domain}.com/wp-content/uploads/2019/10/slackdjango.png",
                    "_links": {
                        "self": [
                            {
                                "href": "https://{domain}.com/wp-json/wp/v2/media/16"
                            }
                        ],
                        "collection": [
                            {
                                "href": "https://{domain}.com/wp-json/wp/v2/media"
                            }
                        ],
                        "about": [
                            {
                                "href": "https://{domain}.com/wp-json/wp/v2/types/attachment"
                            }
                        ],
                        "author": [
                            {
                                "embeddable": true,
                                "href": "https://{domain}.com/wp-json/wp/v2/users/2"
                            }
                        ],
                        "replies": [
                            {
                                "embeddable": true,
                                "href": "https://{domain}.com/wp-json/wp/v2/comments?post=16"
                            }
                        ]
                    }
                }
            ],
            "wp:term": [
                [
                    {
                        "id": 2,
                        "link": "https://{domain}.com/category/back_end/",
                        "name": "Back End",
                        "slug": "back_end",
                        "taxonomy": "category",
                        "acf": [],
                        "_links": {
                            "self": [
                                {
                                    "href": "https://teaglebuilt.com/wp-json/wp/v2/categories/2"
                                }
                            ],
                            "collection": [
                                {
                                    "href": "https://{domain}.com/wp-json/wp/v2/categories"
                                }
                            ],
                            "about": [
                                {
                                    "href": "https://{domain}t.com/wp-json/wp/v2/taxonomies/category"
                                }
                            ],
                            "wp:post_type": [
                                {
                                    "href": "https://{domain}.com/wp-json/wp/v2/posts?categories=2"
                                },
                                {
                                    "href": "https://{domain}t.com/wp-json/wp/v2/projects?categories=2"
                                }
                            ],
                            "curies": [
                                {
                                    "name": "wp",
                                    "href": "https://api.w.org/{rel}",
                                    "templated": true
                                }
                            ]
                        }
                    }
                ],
                [
                    {
                        "id": 16,
                        "link": "https://{domain}.com/tag/python/",
                        "name": "Python",
                        "slug": "python",
                        "taxonomy": "post_tag",
                        "_links": {
                            "self": [
                                {
                                    "href": "https://{domain}.com/wp-json/wp/v2/tags/16"
                                }
                            ],
                            "collection": [
                                {
                                    "href": "https://{domain}.com/wp-json/wp/v2/tags"
                                }
                            ],
                            "about": [
                                {
                                    "href": "https://{domain}.com/wp-json/wp/v2/taxonomies/post_tag"
                                }
                            ],
                            "wp:post_type": [
                                {
                                    "href": "https://{domain}.com/wp-json/wp/v2/posts?tags=16"
                                },
                                {
                                    "href": "https://{domain}.com/wp-json/wp/v2/projects?tags=16"
                                }
                            ],
                            "curies": [
                                {
                                    "name": "wp",
                                    "href": "https://api.w.org/{rel}",
                                    "templated": true
                                }
                            ]
                        }
                    }
                ]
            ]
        }
    }
]

So this file will always hold our collection of data from wordpress, great!. We need to grab the "ID" of every post and IF it has not yet been TWEETED, then we will do so.

One quick and simple solution is to add the id of the post to a csv file that has been selected to be tweeted. This way we can keep a history collection of posts by the id.

id_records.txt


1,2,3

So at this point we opened the json file with the context manager and stored our response schema. So lets grab the Id's of already posted content in the txt file.

def record_check():
    with open("social/id_record.txt", "r") as id_file:
        reader = csv.reader(id_file)
        for row in reader:
            print(row)
            return row

['1', '2', '3']

We will filter the posts by the ids and return an eligible list of new posts for possible tweediness.

def eligible_posts(records):
    post = {}
    eligible = []
    read_json = open("social/wp_posts.json", "r")
    data = json.load(read_json)
    for index in range(len(data)):
        if str(data[index]["id"]) not in records:
            wp_img = wp.get_image(data[index]["_links"]["wp:featuredmedia"])
            post["id"] = data[index]["id"]
            post['title'] = data[index]['title']['rendered']
            post['image'] = wp_img
            eligible.append(post)
    return eligible

Here we parse the json filter and format the posts for publishing. This function will return a list of objects where the posts have not been sent to twitter. Inside this object we call a function called get_image. This is important because you have to download the image to send to twitter. I was suprised you could not just send the source url. This method has been added to the wordpress class.

Here is the complete class as of now.

wordpress.py

class WPGateway:

    def __init__(self):
        self.url = os.environ.get("WP_POSTS")

    def get_posts(self):
        response = requests.get(self.url, verify=False)
        wp_content = response.json()
        return wp_content

    def get_image(self, img_data):
        url = [x["href"] for x in img_data] 
        r = requests.get(url[0])
        if r.ok:
            data = json.loads(r.content.decode('UTF-8'))
            image_url = data['guid']['rendered']
            return image_url

So now we have the image uri. At this point i thought I could just send the image_url with the post object like so...

[{'id': 15, 'title': 'Post title', 'image': 'https://{domain}.com/wp-content/uploads/2019/10/image.png'}]]
INFO: Uploading image
['6', '7', '8', 12] records

This is the list of eligible posts. This is tedious, but we have to download the image to upload to twitter, then delete the image from your path.


def upload_image(url):
    r = requests.get(url, stream=True)
    with open('social/img.png', 'wb') as out_file:
        shutil.copyfileobj(r.raw, out_file)
        photo = open('social/img.png', 'rb')
        media = twitter.upload_media(media=photo)
        logger.info("Uploading image")
        return media

Now I will stop right here and setup your credentials for authenticating to twitter.

twitter = Twython(os.getenv("CONSUMER_KEY"), 
            os.getenv("CONSUMER_SECRET"),
            os.getenv("ACCESS_TOKEN"), 
            os.getenv("ACCESS_TOKEN_SECRET")
        )

Im going to go ahead and post the complete tweeter.py script and well continue to break it down.

from twython import Twython, TwythonError
from wordpress import WPGateway
import json
import csv
import shutil
import os, sys
import logging
import requests
import urllib3
urllib3.disable_warnings()

logger = logging.getLogger(__name__)

wp = WPGateway()

twitter = Twython(os.getenv("CONSUMER_KEY"),
            os.getenv("CONSUMER_SECRET"),
            os.getenv("ACCESS_TOKEN"),
            os.getenv("ACCESS_TOKEN_SECRET")
        )

def record_check():
    with open("id_record.txt", "r") as id_file:
        reader = csv.reader(id_file)
        for row in reader:
            print(row)
            return row

def update_records(records, post):
    records.append(post["id"])
    print(records, "records")
    with open("id_record.txt", "w", newline="") as id_file:
        id_writer = csv.writer(id_file)
        id_writer.writerow(records)

def eligible_posts(records):
    post = {}
    eligible = []
    read_json = open("wp_posts.json", "r")
    data = json.load(read_json)
    for index in range(len(data)):
        if str(data[index]["id"]) not in records:
            wp_img = wp.get_image(data[index]["_links"]["wp:featuredmedia"])
            post["id"] = data[index]["id"]
            post['title'] = data[index]['title']['rendered']
            post['image'] = wp_img
            eligible.append(post)
    return eligible


def upload_image(url):
    r = requests.get(url, stream=True)
    with open('img.png', 'wb') as out_file:
        shutil.copyfileobj(r.raw, out_file)
        photo = open('img.png', 'rb')
        media = twitter.upload_media(media=photo)
        logger.info("Uploading image")
        return media


if __name__ == "__main__":
    logging.basicConfig(
        format="%(levelname)s: %(message)s",
        level=logging.INFO, stream=sys.stdout
    )
    with open("wp_posts.json", "w") as write_file:
        data = wp.get_posts()
        json.dump(data, write_file, indent=4)

    records = record_check()
    new_posts = eligible_posts(records)
    print(new_posts)
    if len(new_posts) >= 1:
        uploaded_media = upload_image(new_posts[0]["image"])
        twitter.update_status(
                status=f"Visit https://www.{domain}.com, to see recent posts like {new_posts[0]['title']}.",
                media_ids=[uploaded_media['media_id']]
            )
        update_records(records, new_posts[0])
        os.remove('img.png')
    else:
        logger.info("There are no new posts")

You can see the additional handlers for updating records -> the txt file.
Finally remove the image from your path. I wonder if you could use Python's tempdir and still capture the image for upload?

There is plenty of room to simplify this code. Even the idea of recording ID's to a txt file is sloppy. In the next post, lets consolidate this code and add a github action to trigger this script periodically.

Test Automation in the Cloud with Python. Contributors Wanted

dillan teagle — Mon, 30 Sep 2019 14:19:40 +0000

Interested in all the services that AWS has to offer. Do you love python? Do you enjoy working with boto3 API to interact with AWS services?

Imagine if you need to write automated tests that interacted with AWS Services like s3, dynamoDB, RDS, and more.

Robotframework is close to the standard for test automation in 2019. Its all python and built by open source with the ability to write python to add custom libraries.

Here is a long list of all of the libraries.
https://github.com/VVilde/RobotFramework-Materials'

Notice the cloud is missing?

If your interested and have experience with Robotframework, boto3, and python. Check it out!

This library already has features to build automated tests to ensure file uploads, downloads, and others with s3. This is only the beginning.

https://github.com/teaglebuilt/robotframework-AWS

Wordpress with React and Apollo

dillan teagle — Sun, 29 Sep 2019 12:47:56 +0000

Ive decided to tryout React with GraphQL, Apollo, and Wordpress.

Repo: https://github.com/teaglebuilt/teaglebuilt.com

index.js


import React from 'react';
import ReactDOM from 'react-dom';
import './index.css';
import App from './App';
import { ApolloClient } from 'apollo-boost';
import { createHttpLink } from 'apollo-link-http';
import { InMemoryCache } from 'apollo-cache-inmemory';
import { BrowserRouter } from 'react-router-dom';
import { ApolloProvider } from 'react-apollo';
import Config from './config';


const client = new ApolloClient({
    link: createHttpLink({
      uri: Config.gqlUrl,
    }),
    cache: new InMemoryCache(),
  });


ReactDOM.render(
    <BrowserRouter>
        <ApolloProvider client={client}>
            <App />
        </ApolloProvider>
    </BrowserRouter>,
    document.getElementById('root')
);

App.js

import React from 'react';
import { Switch, Route } from 'react-router-dom';
import Home from './components/Home';
import Header from './components/Header';
import Sidebar from './components/Sidebar';
import Page from './components/Page';
import Post from './components/Post';
import Category from './components/Category';
import Tagbar from './components/Tagbar';
import Tag from './components/Tag';
import './styles/index.css';


export default () => (
  <div className="">
    <Header />
    <div className="container w-full flex flex-wrap mx-auto px-2 pt-8 lg:pt-16 mt-16">
    <Sidebar />
      <Switch>
        <Route exact path="/" component={Home} />
        <Route exact path="/page/:slug" component={Page} />
        <Route exact path="/post/:slug" component={Post} />
        <Route exact path="/category/:slug" component={Category} />
        <Route exact path="/tag/:id" component={Tag} />
      </Switch>
    <Tagbar />
    </div>
  </div>
);

Tagbar.js

import React, { Component } from 'react';
import { withApollo } from 'react-apollo';
import { Link } from 'react-router-dom';
import gql from 'graphql-tag';


const TAGS_QUERY = gql`
    query {
        tags {
        edges {
            node {
            id
            link
            name
            slug
            posts {
                edges {
                node {
                    id
                }
                }
            }
            }
        }
        }
    }
`;


class Tagbar extends Component {
    constructor(props){
        super(props);
        this.state = {
            tags: []
        }
    }

    componentDidMount(){
        this.executeTagsQuery();
    }

    executeTagsQuery = async () => {
        const { client } = this.props;
        const result = await client.query({
            query: TAGS_QUERY
        })
        let tags = result.data.tags.edges;
        this.setState({
            tags: tags
        })
    }

    render(){
        const { tags } = this.state;
        return(
            <div className="w-full lg:w-1/5 lg:px-6 text-xl text-gray-800 leading-normal">
            <h1>TagBar</h1>
            <ul>
            {tags.map(tag => (
                <li key={tag.node.slug}>
                    <Link to={`/tag/${tag.node.id}`}>
                        {tag.node.name}
                    </Link>
                </li>
               ))}
            </ul>
        </div>
        )
    }
}

export default withApollo(Tagbar);

So you see here in the graphql query that i could pass the posts for each tag in the map function to the tag component but im not sure if i should do that...meaning render the tag component inside of tagbar.js...

tagbar is a sidebar on the right side of the ui with the list of tags.

tag.js

import React, { Component } from 'react';
import { withApollo } from 'react-apollo';
import { Link } from 'react-router-dom';
import gql from 'graphql-tag';


const TAG_QUERY = gql`
query TagQuery($filter: ID!) {
    tag(id: $filter) {
      slug
      name
      posts {
        edges {
          node {
            content
            slug
            title
          }
        }
      }
    }
  }
`;  



class Tag extends Component {
    constructor(props){
        super(props);
        this.state = {
            tag: {
                name: "",
                posts: []
            }
        }
    }

    componentDidMount(){
        this.executeTagQuery();
    }

    executeTagQuery = async () => {
        const { match, client } = this.props;
        const filter = match.params.id;
        const result = await client.query({
            query: TAG_QUERY,
            variables: { filter }
        })
        const { name } = result.data.tag;
        let posts = result.data.tag.posts.edges;
        posts = posts.map(post => {
            const finalLink = `/tag/${post.node.id}`;
            const modifiedTag = { ...post };
            modifiedTag.node.link = finalLink;
            return modifiedTag;
        })
        const tag = {
          name,
          posts
        }
        this.setState({ tag });
    }

    render(){
        const { tag } = this.state;
        return (
            <div className="w-full lg:w-3/5 p-8 mt-6 lg:mt-0 text-gray-900 leading-normal bg-white border border-gray-400 border-rounded">
              <h4><strong>Posts with Tag:</strong> <em>{tag.name}</em></h4>
              <ul className="py-10 mx-auto">
                {tag.posts.map(post=> (
                  <li key={post.node.slug} className="">
                      <div className="flex my-10">
                        <div className="bg-white w-3/4 m-auto border-1  border-dashed border-gray-100 shadow-md rounded-lg overflow-hidden">
                            <img src="https://via.placeholder.com/400x200" alt="" className="w-full object-cover object-center" />
                            <div className="p-4">
                                <Link to={post.node.link} className="mx-auto f text-gray-900 font-semibold">{post.node.title}</Link>
                                <br/>
                                <span className="text-gray-700">{post.node.content}</span>
                            </div>
                        </div>
                    </div>   
                  </li>
                ))}
              </ul>
          </div>
        )
    }
}

export default withApollo(Tag);

Is ruby dead?

dillan teagle — Sun, 22 Sep 2019 14:19:40 +0000

My previous company used ruby and rails throughout the company. I would say in Nashville adleast, Ruby is not a common choice of a technology stack. In Nashville, unfortunately there is so many .NET shops with the healthcare startups. Although, other large tech companies like Lyft, EventBrite, CampaignMonitor, Amazon, etc...do not choose ruby as there foundational language.

I feel like even the open source community and the online learning community on YouTube, Dev.to, Medium, etc... is limited to articles and tutorials about Ruby / Rails ..."""Compared to some of the other most popular languages"".

Is Ruby Dead?

Integrating Slack with Django with Active Directory

dillan teagle — Thu, 19 Sep 2019 13:16:08 +0000

Many companies may use Active Directory for domain authentication. LDAP, a.k.a “Lightweight Directory Access Protocol”, is a communication language that applications use to communicate with other directory services servers.

With that said, we will be integrating two external API’s into the application, python_ldap and SlackAPI. Now before we dig in, I want to clarify the use case and why this makes sense. A companies domain controller that is the gateway for user authentication, will also be listed as a a key: value pair in a Slack API response. Typically, your slack workspace will also identify with the name of your company domain. If that is the case, then using Slack as another form of authentication next to Active Directory makes sense, IF you have plans for integrating Slack within areas of your application!

As far as interacting with Active Directory, fortunately there is already a library that is maintained that simplifies interacting with the python_ldap api.

So lets set up authentication to the domain controller first. Install django_auth_ldap and configure your settings.py.

At this step, you need to configure the settings to connect to the server where Active Directory runs, and configure your settings to bind, connect, and state how to search for user’s in the directory.

AUTH_LDAP_SERVER_URI = config("active_directory_server")
AUTH_LDAP_CONNECTION_OPTIONS = {
ldap.OPT_DEBUG_LEVEL: 1,
ldap.OPT_REFERRALS: 0
}
AUTH_LDAP_BIND_DN = config("DN")
AUTH_LDAP_BIND_PASSWORD = config("PASSWORD")
AUTH_LDAP_USER_SEARCH = LDAPSearch(
"CN=Users,DC=company,DC=test,DC=domain,DC=com", ldap.SCOPE_SUBTREE, "sAMAccountName=%(user)s"
)
AUTH_LDAP_USER_ATTR_MAP = {"first_name": "givenName", "last_name": "sn", "username": "sAMAccountName"}
LDAP_IGNORE_CERT_ERRORS = True
AUTHENTICATION_BACKENDS = (
'django_auth_ldap.backend.LDAPBackend',
)

One thing that is important to mention is that I took out the default Django backend “ModelBackend”, for authenticating users, and replaced it with..

django_auth_ldap.backend.LDAPBackend

Once your settings are configured, you should be able to authenticate into your app with your credentials using Django’s login method. Seriously, its that simple.

def login_view(request, *args, **kwargs):
    form = UserLoginForm(request.POST or None)
    try:
        kwargs.get("msg")
        if msg == True:
            messages.add_message(request, messages.SUCCESS, 'Your account has not yet been authenticated with Active Directory '
                                            'Please first register with the domain before using Slack.')
    except:
        pass

    if form.is_valid():
        user = form.login(request)
        if user is not None:
            login(request, user)
            return redirect("/")
    return render(request, "accounts/login.html", {"form": form})

For now ignore the try / except block, but pay attention to how I am using a login method off of our django form, if it is valid.

class UserLoginForm(forms.Form):
    username = forms.CharField()
    password = forms.CharField(widget=forms.PasswordInput)

    def clean(self):
        username = self.cleaned_data.get('username')
        password = self.cleaned_data.get('password')
        user = authenticate(username=username, password=password, backend=LDAPBackend())
        if not user or not user.is_active:
            raise forms.ValidationError("Sorry, that login was invalid. Please try again.")
        return self.cleaned_data

    def login(self, request):
        username = self.cleaned_data.get('username')
        password = self.cleaned_data.get('password')
        user = authenticate(username=username, password=password, backend=LDAPBackend())
        return user

As you can see, I am setting Django’s authenticate method to use the LDAPBackend.

Now its time for the fun stuff, interacting with Slacks API which is a fantastic experience due to great documentation and structure.

So for authenticating with Slack we will be using OAuth2.0 / Sign in with Slack. If you have not already created you slack app, please do so and get your slack sign in button from the api docs. Look below.

<a href="https://slack.com/oauth/authorize?scope=identity.basic&client_id=your_client_id"><img src="https://api.slack.com/img/sign_in_with_slack.png" /></a>

I chose to set the Slack API credentials in a file that is within my settings directory.

SLACK_CLIENT_ID = config("SLACK_CLIENT_ID")
SLACK_CLIENT_SECRET = config("SLACK_CLIENT_SECRET")
SLACK_AUTHORIZATION_URL = 'https://slack.com/oauth/authorize'
SLACK_OAUTH_ACCESS = 'https://slack.com/api/oauth.access'
SLACK_API_TOKEN = config("SLACK_API_TOKEN")

Now it is time to think about our authorization flow. For the most part, Slack will take you through a redirect page that allows the user to authenticate with their Slack credentials against the workspace. So I chose to handle this with a RedirectView / “Class Based View”.

Before digging in to the view, lets set up how we are going to call the API.

class SlackManager:

    def __init__(self):
        self.base_url = "https://slack.com/api/"

    def authenticate(self):
        data = {
            "client_id": settings.SLACK_CLIENT_ID,
            "scope": "identity.basic identity.team identity.avatar:read:user",
            "redirect_uri": "http://127.0.0.1:8000"
        }
        url = settings.SLACK_OAUTH_ACCESS
        response = requests.get(url, params=data)
        print(response)
        return response

    def get_user_data(self, token):
        endpoint = "users.identity"
        data = {
            "token": token
        }
        user_object = requests.get(self.base_url + endpoint, params=data)
        return user_object.json()

    def get_user_profile(self, user_id, token):
        data = {
            "token": token,
            "user": user_id
        }
        endpoint = "users.profile.get"
        response = requests.get(self.base_url + endpoint, params=data)
        print(response.text)
        return response.json()

If you have read through Slack’s API docs and created your application, then you should understand the parameters used in our requests. The token is the access token obtained from ‘https://slack.com/api/oauth.access' and scope defines permissions amongst the available endpoints.

Now setting up the authorization flow is your decision, but when you created and registered the slack app for your workspace, there is an option for a redirect uri. This is how slack knows where to deliver the user after they have handled the authentication.

Below is a RedirectView that handles the logic of the authorization flow.

class OAuthView(RedirectView):
    permanent = True
    text_error = 'Something went wrong with updating your authorization.'

    def dispatch(self, request, *args, **kwargs):
        return super(OAuthView, self).dispatch(request, *args, **kwargs)

    def get(self, request, *args, **kwargs):
        code = request.GET.get('code')
        if not code:
            return self.auth_request()
        self.validate_state(request.GET.get('state'))
        access_content = self.oauth_access(code)
        if not access_content.status_code == 200:
            return self.error_message()
        api_data = access_content.json()
        if not api_data['ok']:
            return self.error_message(api_data['error'])
        user_obj = gather_slack_info(api_data)
        try:
            user = User.objects.get(username=user_obj["ldap_name"])
            if user is not None:
                profile = UserProfile.objects.get(user=user)    
                profile.slack_token = api_data["access_token"]
                profile.name = user_obj["name"]
                profile.email = user_obj["email"]
                profile.slack_id = user_obj["slack_id"]
                profile.extras = user_obj
                profile.save()
                login(request, user)
                messages.add_message(self.request, messages.SUCCESS, 'Your account has been successfully updated.')                                        
            return self.response()
        except ObjectDoesNotExist:
            return self.ldap_initial_auth(request)

    def auth_request(self):
        state = self.store_state()
        params = urlencode({
            'client_id': settings.SLACK_CLIENT_ID,
            'redirect_uri': self.request.build_absolute_uri(reverse('accounts:Oauth')),
            'scope': 'identity.basic identity.email identity.team identity.avatar',
            'state': state
        })
        return self.response(settings.SLACK_AUTHORIZATION_URL + '?' + params)

    def oauth_access(self, code):
        params = {
            'client_id': settings.SLACK_CLIENT_ID,
            'client_secret': settings.SLACK_CLIENT_SECRET,
            'code': code,
            'redirect_uri': self.request.build_absolute_uri(reverse('accounts:Oauth'))
        }
        return requests.get(settings.SLACK_OAUTH_ACCESS, params=params)

    def validate_state(self, state):
        state_before = self.request.session.pop('state')
        if state_before != state:
            raise StateMismatch('State mismatch upon authorization completion.'
                                ' Try new request.')
        return True

    def store_state(self):
        state = str(uuid.uuid4())[:6]
        self.request.session["state"] = state
        return state

    def response(self, redirect='/'):
        return HttpResponseRedirect(redirect)

    def ldap_initial_auth(self, request):
        msg = True
        return HttpResponseRedirect(reverse("accounts:login", kwargs={"msg": msg}))

Lets start with the get method, which is the primary method that combines all the functionality of the other helper methods together. So slack uses an authentication code that is exchanged for an access token. At https://slack.com/api/authorize this code is generated and swapped for an access code at https://slack.com/api/oauth.access.

The state parameter should be used to avoid forgery attacks by passing in a value that's unique to the user you're authenticating and checking it when auth completes. We store our state in the user session because of that reason.

If our API call is successful, then we will extract the information we want and return it in one dictionary.

def gather_slack_info(user_object):
   slack_user = user_object["user"]
   slack_team = user_object["team"]
   print(slack_user)
   ldap_name = ldapify_name(slack_user["name"])
   context = {
      "slack_id": slack_user["id"],
      "name": slack_user["name"],
      "ldap_name": ldap_name,
      "email": slack_user["email"],
      "team_id": slack_team["id"],
      "team_name": slack_team["name"],
      "domain": slack_team["domain"],
      "avatar": slack_user["image_24"]
    }
    return context

ldapify_name will format the slack’s user name to match the format of username’s in our Active Directory.

def ldapify_name(name):
    x = name.lower()
    y = x.replace(" ", "")
    return y

In the last method of the RedirectView, we are sending the user back to the login screen if they do not exist in the database. User’s do not register in this application. So when you sign in for the first time using your credentials, you will then be added as a user with your active directory username and password.

I thought it was important that every user initially logs in to the application by LDAP for the first time, then they may use their workspace as an additional form of authentication in the future.

So I pass a keyword argument over to the login view that tells the django template to display a message to the user to login with Active Directory for their first time.