DEV Community: Steven Randolph

Jumper Cables

Steven Randolph — Mon, 24 Feb 2020 15:50:00 +0000

Have a great idea but have limited to no cash and need help becoming VC, incubator or accelerator ready? jumpstartup.io is worth a look.

realMethods now certified on Nutanix Acropolis Hypervisor

Steven Randolph — Fri, 22 Nov 2019 15:23:23 +0000

Nutanix has certified realMethods Project Generator as Nutanix Ready AHV in Applications on Nutanix Acropolis Hypervisor.

Nutanix Ready identifies recommended applications and solutions that are trusted to enhance the Nutanix delivery infrastructure.

Poll: Please answer a few HPE questions.

Steven Randolph — Fri, 22 Nov 2019 05:54:39 +0000

I am curious to hear what others know and think about HPE. I do not work for HPE. I am simply trying to get an understanding of the company software/DevOps wise. Not knowing or having heard of HPE is important to know too.

Thanks to all.

Do you associate HPE with DevOps?
When you hear Hewlett Packard, what things come to mine?
Have you used an HPE product in the normal course of building a software app? Was that product hardware or software and which one(s).
Would you consider using the HPE container platform in place of your current container platform?

We Built a Project Generator Because DevOps Needs One

Steven Randolph — Sat, 09 Nov 2019 22:28:48 +0000

We, realMethods, are a small System Integrator. As a small SI with limited resources, we needed an advantage to compete for large engagements. We needed a lever we could pull that the bigger SIs could not. Before plowing head first into developing something, it was important to reflect on what we actually needed, had tried and experienced, what worked and did not work, and what mattered today that would still be relevant tomorrow.

We came up with 9 statements.

1. Simple to use, Then Get Out Of The Way
If a tool can serve a single purpose with minimum input and very useful output, they have me as a loyal user. Good tool developers recognize they don’t own the entire toolbelt and find a place to fit for a well defined task. And once used, they know how to get out of the way so work can continue. They add value to the work continuum without needing to be the star of the show.

It must fit a current need and do so without disrupting a Developer’s normal course of work.

2. Overcome “Generator Bias”
Some of the team initially had no interest in heading down the “Generator” path. Generators fundamentally have a tenuous reputation. Most come with a lot of hype and promise followed by a let down.

Although automation is a pillar of DevOps, automation of the required code still has its skeptics. More interesting is how “everything as code” is widely accepted yet using code to create code is not an integral DevOps phase.

We decided to use logic over prior bias. If we could generate entire projects, we could do more with fewer resources for lower cost in less time, leading to us winning more client engagements.

3. We need contextual projects generated, not just applications or code

A bit of common understanding

Code Generator: creates the beginning of an application with no promise of one.

Application Generator: creates code with the expectation of minimally functionality. All the pieces are likely buildable, but the result still lacks the configuration required of an operational environment.

Project Generator: Creates an application along with all the files to build, test, archive, package, containerize, and finally deploy the application.

Model-Driven Project Generator: Creates everything by applying an entity domain wherever appropriate among the generated assets. It is the closest a first commit of a project can come to its final commit.

It must generate all business contextual code and files such that an operational CI/CD pipeline can flow the resulting application from build through deployment.

4. Can't Live Without It
We need to build something that when used once, we will never want to go back to life without it. This is true for us and our clients. Not only because it offers a clear competitive advantage. It provides early project lift we have become professionally and psychologically dependent on. We don’t have to worry about writing the low level plumbing any more. We don’t want to. Just as we have become dependent on the leverage our tool chain offers, having to instead manually reproduce the 10s to 100s of 1000s of lines of code would be demoralizing.

It must provide so much value, once used a user will never want to start a DevOps project without it.

5. Can Never Predict What A Client Needs

We start new projects with a client,and until told, we have no idea about:

Business entity domain
CI platform(s) to use
Cloud or on-prem or hybrid
Relational vs NoSQL
Traditional App vs Serverless vs Restful APIs vs Microservices
Technology Stack / Language
Application specific things like name, description, corporate logo, etc.
Git server and credentials
Docker image creation details
Kubernetes cluster details

It must be fully customizable to accommodate a project’s requirements

6. Support Declarative DevOps
There is “as-code” for containers,orchestration, infrastructure, configuration, and pipelines, so why not a project?

What we built had to follow suit. If a project’s technical requirements can be described, then those requirements can be declared and codified in a YAML file. The tool had to hide the messy details of the "how" in order to create the "what". Just like other "as-code" implementations, it should be easy to use.

Simply stated, Project-as-Code allows us to make the following declarative statement to turn requirements into a running CI pipeline with a functional application flowing through it:

"I have a business model I would like to apply to an Angular7 tech stack. I want >to store data using MongoDB. I would like the resulting project files to be >committed to my GitHub repo, the source files to be built and tested using >CircleCI, and the resulting application pushed as a Docker image to my container >repository then finally deployed to a designated Kubernetes cluster on GCP."

The resulting YAML looks something like this:

    techstack:
       identifier:              Angular7Stack
    model:
       identifier:              Some business model file location
    options:
        application:
            name:             MyAngular7App
            version:           0.0.1
        cicd:
            platform:         circleci    
        git:            
            username:      gitUId
            password:       gitPwd
            repository:       gitRepo
            tag:                  anyTag
            host:                anyGitHost 
        docker:
            userName:      dockerUId
            password:       dockerPwd
            orgName:        dockerOrg
            repo:                dockerRepo
            tag:                  dockerTag
        kubernetes:
            host:                  https://xxx.xxx.xxx.xxx
            hostTarget:        google
            username:         k8UId
            password:          k8Pwd
        artifact-repo:
            type:                   jFrog
            userName:         jFrogUId
            password:          jFrogPwd
            repoUrl:              http://xxx.xxx.xxx.xxx:8081/repository/npm-public
        mongodb: 
            serverAddress:   localhost:27017
            databaseName:  angularDemoDB

The tool must hide the heavy lifting of project generation by accepting a single configuration file containing the project declarations to then execute the intent of the file’s author

7. Being Small and Lazy Is A Good Thing
A small SI needs to be super efficient doing as little as possible while still fulfilling all required tasks. Each individual task is evaluated as a generation candidate and if found not to be, we look to discover if it (or a variant) has already been built. If it has, we either reuse or extend it. That allows a small team to create output at the scale and pace of much larger teams.

Generate all you can, especially all the things nobody wants to write but are needed by the project.

8. Make Accommodations For Smarter People
By design we chose to capture expertise through the templatizing of a tech stack (we call it a tech stack package). This package contains everything needed to represent the intent of its author, along with other information to interact with the tool during generation time.

The tool cannot make assumptions about the purpose of a tech stack package, so long as it is well formed according to documented standards. This allows expertise to be consumed indiscriminately. Although the focus of the tool is on model-driven DevOps projects, a fortunate consequence of our design is that it can generate any type of application, project, and other packages relying on file-based outcomes.

There are too many permutations of languages and technologies to think we could capture them all at once or capture any perfectly.

9. A Client Entity Model and Our Meta Model Make The Difference
Two projects might use the same programming language with similar technical requirements but differ in their entity models. We support standard model definition types to allow the inclusion of business context into the generation. Without this, we would only be able to generate a glorified “Hello World” app.

Importantly, we developed a meta-model with support for factory plug-ins for each model definition type. A plug-in uses the tools’ meta-model API to help translate the input model to the meta-model structure. In turn, a tech stack package consumes the meta-model through the same API. A template within a tech stack package makes output decisions based on the content of the meta-model among other things.

The tool must support an extensible way of including business context into project generation.

Conclusion
We built a tool that we and our clients use to generate DevOps projects with minimal upfront effort. Tens to hundreds of thousands of lines of code we need suddenly appear. With Project-as-Code, there is now something to automate the automation behind DevOps. It serves as a catalyst to DevOps to help lower hurdles to get from idea to container easier and faster. Project-as-Code is a real thing because it makes sense. It fits nicely between discovering the intent of a project and realizing its first commit. It will take time before project generation becomes a standard phase in the DevOps life-cycle. We believe it will eventually and for us and some forward thinking companies, it already has.

realMethods CircleCI Orb Integration

Steven Randolph — Sat, 09 Nov 2019 22:02:02 +0000

Demonstration of the realMethods Orb for integration in to CircleCI

realMethods 101

Steven Randolph — Sat, 09 Nov 2019 22:01:39 +0000

An introduction video to realMethods and project generation. Includes integration to Nexus Sonatype

Project-as-Code: Furthering Declarative DevOps

Steven Randolph — Tue, 22 Oct 2019 19:58:46 +0000

If you are familiar with Docker, Terraform, and the most popular CI platforms (eg: CircleCI, Codeship), you already know the power behind Declarative DevOps. These technologies hide an incredible amount of reuse and detail by supporting simple structured syntax in an easy to create and read YAML file. Need an Ubuntu image? Done. How about a MongoDB instance? You got it. Need to do something completely custom? Got for it.

Automation Makes DevOps Go

I suggest taking a quick read of an article by BMC titled Automate Cloud & DevOps Initiatives Across All Phases. If automation is the orchestration of DevOps, declarative DevOps is its sheet music. It provides a readable format to play out the intent of the composer. Importantly, it simplifies the learning curve of using many traditional tools in the chain. Sure, it takes time to set up all the distinct and disparate pieces, but once it is done, it hums along rather well uninterrupted. It serves the project’s needs. My CI platform orchestrates things and has its own declarative syntax and requirements. Within it, I am able to do custom things to build and test my project, then invoke both my Docker and Terraform declarative requirements.

However, from project to project, nothing is exactly the same. For example, Terraform changes due to cloud provider, CI changes due to tech stack selection, and Docker changes as well. Different URLs and security credentials to connect to certain things, optional use of an artifact repository (like JFrog or Sonatype). The list goes on with a great # of permutations.

A DevOps Project Shortcoming

Again, declarative DevOps it to make life easier. Repeatable, predictable, and fast. Try something. Don’t like it? Make a quick change and try again. It is now a vital part of my development process. I have no interest in returning to how things used to be.

But I don’t get to enjoy the fullest benefits of DevOps and automation without a project. It’s like a factory waiting to receive the raw materials needed to make it actually be a factory. I define a project as a “meaningful code base” along with all the scaffolding required to make the machines in the factory run at least once. Many IDEs and CI platforms produce a simple tech stack specific “Hello World” app and some stubbed out/empty files to get the ball rolling. Forget starting with a fully functioning app. No ready to go Dockerfile, CI YAML file, or Terraform file. These platforms simply do not know enough about a projects requirements to produce a meaningful project.

Before the project starts and hands go to keyboard, there are quite a few things I already know. I “mostly” have my business requirements (these never change, do they…). I know the tech stack I need to leverage. I am directed to use CI platform ABC and cloud vendor XYZ. Finally, I need a Docker image orchestrated on a remote Kubernetes cluster.

If I can describe the project’s requirements then I would like to declare those requirements.

Automating the Automation

We have Container-as-Code with Docker, Orchestration-as-Code with Kubernetes, Infrastructure-as-Code with Terraform, and Pipeline-as-Code with many CI platforms. So why not Project-as-Code?

A single YAML file that turns project declarations into business contextual tech stack specific source code along with CI/container/orchestration config files. That’s a mouthful but worth considering. By infusing the business context it means no more “Hello World”. Instead, start with a deployable application with (likely) 10s to 100s of 1000s of lines of stuff I need and honestly no longer want to write so long as I can edit as required.

Example Project-as-Code YAML

app:
    techstack:
       id:                Angular7MongoDB  # options: ASPNETCore, LambdaNoSQL, Lambda, Angular7MongoDB, Django, GoogleFunctions
                                           #          SparkMicroWeb, SpringCore, SpringMongoDB, Struts2, Apollo
                                           # To see the list of available tech stacks on a realMethods instance using command:
                                           #     realmethods_cli stack_list --output pretty

    model:
       identifier:        shopping-cart.sql           # options:     Unique id of an existing model or 
                                      #              full or relative path to a model file of a supported type
                                      # model types: UML, Eclipse Modeling Framework, JSON, SQL Script, JAR, EAR
                                      # To see the list of available model on a realMethods instance using command:
                                      #      realmethods_cli model_list --output pretty
                                      #

#
# DevOps Project Creation Options
#       
    options:

#            
# CI/CD parameters
#            
        cicd:
            platform: circleci     # options: codeship, circleci, 
                                   # jenkins, azure, aws, gitlab, 
                                   # bitbucket, buddy, semaphore
            AES key:               # codeship project key; ex: XKyy2IDcSptIIvMY8KLMVMcxVs+ZK6AyNu1B4Wu1DPY=
#            
# Application parameters
#            

        application:
            name:                  angular7demo
            description:           Demo Angular7 application generated by realMethods
            author:                Dev Team
            email:                 xxxx.xxxxxxxxx@xxxxxxxx.com
            application logo URL:  ./img/turnstone.biologics.png
            company name:          Turnstone Biologics
            version:               0.0.1


#            
# Docker parameters
#            
        docker:
            userName:     your_docker_user_name
            password:     your_docker_password
            orgName:      your_dockerorg_name
            repo:         your_repository
            tag:          latest

#
# Git repository parameters
#
        git:            
            name:        GitAngularSetting     # an arbitrary name
            username:    your_git_user_name
            password:    your_git_password:  
            repository:  your_git_repo_name
            tag:         latest
            host:        bitbucket.org            

#
# HashiCorp Terraform parameters
#
        terraform:
            inUse:             true 
            provider:          aws                   # options: aws, google, azure, nutanix
            region:            us-east-1             # options: any cloud provider region
            ssh-fingerprint:   xxxxxxxxxxxxxxxxx     # required by CircleCI for SSH


#
# AWS parameters
#
        aws:
            key-pair-name:     xxxxxxxxxxxx      
            vpc:               xxxxxxxxxxxxx        
            ec2-instance-type: t2.medium          # options: any AWS instance type
            access-key:        ASSIGN__ON_CICD_PLATFORM_AS_ENV_VARS
            secret-key:        ASSIGN__ON_CICD_PLATFORM_AS_ENV_VARS

#
# Nutanix parameters
#
        nutanix:
            inUse:         false
            userName:      xxxxxxxxxxxxxxxxxxxxxxxxx
            password:      xxxxxxxxxxxxxx
            endpoint:      xxx.xxx.xxx.xxx
            insecure:      true
            port:          xxxxx    
            clusterId:     xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
            email:         dev@realmethods.com
            vmUserName:    xxxxxxxxxxx
            vmPassword:    xxxxxxxxxxx
            vmHost:        xxx.xxx.xxx.3xxx
            vmOS:          linux    # options linux, windows - windows not yet supported

#
# Kubernetes parameters: If not in use, and Terraform
#                        is in use, physical infrastructure
#                        will be provisioned on the provider
#
        kubernetes:
            inUse:              true
            host:               https://xxx.xxx.xxx.xxx
            project:            xxxxxxxxxxx        # assigned project name
            region:             us-central1-a      # options: any provider region
            hostTarget:         google             # option: google, aws, azure, nutanix
            username:           admin
            password:           5UhfcIPAPsiPXklT  # generic value
            exposeAsService:    true
            serviceType:        LoadBalancer      # options: LoadBalancer, NodePort, ClusterIP
            useSessionAffinity: true


#            
# Artifact repository for build dependencies and binaries            
#
        artifact-repo:
            inUse:              false
            type:               nexus                  # options: nexus, jfrog
            userName:           xxxxxxx
            password:           xxxxxxxx
            email:              xxxx.xxxxxx@xxxxxxx.com
            repoUrl:            http://xxx.xxx.xxx.xxx:8081/repository/npm-public


#            
#  MongoDB parameters - defaults to a local accessible instance    
#
#  If Terraform is in use, will attempt to instantiate a MongoDB instance via Docker image,
#  and use the instance IP address when binding the lambda functions
#  
        mongodb: 
            server address:          localhost:27017
            database name:           angular7demo
            search size:             10
            default collection name: default_collection
            auto-increment id name:  _id
            auto-increment seq name: seq
            mongooseHost:            http://localhost
            mongoosePort:            4000

Conclusion

Project-as-Code moves the project starting line out and the overall timeline in. It gives (new) purpose to Sprint Zero. The upfront work required to get it setup (often less than a day) pays massive dividends in saving weeks of time to the first meaningful project commit. Every project has certain things that are more fun to work on than others. I prefer to automate the mundane and predictable (although much needed) to focus on more important dev. You can view a sample generated project here to get an idea of what is possible from a single Project-as-Code YAML file.

CRUD Is a Great Way To Go Serverless

Steven Randolph — Sat, 19 Oct 2019 18:54:43 +0000

If you don’t know how cool Serverless is, take a look at this article and then this post. AWS is clearly the most popular Serverless implementation so it will be the focus here. To be honest, serverless isn’t exactly void of a server since code must run on something, but not having to deal with the common issues of code deployment/execution is a good thing to at least consider.

On to CRUD, which is short for Create/Read/Update/Delete. Let’s consider the fundamental intra-operations related to each business entity, and the inter-operations for relationships between entities. Doing the latter increases complexity but these relationships are too important to ignore.

Using the example in the figure above, let’s do a little math to determine how many discrete functions are needed. I count thirteen (13). Now let’s be clear, creating a function in a Java class (for example) is, for the most part, less time consuming than creating a corresponding function as an AWS Lambda function. Let’s assume the difference is minimal, but as the # of functions increases, so will time time along with the difficulty in debugging and monitoring. Checkout dashbird for help in this area

Let’s now grow the domain model a bit so we can continue to better size up the energy involved to create discrete functions for each CRUD operation.

With this small change to the domain model, we can now create a table and equation to better estimate the total number of required discrete CRUD functions.

Imagine if we expand our model to 10 business entities each having one (1) single association and one (1) multi association. We suddenly have 240 (100+60+80) CRUD functions in total.

Doesn’t sound like I am actually making much of a case for CRUD going Serverless but I am getting there. If what makes AWS Lambda valuable is still important to us, let’s assume no matter how many functions we have to write, the additional time to create and manage is near zero to negligible when compared to other development/deployment options.

Serverless and Relational DBs Can Be a Go!

CRUD is associated with persistence operations and these operations are primarily accomplished using either an RDBMS or NoSQL solution. An AWS Lambda function is not only a serverless function but it is a stateless function. This means each function knows nothing about the current state of anything dependent on it or anything it depends on. This is critical when it comes to persistence. If you have ever tried connecting to a database like MySQL, you know the first connection is often the slowest with subsequent connections performing acceptably. Unfortunately, since a serverless function is stateless, each call requiring a database connection is considered a cold call first connection and the performance is unbearable. There are some databases that play nice in a stateless scenario, but if you have already invested in a traditional RDBMS these won’t suffice. Fortunately, there are implementation strategies where the database persistence implementations are abstracted out of the serverless function itself, into a remotely accessed stateful RESTful API. The great part is that API can be accessed by other applications too.

Serverless or otherwise, we more often than not need these CRUD functions. They form the core functionality of most modern application architectures, whether microservices, RESTful APIs, or monolithic.

Creating a Lambda Ecosystem is a Snap!

I used a domain model to help illustrate the relationship between entities, operations, and associations. The more complex the domain, the more complex the model and the more CRUD functions. I have worked on projects using UML with a class diagram having in excess of 100 business entities and more than 250 associations. Not visually appealing to look at but it helps to have a single picture to help grasp the scope of an effort. Another reason why I create a domain model is to help illustrate how the details of the model map nicely to what we need in terms of creating a corresponding Lambda function ecosystem. If the model is well defined and understood, and the mapping from the CRUD capabilities can be inferred simply from the model’s entities and associations, and it is understood how each CRUD capability is to be represented as a Lambda function, we have an opportunity to generate it all. No need to write the 100s Lambda functions when they can be quickly generated using a model as input. For this example, I used UML as input to represent a domain model, but I could have just as easily described the model in a JSON file or used database schema notation.

Application Function Refactoring

The latter point of database schema notation being another form of a domain model is very important. If we consider many legacy applications are built on an architecture where a relational database is used to persist data, the schema of that database gives us a very close to perfect representation of that application’s domain model. As stated, if that model can be used to apply as input to then create a corresponding set of Lambda functions, we can consider this a form of refactoring. It might be easier to consider this function refactoring as an easy first step in application refactoring than refactoring an entire application itself.

In Summary

Even though the number of CRUD functions can be large for even a small project, generating them as AWS Lambda functions simplifies things significantly. Feel free to browse this Bitbucket repository to review the output for a sample project generated for CircleCI using Terraform.

The importance of the right partnerships

Steven Randolph — Sun, 13 Oct 2019 17:27:03 +0000

Recently, my company was chosen by CloudBees to be a part of their new Technical Alliance Partner program. Unlike other partnerships that often do not have significant criteria or a well-defined purpose, this partnership was formed among only 27 companies to create new revenue and growth opportunities in the DevOps marketplace.

As a small dev team (~5 people), it is critical we expend energy on the right things, including partnerships. We typically only create partnerships with the largest technology companies that have a certification process. This allows us to demonstrate value to their business, sales and technical teams firsthand. This is one of the best ways to gain traction within the partner organization and open up opportunities to collaborate with the them and others in their partner ecosystem. Ultimately, it can lead to significant business and relationships we would struggle to create due to our size.

Most do not want to go it alone, nor do they have to. Finding the right strategic partnerships can expand and grow your company in ways it might not be able to otherwise.

I'd be interested to hear how others leverage partnerships, even personal ones for career advancement.

CloudBees Announces New Technical Alliance Partner Program

Steven Randolph — Sat, 12 Oct 2019 19:02:36 +0000

Grateful to CloudBees for including realMethods in the select company of other great Technical Alliance partners.

https://devops.com/cloudbees-announces-new-technical-alliance-partner-program/

Share Your Expertise; Earn Commission & Equity

Steven Randolph — Mon, 07 Oct 2019 21:16:58 +0000

It's cool to contribute uncompensated time and energy into an open source effort, but it's also cool to get something for your efforts and share in a common success.

This is a chance to templatize your design and development tactics/strategies into a reusable asset for the DevOps community to leverage!

Your work will become part of a growing library of open source tech stack packages offered by realMethods. Developers use these stacks to generate model-driven DevOps projects using Project-as-Code declarative YAML. These projects include all source/build/test/CI/Docker/Terraform files.

Specific areas of immediate interest include:

Java (Spring, Struts2)
AWS Lambda (Java, NodeJS, Python, C#)
Apollo GraphQL
Angular
Ruby (RAILS)
Python (DJango)
Vue
React
Swift
C/C++
ASP.NET (C#)

Check out the existing tech stack packages here.

Look here for an example of generated output.

Thanks in advance!

contact: support@realmethods.com