DEV Community: realMethods

Jumper Cables

Steven Randolph — Mon, 24 Feb 2020 15:50:00 +0000

Have a great idea but have limited to no cash and need help becoming VC, incubator or accelerator ready? jumpstartup.io is worth a look.

We Built a Project Generator Because DevOps Needs One

Steven Randolph — Sat, 09 Nov 2019 22:28:48 +0000

We, realMethods, are a small System Integrator. As a small SI with limited resources, we needed an advantage to compete for large engagements. We needed a lever we could pull that the bigger SIs could not. Before plowing head first into developing something, it was important to reflect on what we actually needed, had tried and experienced, what worked and did not work, and what mattered today that would still be relevant tomorrow.

We came up with 9 statements.

1. Simple to use, Then Get Out Of The Way
If a tool can serve a single purpose with minimum input and very useful output, they have me as a loyal user. Good tool developers recognize they don’t own the entire toolbelt and find a place to fit for a well defined task. And once used, they know how to get out of the way so work can continue. They add value to the work continuum without needing to be the star of the show.

It must fit a current need and do so without disrupting a Developer’s normal course of work.

2. Overcome “Generator Bias”
Some of the team initially had no interest in heading down the “Generator” path. Generators fundamentally have a tenuous reputation. Most come with a lot of hype and promise followed by a let down.

Although automation is a pillar of DevOps, automation of the required code still has its skeptics. More interesting is how “everything as code” is widely accepted yet using code to create code is not an integral DevOps phase.

We decided to use logic over prior bias. If we could generate entire projects, we could do more with fewer resources for lower cost in less time, leading to us winning more client engagements.

3. We need contextual projects generated, not just applications or code

A bit of common understanding

Code Generator: creates the beginning of an application with no promise of one.

Application Generator: creates code with the expectation of minimally functionality. All the pieces are likely buildable, but the result still lacks the configuration required of an operational environment.

Project Generator: Creates an application along with all the files to build, test, archive, package, containerize, and finally deploy the application.

Model-Driven Project Generator: Creates everything by applying an entity domain wherever appropriate among the generated assets. It is the closest a first commit of a project can come to its final commit.

It must generate all business contextual code and files such that an operational CI/CD pipeline can flow the resulting application from build through deployment.

4. Can't Live Without It
We need to build something that when used once, we will never want to go back to life without it. This is true for us and our clients. Not only because it offers a clear competitive advantage. It provides early project lift we have become professionally and psychologically dependent on. We don’t have to worry about writing the low level plumbing any more. We don’t want to. Just as we have become dependent on the leverage our tool chain offers, having to instead manually reproduce the 10s to 100s of 1000s of lines of code would be demoralizing.

It must provide so much value, once used a user will never want to start a DevOps project without it.

5. Can Never Predict What A Client Needs

We start new projects with a client,and until told, we have no idea about:

Business entity domain
CI platform(s) to use
Cloud or on-prem or hybrid
Relational vs NoSQL
Traditional App vs Serverless vs Restful APIs vs Microservices
Technology Stack / Language
Application specific things like name, description, corporate logo, etc.
Git server and credentials
Docker image creation details
Kubernetes cluster details

It must be fully customizable to accommodate a project’s requirements

6. Support Declarative DevOps
There is “as-code” for containers,orchestration, infrastructure, configuration, and pipelines, so why not a project?

What we built had to follow suit. If a project’s technical requirements can be described, then those requirements can be declared and codified in a YAML file. The tool had to hide the messy details of the "how" in order to create the "what". Just like other "as-code" implementations, it should be easy to use.

Simply stated, Project-as-Code allows us to make the following declarative statement to turn requirements into a running CI pipeline with a functional application flowing through it:

"I have a business model I would like to apply to an Angular7 tech stack. I want >to store data using MongoDB. I would like the resulting project files to be >committed to my GitHub repo, the source files to be built and tested using >CircleCI, and the resulting application pushed as a Docker image to my container >repository then finally deployed to a designated Kubernetes cluster on GCP."

The resulting YAML looks something like this:

    techstack:
       identifier:              Angular7Stack
    model:
       identifier:              Some business model file location
    options:
        application:
            name:             MyAngular7App
            version:           0.0.1
        cicd:
            platform:         circleci    
        git:            
            username:      gitUId
            password:       gitPwd
            repository:       gitRepo
            tag:                  anyTag
            host:                anyGitHost 
        docker:
            userName:      dockerUId
            password:       dockerPwd
            orgName:        dockerOrg
            repo:                dockerRepo
            tag:                  dockerTag
        kubernetes:
            host:                  https://xxx.xxx.xxx.xxx
            hostTarget:        google
            username:         k8UId
            password:          k8Pwd
        artifact-repo:
            type:                   jFrog
            userName:         jFrogUId
            password:          jFrogPwd
            repoUrl:              http://xxx.xxx.xxx.xxx:8081/repository/npm-public
        mongodb: 
            serverAddress:   localhost:27017
            databaseName:  angularDemoDB

The tool must hide the heavy lifting of project generation by accepting a single configuration file containing the project declarations to then execute the intent of the file’s author

7. Being Small and Lazy Is A Good Thing
A small SI needs to be super efficient doing as little as possible while still fulfilling all required tasks. Each individual task is evaluated as a generation candidate and if found not to be, we look to discover if it (or a variant) has already been built. If it has, we either reuse or extend it. That allows a small team to create output at the scale and pace of much larger teams.

Generate all you can, especially all the things nobody wants to write but are needed by the project.

8. Make Accommodations For Smarter People
By design we chose to capture expertise through the templatizing of a tech stack (we call it a tech stack package). This package contains everything needed to represent the intent of its author, along with other information to interact with the tool during generation time.

The tool cannot make assumptions about the purpose of a tech stack package, so long as it is well formed according to documented standards. This allows expertise to be consumed indiscriminately. Although the focus of the tool is on model-driven DevOps projects, a fortunate consequence of our design is that it can generate any type of application, project, and other packages relying on file-based outcomes.

There are too many permutations of languages and technologies to think we could capture them all at once or capture any perfectly.

9. A Client Entity Model and Our Meta Model Make The Difference
Two projects might use the same programming language with similar technical requirements but differ in their entity models. We support standard model definition types to allow the inclusion of business context into the generation. Without this, we would only be able to generate a glorified “Hello World” app.

Importantly, we developed a meta-model with support for factory plug-ins for each model definition type. A plug-in uses the tools’ meta-model API to help translate the input model to the meta-model structure. In turn, a tech stack package consumes the meta-model through the same API. A template within a tech stack package makes output decisions based on the content of the meta-model among other things.

The tool must support an extensible way of including business context into project generation.

Conclusion
We built a tool that we and our clients use to generate DevOps projects with minimal upfront effort. Tens to hundreds of thousands of lines of code we need suddenly appear. With Project-as-Code, there is now something to automate the automation behind DevOps. It serves as a catalyst to DevOps to help lower hurdles to get from idea to container easier and faster. Project-as-Code is a real thing because it makes sense. It fits nicely between discovering the intent of a project and realizing its first commit. It will take time before project generation becomes a standard phase in the DevOps life-cycle. We believe it will eventually and for us and some forward thinking companies, it already has.

realMethods CircleCI Orb Integration

Steven Randolph — Sat, 09 Nov 2019 22:02:02 +0000

Demonstration of the realMethods Orb for integration in to CircleCI

realMethods 101

Steven Randolph — Sat, 09 Nov 2019 22:01:39 +0000

An introduction video to realMethods and project generation. Includes integration to Nexus Sonatype

CRUD Is a Great Way To Go Serverless

Steven Randolph — Sat, 19 Oct 2019 18:54:43 +0000

If you don’t know how cool Serverless is, take a look at this article and then this post. AWS is clearly the most popular Serverless implementation so it will be the focus here. To be honest, serverless isn’t exactly void of a server since code must run on something, but not having to deal with the common issues of code deployment/execution is a good thing to at least consider.

On to CRUD, which is short for Create/Read/Update/Delete. Let’s consider the fundamental intra-operations related to each business entity, and the inter-operations for relationships between entities. Doing the latter increases complexity but these relationships are too important to ignore.

Using the example in the figure above, let’s do a little math to determine how many discrete functions are needed. I count thirteen (13). Now let’s be clear, creating a function in a Java class (for example) is, for the most part, less time consuming than creating a corresponding function as an AWS Lambda function. Let’s assume the difference is minimal, but as the # of functions increases, so will time time along with the difficulty in debugging and monitoring. Checkout dashbird for help in this area

Let’s now grow the domain model a bit so we can continue to better size up the energy involved to create discrete functions for each CRUD operation.

With this small change to the domain model, we can now create a table and equation to better estimate the total number of required discrete CRUD functions.

Imagine if we expand our model to 10 business entities each having one (1) single association and one (1) multi association. We suddenly have 240 (100+60+80) CRUD functions in total.

Doesn’t sound like I am actually making much of a case for CRUD going Serverless but I am getting there. If what makes AWS Lambda valuable is still important to us, let’s assume no matter how many functions we have to write, the additional time to create and manage is near zero to negligible when compared to other development/deployment options.

Serverless and Relational DBs Can Be a Go!

CRUD is associated with persistence operations and these operations are primarily accomplished using either an RDBMS or NoSQL solution. An AWS Lambda function is not only a serverless function but it is a stateless function. This means each function knows nothing about the current state of anything dependent on it or anything it depends on. This is critical when it comes to persistence. If you have ever tried connecting to a database like MySQL, you know the first connection is often the slowest with subsequent connections performing acceptably. Unfortunately, since a serverless function is stateless, each call requiring a database connection is considered a cold call first connection and the performance is unbearable. There are some databases that play nice in a stateless scenario, but if you have already invested in a traditional RDBMS these won’t suffice. Fortunately, there are implementation strategies where the database persistence implementations are abstracted out of the serverless function itself, into a remotely accessed stateful RESTful API. The great part is that API can be accessed by other applications too.

Serverless or otherwise, we more often than not need these CRUD functions. They form the core functionality of most modern application architectures, whether microservices, RESTful APIs, or monolithic.

Creating a Lambda Ecosystem is a Snap!

I used a domain model to help illustrate the relationship between entities, operations, and associations. The more complex the domain, the more complex the model and the more CRUD functions. I have worked on projects using UML with a class diagram having in excess of 100 business entities and more than 250 associations. Not visually appealing to look at but it helps to have a single picture to help grasp the scope of an effort. Another reason why I create a domain model is to help illustrate how the details of the model map nicely to what we need in terms of creating a corresponding Lambda function ecosystem. If the model is well defined and understood, and the mapping from the CRUD capabilities can be inferred simply from the model’s entities and associations, and it is understood how each CRUD capability is to be represented as a Lambda function, we have an opportunity to generate it all. No need to write the 100s Lambda functions when they can be quickly generated using a model as input. For this example, I used UML as input to represent a domain model, but I could have just as easily described the model in a JSON file or used database schema notation.

Application Function Refactoring

The latter point of database schema notation being another form of a domain model is very important. If we consider many legacy applications are built on an architecture where a relational database is used to persist data, the schema of that database gives us a very close to perfect representation of that application’s domain model. As stated, if that model can be used to apply as input to then create a corresponding set of Lambda functions, we can consider this a form of refactoring. It might be easier to consider this function refactoring as an easy first step in application refactoring than refactoring an entire application itself.

In Summary

Even though the number of CRUD functions can be large for even a small project, generating them as AWS Lambda functions simplifies things significantly. Feel free to browse this Bitbucket repository to review the output for a sample project generated for CircleCI using Terraform.

The importance of the right partnerships

Steven Randolph — Sun, 13 Oct 2019 17:27:03 +0000

Recently, my company was chosen by CloudBees to be a part of their new Technical Alliance Partner program. Unlike other partnerships that often do not have significant criteria or a well-defined purpose, this partnership was formed among only 27 companies to create new revenue and growth opportunities in the DevOps marketplace.

As a small dev team (~5 people), it is critical we expend energy on the right things, including partnerships. We typically only create partnerships with the largest technology companies that have a certification process. This allows us to demonstrate value to their business, sales and technical teams firsthand. This is one of the best ways to gain traction within the partner organization and open up opportunities to collaborate with the them and others in their partner ecosystem. Ultimately, it can lead to significant business and relationships we would struggle to create due to our size.

Most do not want to go it alone, nor do they have to. Finding the right strategic partnerships can expand and grow your company in ways it might not be able to otherwise.

I'd be interested to hear how others leverage partnerships, even personal ones for career advancement.

CloudBees Announces New Technical Alliance Partner Program

Steven Randolph — Sat, 12 Oct 2019 19:02:36 +0000

Grateful to CloudBees for including realMethods in the select company of other great Technical Alliance partners.

https://devops.com/cloudbees-announces-new-technical-alliance-partner-program/

Share Your Expertise; Earn Commission & Equity

Steven Randolph — Mon, 07 Oct 2019 21:16:58 +0000

It's cool to contribute uncompensated time and energy into an open source effort, but it's also cool to get something for your efforts and share in a common success.

This is a chance to templatize your design and development tactics/strategies into a reusable asset for the DevOps community to leverage!

Your work will become part of a growing library of open source tech stack packages offered by realMethods. Developers use these stacks to generate model-driven DevOps projects using Project-as-Code declarative YAML. These projects include all source/build/test/CI/Docker/Terraform files.

Specific areas of immediate interest include:

Java (Spring, Struts2)
AWS Lambda (Java, NodeJS, Python, C#)
Apollo GraphQL
Angular
Ruby (RAILS)
Python (DJango)
Vue
React
Swift
C/C++
ASP.NET (C#)

Check out the existing tech stack packages here.

Look here for an example of generated output.

Thanks in advance!

contact: support@realmethods.com