DEV Community: Diego

JIRA Workflow task from YAML files

Diego — Tue, 05 Feb 2019 01:36:49 +0000

Use Case:

In our company we have different areas that work across JIRA tasks and Confluence.

Sometime we need to set around 20 tasks for a common target. This tasks are for example to get up and running some customizations from our customers.

In the example that i'm going to use we need:

[1] Epic Task
[4-5] Main Tasks for the different areas
[4-5] Sub-tasks from each main task
Relate the task to labels
Relate the tasks between them
Attach links (Zendesk, Confluence, docs, etc)

This actions at this moment are executed by one person, but sometimes you forget to do a task, or may be creation the relations between tasks (blocks, relates to). With this we avoid human mistakes, we set the structure and then the script auto-generates all the tasks.

diegonalvarez / jira-workflow-tasks

Jira workflow tasks

JIRA Workflow tasks

This project allows to you create a workflow template to build tasks in Jira. You only need to set the specifications in multiples YML files, and then the script will generate the tasks and their relationships accordingly the YML configuration.

Requirements

This project uses two libraries:

This libraries are already loaded with composer

Getting Started

To get started it's necessary set some environment variables. You need to copy the .env.example file as .env file.

Here set the JIRA properties:

JIRA_HOST="https://your-jira.host.com"
JIRA_USER="jira-username"
JIRA_PASS="jira-password-OR-api-token"

Installing

Firts install composer to get the libraries installed.

composer install

Now it's necessary set the templates

Templates

The project already becomes with a example template. The templates are stored in the folder templates.

Template Folder

To get a project working you can copy and paste the structure of templates/example or create your own First create a folder inside …

View on GitHub

So with this we can set many templates in the projects folder and run automated workflows.

Installing and configuring Jenkins in AWS with SSL

Diego — Wed, 23 Jan 2019 21:06:57 +0000

Description

We are going to install Jenkins in a Amazon Linux 2 AMI, but this set-up can be done in any server.

Installation

#1 First Step - Launching Instance

Selecting the AMI

We are going to go to EC2 in the AWS Console. Select the service and in the section "Instances" select "Launch Instance". We choose, to this set-up an Amazon Linux 2 AMI.

Type of Instance

In the next step you need to select an Instance Type, you can select t2.micro for the eligible free tier or any Instance according to your needs. We selected for this testing the t3.medium Instance.

Configure Instance

Here you need to select the configuration that better work for your needs. The only thing that I set here is an "Auto-assign Public IP", because after the set-up we are going to integrate with github and we are going to need a public IP.

Selecting the storage

For this installation we selected 40gb, you can use the size that you need.

Adding Tags

Set some tags that came be useful for you. I personally use the organization and the name of the instance.

Security Group

Here set the firewall rules, to allow access to your instance. I personally select a security group that I set before for other Instances.

Review and launch

In the last step you can review the total configuration and set the key pair to connect throught SSH to your instance. We going to use this key pair in the next steps.

#2 Second Step - Assigning an Elastic IP address to the instance and setting a domain.

In the EC2 service, select Elastic IPs option from the nav bar, under "Network & Security" option:

Then click in "Allocate new address" and Allocate a new address, I use "Amazon pool" option. Here you receive the new IP, for example 127.0.0.1. Then select the IP and go to the option "Associate address".

In the next form you need to select the Instance and then click in Associate.

With this now you have the new Instance allocated to the new Elastic IP, so the IP of your instance has changed.

Now we are going to set-up a subdomain, if you have any registered in aws is registered in Route 53. Go to Route 53 service and select "Create Record Set".

Complete the subdomain that you want and:

Type: A - IPv4 Address
Alias: NO
Value: Put the IP that your instance have
Routing Policy: Simple

Then click on "Save Record Set" and the subdomain it's ready.

#3 Third Step - Shorcut to connect the instance via SSH

You can use this command in your terminal to access the instance via SSH:



ssh -i /Path/to/file/pem/file.pem user@ipOrServerName

There you have 3 variables:

file.pem, accordingly to the screenshot the name of our file is jenkins-v2.pem
user - If you selected the Amazon Linux 2 the user is ec2-user
IP - The IP assigned to your instance

We are going to set an easy way to connect this to not remember always the command, for this you need to:

Create a new file named "config" in the ~/.ssh folder and paste the next code, replacing with your vars:



Host ci
    HostName InstanceIP or ServerName
    User ec2-user
    IdentityFile /Path/to/file/pem/file.pem

After this we are going to set the proper permissions to the .pem file.



sudo chmod 600 /Path/to/file/pem/file.pem

Now you can access with this command to the server:



ssh ci

Or, for more comfortable way you can create an alias in you .bashrc profile.

#4 Fourth Step - Installing Jenkins

To install Jenkins run the following commands in order:



sudo wget -O /etc/yum.repos.d/jenkins.repo http://pkg.jenkins-ci.org/redhat/jenkins.repo



sudo rpm --import https://jenkins-ci.org/redhat/jenkins-ci.org.key



sudo yum install jenkins -y



sudo yum install java -y

At this point jenkins it's already installed and the config file path is:



/etc/sysconfig/jenkins

Now you can start Jenkins and see it in your browser in the port 8080.



sudo service jenkins start

And finally enable Jenkins for boot with:



sudo chkconfig jenkins on

Now you can see your jenkins runinng in http://domain:8080 you can change the port or set a revers proxy to redirect the traffic to your domain.

#5 Fifth Step - Configuring Jenkins Installation

On this setp you have already Jenkins installed and now it's necessarily configure some steps. You are going to see this image, for Jenkins 2.1.*

To get the initial admin password run in the server:



sudo cat /var/lib/jenkins/secrets/initialAdminPassword

Copy the ouput and pasted to conitnue the Jenkins configuration.

In the next step I selected that Jenkins installed for me some recomended Plugins, but fell free to install the plugins manually.

After this finish, set the Admin user and password and continue, that's it Jenkins it's installed.

#6 Sixth Step - Extra nginx configuration with SSL

We are going to install nginx, if I run:



sudo yum install nginx

I receive this output



Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
amzn2-core                                                                                                                                                                                                                                                     | 2.4 kB  00:00:00
No package nginx available.
Error: Nothing to do


nginx is available in Amazon Linux Extra topic "nginx1.12"

To use, run
# sudo amazon-linux-extras install nginx1.12

Learn more at
https://aws.amazon.com/amazon-linux-2/faqs/#Amazon_Linux_Extras

So, I installed nginx with the suggested command:



sudo amazon-linux-extras install nginx1.12 -y

Now we need to tell to jenkins that run in other port and in another address. To do this we are going to edit the file:



/etc/sysconfig/jenkins

And modified the var JENKINS_ARGS, with the next value:



JENKINS_ARGS="--httpPort=4433 --httpListenAddress=127.0.0.1"

Then in the section server of you nginx config file /etc/nginx/nginx.conf, modify with the following instructions:

Under http add:



upstream jenkins {
  server 127.0.0.1:4433;
}

And in the server section:



server {
  server_name servername.com;
  listen 443;
  resolver 8.8.8.8 valid=360s;
  ssl on;
  ssl_certificate /etc/ssl/certs/ssl-bundle.crt;
  ssl_certificate_key /etc/ssl/certs/domain.com.key;

  ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_ciphers "ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-AES256-SHA:ECDHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES256-GCM-SHA384:AES128-GCM-SHA256:AES256-SHA256:AES128-SHA256:AES256-SHA:AES128-SHA:DES-CBC3-SHA:HIGH:!aNULL:!eNULL:!EXPORT:!DES:!MD5:!PSK:!RC4";
  ssl_prefer_server_ciphers on;

  ssl_session_cache shared:SSL:10m;
  ssl_stapling on;
  ssl_stapling_verify on;

  location / {
    try_files $uri @app;
  }

  location @app {
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_next_upstream error;
    proxy_pass http://jenkins;
    proxy_redirect http:// https://;
    proxy_read_timeout 150;
  }
}

Here you need to ensure that you have your certs file in /etc/ssl/certs/ and set you servername.

After this restart the services:



sudo service nginx restart



sudo service jenkins restart

And now you can go to:



https://domain.com

And you are going to see the jenkins interface and if you try to go https://domain.com:8080 it's going to be unavailable.

Please let me know any issues or suggestions.

Thanks !

AWS Lambda - Step Functions - RDS - S3 - Aurora - Part 1

Diego — Wed, 15 Aug 2018 04:45:41 +0000

This is the first post about integrate Step Functions to an specific workflow. It's a public repo with one of my first python code, learning about that too.

diegonalvarez / bi-step-functions

How it works

A explanation on how it works can founded here:

https://dev.to/diegonalvarez/aws-lambda---step-functions---rds---s3---aurora---part-1-1fe

Requirementes

Chalice 1.1.0
Python 2.7

Json For Unique Machine

{
  "Comment": "State machine to populate databases for reports.",
  "StartAt": "MySql To S3 Files",
  "States": {
    "MySql To S3 Files": {
      "Type": "Task",
      "Resource": "ARN",
      "ResultPath": "$.guid",
      "Next": "S3 Data To Mysql",
      "Retry": [
        {
          "ErrorEquals": [
            "States.ALL"
          ],
          "IntervalSeconds": 1,
          "MaxAttempts": 3,
          "BackoffRate": 2
        }
      ]
    },
    "S3 Data To Mysql": {
      "Type": "Task",
      "Resource": "ARN",
      "InputPath": "$.guid",
      "End": true,
      "Retry": [
        {
          "ErrorEquals": [
            "States.ALL"
          ],
          "IntervalSeconds": 1,
          "MaxAttempts": 3,
          "BackoffRate": 2
        }
      ]
    }
  }
}

Json For Parallel Machines

{
  "Comment": "State machine to populate databases for reports."
  "StartAt": "MySql To S3 Files",
  "States": {
    "MySql To S3 Files": {
      "Type": "Task",
      "Resource": "lambda-arn",
      "ResultPath": "$.guid",
      "Next": "InsertDataToMysql",
      "Retry": [
        {
          "ErrorEquals": [
            "States.ALL"
          ],
          "IntervalSeconds": 1,
          "MaxAttempts": 3,
          "BackoffRate": 2
        }
      ]
    },

…

View on GitHub

Objective:

The objective for this post, it's moving data from a MySQL database to S3, and from there to another database.
The case that we are using is from Amazon RDS to Amazon Aurora, it can be another database of course, the idea is connect our new dataset with Bussiness Inteligences tools without affecting the original dataset and with a custom structure.

The first tought was use AWS Data Pipeline, but there are certains behavior that we need to custom.

The idea is have three or more post to get the objective, this is:

An admin database to handle the params, in our case we use:
- Companies
- Tables
Moving the data from RDS to S3
Moving the data from S3 to Aurora
Schedule to run the functions
An script to auto-create the information, only setting a customer(company in our use case)
Script to truncate tables, change names and detect that the new data is available.
Lambdas working in parallel with step functions and params for big tables.

In this post i'm going trough the points 1-2-3 and 4.

Admin Database

I'm going to put the structure that i'm using for the two main tables and how we use both:

Companies

I called the table companies, but you can change to customer or whatever you need (it's necessary to change the code to). The idea is that we have multiple companies, but only a few need Bussiness Inteligence, so for that companies we set-up a small configuration to get the info in our main database.

Name	Description
company_id	Unique identifier of the element
status	The status represent if it need the report
alias	Used to create csv and the bucket of the company



CREATE TABLE `companies` (
  `id` int(11) unsigned NOT NULL AUTO_INCREMENT,
  `company_id` int(11) DEFAULT NULL,
  `status` int(11) DEFAULT '0',
  `alias` varchar(60) DEFAULT NULL,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=5 DEFAULT CHARSET=latin1;

Tables

This table is designed to get the queries related to what information it's needed to be migrated.

Name	Description
name	Name of the table i.e. users
status	Logical status if it's necessary get the information
query_select	SQL to get the information
params_select	Params for the Select Query
query_insert	SQL to insert the information
params_insert	Params for the Insert Query



CREATE TABLE `tables` (
  `id` int(11) unsigned NOT NULL AUTO_INCREMENT,
  `name` varchar(255) DEFAULT '',
  `status` int(11) DEFAULT '0',
  `query_select` text,
  `params_select` text,
  `query_insert` text NOT NULL,
  `params_insert` text NOT NULL,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=9 DEFAULT CHARSET=latin1;

This is an real example, in the future version I wanna do a better approach to handle params.
The related insert is going to be a detect the table "users" and get the data accordingly the value of query_select using the params_query values, with this information the CSV is generated and inserted in S3.

When the function that insert the data run, it's going to execute the insert in the field query_insert with the params of params_insert. Here you can put any information and play with the structure of the dataset with your business needs.



INSERT INTO `tables` (`id`, `name`, `status`, `query_select`, `params_select`, `query_insert`, `params_insert`)
VALUES
    (1, 'users', 1, 'SELECT id, username FROM users WHERE company = %s', 'company[1]', 'REPLACE INTO users(id, username) VALUES ', '\'(\'+row[0] +\',\"\'+ row[1]+\'\")\'');

Thursday 06 December of 2018 Changes:

I added the feature to get from the database config fields. The first field that we need to customize is the date, so I created two tables:



CREATE TABLE `configurations` (
  `id` int(11) unsigned NOT NULL AUTO_INCREMENT,
  `name` varchar(192) DEFAULT NULL,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=2 DEFAULT CHARSET=latin1;

Here I the first record have the value "date Y-M-D", to identify the config added in the next table:



CREATE TABLE `companies_configurations` (
  `id` int(11) unsigned NOT NULL AUTO_INCREMENT,
  `retail_id` int(11) DEFAULT NULL,
  `configuration_id` int(11) DEFAULT NULL,
  `configuration_value` text,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=latin1;

An example of insert to get the last 60 days in the query it's:



INSERT INTO companies_configurations (id, retail_id, configuration_id, configuration_value)

VALUES

    (1, 1, 1, '60');

Config File

A config file is in .chalice/.config.sample.json

Rename the file to config.json and complete the params. All the params explain for themselves, don't forget to add the IAM Role.

Lambdas

There are two main functions, used as lambdas in this repo:

mysql_csv_to_s3
s3_to_mysql

mysql_csv_to_s3

This Lambda take the information from tables, execute the select query and insert the data into S3.

s3_to_mysql

Here, the data is collected from S3 and with the customs query, do the inserts.

Step Function [State Machine Definition]

At a final step, it's create a new State Machine from AWS Step Function and add the following json. You need to modify the ARN for each new lambda, and runned.



{

  "Comment": "State machine to populate databases for reports.",

  "StartAt": "MySql To S3 Files",

  "States": {

    "MySql To S3 Files": {

      "Type": "Task",

      "Resource": "ARN",

      "ResultPath": "$.guid",

      "Next": "S3 Data To Mysql",

      "Retry": [

        {

          "ErrorEquals": [

            "States.ALL"

          ],

          "IntervalSeconds": 1,

          "MaxAttempts": 3,

          "BackoffRate": 2

        }

      ]

    },

    "S3 Data To Mysql": {

      "Type": "Task",

      "Resource": "ARN",

      "InputPath": "$.guid",

      "End": true,

      "Retry": [

        {

          "ErrorEquals": [

            "States.ALL"

          ],

          "IntervalSeconds": 1,

          "MaxAttempts": 3,

          "BackoffRate": 2

        }

      ]

    }

  }

}

Cloudwatch Rule

Finally, you need to create a rule in cloudwatch related to the Machine State to generate a Schedule, it simply, here a documentation that helps

https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/Create-CloudWatch-Events-Scheduled-Rule.html

Hope the get some feedback and i'll be updating the posts with the second part.

Updating tasks in JIRA

Diego — Fri, 10 Aug 2018 04:44:40 +0000

Here i'm going to share a part of our mix Agile workflow that we have in our company.

The thing is that we have like ~200 task assigned to each member of the team in an unresolved situation. This is an accumulation from 2 years of getting requests and new ideas from all sources. It's happen because we don't wanna loose anything and we think that the most of the times, if it's a request of a customer, it's because there is a necessity to cover, even if the client left we stay with the task open, capitalizing the moment.

So what happen now it's that our team increase the size and it's difficult to see what everyone does and it's difficult for the developer know what to do because they don't know what are the important tasks for our customers. So we set-up a kanban board and clean-up some tasks.

The workflow is very simple:

Selected for development
In progress
Under Review
Done

We manage like a manual sprint, with a week release but with daily post in our chat room.

The issue is that we have too many tasks in "Selected for Development" because of the old tasks, so we sended with a "dueDate" to a "Backlog" status. If the task is in Backlog the developer doesn't see it until it's in "Selected For Development", this helps them to get a clear vision of what they need to do.

For this issue I set-up a script that updates all the tasks with a "dueDate" of 30 days and update the tasks from "Backlog" to "Selected For Development". The number of days can change with every project, but is a general idea and we are iterating to see how this works.

Here it's the repository with the code:

diegonalvarez / automated-tasks

Group of tasks that automate process

Getting Started

To start is necessary rename the file .env.sample to .env.

Set the correct credentials depending on the project that you are running.

Jira Update Status Tasks (Tasks/Jira/UpdateTasks/UpdateTasks.php)

This task allows update Jira issues to a specific transition(status). We used to change the status of the task per Due Date condition, avoiding missings issues in Jira.

Update the .env file

Set the JIRA variables in the .env file

Config File

There is a config file in Tasks/Jira/UpdateTasks/config.sample.cfg. First copy the file accordingly your environment, for example:

config.production.cfg
config.development.cfg

Then set the variables:

[issue_days]

The ammount of days to find issues from the current date.

[users]

The users that need to be analyzed. In a future we are going to use a group and getting directly from Jira.

[project_transition_id]

Set the Project and the transition ID. For example if you have multiples…

View on GitHub

The repo has a Readme file, but basically you only need to set the environment variables and set-up the behavior of Jira.

You can define the id of the status across multiples projects, status to avoid, users to trait and how many days, from the "dueDate" check the tasks to update.

Thanks for read, I hope to come back after a few months and share the feedback on how this work for us.