Eating Tech Debt - One Suggestion a Day

With the CLI of DeepCode, you can build suggestions into your daily routine. I thought I might be of help a bit.

Project Suggestions to Jira

User Story: As a developer, I want to automatically transform the topmost severe suggestion into a decorated Jira issue.

The result will look this:

So, what we will do is:
(1) We get the JSON output of the DeepCode CLI stored in a file as the first parameter
(2) We download the repo in a local temporary directory
(3) We get the most severe suggestion
(4) We build the decorated text for a Jira issue ticket
(5) We record a Jira issue ticket

Note: This is not a ready-to-use tool but rather a starter to hack it yourself :-) I left most error handling out so the concept is more obvious. You can find the source code here

The Code

I used some Python here but it should be easily adaptable to any language you prefer. Let us walk through - step by step:

import os
import json
import pathlib
import copy
import sys

from jira import JIRA
from git import Repo

import urllib.parse


from app_config.Configuration import Configuration

"""
The configuration is a simple data structure

class Configuration:
    Jira = {
        'host' : 'http://192.168.0.6:8080',
        'user' : 'frank',
        'token' : '...',
        'project' : 'demo'
    }
    Git = {
        'repo_url' : 'https://github.com/CU-0xff/deepcode-vuln-logviewer.git',
        'user' : 'cu-0xff',
        'branch' : 'master',
        'repo_dir' : 'e:/temp/cloned_temp'
    }

"""

config = Configuration()

You see, we import some libs. Notably jira and git. I exported the configuration into a separate file. The layout is provided as a comment above. Finally, we instantiate the configuration.

Next, comes a ton of helper functions that will help to flatten the suggestion file, pick the topmost interesting suggestion, slice and dice the text and source code to build a nice issue. There is a demo input file (demo_output.json) provided with the source to play around if needed. The following blob of code is annoying but will make our life much easier later.

# Helper functions

def log_error(msg):
    print("Fatal Error: {msg}".format(msg=msg))

def load_json(filename):
    try:
        with open(filename) as f:
            data = json.load(f)
        return data
    except:
        log_error("Something is wrong with the input file")
        sys.exit(1)
def read_file(filename):
    with open(filename) as f:
        content = f.readlines()
    return content

def retrieve_top_suggestion(suggestions):
    files = []
    try:
        # Flatten the data
        for file in suggestions['results']['files']:
            file_entry = suggestions['results']['files'][file]
            for suggestion in suggestions['results']['files'][file]:
                temp_body = copy.deepcopy(file_entry[suggestion])[0]            
                temp_body["suggestion"] = suggestions['results']['suggestions'][suggestion]
                temp_body["file"] = file
                files.append(temp_body)
        # Search for highest severity
        selected_file = {}
        highest_severity = 0
        for file in files:
            if file['suggestion']['severity'] > highest_severity:
                selected_file = file
    except:
        log_error("Something is wrong with the input file")
        os._exit(1)
    return selected_file

def decorate_source(sourcecode):
    decorated_text = []
    line_counter = 1
    for line in sourcecode:
        #Add line numbers
        new_line = "{Line}:{code}".format(Line=line_counter, code=line)
        decorated_text.append(new_line)
        line_counter += 1
    return decorated_text

def generate_source_excerpt(Row_Start, Row_End, sourcecode):
    TheStart = int(Row_Start) - 5
    if TheStart < 0: 
        TheStart = 0
    TheEnd = int(Row_End) + 5
    if TheEnd > len(sourcecode): 
        TheEnd = len(sourcecode)
    return "".join(sourcecode[TheStart:TheEnd])

def generate_Markers(top_suggestion):
    #Transform markers into list of data entries for easier access plus have a MarkerId
    Markers = []
    MarkerId = 0
    for Marker in top_suggestion['markers']:
        msg = (Marker['msg'][0], Marker['msg'][1])
        col = (Marker['pos'][0]['cols'][0], Marker['pos'][0]['cols'][1])
        row = (Marker['pos'][0]['rows'][0], Marker['pos'][0]['rows'][1])
        newMarker = [MarkerId, msg, col, row]
        Markers.append(newMarker)
        MarkerId += 1
    return Markers

def generate_Suggestion_Text(TopSuggestion, Markers):
    SuggestionText = TopSuggestion['suggestion']['message']
    Markers.sort(key=lambda tup: tup[1][0], reverse=True)
    for Marker in Markers:
        SuggestionText = "{Prelude}({id}) [{Highlight}|#Msg{id}]{Rest}".format(Prelude=SuggestionText[:Marker[1][0]], id=str(Marker[0]), Highlight=SuggestionText[Marker[1][0]:Marker[1][1]+1], Rest=SuggestionText[Marker[1][1]+1:])
    return SuggestionText

def generate_Code_Text(Marker, sourcecode):
    Output = "{{anchor:Msg{id}}}({id}) Code - refer to line {start} to {stop} \n".format(id=Marker[0], start=Marker[3][0], stop=Marker[3][1])
    Output += "{{code}}\n{code}\n{{code}}\n\n".format(code=generate_source_excerpt(Marker[3][0], Marker[3][1], sourcecode))
    return Output

def generate_Jira_Text(TopSuggestion, sourcecode):
    Output = "*Repository:* {repo}\n".format(repo=config.Git['repo_url'])
    Output += "*File:* {file}\n\n".format(file=TopSuggestion['file'])
    Markers = generate_Markers(TopSuggestion)
    Output += "*Suggestion:* {suggestion}\n\n".format(suggestion=generate_Suggestion_Text(TopSuggestion, Markers))
    Markers.sort(key=lambda tup: tup[1][0])
    for Marker in Markers:
        Output += generate_Code_Text(Marker, sourcecode)
    return Output    

The rest is a peace of cake. First, we load the JSON and get the top suggestion:

# - get checks from DC in
dc_suggestions = load_json(sys.argv[1])
top_suggestion = retrieve_top_suggestion(dc_suggestions)

Next we get the code so we can decorate the issue:

# - get source code from Git
Repo.clone_from(url=config.Git['repo_url'], to_path=config.Git['repo_dir'], branch=config.Git['branch'] )
sourcecode = read_file(config.Git['repo_dir'] + top_suggestion['file'])
sourcecode = decorate_source(sourcecode)

...and, well, do a Jira issue out of it:

# - get decoration text
jiraText = "h1. DeepCode Scan - Automatically Generated Ticket\n\n{msg}".format(msg=generate_Jira_Text(top_suggestion, sourcecode))

# - generate Jira ticket
issue_dict = {
    'project': 'DEMO',
    'summary': 'DeepCode - {Suggestion}'.format(Suggestion=urllib.parse.unquote(top_suggestion["suggestion"]["id"])),
    'description': jiraText,
    'issuetype': {'name': 'Bug'},
}

jira = JIRA(config.Jira['host'], basic_auth=(config.Jira['user'], config.Jira['token']))
new_issue = jira.create_issue(fields=issue_dict)

log_msg("New jira issue created {issueid}".format(issueid=new_issue.key))

Voila... Still open is to reset the environment - aka, delete the temporary directories and some error handling. Have fun hacking :-)