DEV Community: DeepCode.AI

JavaScript Promises: All you need to know

cu_0xff 🇪🇺 — Tue, 30 Jun 2020 11:36:21 +0000

Without any doubt: The most frequent suggestion we see corrected in the wild is actually a pretty trivial one (who would have thought). We found roughly 20,000 changes in our training repos addressing one thing: Unhandled rejections in promises. Maybe it is time to provide a fundamental guide.

If you want your code to be scanned, just go to deepcode.ai.

Why are promises needed?

JavaScript provides a single threaded environment: No two pieces of code run at the same time. This reduces issues regarding consistency with mutexes (think race conditions), but add the need for others. JavaScript makes use of callback functions to provide asynchronous calculations. While this in itself is a possible way to write asynchronous code, it leads to what is known as Pyramid of Doom where you have callback in callback in callback in ... until you totally lose track what happens when. Here come promises to help.

A promise is an object that represents eventual completion or failure of an asynchronous operation and its subsequent result value.

Dive deeper with MDN docs on Promise or Javascript.info

Basic Functionality

let promise = new Promise(function(resolve, reject) {
    // Here is the workload

    if( /* Success */) {
        /* result can be a Promise or thenable */
        resolve(result)
    }
    else {
        /* Best practice: reason is instanceof Error */
        reject(reason)
    }
})

Within a promise, two callback functions are provided: resolve() shall be called in the case of a positive outcome. The result can be handed on as a parameter. reject shall be called in the case of an error (including an explanation as a parameter). There is no third option (like cancelled or whatever). A promise is always in one of three states:

pending: Initial state and still work in progress
fulfilled: Completed successfully
rejected: Operation failed

Usage

Promises can be hand-on within the app as long as their value is not directly needed. This gives the system the opportunity to resolve whatever is asked in the background without the need to wait for things to settle. When the application has the need for the result to proceed, it can query the result and react upon it.

Side note: Try to delay the need for a result of a promise as much as possible to gain the maximum benefit of using promises.

Example:

const promise1 = new Promise((resolve, reject) => {
    resolve('Here is the result');
});

promise1.then(/* Success */(value) => {
    // Do something with the result
    },
    /* Fail */ (error) => {
    // React on error case
    }
);

If the success handler is not provided (as a function), it is replaced by the identity function (aka, it returns the parameter).

This is the lifecycle as provided by Mozilla's documentation (yet omitting finally()):

As you can see in the life cycle: There are two options to provide an error handler: Either by providing a callback as a parameter for then or by explicitly providing a callback in catch.

async and await

async in front of a function means the function always returns a promise. If the function returns a different type, it is wrapped in a promise.

async function f() {
    return 42; // We will get a promise with result 42 and success state
}

f().then(console.log) //prints 42

So far, we only stacked on promises but what if we really need the value to be settled. Here comes await. This keyword makes JavaScript wait until the promise is settled. Obviously, while the system stops here and waits with the execution of the code until the promise is settled, it continues to execute other code. We will see more sophisticated methods to combine promises a little later.


async function f() {
    let promise = new Promise((resolve, reject) => {
        // Do something here
        resolve(result);
    });

    let result = await promise;
}

f();

Note: await only works inside an async function. You cannot use await on top level code but you can wrap it in an anonymous async function:

(async () => {
    let response = await promise1;
})();

Chaining of Promises

Nice in-depth article on the topic here
The result of a promise can be piped through subsequent functions. Those functions can provide alternative parameters to their subsequent followers allowing to build a pipeline to manipulate data such as filtering or enriching. The then function returns a promise itself.

new Promise((resolve, reject) => {
    // Promise
    resolve(result);
}).then((result) => {
    // First handler
    return handler_result;
}).then((handlerResult) => {
    // Second handler
    return secondHandlerResult;
}).then((secondHandlerResult) => ...

Objects providing a then function, are called thenable. These objects can be used with then or await as we saw above.

class Thenable {
    then(resolve,reject) {
        if(success) //in case of success 
            resolve(result);
        else
            reject(error);
    }
};

async function f() {
    let result = await new Thenable();
    }

f();

There are two more important functions:

const promise1 = new Promise((resolve, reject) => {
    throw 'Error'; // calling reject() also leads to rejected state
    })

promise.catch((error) => {
    // Handle your error here
    }).finally(() => {
    //Do clean up needed whether success or failure
    });

catch: This is actually a shortcut for then(undefined, onRejected) and provides a promise that handles error cases. It does an implicit try-catch (as shown above).
finally: Again a promise which is called in both end states. Helps to reduce code doubling in then and catch as you can put all clean up needed in both cases in here.

Side note: Since both catch() and finally() return promises, you can chain one them again by using then.

If there is no error handler provided ("unhandled rejection"), the error bubbles up and leads to the script crashing with an error message. This is what DeepCode complains when it finds an unhandled rejection.

Combination of Promises

Promises can be collected and combined in various ways. In general, combining promises in a iterable object and provide this to the combination.

Promise.all([promise1, promise2, promise3, promise4]).then((values) => {
    console.log(values); // Result Array [result1, result2, result3, result4]
    }).catch((error) => {
    // something went wrong
    })

Here is a table of all combination functions:

Function	Explanation	Typical Use	Reaction on Rejection
Promise.all()	Returns a single promise to collect all results of the input promises	Multiple asynchronous tasks that are dependent on each other	Will reject immediately on one input promise reject
Promise.allSettled()	Returns a single promise to collect all results of the input promises	Multiple asynchronous tasks that are not dependent on each other	Will collect all results and rejects
Promise.race()	Returns a single Promise that returns as soon as one of the input promises resolve or rejects	Used in batching or for time-outs	Immediately returns on one input rejects
Promise.any()(still experimental)	Returns a single Promise which resolves as soon as one of the input promises resolves but wait for all promises to reject before reject	When two resources race to provide data (e.g., cache versus network)	Swallow rejections until all input promises reject

Next Steps

From our point of view, knowing the above should equip you to understand and use promises like a pro. Did we miss something, let us know. And, make sure to check your code on deepcode.ai.

Smart Labeling of Suggestions

cu_0xff 🇪🇺 — Fri, 24 Apr 2020 12:14:00 +0000

When using DeepCode, you might see the following effect:

On the first glimpse, it looks like the same suggestion. The difference is actually in the tags. One of them is Info and one is InTest. How comes?

Suggestions in Test Code

When you run tests, you probably want to to check for unsupported conditions and when mocking environments, you will use static elements. As an example, you can test your code by providing a wrong type to a function call, or mocking by using a hard-coded password. Obviously, DeepCode’s rule would point out this problem.
While we are providing the option to exclude certain directories from a scan using .dcignore (see here ) but what if you scan a repo for the first time. It might clutter the whole result set. So, DeepCode applies some smartness to flagging suggestions.

Smart Flags

DeepCode tries to understand if the code is actually part of the test suite. This is done by applying some rules including scanning the path and file name. Since DeepCode cannot finally decide if you want to skip the test code analysis results. If you see the example above, it might make sense to touch the tests, too. Therefore, we add a flag that enables you to decide to have testing code in or not.

Summary

A best practice is to keep all results in but watch out for the InTest flag. If you see a security issue like a hardcoded password in a test, you can relax. But sometimes, you will find nuggets in your test code that you should have a look at because it might break your tests. Btw, simply clicking on a tag, makes it a search query. If you click on the filters in the top right corner, you can deselect InTest to suppress all suggestions in test files.

Ability to Infer Types or What Python Got to do With It

cu_0xff 🇪🇺 — Wed, 22 Apr 2020 08:01:05 +0000

DeepCode recently added a feature to infer types from hints in source code. For "hard-typed" languages like Java, this is pretty straight forward. For languages like Python, it is not. Let us start with a small example and dig into the details after that.

Example First

Our example below uses Python and if you want to follow along, you can use this link.

The suggestion says that a variable that is a list might be accessed using the index operator by providing a string. While this would normally lead to a crash, in our case since we try except pass ... well, it is a kind of creative use of the exception mechanism. I guess not a best-practice. Let us unveil what happens here.

As you can see the variable params is initialized using the function get_params() (see line 113). Since later on we are using the index operator [] with a string (see lines 119, 123, and 127), we would need this to be a dictionary (signalled by {}) or something along that lines.

Let us have a look at get_params() now:

As you can see, line 81 the return value is the variable param which is initialized to a list in line 66. If you follow the paths, you see that param actually becomes a dictionary in line 74. But only if len(paramstring) is greater or equal to two. Otherwise, the param stays a list and we will see the creative use of the exception mechanism kicking in. I spare any comment if someone should use exceptions in that regard and focus on what happens from a language point of view.

Behind the Scenes

The DeepCode engine analysed that piece of code and inferred that the return value of get_params() can be list or dictionary. In Python, is less strict with types like Java is. And in Python, it is quite common to use different types for the same variable to signal different states. Following the flow of the application, DeepCode noticed that later an access mechanism is used that is not compatible with all possible values. This is what is mentioned in the suggestion.

The DeepCode engine solves several tasks here: First, the possible types of an element in memory. Next, the path this element takes within the code. It might be — just as in our case — that variables get assigned to results of function calls or other variables. Last, DeepCode checks the usage of these elements for example in function calls or operators. The engine learns which types are acceptable where by using a combination of open-source code usage, documentation scanning, and human engineers reviewing the rules. Augmented AI at its best.

This feature is especial interest for developers using languages like Python or JavaScript. In these languages, variables can change type. A lot of developers put this to great use as the type is used as a flag to carry information. But sometimes, you can entangle yourself and these faults are hard to trace. In the above’s example, our developer — instead of making a call and catch the exception — could have delivered a None back if there is no parameter, check for this case and act accordingly.

We added several new rules that find patterns like passing linear data structures into **kwargs or map data into *args, attribute address on primitive values, mutation of immutables, iterating over non-iterables, and more.

In DeepCode, we are committed to serve the developer community to achieve the best code quality possible. We provide our service for free for teams below 30 people and for open source projects. Simply, check it out at DeepCode.AI

Stay Home, Stay Safe, Code On

cu_0xff 🇪🇺 — Mon, 20 Apr 2020 09:32:09 +0000

We from DeepCode are committed to our developer community! As seen on Computerworld or Heise, we keep on to provide our service free of charge plus extended it to teams larger than 30 people.

We are encouraging all developers to do the right thing and stay home to stay safe. Also, code on as the world needs more good software than ever. Let us lend you a hand to be productive early on by using our Visual Studio Code or Atom Package. Make sure to use DeepCode so your code stays safe, too.

We are also here to help whenever you need us. Send us an email on hello@deepcode.ai or visit our website to chat, DM on @DeepCodeAI, LinkedIn … just no office visits for now :-)

DeepCode's Top Findings #13: Writing Immutable Objects in Python

cu_0xff 🇪🇺 — Fri, 10 Apr 2020 08:57:49 +0000

DeepCode offers an AI-based Static Program Analysis for Java, JavaScript and TypeScript, C/C++ and Python. As you might know, DeepCode uses thousands of open source repos to train our engine. We asked the engine team to provide some stats on the findings. On the top suggestions from our engine, we want to introduce and give some background in this series of blog articles.

Language: Python
Defect: X is an immutable object and one of its elements gets assigned
Diagnose: The code tries to write to an element within an instance of an immutable data type

I found this example in monkeylyf / interviewjam and as usual, you can open the repo in DeepCode and follow along. Below is the snippet of the dashboard in DeepCode:

def findCelebrityWithExtraSpace(self, n):
    """
    :type n: int
    :rtype: int
    """
    stack = range(n)
...
            stack[-2] = a
...

DeepCode gives us the feedback "Trying to store a value in element of immutable type range (from call to range) will lead to a crash." But does range() not bring a list back? Well, that used to be...

Let us replay what happens in the Python interpreter:

Python 3.8.1 (tags/v3.8.1:1b293b6, Dec 18 2019, 22:39:24) [MSC v.1916 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> stack = range(10)
>>> stack
range(0, 10)
>>> a = stack[-1]
>>> stack[-2] = a
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'range' object does not support item assignment
>>>

What we are calling here is the range() function. In Python3 it is a renamed version of Python2's xrange() function. This means it does not actually produce a list with a sequence of values. Rather, it uses yielding to produce the specific value when it is needed instead of storing vast amounts of data. That is the reason why Python says the range object does not support item assignment. And by the way, it also does not support the pop() function which is used in the code example.

In the above code example, the application would crash when trying to write into the range object. Yet, as ranges are used often as a sequencer for loops, ranges can get quite large. Thus it makes sense to use such a range object instead of a list. Yet, if you need the list as a datastore - such as in our example here - there is an easy way to get lists with value sequences:

stack = list(range(n))

For the static code analysis, the task here was to understand the underlying type returned by the range() function and infer the abilities the type has (like supporting index operators).

Test it yourself and run analysis over your code. It is very fast and free to check out at deepcode.ai.

Java Syntax Versus Semantics by Using Some GitPod...

cu_0xff 🇪🇺 — Thu, 02 Apr 2020 06:38:12 +0000

DeepCode offers an AI-based Static Program Analysis for Java, JavaScript and TypeScript, C/C++ and Python.

Let us have fun and do something cool to start, just follow along and I promise you will enjoy it. No need to install anything on your machine.

Step 1: Let us open the project in an IDE

First, let us open the project in a Visual-Studio-Code-like IDE. You can do so by simply clicking on the following link:

https://gitpod.io/#github.com/confluentinc/ksql

You might have to log in with you GitHub-Account. Then it will open the repo on GitPod. This means GitPod builds a container with the repo for you and provides a browser-based IDE on top. It only takes a few seconds.

Step 2: Get DeepCode analysing the repo

Now, simply follow this link

https://github.com/DeepCodeAI/vscode-extension/releases

You will find our latest release for the Visual Studio Extension (the vsix-file) there. Just download it somewhere on your machine where it comes handy for later drag-n-drop.
Back on your GitPod, find the Extensions Icon (shown below) and click on it

In there, you have two options: Either install for the project only (upper box) or for your whole account (lower box). The decision is yours in which field you want to drag-n-drop the vsix-file that we just downloaded.
As soon as you did that, the extension is uploaded and you will the following message on the lower right side:

Please click Confirm and start analysis to start the analysis. It should take only a few seconds.
Cool, you made it.

Step 3: Let's inspect the code

Within a few seconds, DeepCode should have done the analysis and you can find the results listed in the Problems tab:

Now, let us talk about one issue at hand here. By following the red circles on the icons, navigate to the following file: ksqldb-engine/src/main/java/io/confluent/ksql/function/UserFunctionLoader.java

In line 87, you find the following problem:

The code highlighted is shown below:

...
  public void load() {
    // load functions packaged as part of ksql first
    loadFunctions(parentClassLoader, empty());
    if (loadCustomerUdfs) {
      try {
        if (!pluginDir.exists() && !pluginDir.isDirectory()) {
          LOGGER.info(
              "UDFs can't be loaded as as dir {} doesn't exist or is not a directory",
              pluginDir
          );
          return;
        }
...

Step 4: What do we see here?

The if-conditional is the interesting part here. It seems the developer wanted to check if the pluginDir exists and if it is a directory. What he did was that first, it tests if pluginDir does not exist and that it is also not a directory - which in the case it does not exist, is kind of trivial.
From a pure syntactical perspective, there is nothing wrong with this code. But from a semantical perspective, it looks fishy. DeepCode's suggestion highlights this.

The strength of DeepCode showing here is the knowledge of the relationship between exists() and isDirectory(). Not exists() defines the result of isDirectory(). DeepCode learned that a negative result of the first automatically results in a negative result in the latter. Therefore, the logical combination we see in this conditional makes no sense.

DeepCode, therefore, argues that the code pretty likely does not reflect the intention of the developer.

We are happy to help! Try it yourself on your code by using our web-dashboard at deepcode.ai or using one of our plugins for Visual Studio Code or Atom.

DeepCode's Top Findings #12: Integer Promotion on Bitwise Operations in C

cu_0xff 🇪🇺 — Thu, 19 Mar 2020 08:14:22 +0000

Language: C
Defect: Integer Promotion on Bitwise Operations
Diagnose: Cast the result of shift left to unsigned short to avoid unexpected behavior because of integral type promotion. The shifted expression is promoted to unsigned int, which may introduce a number of unknown bits.

This example is sponsored by Linux in the Alpha architecture (see here ). Obviously, you can load them also in your own dashboard.

So, here is the code:

static __inline__ int get_dma_residue(unsigned int dmanr)
{
    unsigned int io_port = (dmanr<=3)? ((dmanr&3)<<1) + 1 + IO_DMA1_BASE
                     : ((dmanr&3)<<2) + 2 + IO_DMA2_BASE;

    /* using short to get 16-bit wrap around */
    unsigned short count;

    count = 1 + dma_inb(io_port);
    count += dma_inb(io_port) << 8;

    return (dmanr<=3)? count : (count<<1);
}

What we should observe is there count variable which type is unsigned short. The function returns the type int and in the return statement, we have a conditional that either returns count(which would need a typecast to int) or a bitwise shift left of count.

The type conversion rules for bitwise operations in C are actually not that straight forward. The long explanation - as pointed out by DeepCode with the link More Info - is here. Under the hood, C converts operants oftentimes to int, applies the operation, and truncates the result to fit into the target variable. Bitwise operators are such an example.

Given the specific architecture, the implicit typecast between unsigned short and int might just work out perfectly. I would argue it is hard to follow the intent of the developer and he was aware (therefor the comment). Still, following the best practices, he should do explicit type conversions (see also the link above).

DeepCode - UX updates...

cu_0xff 🇪🇺 — Tue, 10 Mar 2020 15:42:18 +0000

Hey,
we constantly get feedback that we incorporate into the product. Based on your feedback, we changed some aspects on our UX and we will do some more in the coming days. Here is a first glance...

Suggestions Page

This is the new layout of the Suggestion page. On the top left (Number 1), you see the current repo and you can select which branch to work on using the drop down. Mid top (Number 2), you find how many different suggestions and their category were found. Top right (Number 3), you can find how long it took DeepCode to scan and what amount of files were covered. By clicking on File Coverage, you can drill down on what file types were scanned or not. Lastly (Number 4), we have the typical Suggestion tiles.

Suggestion Detail Page

Also a few changes in Suggestion Details. We implemented the option to flick through the suggestions quickly (top right, Number 1) to get an overview. Also, we included help to add the ignore comments (lower right, Number 2). As you might know, you can switch off suggestions made by DeepCode by using comments. We provide these comments that would switch off the suggestion.

We hope you find these changes helpful. Also, check out the Visual Studio Code Extension and our Atom Package. But - for sure - check us out at deepcode.ai.

DeepCode's Top Findings #11: Synchronizing Strings

cu_0xff 🇪🇺 — Tue, 10 Mar 2020 06:12:32 +0000

DeepCode offers an AI-based Static Program Analysis for Java, JavaScript and TypeScript, and Python. As you might know, DeepCode uses thousands of open source repos to train our engine. We asked the engine team to provide some stats on the findings. On the top suggestions from our engine, we want to introduce and give some background in this series of blog articles.

Language: Java
Defect: Do not use Strings for synchronization.
Diagnose: Do not use Strings for synchronization, this could lead to unwanted deadlocks and race-conditions. Strings with the same value might be represented as the same object in the JVM.

I found this one in the JDK 14 code. I copied parts of it out to this repo here https://github.com/CU-0xff/jdk14_parts to make things manageable. So, if you want to follow along, open that repo in DeepCode.ai. It is a really nasty one, as it forces you to know some of the implementation details of the JVM to understand the problem. Here is the code of interest:

class WindowsPath implements Path {
...
    // normalized path
    private final String path;
...
        // cache the resolved path (except drive relative paths as the working
        // directory on removal media devices can change during the lifetime
        // of the VM)
        if (type != WindowsPathType.DRIVE_RELATIVE) {
            synchronized (path) {
                pathForWin32Calls = new WeakReference<String>(resolved);
            }
        }
...

Background:

Synchronized

Let us introduce the idea of synchronized() first. In a multi-threaded environment, we have the issue to ensure that access to variables is managed properly. Java provides synchronized as functions or blocks to manage concurrent access. Below is a simple example:

class ListOfNumbers {
    int number;

    void printListOfNumbers(final int n) {
        this.number = 0;
        for (int i = 1; i <= 10; i++) {
            try {
                // Simulate some substantial work
                Thread.sleep(400);
            } catch (final Exception e) {
                System.out.println(e);
            }
            this.number += n;
            System.out.print(String.format("%d ",this.number));
        }
    }
}

class MyThread extends Thread {
    ListOfNumbers l;
    int flag;

    MyThread(final ListOfNumbers l, int flag) {
        this.l = l;
        this.flag = flag;
    }

    public void run() {
        l.printListOfNumbers(flag);
    }
}


class TestSynchronization1 {
    public static void main(final String args[]) {
        final ListOfNumbers obj = new ListOfNumbers();// only one object
        final MyThread t1 = new MyThread(obj, 1);
        final MyThread t2 = new MyThread(obj, 7);
        t1.start();
        t2.start();
    }
}

Here we have a class ListOfNumbers, that stores an integer and prints a series of it by looping and adding it. Now, we have two threads that are working on one object instance of ListOfNumbers and it happens as you would expect it. We would like to see 1 2 3 4 5 6 7 8 9 10 7 14 21 28 35 42 49 56 63 70on the console, but we get 8 8 9 16 24 24 25 32 39 39 46 47 55 55 62 62 69 69 76 77, then 1 1 9 9 17 10 24 25 32 32 40 40 48 48 55 56 57 57 65 65, then...

This teaches us two things: (1) Obviously, the two threads are overwriting each other as they are pointing to the same object. (2) The outcome is unpredictable as it depends on the sequence of how the threads are called.

With a simple change, we can sort the issue:

class MyThread extends Thread {
...
    public void run() {
        synchronized(l) {
            l.printListOfNumbers(flag);
        }
    }
...

With this, a thread asks for control over the instance before changing its state by calling the method. So far, so good.

Why not using `synchronized()` with strings?

In Java, Strings are immutable; they cannot be changed. If you overwrite an existing one, you will generate a new instance. There are several reasons why the language designers decided to do so, amongst them is that Strings are thread-safe (as they do not change, several threads can use them concurrently). But there is something that is called String Pool. Under the hood, the JVM optimizes the data usage. If the program asks for a string to be created and one with the same content already exists in the pool, the JVM returns the already existing instance instead of creating a new one.

Now, you get the larger issue of using a String in synchronized. You could end up locking your application because two strings have the same content (aka point to the same object) but within the code, they are different variables. Good luck debugging this one. By the way, see the DeepCode-provided explanation in this regard. Spot on I would say. In our example above, the path variable is a string and it is not guaranteed to be unique.

If you are using synchronized(), make sure to use an object that is not a string. One way for our example could be to use synchronized(this) as it would use an instance of a complex class.

This example also shows the difference between dynamic and static analysis. In the dynamic case, we would have to have the case of having two strings with the same content as a test case or by running the system, falling over it. In the static case, we test if the case is within all possible cases and ring the bell.

As you see from the text, DeepCode found this issue fixed in 57 projects, so it is not that uncommon. Have a look at your own code on deepcode.ai.

DeepCode's Top Findings #10: Confusing Use of '!'

cu_0xff 🇪🇺 — Wed, 04 Mar 2020 06:27:19 +0000

Language: JavaScript
Defect: Confusing Use of '!' (Category Defect)
Diagnose: Negation applied before instanceof. Check operator priorities - negation has higher priority than instanceof.

This instance is sponsored by npm cli - as usual, you can open the project in deepcode and follow along.

Have a look at the code below:

  if ((!strings instanceof Array))
    throw new TypeError('argument must be an Array[String]');

For me, this is a typical issue of syntax versus semantics. As from a syntactical perspective, this is fine code, it compiles, and kind of works. But seeing the double parenthesis, I guess the developer intended a different behaviour. Per definition, the condition will always be false (see MDN see Not Instance Of). It is pretty likely the developer made a mistake here.

DeepCode correctly provides an explanation of the problem. The negation operator (!) has a higher priority than instanceof and is there executed before. ! reviews the operand and if the value is truthy (see MDN Truthy ) negates that and delivers a false. Otherwise a true. Then comes instance of and obviously, both of them can never be an Array. The condition will always be false. The code should look like this:

  if (!(strings instanceof Array))
    throw new TypeError('argument must be an Array[String]');

DeepCode has seen this in 202 projects and also provides examples below. For me, these examples are extremely helpful as they nicely show the before / after. Especially when parenthesis and their ordering comes into play, I tend to get blind sometimes. So this helps enormously.

If you have another 5 minutes to spare, have a look at this table here MDN Operator precedence. Always good to refresh things a bit. My general advise: Instead of trusting on operator precedence wizardry - use correct parenthesis :-)

And if you are not sure if you have seen this in your own code, give it a check - deepcode.ai.

DeepCode's Top Findings #9: Deadlocks

cu_0xff 🇪🇺 — Wed, 26 Feb 2020 06:28:41 +0000

Language: Python
Defect: Deadlock (Category Defect)
Diagnose: Using wait() can deadlock if the child process prints larger output. Use communicate().

This finding is sponsored by an ansible extension for spacewalk. As usual, you can open it in (deepcode.ai)[https://deepcode.ai]. The repo is ansible/ansible.

def spacewalk_report(name):
    """Yield a dictionary form of each CSV output produced by the specified
    spacewalk-report
    """
    cache_filename = os.path.join(CACHE_DIR, name)
    if not os.path.exists(cache_filename) or \
            (time.time() - os.stat(cache_filename).st_mtime) > CACHE_AGE:
        # Update the cache
        fh = open(cache_filename, 'w')
        p = subprocess.Popen([SW_REPORT, name], stdout=fh)
        p.wait()
        fh.close()

    ...

Let us follow along with the code above: First, it generates a filename and checks if either the file does not exist or is outdated. If so, it generates the file. Then it asks subprocess to spawn a command and write the results into the file as the output pipe. Then it waits until this is done and closes the file. The rest is not of importance to us for now.

Background:

It made me think about pure functions. Let me explain a bit. A pure function is defined by two aspects: First, it returns the same value whenever it is called with the same parameters. So, it does not depend on the state of the program or environment. Secondly, it has no side-effects, so no changes to non-local variables or I/O streams. One nice thing on these pure functions is that we can easily test them, right? We can write unit tests easily for example.

I guess the problem is that we are actually paid for the not pure parts of our applications. Changing the external state by writing data or switching off the oven is what our customers actually want. We hardly can stay in a pure world only.

The unpure world though is a jungle, my friends. Because unpure means physics kicks in. As an example: In the theory of relational databases no indexes are needed. Because in theory, the access time is zero. In reality, hard drive heads need to move and that takes time. So, indexes speed up access time.

In our little example, we are calling an external command and wait for it to finish. I mean what could go wrong?

A deadlock means that two processes are waiting on different resources for each other. The child waits for the parent to do something, while the parent waits for the child. Both of them are in a stalemate. As you can see, DeepCode warns us that wait can deadlock the child process. In between child and parent, there is a limited pipe I/O buffer. Given the child would fill this buffer with data, the system would force the child to wait until the parent process reads off significant amounts of data from the buffer. With typical mutexes, the developer is normally aware of deadlock situations but with these I/O buffers, developers tend to forget about it.

It was found to be an issue that was corrected by 41 projects out of our training set. Actually, if you click on more info, it brings you here which is the official Python documentation telling you the same. It states the same as DeepCode: To use Popen.communicate() instead. Also, see the examples DeepCode shows to see how to use the command. The communicate method takes responsibility for avoiding deadlock; it only sends the next chunk of the stdin string when the subprocess is ready to read it, and it only tries to read the next chunk of stdout or stderr when the subprocess is ready to provide it.

Don't get me wrong: communicate is not the silver bullet. You should think about the size of your data and choose the proper means to communicate between the sub-processes.

Yet, there is even more to this story: Obviously, we have no error management here. No time out, no way to find out that the hard-drive just filled up (yeah, physics with these inconvenient issues :-) ). That is simply a bad style.

Whenever interacting with the external, unpure, world, make sure to ask yourself these questions:
(1) Do I call something that might not come back in time or maybe never?
(2) Do I receive data from an external, untrusted source?
(3) What error conditions might arise resulting from the size or makeup of the data, timing issues?

With the list in hand, decide what aspects to address in code by handling them and whatnot (as it is too unlikely or pain-versus-gain does not favour a handler). But make sure, you write them down as not handled error cases.

Next, maybe you want to add some testing on the ones you addressed.

Introducing: The DeepCode CLI

cu_0xff 🇪🇺 — Tue, 25 Feb 2020 08:35:49 +0000

Hey,

DeepCode provides plugins and extensions for Visual Studio Code and Atom. But what about including it into your CI pipeline? Well, there comes the CLI quite handy.

System: Ubuntu 18.04.4 LTS

Preparations:

There are a few steps I would suggest to do before installing the CLI.

(1) Make sure you have Python 3 and Pip3 installed.

~$ python3 --version
Python 3.6.9
~$ python3 -m pip --version
pip 9.0.1 from /usr/lib/python3/dist-packages (python 3.6)

If it is not installed, do so by calling:

sudo apt install python3.8
sudo apt install python3-pip

Note: You might want to use a separate environment (see pyenv) to separate it and keep things organized. With this, you can easily switch between environments - one of the great strengths of Python.

(2) Get an API key. On your DeepCode Account Dashboard you can generate new access tokens (scroll to the bottom).

(3) You can store the API key in the config file so you do not have to handle it again. You can also provide the API key using the -a parameter when calling the CLI. To store it in the config file, use the editor of your choice and generate a text file with the following content:

{"api_key": "[Your key goes here]"}

Per default, the name .deepcode.json is used.

Note: The CLI also supports the command config that records this file for you.

Installing DeepCode CLI:

(1) Installation is rather simple

pip3 install deepcode

You might have to restart the shell so that the path is set correctly and the deepcode command is found.

First Look Around:

Note: The parameter --help is always there for you.

In essence, the CLI supports three main commands: analyze, config, login.

With deepcode login, the command starts a browser and guides you through the login process. As we have an API key already, we do not need this for now.

deepcode config asks you two main questions (the service URL and the API code) and writes them to the default config file .deepcode.json. The service URL (default is DeepCode's internet-facing service) enables you to point towards your internal DeepCode server. The API key is the key you gathered as described above.

deepcode analyze provides the interface to send your code for analysis. There are three main things to be aware of:

Sources: You can provide a local path using the -p option or a remote, GIT repository using the -r option.

Linters: Providing the -l option asks DeepCode to run the optional linters. One word of warning: The speed of DeepCode analysis is way(!!) faster than the linters. Sending big chunks of source code may slow the answer down significantly when enabling the linters.

Output Format: Per default DeepCode returns a JSON format described here: REST API Bundles (see GetAnalysis Results). It has the following structure:

{ "id": string,
  "results": {
  "suggestions": {
    suggestionIndex: {
      "id": string,
      "message": string,
      "severity": 1 | 2 | 3
    },
    ...
  },
  "files": {
    filePath: {
      suggestionIndex: [
        {
          "cols": [number, number],
          "rows": [number, number],
          "markers": [marker, ...]
        },
        ...
      ],
      ...
    },
    ...
  }
 }
 "url", url
}

To map the many-to-many relationship between suggestions and files, each suggestion is identified by a short suggestionIndex string (e.g. "0", "1", etc). Every suggestion object contains the respective id, the description message, and a numeric severity (1 for Info suggestions, 2 for Warnings, or 3 for Critical issues). Each file is identified by its path and maps all the matching suggestions with the respective array of positions in the files. A position uses two ranges [ from , to ], one for the rows and one for the columns (cols), to describe an area of the file included between two characters. Each position also has an array of markers with the following structure:

{
  "msg": [number, number],
  "pos": [
    {
      "cols": [number, number],
      "rows": [number, number]
    },
    ...
  ]
}

Each marker object maps a substring of the suggestion's message, identified by the msg property as a [ from , to ] range between two characters, to an array of positions in the code (pos) where that substring has a meaningful match.

Additionally, you can find a unique identifier (id) and the URL (url) under which you can find the DeepCode dashboard for your request. The URL is valid for a day from the time you ran the analysis.

On top, the CLI provides exit codes:

0 - no issues found
1 - some issues found
2 - Execution was interrupted by the user
3 - Some error happened while executing

Example:

When running this command, you request a public demo repo from GitHub to run which results in three suggestions:

deepcode analyze -r https://github.com/CU-0xff/deepcode-vuln-logviewer.git@cu-0xff

Resources:

PyPI deepcode CLI
Developer docu

DEV Community: DeepCode.AI

JavaScript Promises: All you need to know

Why are promises needed?

Basic Functionality

Usage

async and await

Chaining of Promises

Combination of Promises

Next Steps

Smart Labeling of Suggestions

Suggestions in Test Code

Smart Flags

Summary

Ability to Infer Types or What Python Got to do With It

Example First

Behind the Scenes

Stay Home, Stay Safe, Code On

DeepCode's Top Findings #13: Writing Immutable Objects in Python

Java Syntax Versus Semantics by Using Some GitPod...

Step 1: Let us open the project in an IDE

Step 2: Get DeepCode analysing the repo

Step 3: Let's inspect the code

Step 4: What do we see here?

DeepCode's Top Findings #12: Integer Promotion on Bitwise Operations in C

DeepCode - UX updates...

Suggestions Page

Suggestion Detail Page

DeepCode's Top Findings #11: Synchronizing Strings

Background:

Synchronized

Why not using synchronized() with strings?

DeepCode's Top Findings #10: Confusing Use of '!'

DeepCode's Top Findings #9: Deadlocks

Background:

Introducing: The DeepCode CLI

Preparations:

Installing DeepCode CLI:

First Look Around:

Example:

Resources:

Why not using `synchronized()` with strings?