DEV Community: Elemar Júnior

How Static Analysis Tools are Helping me to Produce Better Software: Understanding the Design

Elemar Júnior — Sat, 05 May 2018 15:46:50 +0000

As a consultant, I often need to work with the code that I do not know. I need to understand design decisions that were not mine. I need to get acquainted with architectures I did not help to conceive. Static analysis tools, like NDepend and Visual Studio Code Map, help me do my job more effectively and efficiently.

Today, I am starting a series where I will share how Static Analysis tools are helping me to produce better software. In this post, I will share with you my first steps when getting in touch with some unfamiliar code base.

STEP #1: Recognize the components and their relationships

As an example, let’s assume that I need to start to work with the Microsoft’s eShopContainer project.

Just after downloading the code and get it up and running, my first step would be to generate a Code Map from the solution (using Visual Studio).

This simple view allows me to get a first understanding of the software components and their relationships. By default, Visual Studio uses the solution’s folder structure to group the projects.

Note that there are no “explicit dependencies” between the web apps and the services. Also, there are no “explicit dependencies” between the services as well. The reason why this is happening is that the connection only exists in run-time, via HTTP/messaging (which is good, considering this as a Microservices architecture implementation). But it requires me to talk with people to understand the business.

This view also helps me to discover some “technical complexities” added to the project. For example, there are at least two implementations of EventBus. Is that needed? What are the business motivations to this? In the “real world” would exist two “release” strategies, two support strategies, … (I know this is a sample project, but, …)

STEP #2: Discover the size of each component

My second step would be to generate a Dependency Graph with the box sizes representing the number of lines of code (using NDepend).

Now I have a good idea of the proportional size of the projects. Also, I can estimate the amount of time I would need to read the code (It’s not precise, but generally it is good enough).

This graph has been helping me to identify disproportional sizes in the components of the applications. Disproportionally large assemblies are a good indication of poor architecture/design choices. For example, it is strange to have a “so big” identity project. Also, it is strange that the Ordering API is twice more prominent than the Ordering Domain and Ordering Infrastructure. Would be the API doing more than it should?

In the “real world,” I have seen “big” assemblies as a symptom of coupling and difficulties to meet the business needs.

STEP #3: Discover the core (hard to change) components

My third step would be to generate the Dependency Graph, this time using the afferent coupling to define the box sizes.

Analyzing the resulting graph, I know that the EventBus is a core component and concept in this project. Also, I am aware that changing EventBus is dangerous because it can affect all the system.

Also, at this moment, I start to think and ask about the “choreography” of the services. It seems that there are no “Orchestrators” here. Would I be right?

It is important to remember that this graph does not include “implicit dependencies.” When analyzing code, no tool will do all your job. Charts, reports, matrices will only help you to be more precise and to save (some) time. You will still need to talk with the domain people.

STEP #4: Get a better (detailed) understanding about the relationships

My fourth step would be to generate a Dependency Matix.

The blue cells indicate that the assembly in the column is using the assembly in the row. The green cells suggest that the assembly in the row uses the assembly in the column.

This matrix allows me to see that all the web projects are using the “Microsoft.AspNetCore.HealChecks” package. Nice, huh?!

In the real-world, I have seen assemblies with thousands of dependencies to another one.

STEP #5: Create a list of critical issues

Finally (for a while), I use NDepend to check critical potential problems in the code.

NDepend uses generic rules to try to identify pain points in the code. It is usually really efficient.

The “avoid methods too big …” rule is the more commonly violated one.

Last words

In this post, What I recommend you do when getting in touch with unfamiliar code using proper tools. Next post, I will start sharing some techniques I use to dig deep into the code discovering and removing pain points.

Could I help you? Let me know.

The post How Static Analysis Tools are Helping me to Produce Better Software: Understanding the Design appeared first at Elemar JR.

Is the Managed Code Slower than the Native Code?

Elemar Júnior — Thu, 19 Apr 2018 16:45:12 +0000

That is a question that I have been answering for years. The answer is an emphatic “NO” in most cases.

What is true is that .NET make it very easy to write slow code. Being more friendly, it is pretty standard for .NET developers being sloppy and uncritical about performance and that, I think, is the main reason why some .NET applications performance sucks.

RavenDB team is doing a great job proving, again and again, that managed code can be so fast as Native Code. Why? Because we pay attention to performance starting from the architecture to the implementation.

When you build your .NET code, the compiler translates it to Intermediate Language and metadata. Then, when you run your code, whenever a method is executed for the first time, the Just in Time Compiler convert it to assembly code. Most of the code optimization happens at this stage (If you want to understand this process, I recommend you to read this post).

The quality of the JITted code is fantastic. Microsoft has improved the JIT, especially quite recently.

In fact, there are advantages in the Managed World:

Since the code is JITted as it is executed, its location in memory is better than that of native code. The related code will be often closer, commonly in a single memory page or the cache. So we have fewer page faults;
Different from Native Code, .NET applications are less susceptible to memory fragmentation and. When it happens, the Garbage Collection compacts the heap solving the problem.
The memory allocation occurs with no contention.

Remember: Garbage Collection in .NET is deterministic. We can significantly affect how often it runs by controlling the memory allocation patterns, object scope, etc.

Is the Managed Code Slower than the Native Code? appeared first at Elemar JR.

3 Mistakes Developers Make that Prevent Them from Improving the Performance of Applications

Elemar Júnior — Fri, 23 Mar 2018 10:20:15 +0000

In these days, performance is a feature! So, you should start, right now, to refactor your code to achieve better performance. Right? No. you shouldn’t!

Before starting to change your code, take some time and think. Be sure you are not making one of the following mistakes.

Mistake #1 – Do not define what is important

What do you mean by performance? Are you looking for better use of the CPU? Are you trying to reduce the latency (time to run a single operation from start to finish)? Throughput (number of operations your application can handle in a certain amount of time)? Do you need to reduce the amount of memory necessary to run? Are you interested in better disk usage? Do you want to reduce the network bandwidth required to transfer your data?

You need to define what aspects you are trying to improve. You need to be precise. The right moment is now!

Mistake #2 – Try to optimize code without a reliable measurement

No matter how experienced you are, you should NOT rely on your gut feelings.

There are excellent tools that you could and should use. You will need accurate performance metrics to get your work done. Do NOT ignore this action.

Benchmarking is hard (especially micro-benchmarking). Do NOT try to be smart creating your tools.

Mistake #3 – Do not verify the results

How do you know you succeed? The only way is measuring again and again.

One more time, no matter how experienced you are, you should NOT rely on your gut feelings.

Sometimes, to improve one aspect, you will make another worse. Double check if you are ready to handle the “improvement” consequences.

Time To Action

Again, performance is a feature. Take it seriously.

Define your performance goals precisely
Choose right performance metrics and select proper tools to provide them for you.
Double check your results

I want to help you. Let’s get in touch! Subscribe mymailing list.

O post 3 Mistakes Developers Make that Prevent Them from Improving the Performance of Applications apareceu primeiro em Elemar JR.

Implementing Parallel Reduction in CUDA

Elemar Júnior — Thu, 22 Mar 2018 04:31:39 +0000

Reduction operations are those that reduce a collection of values to a single value. In this post, I will share how to implement parallel reduction operations using CUDA.

Sequential Sum

Compute the sum of all elements of an array is an excellent example of reduction operation. The sum of an array which values are 13, 27, 15, 14, 33, 2, 24, and 6 is 134.

The interesting question is: How would you compute it? Probably your first answer would be doing something like this (((((((13+27)+15)+14)+33)+2)+24)+6). Am I right?

The problem with this approach is that it is impossible to parallelize. Why? Each step depends on the result of the previous one.

Parallel Sum

Adding values is an associative operation. So, we can try something like this ((13+27)+(15+14))+((33+2)+(24+6))

This way is much better because now we can execute it in parallel!

CUDA

Let’s figure out how to do it using CUDA.

Here is the main idea:

Assuming N as the number of the elements in an array, we start N/2 threads, one thread for each two elements
Each thread computes the sum of the corresponding two elements, storing the result at the position of the first one.
Iteratively, each step:
- the number of threads halved (for example, starting with 4, then 2, then 1)
- doubles the step size between the corresponding two elements (starting with 1, then 2, then 4)
after some iterations, the reduction result will be stored in the first element of the array.

Let’s code

#include "cuda_runtime.h"
#include "device_launch_parameters.h"

#include <iostream>
#include <numeric>
using namespace std;

__global__ void sum(int* input)
{
    const int tid = threadIdx.x;

    auto step_size = 1;
    int number_of_threads = blockDim.x;

    while (number_of_threads > 0)
    {
        if (tid < number_of_threads)
        {
            const auto fst = tid * step_size * 2;
            const auto snd = fst + step_size;
            input[fst] += input[snd];
        }

        step_size <<= 1; 
        number_of_threads >>= 1;
    }
}

int main()
{
    const auto count = 8;
    const int size = count * sizeof(int);
    int h[] = {13, 27, 15, 14, 33, 2, 24, 6};

    int* d;

    cudaMalloc(&d, size);
    cudaMemcpy(d, h, size, cudaMemcpyHostToDevice);

    sum <<<1, count / 2 >>>(d);

    int result;
    cudaMemcpy(&result, d, sizeof(int), cudaMemcpyDeviceToHost);

    cout << "Sum is " << result << endl;

    getchar();

    cudaFree(d);
    delete[] h;

    return 0;
}

Time to action

In this post, we implemented a primary example of parallel reduction operation in CUDA. But, the pattern we adopted can be used in more sophisticated scenarios.

I strongly recommend you to try to implement other reduction operations (like discovering the max/min values of an array). Now, it would be easy. Right?

Share your thoughts in the comments.

The Implementing Parallel Reduction in CUDA comes from Elemar JR.

About “Difficult Conversations” (book recommended by Phil Haack)

Elemar Júnior — Fri, 16 Mar 2018 16:57:53 +0000

Some days ago, I heard a fantastic interview with Phil Haack on the IT Career Energizer Podcast. Here is the episode description:

In this episode, Phil Haack tells us why we need to be prepared to have difficult conversations and why this can help your career. Phil also talks about the importance of taking care when writing code and why you should test your code carefully.

What amazing show!

In this interview, Phil recommended an excellent book: Difficult Conversations: How to Discuss What Matters Most. I listened to the audio version of the book, and it is impressive. I strongly recommend it.

The book gives a lot of good suggestions about how to succeed in challenging conversations. Here is a short description (from Amazon):

We attempt or avoid difficult conversations every day-whether dealing with an underperforming employee, disagreeing with a spouse, or negotiating with a client. From the Harvard Negotiation Project, the organization that brought you Getting to Yes, Difficult Conversations provides a step-by-step approach to having those tough conversations with less stress and more success. you’ll learn how to:

Decipher the underlying structure of every difficult conversation
Start a conversation without defensiveness
Listen for the meaning of what is not said
Stay balanced in the face of attacks and accusations
Move from emotion to productive problem solving

I wish I had the opportunity to read (or listen) it some years ago. For sure I would have less stressful moments.

Thanks Phil!

O post About “Difficult Conversations” (book recommended by Phil Haack) apareceu primeiro em Elemar JR.

The #1 Rule for .NET Performance

Elemar Júnior — Fri, 16 Mar 2018 14:27:48 +0000

If you ask me one tip to improve the performance of your applications, it would be:

Design your objects to be collected on gen #0 or not at all.

Naturally, following this recommendation demands you to know, at least the basics, of how the garbage collection works. But, if you are interested in improving the performance of your applications, this is necessary at all.

In fact, the garbage collector was explicitly designed to be very efficient performing gen #0.

Garbage collector gets more expensive in each generation. Even with background processing for gen #2, there is still a very high CPU cost to pay. Consider that you should avoid gen #1 collections too. In most scenarios, gen #1 objects are directly promoted to gen #2.

Ideally, every object you allocate goes out of scope by the time next gen #0 comes around. You can measure how long is that interval and compare it to the duration that data is alive in your application using tools such as PerfView (By the way, take some time and learn how to use PerfView – this is such great tool).

Time to action

Following this recommendation requires a shift in your mindset. It will inform nearly every aspect of your application (I strongly recommend you this Ayende’s talk about this topic).

Here are some guidelines to keep in mind when writing your code:

Pool Long-lived and large objects – Consider following the same patterns adopted by the Roslyn team.
Avoid copying data whenever you can. Use classes as ArraySegment and, now, the new Span<T> (Please, take some time and read this MSDN Magazine written by Stephen Toub).
Reduce References between Objects – That would make the GC work less expensive. Also, it makes easier to predict object lifetimes.
Be aware of the cost of finalizers.
Use Weak References for Caching

Isn’t that enough? In the next weeks, I will share in-depth information about how you could follow these recommendations to get improved .NET applications performance. Sign in my contacts list, and you will receive notification when new content arrives.

Last but not least, remember that Performance is a feature. If the performance of your application is not that good, chances are you missing opportunities to deliver business value.

Cover image: chuttersnap

The post The #1 Rule for .NET Performance first appeared on ElemarJR.com.

3 mistakes that Developers make that prevent them from creating software that meets the business needs

Elemar Júnior — Sun, 11 Mar 2018 14:14:12 +0000

As professionals, we should focus on developing great technical solutions. But, these solutions need to solve real problems. At the end of the day, if we don’t deliver value, no matter how much effort we made, we will fail.

Let’s see three common mistakes developers do that prevent them from meeting the business needs.

Mistake #1 – Pay attention only to the technical aspects

Yes, I know! We are technical people, and our primary concern should be technical stuff. Right? No. I don’t think so.

We are technical people, that’s true! But our mission is to use technology to solve business needs. All technology that we use that business does not recognize as a benefit is wast of time, talent, and money.

Recommended solution: Justify any technical solution you choose with a business need.

For example,

You want to reduce the cost of adding new features. So, you will adopt TDD.
You want to save user’s time. So, you use Angular/React/Vue.

Ensure that you will deliver the business benefits. If you are are not ready, don’t make promises _ and _ do not adopt the technology.

Mistake #2 – Do not (even try to) speak the language of the business

If you want to justify your technical decisions with business arguments, you need to speak the business language.

Accept this fact: YOU ARE NOT A DOMAIN EXPERT, no matter how experienced you are. If you are not in front of the business, you are not able to define what the business needs are.

Recommended Solution: Pay attention to the business vocabulary and adopt it. Develop and use a Ubiquitous Language.

Do not even try to make business people understand technical vocabulary. It is waste of (valuable) time.

Mistake #3 – Resist to accept changes in the requirements

This one is difficult. Believe me; I know how frustrating it is. Sometimes, we need to change everything. Sometimes we need to throw away many hours of work and thousands of lines of code.

Yes! The business world is dynamic and changes every time. Besides of that, people don’t look like understanding how costly it is to change software.

No matter how much effort you put to understand the requirements. When the business people start using your software, the changes will come.

What you need to understand is: It is pointless to resist the change.

Recommend Solution: Start your work knowing that the changes will come. Write code that is easy to read, understand and change. And, above all, stop segregating people. There is no “us and them.” Be professional and accept that everyone is in the same boat.

It’s time to action

Of course, this is not an exhaustive list. What we have here is just a starting point. Share with me your opinions! I want to make a step in the right direction every day, and you are welcome to help me.

Also, You can subscribe the newsletter. I will send you a lot of technical and non-technical content. Only things that are helping me to be a better professional.

Cover image from: Nathan Dumlao

O post 3 mistakes that Developers make that prevent them from creating software that meets the business needs apareceu primeiro em Elemar JR.

Getting started with CUDA (using VS2017)

Elemar Júnior — Tue, 06 Mar 2018 17:05:21 +0000

If you are interested in performance, you need to know more about CUDA.

From the official website:

CUDA® is a parallel computing platform and programming model developed by NVIDIA for general computing on graphical processing units (GPUs). With CUDA, developers are able to dramatically speed up computing applications by harnessing the power of GPUs.

In GPU-accelerated applications, the sequential part of the workload runs on the CPU – which is optimized for single-threaded performance – while the compute intensive portion of the application runs on thousands of GPU cores in parallel. When using CUDA, developers program in popular languages such as C, C++, Fortran, Python and MATLAB and express parallelism through extensions in the form of a few basic keywords.

In this post, I will guide you through your first steps with CUDA. Also, I will show you how to move some basic processing from the CPU to the GPU.

Let’s start getting some bytes.

Environment setup

I am using Visual Studio 2017 (version 15.6.0). To code with CUDA, you will need to download and install the CUDA Toolkit. The Toolkit includes Visual Studio project templates and the NSight IDE (which it can use from Visual Studio).

Also, you will need to install the VC++ 2017 toolset (CUDA is still not compatible with the latest version of Visual Studio).

Starting a new project using CUDA

The easiest way to start a project that uses CUDA is utilizing the CUDA template.

To be able to compile this, you will need to change the Project Properties to use the Visual Studio 2015 toolset.

I recommend you to clean the template’s boilerplate changing the content of the file kernel.cu to this_:_

#include "cuda_runtime.h" 
#include "device_launch_parameters.h"

Basic CPU processing

Let’s add two arrays (a and b) putting the results in a third array (c).

#include "cuda_runtime.h" 
#include "device_launch_parameters.h" 

#include <stdio.h> 

void add_arrays_cpu(int* a, int* b, int* c, const int count)
{
    for (auto i = 0; i < count; i++)
    {
        c[i] = a[i] + b[i];
    }
} 

int main()
{
    const auto count = 5; 
    int a[] = { 1, 2, 3, 4, 5 }; 
    int b[] = { 10, 20, 30, 40, 60 }; int c[count]; 
    add_arrays_cpu(a, b, c, count); 
    for (auto i = 0; i < count; i ++)
    {
        printf("%d ", c[i]);
    } 
    getchar(); 
    return 0;
}

This code is pretty simple. But it is tough to parallelize.

Whenever we want to use parallelism, we need to create functions that could be executed independently. The function add_arrays_cpu has a for-loop that runs the add process in a sequential fashion. But, there is no reason to work in this way. Let’s change it:

#include "cuda_runtime.h"
#include "device_launch_parameters.h"

#include <stdio.h>;

void add_array_element_cpu(int* a, int * b, int* c, const int index)
{
    c[index] = a[index] + b[index];
}

int main()
{
    const auto count = 5;
    int a[] = { 1, 2, 3, 4, 5 };
    int b[] = { 10, 20, 30, 40, 60 };
    int c[count];

    add_arrays_cpu(a, b, c, count);

    for (auto i = 0; i < count; i++)
    {
        add_array_element_cpu(a, b, c, i);
    }

    for (auto i = 0; i < count; i ++)
    {
        printf("%d ", c[i]);
    }

    getchar();

    return 0;
}

The function add_array_element_cpu can be executed independently, and that is great. We could start a thread for each position of the array, and that would work fine (I am not saying that would be the right thing to do with CPUs, just as an example).

Bringing CUDA to the game

The next logical step is to start using the GPU to run our code. We will not move all the functions to the GPU, but only the functions we think we could be parallelized.

#include "cuda_runtime.h"
#include "device_launch_parameters.h"

#include <stdio.h>;

__global__ void add_arrays_gpu(int* a, int *b, int* c)
{
    c[threadIdx.x] = a[threadIdx.x] + b[threadIdx.x];
}

int main()
{
    const auto count = 5;
    int host_a[] = { 1, 2, 3, 4, 5 };
    int host_b[] = { 10, 20, 30, 40, 60 };
    int host_c[count];

    int *device_a, *device_b, *device_c;
    const int size = count * sizeof(int);
    cudaMalloc(&amp;device_a, size);
    cudaMalloc(&amp;device_b, size);
    cudaMalloc(&amp;device_c, size);

    cudaMemcpy(
        device_a, host_a,
        size,
        cudaMemcpyHostToDevice
    );

    cudaMemcpy(
        device_b, host_b,
        size,
        cudaMemcpyHostToDevice
    );

    add_arrays_gpu <<< 1, count >>> (device_a, device_b, device_c);

    cudaMemcpy(
        host_c, device_c,
        size,
        cudaMemcpyDeviceToHost
    );


    for (auto i = 0; i < count; i ++)
    {
        printf("%d ", host_c[i]);
    }

    getchar();

    return 0;
}

We marked the add_arrays_gpu function as global. Using CUDA terminology, this function is a kernel, and that will be executed from the GPU. It runs on the GPU, called from the CPU (there is another qualifier location qualifier, devicethat should be used with functions that run on GPU, called from the GPU).

global functions are invoked using the special <<<…>>> syntax. The parameters we used indicates that we are running the kernel function in 1 block of count (5) threads. We will talk more about it in the future.

Note that we don’t need to inform an index anymore – we retrieve the index position from threadIdx.x. threadIdx is a special variable, provided by CUDA runtime, that informs the position of the current thread in the thread block (we are using five threads, one thread block to run our code. Because of that, we were able to use this information as the index in the array).

I would like to make a special consideration about an vital pattern that you will see a lot of times when doing CUDA programming. To run, GPU code is not allowed to access CPU memory (and vice-versa). So we will need:

copy the data we need to process from the CPU to the GPU memory
do the processing
copy results from the GPU to the CPU memory.

Final words

I know, I know! Adding two arrays with only five elements each is not an exciting example. But, in this post, we did the setup to use CUDA on our machines. Then, I helped you to move your could from sequential running on CPU to parallel on GPU. In the future, let’s return to it and get some real benefits.

The post Getting started with CUDA (using VS2017) appeared first on Elemar JR.

How to parse, simplify, differentiate and evaluate/solve (using Newton's method) equations and expressions in F#.

Elemar Júnior — Mon, 05 Mar 2018 13:48:32 +0000

This was orginally posted in my blog

The Object Model

The first step to parse expressions and equations is to create a "good-enough" object model.

type Expr =
  | X
  | Const of value: double
  | Add   of Expr * Expr
  | Sub   of Expr * Expr
  | Mult  of Expr * Expr
  | Div   of Expr * Expr
  | Pow   of Expr * Expr
  | Neg   of Expr

// 2 + 2 / 2
let sample = Add(Const(2.), Div(Const(2.), Const(2.))

It is fantastic how easy it is when using F#.

Parsing

In this implementation, I use F# Active Patterns to do the parsing.

open System

let (|Digit|_|) = function
    | x::xs when Char.IsDigit(x) ->
        Some(Char.GetNumericValue(x), xs)
    | _ -> None

let (|IntegerPart|_|) input =
    match input with
    | Digit(h, t) ->
        let rec loop acc = function
            | Digit(x, xs) -> loop ((acc * 10.) + x) xs
            | xs -> Some(acc, xs)
        loop 0. input
    | _ -> None

"10" |> List.ofSeq |> (|IntegerPart|_|)

let (|FractionalPart|_|) = function
    | '.'::t->
        let rec loop acc d = function
            | Digit(x, xs) -> loop ((acc * 10.) + x) (d * 10.) xs
            | xs -> (acc/d, xs)
        Some(loop 0. 1. t)
    | _ -> None

"10" |> List.ofSeq |> (|FractionalPart|_|)
".34" |> List.ofSeq |> (|FractionalPart|_|)

let (|Number|_|) = function
    | IntegerPart(i, FractionalPart(f, t)) -> Some(i+f, t)
    | IntegerPart(i, t) -> Some(i, t)
    | FractionalPart(f, t) -> Some(f, t)
    | _ -> None

"10" |> List.ofSeq |> (|Number|_|)
".35" |> List.ofSeq |> (|Number|_|)
"10.35" |> List.ofSeq |> (|Number|_|)

let parse (expression) =
    let rec (|Expre|_|) = function
        | Multi(e, t) ->
            let rec loop l = function
                | '+'::Multi(r, t) -> loop (Add(l, r)) t
                | '-'::Multi(r, t) -> loop (Sub(l, r)) t
                | [] -> Some(l, [])
                | _ -> None
            loop e t
        | _ -> None
    and (|Multi|_|) = function
        | Atom(l, '*'::Powi(r, t)) -> Some(Mult(l, r), t)
        | Atom(l, '/'::Powi(r, t)) -> Some(Div(l, r), t)
        | Powi(e, t) -> Some(e, t)
        | _ -> None
    and (|Powi|_|) = function
        | '+'::Atom(e, t) -> Some(e, t)
        | '-'::Atom(e, t) -> Some(Neg(e), t)
        | Atom(l, '^'::Powi(r, t)) -> Some(Pow(l, r), t)
        | Atom(e, t) -> Some(e, t)
        | _ -> None
    and (|Atom|_|) = function
        | 'x'::t -> Some(X, t)
        | Number(e, t) -> Some(Const(e), t)
        | '('::Expre(e, ')'::t) -> Some(e, t)
        | _ -> None

    let parsed = (expression |> List.ofSeq |> (|Expre|_|))

    match parsed with 
        | Some(result, _) -> result
        | None -> failwith "failed to parse expression"


parse "2+2"  // Add(Const(2.), Const(2.))
exec 0. (parse "2+2") // 4
exec 2. (parse "x^3") 
parse "x^2-3" // Sub(Pow(X, Const(2.)), Const(3.)

Simplifying and Evaluating

The following code can simplify equations/expressions removing steps to solve it.

let rec simplify e =
  let result =
      match e with
      // add
      | Add(Const(0.), r) -> simplify r
      | Add(l, Const(0.)) -> simplify l
      | Add(Const(l), Const(r)) -> Const (l + r)
      | Add(l, r) -> (Add(simplify l, simplify r))
      // sub
      | Sub(Const(0.), r) -> Neg (simplify r)
      | Sub(l, Const(0.)) -> l
      | Sub(Const(l), Const(r)) -> Const (l - r)
      | Sub(X, r) -> Sub (X, simplify r)
      | Sub(l, X) -> Sub (simplify l, X)
      | Sub(l, r) -> (Sub(simplify l, simplify r))
      // mult
      | Mult(Const(0.), _) -> Const(0.)
      | Mult(_, Const(0.)) -> Const(0.)
      | Mult(Const(1.), r) -> r
      | Mult(l, Const(1.)) -> l
      | Mult(Const(l), Const(r)) -> Const (l * r)
      | Mult(l, r) when l = r -> (Pow (simplify l, Const(2.)))
      | Mult(Pow(b, p), r) when b = r -> Pow (simplify b, (simplify (Add((simplify p), Const(1.)))))
      | Mult(X, r) -> Mult (X, simplify r)
      | Mult(l, X) -> Mult (simplify l, X)
      | Mult(l, r) -> (Mult(simplify l, simplify r))
      // div
      | Div(Const(0.), _) -> Const(0.)
      | Div(l, Const(1.)) -> l
      | Div(Const(l), Const(r)) -> Const (l / r)
      | Div(X, r) -> Div (X, simplify r)
      | Div(l, X) -> Div (simplify l, X)
      | Div(l, r) -> simplify (Div(simplify l, simplify r))
      // pow
      | Pow(_, Const(0.)) -> Const(1.)
      | Pow(b, Const(1.)) -> simplify b
      | Pow(Const(l), Const(r)) -> Const(System.Math.Pow(l, r))
      | Pow(X, r) -> Pow (X, simplify r)
      | Pow(l, X) -> Pow (simplify l, X)
      | Pow(b, p) -> (Pow(simplify b, simplify p))
      // neg
      | Neg(Const(k)) -> Const (-k)
      | Neg(X) -> Neg(X)
      | Neg(x) -> (Neg(simplify x))
      //
      | other -> other

  if (result = e)
      then result
      else simplify result

simplify (Mult(Mult(X, X), X))
simplify (Pow(Const(2.), Const(3.)))
simplify (Mult(Const(2.), X))
simplify (Add(Const(2.), Div(Const(2.), Const(2.) )))

I love local functions! The simplification process works as an evaluator for expressions. With equations, the process will stop when there are no more possible simplification steps to take.

let exec x expr = 
    let rec replaceX = function 
        | Add(l, r)  -> Add(replaceX l, replaceX r)
        | Sub(l, r)  -> Sub(replaceX l, replaceX r)
        | Mult(l, r) -> Mult(replaceX l, replaceX r)
        | Div(l, r)  -> Div(replaceX l, replaceX r)
        | Pow(l, r)  -> Pow(replaceX l, replaceX r)
        | Neg(e)     -> Neg(replaceX e)
        | Const(v)   -> Const(v)
        | X          -> Const(x)

    match simplify (replaceX expr) with 
    | Const(result) -> result
    | _ -> failwith "impossible to execute"

// resulta 8
Pow(Const(2.), X) |> exec 3.

Differentiating

Newton's method will need derivatives to work. So, let's produce it.

let rec deriv = function 
    | X          -> Const(1.)
    | Const(_)   -> Const(0.)
    | Add(l, r)  -> Add(deriv l, deriv r)
    | Sub(l, r)  -> Sub(deriv l, deriv r)
    | Mult(l, r) -> Add(Mult(deriv l, r), Mult(l, deriv r))
    | Neg(v)     -> Neg(deriv v)
    | Pow(b, e)  -> Mult(e, simplify (Pow(b, Sub(e, Const(1.)))))
    | _          -> failwith "expression not supported."

deriv (Pow(X, Const(3.)))

Welcome, Mr. Newton

We now already have all the elements we need to solve equations using Newton's method.

Here is my implementation.

let newton expr guess error maxdepth =
    let o = parse expr
    let d = deriv o
    let eq = Sub(X, Div(o, d))

    let rec iter g depth =
        if depth > maxdepth
        then failwith "maxdepth exceeded."
        else 
            let newg = exec g eq
            printfn "%A" g

            if (abs (newg - g) < error)  
            then newg 
            else iter newg (depth + 1)

    iter guess 0

newton "x^3-27" 5. 0.000001 100 // 3

The parameters are the equation we need to solve, a solution guess, an acceptable error, and iterations.

We can make it simpler to use:

let solve expr = 
    newton expr 100. 0.00001 100

solve "x^2-9"        // 3
solve "3*x^2-4*x+1"  // 1

That's it

This post was written just for fun. I did it trying to learn F#. Do you like? Could I do something different? Please, give me your feedback.

Computing the Levenshtein (Edit) Distance of Two Strings using C#

Elemar Júnior — Sat, 03 Mar 2018 17:29:51 +0000

Sometimes it is not enough to know if two strings are equals or not. Instead, we need to get an idea of how different they are.

For example, ant and aunt are two different words. However, not as different as ant and antidote. The “edit distance” between ant and aunt is smaller than the “edit distance” between _ant and antidote. _ By edit distance, I mean the number of edits (removals, inserts, and replacements) needed to turn one string into another.

In this post, I share an implementation of the Levenshtein’s algorithm that solves the edit distance problem.

The Levenshtein Algorithm

In 1965, Vladimir Levenshtein created a beautiful distance algorithm. Here is a C# implementation.

using System;

public static class Levenshtein
{
    public static int Compute(string first, string second)
    {
        if (first.Length == 0)
        {
            return second.Length;
        }

        if (second.Length == 0)
        {
            return first.Length;
        }

        var d = new int[first.Length + 1, second.Length + 1];
        for (var i = 0;
            i <= first.Length;
            i++)
        {
            d[i, 0] = i;
        }

        for (var j = 0;
            j <= second.Length;
            j++)
        {
            d[0, j] = j;
        }

        for (var i = 1;
            i <= first.Length;
            i++)
        {
            for (var j = 1;
                j <= second.Length;
                j++)
            {
                var cost = (second[j - 1] == first[i - 1]) ? 0 : 1;
                d[i, j] = Min(d[i - 1, j] + 1, d[i, j - 1] + 1, d[i - 1, j - 1] + cost);
            }
        }

        return d[first.Length, second.Length];
    }

    private static int Min(int e1, int e2, int e3) => 
        Math.Min(Math.Min(e1, e2), e3);
}

And here it is some tests to ensure this implementation works:

public class LevenshteinShould
{
    [Theory]
    [InlineData("ant", "aunt", 1)]
    [InlineData("fast", "cats", 3)]
    [InlineData("Elemar", "Vilmar", 3)]
    [InlineData("kitten", "sitting", 3)]
    public void ComputeTheDistanceBetween(string s1, string s2, int expectedDistance)
    {
        Assert.Equal(expectedDistance, Levenshtein.ComputeDistance(s1, s2));
    }
}

How it works

Levenshtein is a dynamic programming algorithm.

The main idea is to fill the entries of a matrix m, whose two dimensions equal the lengths of the two strings whose the edit distance is being computed.

At the end of the execution, each entry (i, j) holds the edit distance between the strings consisting the first i _ characters of the first string and first _j characters of the second string.

Let’s see an example:

As you can see, the cell at the bottom-right position contains the edit distance for the two string that we are comparing.

Let’s see how the algorithm works, step-by-step. This time let’s compare the strings “ant” and “aunt” (edit distance = 1)

To get started we can quickly fill the first row and column.

Then, for each cell we will compare the corresponding character from the first string and the second string, then select the minimum one of these three values:

the value of the left cell plus one
the value of the cell above plus one
The value of the cell on the left and above, plus one if the characters being compared are different

And the process will continue filling the matrix.

Optimization Opportunity

As you probably noted in the step-by-step process. We only use two rows of the matrix to compute each cell value. So, we don’t need to keep all the costs, all the time, in memory.

using System;

public static class Levenshtein
{
    public static int ComputeDistance(string first, string second)
    {
        if (first.Length == 0)
        {
            return second.Length;
        }

        if (second.Length == 0)
        {
            return first.Length;
        }

        var current = 1;
        var previous = 0;
        var r = new int[2, second.Length + 1];
        for (var i = 0;
            i <= second.Length;
            i++)
        {
            r[previous, i] = i;
        }

        for (var i = 0;
            i < first.Length;
            i++)
        {
            r[current, 0] = i + 1;
            for (var j = 1;
                j <= second.Length;
                j++)
            {
                var cost = (second[j - 1] == first[i]) ? 0 : 1;
                r[current, j] = Min(r[previous, j] + 1, r[current, j - 1] + 1, r[previous, j - 1] + cost);
            }

            previous = (previous + 1) % 2;
            current = (current + 1) % 2;
        }

        return r[previous, second.Length];
    }

    private static int Min(int e1, int e2, int e3) =>
        Math.Min(Math.Min(e1, e2), e3);
}

Much, much better!

Last words

Levenshtein is a very interesting and robust algorithm. In the future, I will use this implementation to improve the querying experience in the simple search library that I have been working.

Again, if you have any suggestions, feel free to share it in the comments.

The post Computing the Levenshtein (Edit) Distance of Two Strings using C# appeared first on Elemar JR.

Let me help you to start using RavenDB4: Querying with RQL

Elemar Júnior — Thu, 01 Mar 2018 12:10:45 +0000

In the previous post, you learned how to install RavenDB on your computer, create a database and load sample data. Also, you learned some fundamental concepts about document databases.

Now, you will learn how to write your queries using RQL.

RQL? What is it?

RQL is one of the most exciting features of RavenDB 4. It is a powerful, easy to use and learn, language that we design to make your life simpler.

From the Documentation:

RQL, the Raven Query Language, is a SQL-like language used to retrieve the data from the server when queries are being executed. It is designed to expose the RavenDB query pipeline in a way that is easy to understand, easy to use, and not overwhelming to the user.

The RQL documentation is really good. You should consider reading it.

Getting ready to write queries

It’s time to stop talking and to write some code. So, let’s do that.

Open the RavenDB Management Studio (http://localhost:8080 by default. Remember?)
In the left panel, click on Databases
Open the database we created in the previous post (Northwind, if you followed my recommendation)
In the left panel, select the Documents section.
Click on Query.

There are other paths to this. I will let you discover it.

You first query

Let’s start easy.

Assuming you are already in the Query editor (inside the RavenDB Management Studio). Type the following query.

from Employees

Click on the Run button.

Yes. You are right. This query returns all the documents inside the Employees collection. I think you got it.

Now, go ahead and try other queries like these. Get all the documents from the Products collection.

Filtering

Getting all documents from a collection is a nice thing but quite useless. Let’s make something more exciting.

from Employees
where FirstName=="Nancy"

Yes! I think you got it. FirstName is the name of one of the properties present

in the documents from the Employees collection.

{
    "LastName": "Davolio",
    "FirstName": "Nancy",
    "Title": "Sales Representative",
    "Address": {
        "Line1": "507 - 20th Ave. E.\r\nApt. 2A",
        "Line2": null,
        "City": "Seattle",
        "Region": "WA",
        "PostalCode": "98122",
        "Country": "USA",
        "Location": {
            "Latitude": 47.623473,
            "Longitude": -122.306009
        }
    },
    "HiredAt": "1992-05-01T00:00:00.0000000",
    "Birthday": "1948-12-08T00:00:00.0000000",
    "HomePhone": "(206) 555-9857",
    "Extension": "5467",
    "ReportsTo": "employees/2-A",
    "Notes": [
        "Education includes a BA in psychology from Colorado State University in 1970. She also completed \"The Art of the Cold Call.\" Nancy is a member of Toastmasters International."
    ],
    "Territories": [
        "06897",
        "19713"
    ],
    "@metadata": {
        "@collection": "Employees",
        "@flags": "HasAttachments"
    }
}

Shaping the query result

Until now, we are just getting documents. Let’s say we want to shape what we get. Consider the following query.

from Orders
where Lines.Count > 4
select Lines[].ProductName as ProductNames, OrderedAt, ShipTo.City

Again, I am sure you understand what is going on. We are not interested in all data from the Orders documents. So, we are specifying a shape.

One of the results will look like that:

{
    "ProductNames": [
        "Ikura",
        "Gorgonzola Telino",
        "Geitost",
        "Boston Crab Meat",
        "Lakkalikööri"
    ],
    "OrderedAt": "1996-08-05T00:00:00.0000000",
    "ShipTo.City": "Cunewalde",
    "@metadata": {
        "@flags": "HasRevisions",
        "@id": "orders/26-A",
        "@last-modified": "2018-02-28T11:21:24.1689975Z",
        "@change-vector": "A:275-ZzT6GeIVUkewYXBKQ6vJ9g",
        "@projection": true,
        "@index-score": 1
    }
}

We will talk about the metadata in the future.

Using Javascript in the query projections

The last query was nice. But, let’s say you want to do more customization.

from Orders as o
load o.Company as c
select {
    Name: c.Name.toLowerCase(),
    Country: c.Address.Country,
    LinesCount: o.Lines.length
}

RavenDB allows you to use Javascript (Everybody knows the basics of Javascript, right?) when defining projections for the query results.

There is another interesting thing in this query, as you probably noted. The Company field of an Order document contains the ID of another document stored in the database. The load instruction is smart enough to get that document for you, so you can use it to project data as well.

Map and reduce (Oh Yeah!)

Consider the following query:

from Orders
group by Company
where count() > 5
order by count() desc
select count() as Count, key() as Company

What are we doing here? We are grouping the Orders using the Company field as grouping key. So we are adding a filter to get only groups with five documents at least, and then, ordering this groups by the number of elements in descending order. Finally, we are projecting the number of documents per group and the group key.

In “business words” this query results in a list of top buyers companies.

How it works

For a while, you shouldn’t care about our implementation details. But, it’s important to say that we are concerned about performance and we use a bunch of techniques to deliver results as fast as possible (even more!).

All queries in RavenDB are supported by a sophisticated and efficient indexing mechanism. In simple words, we use indexes for all the queries. But, I will explain it with more details in the future.

Last words…

Awesome! Now you know the basics about Querying with RavenDB. It’s easy, right?

The post Let me help you to start using RavenDB4: Querying with RQL appeared first on Elemar JR.

Let me help you to start using RavenDB4

Elemar Júnior — Wed, 28 Feb 2018 13:14:00 +0000

It’s time to start learning RavenDB 4. Even if you know previous versions of RavenDB, you will probably get good surprises and learn some new tricks. For example, did you know that RavenDB have a community (free) license?

In this post, you will learn how to install and start using it on your computer.

Getting RavenDB up and running

In this post, I am assuming that you are a developer trying to get an instance of RavenDB up and running on your computer. I will not discuss how to deal with production scenarios (at least, for a while). If you need information about how to setup RavenDB for production, I recommend you to read the online documentation.

Running on the live demo instance

Don’t want to download bits and bytes? No problem!

Without installing anything, you can point your browser to http://live-test.ravendb.net and access the public demo instance that I have available.

Using the live demo version is useful for quick checks and verifications, but it isn’t meant for anything more serious than that. Obviously, all data in the live instance is public, and there are no guarantees about availability. We use this instance to try out the latest versions, so you should consider that.

Running on Docker

Using docker is a fast way to start using RavenDB on your computer. If you have Docker installed, all you need to do is run the following command:

docker run -e UNSECURED_ACCESS_ALLOWED=PublicNetwork -p 8080:8080 ravendb/ravendb

Docker will now get the latest RavenDB version and spin up a new container to host it. Note that we run it in developer mode, without any authentication.

Running on Windows, Linux or MacOS

To set up RavenDB on Windows, Linux or MacOS, you will need first to download it from http://ravendb.net/download (selecting the right distribution for your platform). You will get a compressed file that you can extract to a folder of your preference.

Go ahead and do it, I can wait.

Done?! Now, all you need to do is run the ./Server/Raven.Server.exe located in RavenDB folder. This will start a console application in interactive mode, inside a console application. Also, the script will open your browser and start the RavenDB Management Studio

By default, RavenDB will try to use http://localhost:8080 as its endpoint. But, if something is already using port 8080, RavenDB will fail to start and give you an “address in use” error (more specifically, EADDRINUSE). In this case, you will need to configure RavenDB (You can follow the documentation to get it done).

Running RavenDB for the first time, you will need to do a little setup. You just need to answer the questions to get it done.

Your first database

RavenDB Management Studio was completely re-designed. It’s looking beautiful, and we did a lot of work to make it simpler and easier than ever.

Creating a Database

To create your first database:

Select the Database option in the left panel.
Click on the New Database button.
Type a name for the new database (I recommend Northwind, for this post)
Click on the Create button

Congratulations! You just created your first RavenDB database… but, it is empty.

Loading sample data

For learning purposes let’s load some sample data into our database.

Select Databases on the left panel
In the right panel, click on the name of the database you just created (that is empty for a while)
In the left panel, click on settings, and then Create Sample Data
Click on the big Create button

The Northwind database is the sample database that came with SQL Server; it has been used for decades as the sample database in the Microsoft community. We chose this database as our sample data because it is likely already familiar to you in its relational format.

Going to the Documents session (left panel), you will see that RavenDB created a lot of documents for you.

Exploring the database

That’s great. We just launched RavenDB in interactive mode, created our first database and loaded some sample data. But, wait! It looks remarkably similar to what you see in a relational database. Right? The data is shown in a grid format with the tables on the left. If you click on any “record”, you will start to see the NoSQL magic!

Yes! All RavenDB data is stored as JSON.

Understanding the `Document` concept

Using the Go to document feature (the text box in the Studio toolbar), go to the document orders/101.

{
    "Company": "companies/86",
    "Employee": "employees/4",
    "OrderedAt": "1996-11-07T00:00:00.0000000",
    "RequireAt": "1996-12-05T00:00:00.0000000",
    "ShippedAt": "1996-11-15T00:00:00.0000000",
    "ShipTo": {
        "Line1": "Adenauerallee 900",
        "Line2": null,
        "City": "Stuttgart",
        "Region": null,
        "PostalCode": "70563",
        "Country": "Germany"
    },
    "ShipVia": "shippers/2",
    "Freight": 0.78,
    "Lines": [
        {
            "Product": "products/1",
            "ProductName": "Chai",
            "PricePerUnit": 14.4,
            "Quantity": 15,
            "Discount": 0.15
        },
        {
            "Product": "products/23",
            "ProductName": "Tunnbröd",
            "PricePerUnit": 7.2,
            "Quantity": 25,
            "Discount": 0
        }
    ]
}

It is very different from what we’re used to in relational databases.

A document is a self-describing, hierarchical tree data structure which can consist of maps, collections, and scalar values.

Being practical, RavenDB database stores documents, which are plain JSON formatted data. So, we can aggregate related information into a common object, as in the case of the ShipTo property which has all the shipping information.

In a Document Database, documents are organized in collections.

Understanding the `Collection` concept

The collections provide a good way to have some level of organization. For example, documents holding customers data are very different from documents holding products information, and you want to talk about groups of them. RavenDB allows for a document to be stamped with a string value that will be evidence of its type (like “Customers” and “Products”).

Note that documents that are in the same collection can have a completely different structure, which is fine because RavenDB is schema-less.

Exploring the Northwind collections

Open the RavenDB Management Studio at http://localhost:8080
Open the Northwind Database
In the Documents session, explore all the collections.

The user interface is pretty simple. It is important to you to get familiar with it. I strongly recommend you to try to create your documents, edit and so on. You can create another database, load the sample data (if you want).

Don’t be afraid. Test your self!

Final words…

Awesome! You have just made your first steps with RavenDB 4. If you are getting started with document databases and want to learn more about document modeling, we recommend you to watch this Ayende’s talk.

Cover: unsplash-logo
Tyler Quiring

The post Let me help you to start using RavenDB4 appeared first on Elemar JR.

DEV Community: Elemar Júnior

How Static Analysis Tools are Helping me to Produce Better Software: Understanding the Design

STEP #1: Recognize the components and their relationships

STEP #2: Discover the size of each component

STEP #3: Discover the core (hard to change) components

STEP #4: Get a better (detailed) understanding about the relationships

STEP #5: Create a list of critical issues

Last words

Is the Managed Code Slower than the Native Code?

3 Mistakes Developers Make that Prevent Them from Improving the Performance of Applications

Mistake #1 – Do not define what is important

Mistake #2 – Try to optimize code without a reliable measurement

Mistake #3 – Do not verify the results

Time To Action

Implementing Parallel Reduction in CUDA

Sequential Sum

Parallel Sum

CUDA

Time to action

About “Difficult Conversations” (book recommended by Phil Haack)

The #1 Rule for .NET Performance

Time to action

3 mistakes that Developers make that prevent them from creating software that meets the business needs

Mistake #1 – Pay attention only to the technical aspects

Mistake #2 – Do not (even try to) speak the language of the business

Mistake #3 – Resist to accept changes in the requirements

It’s time to action

Getting started with CUDA (using VS2017)

Environment setup

Starting a new project using CUDA

Basic CPU processing

Bringing CUDA to the game

Final words

How to parse, simplify, differentiate and evaluate/solve (using Newton's method) equations and expressions in F#.

The Object Model

Parsing

Simplifying and Evaluating

Differentiating

Welcome, Mr. Newton

Computing the Levenshtein (Edit) Distance of Two Strings using C#

The Levenshtein Algorithm

How it works

Optimization Opportunity

Last words

Let me help you to start using RavenDB4: Querying with RQL

RQL? What is it?

Getting ready to write queries

You first query

Filtering

Shaping the query result

Using Javascript in the query projections

Map and reduce (Oh Yeah!)

How it works

Last words…

Let me help you to start using RavenDB4

Getting RavenDB up and running

Running on the live demo instance

Running on Docker

Running on Windows, Linux or MacOS

Your first database

Creating a Database

Loading sample data

Exploring the database

Understanding the Document concept

Understanding the Collection concept

Exploring the Northwind collections

Final words…

Understanding the `Document` concept

Understanding the `Collection` concept