DEV Community: Shekhar Chandra

Common Mistakes to Avoid When Starting a SideProject

Shekhar Chandra — Fri, 27 May 2022 07:36:52 +0000

Time estimation in software engineering seems to be the most difficult task.

Time estimation becomes a bit easier for well-managed projects where most of the design decisions have been made and there is enough internal tooling and resources. But it is still not perfect. Things become more difficult for teams with limited resources and external pressure on deadlines. More often than not, the projects get delivered, some more polished than others.

When it comes to time estimation, solo developers working on their side projects face a real challenge. It becomes worse if there is no external motivation or revenue potential for the project. Building something for fun or learning might end up in an unfinished project that never sees a launch day.
Although launching a side project is not necessary, it does help in closure and moving on to the next big thing. If you do want to finish your next side project and cannot find the time or motivation, try avoiding these pitfalls.

Too Many Features

Side projects should be straightforward by design. If it has dozens of trivial and non-trivial features, it is on its way to becoming a startup. Think about the reason you are building your project. If it is for fun, then pick the top three to five features to include in the first version and move ahead.

No Delegation

Think of yourself as a resource even when you are working alone. Ask yourself how much you can deliver while working a few hours a week and assign achievable tasks to yourself. Don't forget to look for tools and services outside of your site that you can use to handle some of the work.

No Deadline

Set a soft and hard deadline for yourself once you have a clear idea of what you are trying to build. Give yourself enough buffer to catch up.

Unnecessary Marketing

Don’t fall into the trap of doing irrelevant market research and user surveys before coding. As a developer, your strength lies in development. Creating a production-ready product would be easier to market than a landing page.

Busy Work

Too much planning will ruin the flow and give you a false sense of accomplishment. Don’t create an extensive task list and roadmap before you start coding. Once you have your first few features finalized and an estimation of how much time it will take, start working on your project. You can always revise the estimate and drop or add another feature along the way.

Burden of Launch Day

Don’t worry too much about the launch. You don’t have to showcase your launch and post links everywhere. Keep it simple and ensure that your project ends up on an accessible platform. You can share the links to your project later.

In conclusion, if you are a developer working on a side project for fun or learning purposes, Then it's better to jump ahead and target finishing the project and avoid these pitfalls.

HowTo Examine & Modify Executables

Shekhar Chandra — Mon, 26 Jul 2021 21:20:32 +0000

Introduction

I have been working with C all my professional and student life. There have been times when I had to look a little deeper to understand what is going on with my buggy program. There are many tools and techniques to examine an executable and this post is about that and a little bit of reverse engineering.

Test Code

As an example I wrote a pretty basic piece of code with some intentional inclusions. There are two global variables msga and msgb. Two user defined routines allow and deny that get executed inside the main function. One conditional call to an external program using execvp. The idea here is to examine the executable this program creates. Find out where the code I wrote lands in the executable and what compiler adds on top of it. Later I'll showcase some basic reverse engineering that can be done by pretending we haven't seen the code.

GitHub

#include <stdio.h>
#include <unistd.h>

char *msga = "Allow";
char *msgb = "Deny";

void allow() {
    printf("%s\n", msga);
}

void deny() {
    printf("%s\n", msgb);
}

int main(int argc, char **argv) {
    deny();
    int runExternal = 0;
    if (runExternal) {
        char* lsargs[] = {"ls", "-l", NULL};
        execvp("ls", lsargs);
    }
}

While dealing with executable we'll encounter a lot of hexadecimal values, I prefer using Python to do quick arithmetic whenever the need arises.
Also, some of the output is going to be too big to paste here in the post so I'll link them in the end.

Goal

Let's compile the program and get our a.out.

gcc examinebin.c

As we see in the code the execution of the program basically runs the deny and halts. I am creating a goal for myself that I'll identify the instruction inside the executable and change it to make sure that allow is called and then 'lsis executed with-a` argument.

PS: The tools used to do analysis and their output are listed at the bottom of this post for reference.

Call Sequence

If we look at the objdump output, it is very neatly divided into segment and clearly labeled with symbol names. The code we are interested in is the one we wrote but it's nice to know what everything else is.

The short version is every C program needs a main routing which marks the start and end of user written code. C runtime executes main within its framework and takes care of all static and runtime dependencies. The order of execution can be determined very easily by hooking up the executable with gdb and adding breakpoint to all symbols defined in .text section and _init & _fini. Let's see what happens.

breakpoint : _init, _start, deregister_tm_clones, register_tm_clones, __do_global_dtors_aux, frame_dummy, allow, deny, main, __libc_csu_init, __libc_csu_fini, _fini

This is the call sequence labelled by me based on what my understanding of the usual meaning of these symbols

// Initialisation
_init (argc=1, argv=0x7fffffffdfd8, envp=0x7fffffffdfe8)
_start ()
__libc_csu_init ()
_init ()
frame_dummy ()
register_tm_clones ()

// User Code
main ()
deny () // we want to call allow and execvp here instead

// Deconstruction and finalisation
__do_global_dtors_aux ()
deregister_tm_clones ()
deregister_tm_clones ()
_fini ()

Identification

Now that we know what we don't have to explore we can focus on the task at hand, calling allow and ls with -a. To do that we will specify our goal properly, basically we want to:

call allow instead of deny.
change runExternal flag value to non-zero.
change "-l" to "-a" in lsargs`

To do that we have to know where these values are in binary and then change them manually without disturbing everything else.

Replace `deny`

The hexadecimal code calling deny from objdump output

0000000000001189 <allow>:

00000000000011a3 <deny>:

00000000000011bd <main>:
    11e4:    e8 ba ff ff ff           callq  11a3 <deny>
    11e9:    c7 45 dc 00 00 00 00     movl   $0x0,-0x24(%rbp)

from the callq reference we know that opcode e8 takes the operand ba ff ff ff (0xffffffba) which is basically the offset from next instruction 0x11e9. So, it should point to (0x11a3)

offset = hex(0xffffffba - 0x100000000) # getting the negative value
deny_addr = hex(0x11e9 + int(offset, 16))
print(deny_addr)

To call allow instead we will have to change (0xffffffba) to something that gives (0x1189) instead.

allow_addr = hex(0x1189)
offset = hex(int(hex(int(allow_addr, 16) - 0x11e9), 16) + 0x100000000)
print(offset)
# 0xffffffa0 -> a0 ff ff ff

So all we need to do is change ba to a0 in the binary

Change `runExternal`

This is quite simple, all we need to do is locate the mov instruction that is putting the value in the flag.

    11e9:    c7 45 dc 00 00 00 00     movl   $0x0,-0x24(%rbp)

and change the value to any non-zero one. ref

00 00 00 00 -> 01 00 00 00

Change `"-l"`

We basically want to change the arguments going into execvp function call. In the assembly we can see the location where the callq to execvp has been made and there should be push or lea instruction before that to add the argument into the stack. Since these values are hardcoded in binaries all we need to do is get the location of -l and change it to -a.

    11f6:    48 8d 05 12 0e 00 00     lea    0xe12(%rip),%rax        # 200f <_IO_stdin_used+0xf>
    11fd:    48 89 45 e0              mov    %rax,-0x20(%rbp)
    1201:    48 8d 05 0a 0e 00 00     lea    0xe0a(%rip),%rax        # 2012 <_IO_stdin_used+0x12>
    1208:    48 89 45 e8              mov    %rax,-0x18(%rbp)
    121b:    48 8d 3d ed 0d 00 00     lea    0xded(%rip),%rdi        # 200f <_IO_stdin_used+0xf>
    1222:    e8 69 fe ff ff           callq  1090 <execvp@plt>

lea instruction is basically calculating the effective address which in every case here is an offset to the next instruction pointer. So we have three addresses, which can be calculated or seen in the objdump output as well.

print(hex(0xe12 + 0x11fd)) # 0x200f
print(hex(0xe0a + 0x1208)) # 0x2012
print(hex(0xded + 0x1222)) # 0x200f

from the hexdump output we can clearly see that our strings are really there.

00002000: 0100 0200 416c 6c6f 7700 4465 6e79 006c  ....Allow.Deny.l
00002010: 7300 2d6c 0000 0000 011b 033b 5400 0000  s.-l.......;T...

Changing the fourth byte from the right 6c -> 61 will make l->a.

Changes

Let's summarize and do all the necessary changes to the text output provided by xxd utility.

Changes for allow

000011e0: 0000 0000 e8(ba) ffff ffc7 45dc 0000 0000 
000011e0: 0000 0000 e8(a0) ffff ffc7 45dc 0000 0000

Changes for runExternal flag

000011e0: 0000 0000 e8ba ffff ffc7 45dc (00)00 0000 
000011e0: 0000 0000 e8ba ffff ffc7 45dc (01)00 0000

Changes for l -> a

00002010: 7300 2d(6c) 0000 0000 011b 033b 5400 0000 
00002010: 7300 2d(61) 0000 0000 011b 033b 5400 0000

Create new executable

Using xxd utility

xxd -r modified-xxd.txt > a2.out

change permission

chmod +x a2.out

Run

Well I can tell you that it actually works but it's better to try yourself. The output is:

Before

Deny

After

Allow
.  ..  a.out  a2.out

Tools

Lets review some tools that tell us about the file from the outside.

file

utility that gives the file name, file type and other format related information.

output

sum

Get the checksum and number of blocks in the file. Once we do some reverese engineering this output will tell us that the new executable is not genuine.

output

ldd

Gives the list of shared objects required by the executable

output

There are some utilities that give a quick peek about the executable if in depth examination is not something you need.

strings

Displays all printable characters and strings in the file. Works on any file in fact not just executable

output

nm

Lists all the symbols present in the executable file address map.

output

Now comes the in depth analysis of executable, this includes intrepreting the machine code into human readable form and also figuring out a way to edit the file.

objdump

using the -d option you can get the detailed version of each section and segment of your executable along with the interpreted assembly instruction.

output

xxd or hexdump

These are plain read-write tools to deal with binary files and not just executables. Reading part creates a text file showing hexadecimal values at each byte and if possible there is a printable version side by side. Any changes to this output text file can be fed back to the tool, which can then create a binary file.

I am using xxd for reading and writing the executable here.

output

Feel free to send me an email for any suggestion or feedback. Follow me on twitter and github.

Visualizing Artificial Ant Problem using pygame

Shekhar Chandra — Tue, 20 Jul 2021 20:49:15 +0000

DEAP

I recently started learning about genetic algorithms and programming. I came across several easy to start tools, libraries and tutorials.There are many cool visualizations available online to showcase how the algorithm improves the population generation over generation. Seems like a nice way to grasp the concept and build intuition.

Since I mainly code in Python, I ended up using DEAP for prototyping and learning. Documentations are very clear and with lots of examples to study from.

Artificial Ant Problem

One particular example was, Artificial Ant Problem. The goal is to evolve an individual that can control an ant and eat as many food items from the environment as possible given a limited amount and types of moves.

The code runs seamlessly and you could see the output in the console that generation over generation the population is actually able to increase the amount of food items eaten.

Code Changes

That’s all well and good, but I wanted to visualise the solution to see for myself and get a better understanding. First step was to actually read the best individual attributes hall of fame.

Original Base Code for Reference
I captured the best individual after running the program and recompiled the attribute for that individual.

from deap import gp
hof_promitive_tree = gp.PrimitiveTree(hof[0])
routine = gp.compile(hof_promitive_tree , pset)

Modified the AntSimulator class a little bit to record every move for later use.

class AntSimulator(object):
def __init__(self, max_moves):
...
self.recorded_moves = []
        ...
    def record_move(self):
        self.recorded_moves.append(self.position)

Next step was to run the routine again on modified AntSimulator and get the recorder moves for simulation

ant.run(routine)
# contains the initial food item and ant location
grid = ant.matrix
# recorder moves of best individual
moves = ant.recorded_moves

Pygame

Given this much information, initialisation involved placing ant and food items on the pygame screen. The main loop consists of playing each move one by one and removing food items once eaten. Complete modified code can be found here

And here is the end result. Shows the individual controlling the first 600 moves of our artificial ant.

one generation

Ten Generations

Forty Generations

Hugo Shortcodes to Embed Anything

Shekhar Chandra — Thu, 15 Jul 2021 18:08:15 +0000

Introduction

Hugo is a static site generator suitable for personal websites and blogs. Being an open source project it lets users to manipulate the presentation layer to great extent. Strong theme support and customization options makes it a perfect platform to host almost any kind of web object.

Shortcodes

Hugo shortcodes lets developers embed any snippet into a markdown article. It comes with a lot of in built shortcodes to embed widgets from most popular sharing platforms. Shortcodes can embed a complete HTML document as well not just script tags and small snippets. This makes it possible to natively embed a small standalone project.

Shortcodes are html files which are, by convention, placed in /layouts/shortcodes directory. Once placed the shortcode can be included in any page using {{\< filename \>}}

Passing argument

It is possible to pass arguments while including a shortcode. Just add any number of positional arguments like in a command line and hugo will capture them as string on rendering.

For example, the first argument can be captured as a zeroth positional argument inside the shortcode file.

{{- $arg0 := .Get "arg0" | default (.Get 0) -}}

and can be accessed anywhere in the html document as such

<a href={{ $arg0 }}></a>

Adding Style

Shortcodes get the style definition from the selected theme of custom css. A shortcode specific CSS can be included in the HTML as usual and place in /static/ folder

JavaScript and Minification

The generated HTML gets minified naturally while compiling for production using --minify option. To make sure the javascript used in the shortcode is minified as well, place them in the /assets folder and include using hugo minification pipe.

    {{ $customjs := resources.Get "/custom.js" | resources.Minify }}
    <script src="{{ $customjs.Permalink }}"></script>

Hugo Shortcode to Render HTML

Shekhar Chandra — Wed, 14 Jul 2021 20:27:34 +0000

Problem

Adding raw HTML in hugo content doesn't render at all in static page.

Solution

Add a shortcode in layout folder to handle HTML in a safe manner.

<!-- renderhtml.html -->
{{- $content := .Get "content" | default (.Get 0) -}}
<div>
    {{ $content | safeHTML }}
</div>

Now it can be included in any mardown content file as, Just put the raw HTML in between the ticks.

{{ < renderhtml `<div></div>` >}}

DEV Community: Shekhar Chandra

Common Mistakes to Avoid When Starting a SideProject

Too Many Features

No Delegation

No Deadline

Unnecessary Marketing

Busy Work

Burden of Launch Day

HowTo Examine & Modify Executables

Introduction

Test Code

Goal

Call Sequence

Identification

Replace deny

Change runExternal

Change "-l"

Changes

Create new executable

Run

Before

After

Tools

file

sum

ldd

strings

nm

objdump

xxd or hexdump

Visualizing Artificial Ant Problem using pygame

DEAP

Artificial Ant Problem

Code Changes

Pygame

one generation

Ten Generations

Forty Generations

Hugo Shortcodes to Embed Anything

Introduction

Shortcodes

Passing argument

Adding Style

JavaScript and Minification

Hugo Shortcode to Render HTML

Problem

Solution

Replace `deny`

Change `runExternal`

Change `"-l"`