DEV Community: Vinay NP

40,000+ Users in 3 months... Story of a Product I built

Vinay NP — Mon, 12 Jun 2017 16:46:38 +0000

Here is the story of the time when Poornima and I built a product from ground up and then sadly had to shut it down roughly after 3 months from the time we got our first user.

It was early summer of 2010. I was watching a late night episode (Season 4 Episode 20) of Bones on Star World. The plot basically revolves around the team finding a female body in a cardboard box recycling center and the usual grind of going about finding the killer. Somewhere in the middle of the episode, when one of the doctors switches on the victim's phone, another doctor's phone in the room rings with their profile photos on each others' phone. Eureka!! I just thought to myself, I am going to build this application. I called up my uncle who was kind enough to pick up the phone and listen to me ramble about some crazy idea and even give some helpful tips. I met Poornima the next morning and explained the idea to her. She was skeptical but she was in.

Basically, taking inspiration from the episode, I sat up all night thinking about an application (not the typical mobile app) which...

is location based
should work on every phone (smart or dumb)
should be fun and addictive to use
is free for users
has a solid revenue model (I wasn't a fan of get users first and think about revenue later strategy)

I had only one use case (dream) that the application needed to fulfill. Just imagine... 'A guy sitting in a cafe, alone, bored; sends a message to Date or Hate and in 5 minutes the application connects him to a gal sitting right behind him in the cafe or someone a couple of blocks away' the possibilities would be endless.

Then we sat and refined the idea. It was fairly straightforward. A user had to send an SMS giving their location in terms of some pre defined and understood keywords or PIN codes (ex: Market Street or 91785) and the system would apply a matching algorithm and connect 2 users based on the closeness of their locality and place a call back to them, take them in conference and allow them to talk for 3 minutes. Since, it was a call service it would be available from 6 AM to 10 PM only. No hassles. Completely anonymous as there is no sharing of numbers. It's free. Simply connects people who are looking for other people close by.

The revenue model was simple too. We would take the route of placing short in call ads with location specific content and offers etc. during wait times and once in between the call. Roughly, the beginning of the call would be something in the lines of... "Welcome to Date or Hate. This call is sponsored by xyz".

The technicalities of setting up the system was as follows...

The server was Apache.
The database was MySQL (partly because I had not worked with any other database till that point in time).
All API end points were written in PHP.
SMS service was provided by an external vendor, who gave us a 10 digit number and all SMS sent to that number would be pushed on our API.
The calling API was built on top of Asterisk. This was deployed on a separate server cluster (of 2 servers, set up in house accessed via a static IP) which had a 2 port Sangoma PRI card. In a nutshell, this API...
- would take 2 10 digit mobile numbers, an ad file name as input.
- place a call to both these numbers and conference them once it connects.
- play a welcome message
- play the ad identified by the file sent as input before beginning the conference.
- record the call if abuse is reported (pressing * during the call would trigger abuse report).
- returns the state post call termination
The ad files were stored locally on call servers for optimized/speedy delivery.
The telephone lines were provided by Airtel. A total of ~40 lines were used (over 3 PRI lines).
The number matcher would run every five minutes or if there are enough pairs of numbers waiting for the call as there are free telephone lines, match the numbers based on the location codes and place a request to the call server with pairs of numbers and ad files.
The matching algorithm would employ a radius based matching technique.
I had built a crude reporting cum monitoring cum make us feel good dashboard, which would query db and asterisk server and display some numbers under various headings.

All of this took us roughly about 6 weeks to achieve. Poornima called up a journalist friend of ours and explained to him the whole concept and gave a demo. He was excited to hear about it as we were probably the first ones in our country to build such a product at the time. He agreed to write a small piece on our product in the newspaper where he was working.

On the morning of 30 July, 2010 the article (shown above), introducing our product to the world, appeared. Since we had an intimation about when the article would be out, we were awake the whole of previous night checking and cross checking the setup etc. I had optimized the hell out of the MySQL server. Naturally, we went home at around 6 in the morning to freshen up, get some news paper copies and planned to come back to office by around 11 AM. I had forgotten to take the laptop home.

Before I go further, let me tell you what kind of numbers were we expecting and plans we had made. Since we did not have a precedent similar to our product we estimated that we would probably get about 3,000 to 5,000 users in the first 3 months. Post which we could take the service to other cities. We also thought that until this stage we will not play ads but only welcome messages. This would also give us enough time to prepare a proper advertisers kit and back end similar to Ad Words and take it to companies who can advertise with us.

We came back to office around noon. I switched on the laptop only to find that the monitoring dashboard was showing 670 users. This was a pleasant shock as we had never expected so many users. But, we thought "ok, people might be curios and trying out". The dashboard was set to refresh itself every 10 minutes. But 20 minutes passed and the number of users was still the same. We thought our product was a flop and people did not like it. I was disappointed. Then I logged in to the VPS server to shut it down as it was costing us money, in dollars, and to our horror we discovered that there were 1700 api calls rejected as we had crossed our bandwidth limit. HORROR!!! My estimate was wrong. we were having 100s of users using our product. I immediately upgraded the plan, restarted the server and we were back online. At day end we were 1400+ users with 1000s of conferences served. Also, I had been called stupid for the first time by Poornima. Sigh!!

You must be feeling, that this is a happy problem right?! Maybe true for a well funded start up but not for us, just passed out graduates with limited funding and maturity in business. We could've gone to an angel investor with this project but I am not ashamed to admit that, at the time, the only definition of Angel I knew was of Goddess with wings who appear in cartoons and fiction work. Also, one other reason we were in a crunch was because, I had kept the idea very close to my heart and had refused to discuss this even with my friends.

The users kept growing day on day and hour on hour. We did not know what to do?? We had some money of our own which soon got over. We started borrowing from F&F and after about 3 months, we had borrowed, let's say an average government employees' lifetime salary, and still did not have a clear strategy or a way forward to approach advertisers or any other ideas of raising funds.

One day (don't remember exactly which) in November 2010, there we sat, thinking of ways to foot the previous months' telephone bill. This was the moment. The moment when, my life, for years to come flashed in front of my eyes. Sort of like a flash forward. The Entrepreneur in me kept telling... You have a super idea at hand, push forward, things will fall in place. The Engineer in me kept telling... If you can build this, you can build something else too, think about the numbers, it is just not there. Shut it down. I got up, walked to the room where we had kept the call servers, kept my hand on one of the servers, took a moment, a deep breath and pulled out the wire from the wall socket. Poornima was right there behind me. Both of us did not utter a word and went back home in silence.

A lot of users had written to us asking why nothing was working anymore. So, sometime shortly thereafter, I did put up a web page explaining that we had shut down and methodically closed down all the servers/services. We did try to revive the project, sometime towards the end of 2012, as a lot of investor friends and consultants who had heard about the project, had shown some level of interest in the project. But this sort of an application now needed a radically new thought process and significant amount of time commitment. As Poornima was working on her start up and I was with CommonFloor full time, this was not possible. Therefore we finally gave up on this project.

This project, Date or Hate, is a constant reminder of the time I had failed, terribly. But, I did learn a lot. A lot. I have carefully preserved all the code, database dumps etc as a reminder of what could've been. I am still trying, in various ways, to put this behind me but it is really really difficult. It is so difficult because, even though this project had/has altered my life, my outlook, my attitude for many years to come, I never got a chance to see that use case fulfilled. Even though I had gotten 42,524 users for my product in less than 3 months, with bare minimal marketing, I still had to shut it down.

The day I get over this is probably when I will become a full time entrepreneur again, I guess!!

This post was first published on my blog

Building a single page application with vanilla js

Vinay NP — Wed, 17 May 2017 07:04:04 +0000

Often times I've come across this framework vs that framework debate. Many times as an observer, some times as a participant and occasionally as the person who *cough* started the debate *cough*. Most frequent arguments in these debates are around the comparatively easy ways to do stuff or the lesser code needs to be written points. What I don't see people talking about is writing vanilla js and structuring your project better instead of using a framework. What comes across as a shock to me is that many developers don't know anything more outside of their favorite frameworks and are scared to write plain js code. If you decide to code for a living, how can you not know the building blocks of the plugin you use??

A lot of good folks have talked about this in the past. Some of my favorite reads are zero framework manifesto and Look ma, no frameworks. Many of these frameworks have a wonderful way of marketing themselves by presenting their top features or perceived benefits of usage on their websites or through developer's blogs, however, I don't see as many folks showing ways of building SPA with vanilla js. I therefore decided to refactor my personal website as a SPA without using any framework. I hope that this post will serve as a good first step when you are building an app on your own without using any frameworks. All code referenced here is available at vinay20045.github.io repo and this website itself acts as a live demo.

Design
Prior to refactoring my website was a typical blog written in PHP. Every page request used to do a round trip to a server for all html content and assets, it had a management console etc. During refactoring some of my considerations were...

No page loads for every post i.e. it should be an SPA
Posts to be written using markdown syntax.
The blog should be written only in HTML+CSS+JS
Hosting to be done on github pages or AWS S3
It had to be mobile friendly

with these things in mind, the high level design of the blog looks like this...

Basic Structure
One of the primary things that you should be looking at while developing any application is the organization of the code. This includes everything right from your folder structure and naming conventions to declarations and definitions. A lot of developers I've seen argue over 2 line breaks vs 1 line break but are ok with having business logic in the views or templates. Anyways, once you do this for one project, it sort of acts like a boilerplate and will be very easy to replicate and extend for your future projects.

The basic structure of the blog application looks like this...

|-- assets
|   |-- css -- All site styles go here
|   |-- images -- All images used in the templates or page shell go here
|   `-- js
|       |-- config.js -- Environment specific config file
|       |-- init.js -- Contains all instructions on load
|       |-- controllers -- Business logic and view manipulation functions
|       |-- templates -- context based reusable snippets of HTML
|       |-- utils -- All internal and 3rd party libraries
|       `-- views -- Views exposed to the user
|-- index.html -- Page shell. Acts like a container. Actual content is populated based on route
|-- posts -- All posts markdown files go here
`-- uploads -- All assets used in posts go here

Routing
It becomes very important to have proper routing in place to facilitate deep linking, book marking and better SEO. Many techniques can be used for routing but hash based routing works really well and is easy to implement. On load of the application a routing function is registered against the hashchange event.

The routing function, part of utils library, looks like this...

router: function(route, data){
    route = route || location.hash.slice(1) || 'home';

    var temp = route.split('?');
    var route_split = temp.length;
    var function_to_invoke = temp[0] || false;

    if(route_split > 1){
        var params  = extract_params(temp[1]);
    }

    //fire away...
    if(function_to_invoke){
        views[function_to_invoke](data, params);
    }
}

extract_params function looks like this...

var extract_params = function(params_string){
    var params = {};
    var raw_params = params_string.split('&');

    var j = 0;
    for(var i = raw_params.length - 1; i >= 0; i--){
        var url_params = raw_params[i].split('=');
        if(url_params.length == 2){
            params[url_params[0]] = url_params[1];
        }
        else if(url_params.length == 1){
            params[j] = url_params[0];
            j += 1;
        }
        else{
            //param not readable. pass.
        }
    }

    return params;
};

The event listener is registered in init.js...

window.addEventListener(
    "hashchange", 
    function(){utils.router()}  // the router is part of the utils library
);

Dissecting controllers
Controllers hold the business logic. You can use the functions in here to manipulate your views. These functions are not exposed to the user directly. They can access only templates and the libraries available in utils. They can be invoked by a view or another controller.

The controller taking care of the home page looks like this...

controllers.home_page = function(data, params){
    var all_posts = JSON.parse(data);

    var posts_to_show = 3;
    var template_context = [];
    for (var i = 0; i < posts_to_show; i++){
        var post = all_posts[i];
        var item = {
            'link': '#post?'+post.post,
            'title': post.post.replace(/-/g, ' '),
            'snippet': post.snippet,
            'published_on': post.added_on,
        };
        template_context.push(item);
    }

    //get recent posts
    var recent_posts = templates.recent_posts(template_context);

    //get hello text
    var hello_text = templates.hello_text();

    var final_content = hello_text + recent_posts;
    utils.render(
        'page-content',
        final_content
    );    
};

Dissecting templates
Templates hold HTML markup for the actual page content. It helps in reusability when you can have functions generating the HTML you want based on some context passed. All functionality for the templates have to be provided by the controller invoking it by using data binding and event registration techniques. The only exception that I've allowed are the hrefs.

The template for the hello section of the home page is...

templates.hello_text = function(data){
    var content = `
        <div id="hello_text">
            <h2>Hello...</h2>
            <img src="assets/images/Vinay.jpg" align="left" style="width:70px;">
            <p>
                Thank you for visiting my blog. I am Vinay Kumar NP. I am a passionate techie...
            </p>
            <p>
                I am currently working on a <a href="http://www.int.ai/" target = "_BLANK">startup</a> of my own. I have previously worked in various engineering leadership positions at...
            </p>
        </div>
    `;

    return content;
};

Dissecting views
Views are the functions that are directly exposed to user. i.e. they are invoked by the router and are part of the url. There is no other difference between view functions and controllers. You could expose controllers too, but that might hurt modularity.

The view for all posts page looks like this. It simply passes the request to load show_posts controller after making an ajax call to get the posts index file.

views.all_posts = function(data, params){
    var api_stub = 'posts/index.json';

    utils.request(
        api_stub,
        'show_all_posts',
        'show_all_posts_error'
    );
};

Making API Requests
This is the holy grail of any SPA (sic). Though my blog does not need a mechanism to make outside api calls as all my posts are hosted within, I've written it to illustrate the concept. The request method takes the api stub, call back functions, params and fires the request. This is also part of the utils library. (Please be careful of CORS here).

The function to make api calls looks like this...

request: function(api_stub, success_callback, error_callback, callback_params){
    api_stub = api_stub || '';
    callback_params = callback_params || {};

    controllers.show_loader('page-content');

    var url = config.api_server + api_stub;

    var x = new XMLHttpRequest();
    x.onreadystatechange = function(){
        if (x.readyState == XMLHttpRequest.DONE) {
            if(x.status == 200){
                controllers[success_callback](
                    x.responseText, 
                    callback_params
                );
            }
            else{
                controllers[error_callback](
                    x.status, 
                    callback_params
                );
            }
        }
    };
    //other methods can be implemented here
    x.open('GET', url, true);
    x.send();
}

I haven't gotten a chance to do a comparative bench marking but at first glance all my repaints are done very fast with little or no jank. For the question of too many network calls on first load, I am planning to build a python based site packager for one of my other projects, I will post it once done.

There you go, wasn't that easy?? If you're still not convinced, fire up a browser, clone my repo, make the necessary changes (config, templates etc.) and play around. I'm pretty sure that not only will you come around to start building your own js applications, framework free; you will also be contributing more libraries to the open source world... The world needs more people who share :)

I've tested the code on all modern browsers (except IE) and it seems to work without any glitches. Watch out for JS api compatability while building your own applications (For ex, I've used back ticks which are not compatible with older browsers). Let me know if you find any bugs or issues with the code.

This post was first published on my blog

Converting Nested JSON to CSV

Vinay NP — Wed, 17 May 2017 06:32:33 +0000

The first time I came across JSON, I was really happy. It is a very light and fluffy object representation in plain text. The beauty was that there were no new or extra specs; existing concepts of lists, objects, strings, numbers etc. were taken and put together in it's own clean way. I use JSON widely when I am working with APIs. In my playbook it is second only to single word/number replies.

Apart from becoming a standard for data interchange between clients and server calls, JSON has also become popular for intermediate representation and storage of data. This is pretty evident when working with reporting tools and frameworks. Why I say intermediate is because, the origin might be some XML or data from some DB and the actual consumption format might be some chart, graph or CSV data.

JSON allows expression of hierarchical, well formed and structured data by nesting objects and arrays within one another up to multiple nesting levels. Often, this sort of representation is not very easy or almost impossible to express in row/cols based data structures like CSV. However, when you talk about reporting and making dashboards it is almost always required to convert such structured data to more flatter key/value data for interpretation and exporting. This interpretation allows data analysts to import the data into tools like excel and work with it.

Here I am going to discuss about converting multiple nested JSON which might or might not contain similar elements to CSV for usage with tools like excel or open office calc. The script is written in Python2.7.

Let's take a valid multi-level JSON and start off...

{
 "fruit":[
  {
   "name":"Apple",
   "binomial name":"Malus domestica",
   "major_producers":[
    "China", 
    "United States", 
    "Turkey"
   ],
   "nutrition":{
    "carbohydrates":"13.81g",
    "fat":"0.17g",
    "protein":"0.26g"
   }
  },
  {
   "name":"Orange",
   "binomial name":"Citrus x sinensis",
   "major_producers":[
    "Brazil", 
    "United States", 
    "India"
   ],
   "nutrition":{
    "carbohydrates":"11.75g",
    "fat":"0.12g",
    "protein":"0.94g"
   }
  },
  {
   "name":"Mango",
   "binomial name":"Mangifera indica",
   "major_producers":[
    “India", 
    "China", 
    "Thailand"
   ],
   "nutrition":{
    "carbohydrates":"15g",
    "fat":"0.38g",
    "protein":"0.82g"
   }
  }
 ]
}

This is a valid JSON of 3 Fruit objects. Each fruit object has multiple valid data structures nested within.

Each object has to be parsed individually and reduced to a more flatter structure before putting it back together and writing it as a CSV. This is also essential for determining the headers. A few key things for doing this are...

Deducing key/value pairs for each object
I've visualized this structure as a tree and then the key and value is decided as follows...
Value: This is the leaf node.
Key: This is the concatenation of all parent node names with an '_' as separator. In case of Arrays, the immediate parent of the value is the index.

Taking the Apple object as example the tree can be visualized as...

Therefore, the key/value pairs become...
Fruit_Name -> Apple
Fruit_Major Producers_0 -> China
and so on. The same thing is done for all Fruit objects. Here the array indices are ignored. Otherwise it would be Fruit_0_Name -> Apple as against Fruit_Name -> Apple

After this exercise we will be left with a much flatter set of the required key/value pairs derived from the original JSON object. Same thing has to be done for all the objects in the original data to obtain reduced data.

Determining Headers
The headers are decided by adding all keys of all objects together and removing duplicates among them. This will ensure that all the keys across objects are covered. Further, headers are sorted alphabetically so that all the similar keys stay next to each other making it easier to read the CSV file. However, this also means that the original ordering of the JSON is not maintained. This is a current limitation, I am working on this and will update when I fix this.

Some key/value pair(s) existing in one object may not be present in another object. In this situation, the value will be empty wherever the key does not exist.

I am also working on writing multi-row, pretty headers without the '_'. If done, this can make reading and using this CSV even more easier. This will also enable usage of Merge and Center functionality of excel and allow for easier filter/formula application.

Writing the CSV
Finally, after reducing all original objects to flat objects as discussed above, the headers are written followed by every object as a row. the csv.DictWriter is used to write the CSV which will ensure that the rows are written as per the header and in the same order.

All code is available at Github. I am yet to setup an online demo where you can upload a JSON file and get a CSV for download. Will update once this is live.

Please drop me a mail if you have any suggestions or come across any bugs in the code.

This post was first published on my blog