DEV Community: Ankit Vijay

C# dynamic – A friend you may want to keep a distance

Ankit Vijay — Sat, 17 Aug 2019 22:00:39 +0000

Recently, after one of my PRs which was merged to master, my teammates started complaining about a weird scenario. On some occasions, the ASP.NET Core app hosted inside the IIS worker process (w3wp.exe), would simply die without any exception/ warning. There were no clear repro-steps and it was difficult to pinpoint what was causing the issue. It took me some time to figure out the root cause of the issue and it turned out to be quite an interesting issue.

Background

As part of my PR, I had added a new code to map a Repository object/ entity to a Dto. It was quite complex and a multi-level nested entity. I did not use the library such as Automapper to automatically map the entities. Why I chose to do mapping manually could be a discussion for another day. Anyways, the culprit code went something like this.

We had an abstract base class, let us call it AbsractBaseRepository and this class was derived by as many as 18 different child classes. Something similar to below:

The AbsractBaseRepository was then used by one of the nested child Repository as below:

The Dto which was mapped from this Repository had a similar structure.

Why and how we ended up with this structure is again out-of-scope of this post. To map the Dto from Repository, I tried to be a little bit smart/ lazy and used C# dynamic as below:

In the above code, casting the repository parameter to dynamic implicitly converted the base repositoryobjectto the derivedrepositoryand call the correct overload ofToDto` method.

The Issue

Unfortunately, I turned to be too smart for my own good. I missed the mappings for one of the derived Repository class. In a scenario where the missed derived Repository was present in the entity, the code execution fall-back to base class overload method, that is, ToDto(SomeAbstractRepository repository). This resulted in an infinite loop causing the process to crash during debugging. Since there were as many as 18 derived Repository classes this was somehow missed in integration and unit tests as well. The easiest way to fix this was to simply add the mapping for the missed Repository class. However, it presented an additional risk, what if we add another derived Repository and we miss adding the mapping for that Repository? In that scenario, we would land up in a similar situation.

The Fix

As a fix for this issue, I decided to go back to basics and use explicit conversion to map the Repository and Dto, even if it meant more lines of code. The explicit conversion helped us to identify the issue at the compile-time or in the worst case throw a clear exception at run-time instead of blowing the entire process without any exception at the run-time. The updated code looked something like below:

{%gist https://gist.github.com/ankitvijay/faa37402c6b3d5d441259e24ceb34d12 %}

Lessons Learnt

An important lesson which I learned while resolving this issue was to be extra cautious while using dynamic. For all the power dynamic brings, it comes at a cost. Personally, I try to avoid dynamic as much as possible and this issue just gave another reason why I would continue to do so.

SAST Tooling – Part 3: The Winner

Ankit Vijay — Sat, 29 Jun 2019 21:06:10 +0000

Disclaimer: This post is not an endorsement or opposition of any product or tool. Opinions present here is based on our experiences. Please exercise your own independent skill and judgement before you rely on the information in this post.🙂

This is Part-3 and final part of my blog series on Static Analysis Software Testing (SAST) tooling.

In the Part-2, I described our selection criteria to select an alternate to Veracode and how we narrow down our search to just few tools. In this post I will describe how we came about selecting the winner.

The Dilemma

Its good to have choices but it also makes it difficult to choose just one. While, our selection criteria was good enough to shortlist 5 SAST tools, we realized that we needed to do a lot more to select the one which would work best for us.

Documentation and public information did not help us get an answer to all our questions. Hence, we scheduled a demo with each organization to understand more about their product. We prepared a common questionnaire for each vendor. Additionally, we invited folks from different teams such Development, Security, Operations and management so each team had their say. All the demos were organized in a single week so that experience from previous demo was still fresh in our minds.

The weighted Matrix

Product demo was a great way for us understand the capability of each tools and further shorten the list. However, we still did not have a clear winner. We needed to be more objective with our approach. That’s when we came up with an idea of weighted matrix. We wrote down our selection criteria in an Excel sheet and gave a weight to each criteria on a scale of 1 to 10, where 10 would mean an absolute must have and 1 would mean that no one would die if the product does not have that capability. For example: Thoroughness of Security Checks was weighted as 10, Product Support was weighted as 8, while ability to check licensing in 3rd party open source dependencies was weighted just 2.

After that, we rated tools on a scale of 1-5 on each selection criteria and then came with with a weighted score as below:

Tool Criteria score = Criteria Weighting (1-10) * Tool rating (1-5)

I would not go into the details of total criteria score of each tool as it was our internal research and specific to our organization needs.

The Winner

The weighted matrix helped us to come get a total score for each tool and the results were a bit surprising. The SAST tool we ended up selecting was less known tool Kiuwan. Checkmarx and Coverity should also get a special mention

Once, the tool was selected we worked with Kiuwan development team to do a POC with few of our applications.

Why Kiuwan

Here are few reasons why we chose to go for Kiuwan.

While Kiuwan did not have a 24 * 7 support, we found that they were quite prompt in their response. During the POC, one of our developers questioned few of their security findings. Kiuwan development team accepted the one of the findings and agreed to fix it. For the others, they provided an explanation why they think it was valid.
During the demo we found team from Kiuwan to be quite transparent. They answered our questions with honesty without making tall promises. There were few (cough.. Fortify) who tried to convince us that we were not asking the right question.
Kiuwan turned out to be the least expensive among all the products we evaluated.
Due to the value for money it offered, we were able to a get great bundling deal on Code Security (SAST), Insights, and Code Analysis features from Kiuwan.
We found its UI modern and intuitive. It came up with some good features such as security rating, estimated number of hours to improve the security rating, action plan, grouping of vulnerabilities etc.
Kiuwan allowed us to scan the source code locally and upload only the findings along with impacted line instead of uploading the entire source code. This was an important requirement for us, more so because we are using their cloud offering.
Kiuwan was the most up-to-date when it came to supporting the latest .NET Core frameworks or the modern JavaScript frameworks.
We found its reporting to be the best among all the products.
Kiuwan documentation was up-to-date and easy to follow.
We found Kiuwan updates and its release cycle to be the fastest among all the tools.

Whether or not Kiuwan would turned out to be a true to its hype, only time will tell. But as they say, well begun is half done. So we are keeping our fingers crossed.

Photo by Fauzan Saari on Unsplash

The post SAST Tooling – Part 3: The Winner appeared first on Hi, I'm Ankit.

SAST Tooling – Part 2: The selection criteria

Ankit Vijay — Tue, 25 Jun 2019 21:46:55 +0000

Disclaimer: This post is not an endorsement or opposition of any product or tool. Opinions present here is based on our experiences. Please exercise your own independent skill and judgement before you rely on the information in this post.:)

This is Part-2 of my blog series on Static Analysis Software Testing (SAST) tooling.

In the Part 1, I described our pain-points using Veracode and what motivated us to look elsewhere. In this part , I will describe how we went about looking for a tool better suited for our needs.

A “ perfect” SAST tool

Buying a third-party tool is not only expensive but also a huge investment for an organization. It can take considerable amount on effort to customize the tool according to organization needs and visa-versa. It is equally hard to build a ecosystem around it. Since, we had already burnt our hands with Veracode, it was extremely important that we were second time right with our selection. There was no room for mistake. We were aware that if we do not do a due diligence, then it could be fatal and entire IT would need to carry the burden of our mistake for the years to come.

With that in mind, my colleague, Chris from SecOps (Security Operations) and I came up with a selection criteria to shortlist the SAST tools. Based on the selection criteria, we did an independent research on various SAST tools available in the market. The selection criteria helped us remove our unconscious bias while trying to short-list the SAST tools.

Our selection criteria

We divided our criteria into Must-Have, Good-To-have and Should-have. We also categorized the our requirements based on the stakeholders like Dev, Security, IT management. Below are few points we considered during our selection process at a very high level:

Support for security risk and software error such as SANS 25, CERT, OWASP Top 10
Provide a detailed, executive report summary customized as per stakeholder such as Dev, Security, IT management.
More control to user – Allow user to configure scan policy as per project.
Allow user to export reports outside portal via well-known formats like email, pdf and not just propriety formats.
High accuracy rate and low false positive rates.
Good explanation of vulnerabilities within the context of the code.
Provide sample code and guidelines to fix the issue.
Good integration with tools such JIRA, Visual Studio, Visual Studio code.
Integrate well with our build pipeline and CI/CD process.
Easy to understand, navigate and a modern User Interface
Scan turnaround time
Ability to scan on locally
Support for JavaScript frameworks like Angular, VueJS etc and the latest .Net Core framework.
Report vulnerabilities for 3rd party dependencies added through NuGet and NPM.
Provide a rich set of APIs to perform the operations like viewing scan, downloading reports etc
Excellent documentation
Good Support
Cost effective
Capabilities beyond security such as code efficiency, duplicate code, etc

What we did not consider

Reviews from sites such as ITcentralstation. We found the reviews about the product could be misleading and they were more focused on IT operations.
The size of the organization. Our experience from Veracode made us realize that bigger is not always better. Hence, we kept it out of our selection criteria.
We tried to keep a distance from the product marketing teams as from our experience it does not help much in getting answers to our query.

The above criteria helped us narrow down our search to following products:

Checkmarx
Fortify on Demand
Kiuwan
SonarQube
Coverity + Black Duck

In the next and final post, I will talk about the tool how we further reduce our long list of SAST tools and what we ended up procuring.

Photo by Vladislav Babienko on Unsplash

The post SAST Tooling – Part 2: The selection criteria appeared first on Hi, I'm Ankit.

SAST Tooling – Part 1: Why we ditched Veracode

Ankit Vijay — Fri, 21 Jun 2019 23:37:32 +0000

This post is Part-1 of multi-part series describing our journey to ditch popular Static Application Security Testing (SAST) tool Veracodeand our quest for a better security tool.

Background

Until recently, our organization used Veracode for security analysis for few our applications. Veracode came with a lot of reputation. It is considered a leader in Application Security by Gartner and is used by hundreds of organizations across the globe. Unfortunately, our experience with Veracode was quite opposite.

Disclaimer : This post talks about our experience with Veracode and other SAST tools based on our needs last year. Things may have changed since then. It should not be seen as endorsement/ recommendation for or against any SAST tools.

Veracode Pain-Points

Veracode Documentation did not keep up the pace with their updates (which anyway happened once in a blue moon). At few places even we found it misleading.
We found the Veracode APIs very hard to use and integrate with our build pipeline.
Veracode UI portal left lot to be desired. Its UI looked dated and far from intuitive. It did not support some basic integrations. For example, reports, scan results etc. could not be emailed.
Veracode Support took weeks to respond and when they did respond, the responses were short with no help.
- Support from tickets raised almost always included nothing more than a link to outdated documentation. This was even the case when the ticket raised was in regards to the same outdated documentation.🙂
- Often it took follow up emails to even get a response from them.
Integration with JIRA was not usable and did not solve our purpose. We eventually fall-back to creating tickets manually in JIRA due to following reasons:
- Veracode would raise dozens or even hundreds of tickets for the same finding if it was found in more than area such as a text box. For example, a client side validation issue that could be addressed by a single change to the validation library in place was flagged as a new ticket for every text box was created. This resulted in hundreds of new tickets being raised in JIRA creating a clutter.
- Its JIRA plugin itself was not regularly updated. And one of our JIRA updates broke the Plug-in.
The scans incorrectly identified the 3rd party library as our internal organization code. As a result, scans showed the incorrect results. This resulted in our code failing the scan. Developers cannot rewrite the code for 3rd party developers. It took months for Veracode to confirm that it was a bug and then another few to fix it.
A number of dependencies our applications like Castle Windsor and Autofaq were not supported. Veracode mentioned they no plans to support it.
Veracode API Keys had a finite period of validity. We found that Veracode appears to only send 1 notification regarding the pending expiration, just 1 day prior to the API key expiring. There were no additional follow up emails or notifications in the portal etc, that warn of the expired API key. As a result, our Veracode scans had been failing as a result of the expired API keys. Overlooking a single email resulted in all associated scans from being unable to complete.
If there was any issue while uploading the DLLs (eg. due to API key expiration), then the error we get on the portal are very cryptic and it was difficult to understand the root cause and possible solution.
Veracode neither integrated with our source control nor offered local analysis. The only way to scan the code was to upload the DLLs to the VeraCode portal.
Scan Times were quite lengthy. Approximately 30 mins for a small size application. If a minor change is made to code and then all the DLLs were required to be uploaded and the full scan needed to run once more. We could not find a way to integrate it fully with our build pipeline.
The features added to Veracode appear to be focused on creating new income streams via new paid products without addressing any of the known issues with the platform.
VeraCode came up with two Visual Studio plugins: Veracode and Veracode Greenlight. Veracode Greenlight was licensed separately. But more importantly, both these plugins were equally useless.
- Veracode Plugin: Did not provide static scan with-in the editor. We still had to upload the DLLs and then download the report. While the developer uploaded files from their IDE, the machine was unusable and unresponsive. As each file was uploaded, focus returned to the upload window making performing any additional task at the same time impossible. Again, the upload could take a several minutes to complete.
- Veracode Greenlight: While this plugin _supposedly _scanned the source code inline. It could only do it at the file level and not the DLLs. From our experience it could not even identify the vulnerabilities with in the file.

These pain-points were felt by us every day and forced us to look for an alternate. In the next post, I will talk about our approach to evaluate different SAST tools out there in the market.

The post SAST Tooling – Part 1: Why we ditched Veracode appeared first on Hi, I'm Ankit.

My first major Open Source contribution

Ankit Vijay — Fri, 26 Apr 2019 10:44:44 +0000

Without a doubt Open Source is great. It has touched the lives of millions of developers and through them almost everyone living on this planet someway or the other.

The magic of Open Source

For all the power Open Source has given to developers, the sad reality is that only very few people are actually the contributors. Most of the devs just consume what a very few build, mostly in their own free time. I hate to admit, I’m also among those who enjoy using Open Source software, applications, packages but very rarely contribute. Until recently, my major contribution to GitHub was restricted to a few documentation updates, typos, minor code changes or raising issues on the repository. While it is not an excuse, one of the reasons, I have not been able to contribute to the Open Source as much as I would like is because I have mostly worked with enterprise customers in a closed environment which provides a very little opportunity to work beyond the usual tasks. This, however, changed a few days back.

The Problem

Recently, we upgraded our work-in-progress application to ASP.NET Core 2.2. As part of the upgrade, I migrated our HealthChecks to the new out-of-the-box health check feature of ASP.NET Core 2.2. To read more about this feature, refer to this link. The BeatPulse project on GitHub provides HealthCheck NuGet packages for many service providers such as SQL Server, Redis, Raven DB etc. which makes the integration nice and easy. For our application, I needed to integrate with RavenDB. However, the NuGet package had the following limitations:

It did not allow a user to pass an array of connection strings/ URLs which is a must-have for configuring failover.
Additionally, the implementation did not support Certificate Authentication, again a must-have of the production scenario. I went through the implementation and realized it is quite a simple fix.

My Options

I had two options to go about this.

Option 1: Raise an issue with the contributors and wait for this to be fixed someday. In the meantime, have my own implementation for our application. However, that would have meant an additional avoidable code in our repository which would need to be maintained. Also, if someone else from my organization needed that implementation, they would either copy from my code or create our own internal NuGet package. Again, waste of time and effort.
Option 2: Submit a PR with the changes and wait for it to get merged in the mainline before leveraging it in my application. I was initially a bit sceptical about this. I was not sure how much time it would take to get my changes approved. So many times, this apprehension has made me and I’m sure many others like me take a shortcut (read “Lazy”) approach that is the Option 1.

Solution

This time I decided to do what is right. I forked the repository, made the relevant changes and raised this Pull Requestto fix the issue. To my surprise, I found the process very easy and straightforward. The repository owner @unaizorrilla was very prompt in his response. The entire process took less than a week and I was able to use the updated NuGet package in no time once the changes were merged. This experience was very extremely motivating. Most importantly, it gave me an inner satisfaction that my code now will be used by hundreds of other developers and not just within my organization. If you have not yet contributed to Open Source, I encourage you all to look for the opportunities where you can contribute. Trust me, you will find plenty. :)

The post My first major Open Source contribution appeared first on Hi, I'm Ankit.

Team City – Curious case of failed tests but passed build

Ankit Vijay — Sun, 24 Mar 2019 22:07:00 +0000

Recently, I had written a post on how I managed bring down build pipeline for entire organization. While I was at fault during that time, Team City should take the blame for this one.

Background

I had recently created a CI/CD pipeline for our new .NET Core project. As part of the build pipeline, I had usual build steps to Build the solution, Run Unit Tests, Run Integration Tests, and then Deploy to Octopus.

Team City dotnet CLI Plugin

As mentioned in my previous post on Team City, I have been try to avoid Team City plugins as much as possible and use scripts or command line instead. To run the Tests for my new project my first preference was to leverage “dotnet” command out of the box instead of using Team City dotnet plugin. However, dotnet test command does not take list of multiple projects by default. There are obviously way around this but I chose to go for Team City plug-in instead due to simplicity and ease of use. The configuration was simple and straightforward and I was up and running in minutes.

Team City dotnet CLI plugin

Issue

Our CI/CD pipeline seem to be working as expected with no dramas. Once, the PR branch was approved, master branch build was trigger and on successful build, the solution would deploy to Octopus. On one fine day our QA reported a bug which should have ideally been caught by our integration tests. When I looked Team City build, I noticed that build was successful even when integration tests failed. That was weird in many ways and I double-checked the build configuration and verified that they were all set correctly. The “Execution step” for all the build steps was set to “If all previous steps finished successfully”. But still the build steps after the failed build step appear to be getting executed.

Root Cause

One further digging, I found the root cause of the issue to be this issue on Team City dotnet plugin. There were other devs who were also caught off-guard like me. As per one of the plugin contributors:

It is original behavior. For most of runners a process returns non zero exit code which means that a running was not successful. For runners like NUnit, VSTest, dotnet test, msbuild /t:VSTest, dotnet vstest positive exit code just an amount of failed tests. See this thread for details

However, this explanation for me is not enough to NOT fix a wrong behavior/ bug in the plugin. The behavior is not intuitive and can lead to issues like mine and other developers.

If you want JetBrains to prioritize this issue you can vote here.

Workaround

One of the suggested workaround mentioned for this issue was to set “Execution Step” for the each build step as “Only if build status is successful”. But I did not like this solution as it means we are going away from default for no reason and we would need remember yet another thing to change from default every time we add a new build step.

I managed to solve to this issue with the slightly cleaner work-around by adding following “Failure Condition” to my Team City build configuration.

This failure condition worked as build log messages contain text “Tests Failed” each time test build step failed.

Hope this tip helps you avoid making the same mistake and save few hours. 🙂

The post Team City – Failed Tests but Build Pass appeared first on Hi, I'm Ankit.

.NET Core – Running Background Worker on IIS

Ankit Vijay — Mon, 18 Mar 2019 22:09:15 +0000

One of the crucial pieces of the new solution that I’m working on is RabbitMQ. For those who have never heard of RabbitMQ, it is one of the most widely used open source message broker. Since our solution is hosted on-premise, RabbitMQ was one of the most natural fit for us. Our entire solution architecture is built on .NET Core 2.1 and recently migrated to .NET Core 2.2. To run the RabbitMQ we had 3 options:

Run RabbitMQ as a traditional .NET Framework “Windows Service”
Run RabbitMQ as a .NET Core “Windows Service”
Host RabbitMQ on IIS as a .NET Core application

Creating a .NET Framework based Windows Service was a known beast, as we have done it hundred times. However, since our entire solution was based on .NET Core, it was a step backward. As a result, we started exploring option 2, that is hosting a .NET Core based Windows Service. Our solution was largely inspired (read “Copied”) from Steve Gordan’s post on “Running a .NET Core Generic Host App as a Windows Service“

Based on his post, we configured the solution to run as console application during development, and then as windows service at the time of deployment. However, we did not find deploying the windows service on .NET Core as simple as we initially thought. We had to modify project file to add RunTimeIdentifier or RID. RID is OS and architecture specific. It is a different string based on OS, Version, Architecture. We did not want OS specific dependencies in our solution. We did not OS specific dependencies in our solution. In addition to this, a Windows-service also come up with its own set of complexities. It is difficult to debug and monitor as compared to a hosted-service.

This is where, we looked into the third option: Host our Background Worker on IIS. Hosting the Background Worker on IIS meant that we were able to deploy our Background Worker same as we would deploy a web app. Instead of using a “Generic Host” we were able to use default “Web Host”. There was less custom code. It was easier to debug and monitor. In addition to this, we were also able to leverage .NET Core Health Checks.

To run the background worker on the IIS, we had to tweak IIS settings to keep it always up and running. Here is the PowerShell script which we run on Octopus to update the IIS settings.

Hope you find this useful.

The post .NET Core – Running Background Worker on IIS appeared first on Hi, I'm Ankit.

When I brought down build-pipeline for entire organization

Ankit Vijay — Thu, 14 Feb 2019 21:42:14 +0000

December last year, last week before most of IT staff go on well-deserved vacation, when every team was trying to do one last deployment before code freeze, I brought down build-pipeline for almost every project in the company. Sounds scary? Here’s what happened.

How it all started

We are mostly a .NET shop and use Team City for our builds. We have a number of Build Agents for running builds for our projects in Team City. A couple of days back before the incident we changed the repository name of our project. Subsequently, we updated the repo path in Team City for our project. After that, we did a dry run of the build and confirmed that it was all green.

What caused the issue

However, once those changes were made, I noticed that few of our feature-branch builds, only on certain Build Agents started to fail with a weird error Cannot find master branch.. something something

I did a little bit of reading about the issue and it appeared to be a caching issue where Team City for some reason was not able to pick up the updated repository path name. It looked quite reasonable to clear relevant Team City cache variables. So, I went ahead and cleared some of the cache variables. After that, I ran my builds on all the Agents and everything was green again.

The Chaos

Next day, when I reached the office, there was a choas. People were complaining about failing builds on Team City. Since I was the one who last touched the Team City configuration it was highly likely to be caused my changes.

We started digging deeper into the issue. We found out that the Team City “Nuget Plugin” was not working correctly and threw some weird errors for almost all the builds. So, the first thing we did is what every IT specialist does, try restarting the Build Server and Build Agents :). Unfortunately, this time it did not help. There was absolutely no help on the internet.

The Fix

After several trials and errors, we finally managed to fix the major issue with NuGet Plugin.

Clearing the cache had somehow uninstalled NuGet plugin for all the NuGet Versions and Build Agents. To fix this, we had to manually stop Build Agent, reinstall the NuGet Version and start the Build Agent again. Then, repeat the process for all the required NuGet Versions (each project was using different version – may be the lastest one at the creation of build). To add to our pain, Team City Nuget plugin turned out to be case sensitive and for some mysterious reason, the case of one of the NuGet Version file was different from what Team City would have liked. Fortunately, we were able to resolve all the issues related to Nuget plugin.

Fixing the NuGet plugin issue fixed the build pipeline for most of our projects. But, few others were still failing because they had taken the dependency on NuGet packages cached on the Build Server. To fix we had to add the relevant NuGet source feed in the build steps.

It was not over yet….

Just when I thought that all the issues were resolved and I could go home peacefully, life came full circle and we started getting the original error on my build. At this point in time, there was no way I could take the same risk to clear the Team City cache.

So, I did some further digging and found out this issue. Here is the summary of the issue:

We were using GitVersionfor semanticversioning for our builds. GitVersion needs access to master/develop branch to calculate the version number but Team City by default does not fetch the master branch (unless already fetched). To resolve this issue, all I had to do is to add a configuration parameter numbers.git.fetchAllHeads=true. Adding this parameter fixed the issue.

Lessons Learnt

We had many lessons learnt from this incident. First and foremost, was to be careful when dealing with Team City cache. However, one thing key thing here was that all the builds which did not have a dependency on Team City plugins (like mine) did not have any issue. As an organization, we are moving more towards scripting the build steps as opposed to using plugins. This incident just validated our decision. For all the goodies Team City provide, I feel it still has a long way to go. While the other build management tools support YAML, configuring the builds through UI is still the preferred way for Team City. You cannot version control your build definition. Maybe as an organization, we need to start evaluating other options out there in the market.

Final Thoughts

This incident made me realize how awesome our IT team is. There was no finger-pointing and everyone was rather focused on fixing the issue. I’m lucky to be working here. And we are expanding, so if you are interested in joining our team, please email me at vijayankit@outlook.com :)(https://s.w.org/images/core/emoji/11/72x72/1f642.png)

The post When I brought down build-pipeline for entire organization appeared first on Hi, I'm Ankit.

My Stack Overflow journey

Ankit Vijay — Wed, 09 Jan 2019 22:17:22 +0000

Stack Overflow does not need an introduction. It is one platform that has helped millions of developers across the world to be a better programmer. It is Google of software development. Maybe even more, because, Google still has competitors, Stack Overflow has none.

Recently, I managed to cross 1000 reputation points on Stack Overflow. You may be thinking, what’s the big deal? Who writes a blog post for that? There are hundreds of people out there with over 100K reputations and many who may be adding 1000 reputation in a day. You are right, it is not a big deal to reach have 1000 reputations after 10 years of being a programmer. But still, it is somewhat special for me.

My first question

I have been using Stack Overflow since forever but I have never been very active on the platform I asked my first question way back in 2011 when Windows Phone was still a rage.

What motivated me

What really changed my outlook towards the platform was this question. I was at a customer location, highly stressful environment desperately looking for a solution to a not-so-easy problem. However, probably within 15 mins, my question was downvoted with a comment that I need to provide more information. I edited my question, provided more information but all I got was two not very helpful answers. This made me feel, maybe Stack Overflow is not so helpful platform after all, especially, if you were a new user. I found Stack Overflow to be hostile where you had to be lucky, very descriptive (difficult for a new user) or have a good reputation to get an answer to your question. The barrier of entry was just too high.

What I wanted to change with my small contribution to the otherwise great community, was to help the folks who felt the same way as me. I never had time, energy or motivation to spend hours on Stack Overflow. But, spend a few minutes here and there in a week.

Other such example was this question. I spent 20 mins framing the question, trying to explain my problem in-depth. But, it was downvoted within a minute without any explanation. Luckily, by that time I had gained enough reputation to put a bountyand get a good answer from the community.

My contribution in numbers

Asked 15 questions on the platform, with 26 replies and 5 marked as answers.
Answered 96 questions, of which 35 answers were upvoted at least once, 22 were marked as answer and 3 were downvoted.
My most popular question till date is not from Stack Overflow but Stack Engineering. It has received 51K views and 36 upvotes.
Edited 41 posts, raised 27 flags and cast 114 votes, mostly upvotes.

How Stack Overflow has helped me

Contributing to Stack Overflow has helped me in many ways. It has helped me to better frame my problem statement. I feel I’m able to explain my problem to others much better than before. When I answer others question, I need to do a little bit of research myself, helping me to learn and grow. It gives immense satisfaction and joy to know that I could help a fellow developer somewhere in the world with his/ her problem. When I find questions for which I have no idea, it makes me realize where I stand. It keeps me grounded. Few of my questions/ answers have turned into a blog post and visa-versa. But, most importantly, Stack Overflow has helped me become a better programmer.

If you are a programmer to then I encourage you to contribute to Stack Overflow or similar platform. I’m sure you will enjoy the experience and learning which comes along with it.

The post My Stack Overflow journey appeared first on Hi, I'm Ankit.

Override appSettings during development – .NET Core

Ankit Vijay — Thu, 27 Dec 2018 22:13:31 +0000

Sometime back, I had written about how to override appSettings during development in traditional ASP.NET application.

Recently, we started development of a ASP.NET Core application and had a similar challenge. Our developers work on different operating systems (Windows and Mac). They have different local connection strings and application settings. We had a same problem as earlier: How do we ensure that we do not store developer specific app settings/ connection string in source control.

Configuration in ASP.NET Core is quite advance and extensible as compared to traditional ASP.NET Framework. We have multiple ways to solve this problem. From the documentation:

App configuration in ASP.NET Core is based on key-value pairs established by configuration providers. Configuration providers read configuration data into key-value pairs from a variety of configuration sources:

Azure Key Vault

Command-line arguments

Custom providers (installed or created)

Directory files

Environment variables

In-memory .NET objects

Settings files

Also, ASP.NET Core offers following two recommended ways to store app secrets during development:

Environment variables
Secret Manager

One of the issue, with the above two approaches is that the settings are stored outside the IDE. Also, since these settings override the appSettings.json file it can cause confusion while debugging as the developer may not realize that settings are being overridden.

Our Approach

For our project we were looking for the simplest possible solution without any additional overhead. The easiest possible solution us for was to use appSettings.Developement.json file to store local settings and exclude it from source control by adding to .gitignore (we use Git).

However, we already have a separate “Dev” environment where we first deploy our application before pushing to Test and Live environments. Using the default Developement environment for the purpose of storing the local settings could have confused the developers.

As a result, we ended up adding a new appSettings.Local.json file to the project. This setting file was then added to .gitignore to exclude this file from source control.

Here is a sample appSettings.Local.json

We then, changed our launchsettings.json and updated ASPNETCORE_ENVIRONMENT to point to Local instead of Development

Last but not the least, we updated Startup.json to use developer exception page while doing development.

That’s it! This approach helped us to keep all our settings in one place (within our IDE) and it turned out to be a pretty neat solution to our problem.

Hope you find this tip useful. Please share your thoughts in comments section below.

The post Override appSettings during development – .NET Core appeared first on Hi, I'm Ankit.

DDD Brisbane – A weekend well spent

Ankit Vijay — Sun, 02 Dec 2018 00:03:16 +0000

This weekend (that is December 1st, 2018, Saturday), I along with a few of my colleagues from the office got an opportunity to attend DDD Brisbane conference. All thanks to my Employer, Youi, which was also the gold sponsor of the event. (We are hiring). If you do not know what is DDD conference is – It is short for Developer, Developer, Developer. It is a conference for, from, by, to, with, along, only, whatever…. Developers.

Attending a conference on a weekend can be a challenging discussion with the spouse. It is not easy to justify why I would wake up 5 o’ clock on a Saturday morning to go to Brisbane and return late at night. Really grateful to my wife for her understanding and support. I have promised that I will make it up to her

This was my first time at a DDD conference, but certainly not the last. I thoroughly enjoyed the experience, listening to excellent speakers, food, free coffee, and goodies.

My DDD attendee card

Here are my few key takeaways from the conference:

The conference was very well organized. At the registration process was smooth, the organizers were friendly and helpful.
Excellent food, free coffee, a lot of goodies and so many prizes to be won (unfortunately, I did not win any).

It was a “green” conference in the true sense. The attendees were encouraged to bring their own coffee mugs. The coffee mugs, including the lid, were recyclable. The food plates, cutlery was made of bamboo, hence compostable. The reusable bags (or as they called it “swag bags”) were made of waste cloth by the local community.

My Swag Bag

It was refreshing to see so many women attendees and speakers. DDD had by far the highest ratio of women attendees and speakers of any of the conferences I had previously attended.

@DDDBrisbane quick spot of feedback. I had to WAIT in line for a ladies bathroom to be free. I have NEVER had to do that while attending a software conference before.

So I guess what I’m trying to say is nice work, keep it up #DDDBne

— Emily Taylor (@_emilol) December 1, 2018

DDD offered Free Child Care facility. Again, this was the first time I saw any conference which offerred such facility.

Thanks to our Silver sponsor @thoughtworks for sponsoring this year! We've said it before and we'll say it again – we can't thank them enough for funding and organising child care for us. #dddbne

— DDD Brisbane (@DDDBrisbane) December 1, 2018

The speaker line up was a mix of first-timers and experienced speakers. I enjoyed all the sessions which I attended.

Enough gap between the two sessions – there was no information overload.
There was something to learn in every session. None of them were a waste of time.
Few sessions also validated that we follow the industry-wide and recommended development practices at work. In fact, we are probably ahead of rest of industry in some area :). In particular, code snippets from clean architecture with .NET Core 2.2 session looked to be straight out of our private repo.
It was great opportunity to interact with fellow developers
Special mention to keynote speaker Jessica Kerr. But the highlight of the day for me was locknote from Neal Ford on “Support Constant Change”. There was just so much to learn in his 45 mins talk.

Hit the nail..

All-in-all it was a great weekend spent learning new things.

The post DDD Brisbane – A weekend well spent appeared first on Hi, I'm Ankit.

Journey towards UI tests greatness – Lessons Learnt

Ankit Vijay — Tue, 30 Oct 2018 22:08:48 +0000

A bit of a background

I work on a long-running running project in our organization. We have a CI/CD pipeline for our project following the GitHub flow and backed by thousands of unit tests, hundreds of integration tests. However, until recently we were missing a key piece to the puzzle. UI Tests!

It’s not that we did not have UI tests, we had plenty (rather just too many) but we were not able to run them on our build server with confidence for the reasons I will go into later. And since UI tests were not running on every build, we still needed to verify each build for regression. In short, we were not really doing a Continuous Delivery. Our QA as smart he is, then built his own set of UI test suite which he would run on every build to validate the quality of the build.

Why we were not able to run UI tests with Confidence

Our UI tests were flaky. The tests would fail often to due _ external factors _ especially on the build server_ . _ On top of it, the very nature of UI tests is that they are very slow as compared to unit tests or integration tests. To run the entire test suite every time a test failed due to some external factors wasted a lot of time.
Our UI tests were brittle. They were trying to do more than a UI test. They were heavily reliant on the HTML page structure. As a result, they would fail even with minimal changes.
The flaky and brittle nature of UI tests meant that the developers lost trust in the UI test framework. As a result, they were not well maintained.
The UI tests were flooded with a lot of a Thread.Sleep. And Thread.Sleep led to two bigger issues:
- There was no way we could know what was the right delay. Each second delay added had a ripple effect on the already slow tests
- In spite of adding the delays, there was no guarantee that the tests would pass.
The UI tests did not really represent the business scenarios. They were written more with an idea of testing the UI elements. Hence, at times we end up testing not the actual business use case.
The tests did not follow SOLID and DRY principles. They never got the same love and treatment as application code. This made them difficult and harder to maintain.

Fixing the broken tests

We are a very small team of dev and single QA. However, we deliver the new business features and bug-fixes at a very high velocity. As the complexity of the solution started growing it became more evident and obvious to that we cannot deliver the solution with a high confidence without the UI tests. Our QA was forced to focus more on mundane testing rather than planning ahead for the next User Stories.

That’s when we took a step back and decided to do something about it. We listed down our pain-points with existing UI framework and worked upon each point to develop UI framework 2.0. The ultimate aim of this exercise was to resolve the above issues and start running the tests as part of our CI/CD pipeline.

The Solution

Put business first – To ensure that we do not repeat our mistakes we started our UI tests with the question how would our consumers use our application. We worked with our QA to get the list of important business scenarios. We planned our test cases BDD style and chose SpecFlow to write our tests. SpecFlow allowed us to write tests in such that both our BA and QA could easily understand the test cases and even contribute to the tests. While planning for the test cases we kept reminding ourselves of Test Pyramid.
Follow good development practices – The UI test framework was no longer a rejected child. We developed the framework same way as we would write an application code. The tests followed SOLID and DRY principles. All the components such as SpecFlow, Business Logic, Selenium were loosely coupled. The framework allowed Dev to not worry about the low-level UI elements but only what steps needed to be executed in a scenario.
No Thread.Sleep – Thread.Sleep in UI tests is evil. At the start, it looks to be the solution to every problem. But soon it starts creating problems. The framework 2.0 did not use Thread.Sleep anywhere (or may just one place :)). The Thread.Sleep was replaced by Selenium implicit and explicitwaits. This helped us keep the tests fast and reliable.
Compatible with Feature Flags – I mentioned in one of my older posts that we use Launch Darkly for feature flag management. The new UI framework allowed us to test scenarios with different feature flag variations (example: feature turned OFF/ ON). Feature flags were an essential part of our tests as they could vary depending on the environment, flow etc. Hence, it was important for us to be able to test all the variations easily.
Retry Policy – In spite of all the goodness mentioned in the above points, the UI test suite could still fail due to factors beyond our control. That’s where we decided to add retry policy. The retry policy was defined at two levels:
- Retry the _ action, _ if the action failed in the first attempt.
- Retry the _ test case _, if the test failed in the first attempt.

Developing UI framework 2.0 was a journey and it took quite some time and effort for us to reach to the point where we could start running our UI tests as part of our CI/CD pipeline again. As we worked on the framework to get the constant feedback, we ran the UI tests on the build server separate to our CI/CD pipeline. Running the UI tests in a separate build allowed us to make improvements to the framework without impacting the business-as-usual. And once we had enough confidence in our UI tests we got rid of the separate build and made the UI tests part of the mainline build.

What’s Next

There is no such thing as a perfect solution or framework. We intend to improve the framework further. Few things which we are looking to improve in future are:

Run tests in parallel to reduce the build time on the server
Add more complex business scenarios to bring the test effort to minimal
Integrate tests with BrowserStack to across the platform and devices. Check out my post to know more about BrowserStack.

The post Journey towards UI tests greatness – Lessons Learnt appeared first on Hi, I'm Ankit.