DEV Community: Niraj Kumar

Application Security Best Practices / Defensive Programming

Niraj Kumar — Sat, 05 Sep 2020 19:09:34 +0000

My Other lists

Application security is a critical topic. Being a good engineer requires being aware of Application security best practices. You should practice defensive programming to ensure a robust, secure application. I have collected points and created this list for my reference. Hope, you too get benefitted out of this. This is a big list, but worth reading to the end.

Always Handle Exceptions
Validate user input to avoid common vulnerabilities (e.g. XSS, SQL-Injection, Remote Code Execution, etc.)
Escape HTML, JS and CSS output
HTML Encode Before Inserting Untrusted Data into HTML Element Content
JavaScript Encode Before Inserting Untrusted Data into JavaScript Data Values
URL Encode Before Inserting Untrusted Data into HTML URL Parameter Values
Implement type conversion with strict exception handling
Apply minimum and maximum value range check for numerical parameters and dates, minimum, and maximum length check for strings.
Use an array of allowed values for small sets of string parameters (e.g. days of week).
Implement CAPTCHA
Don't use Basic Auth. Use standard authentication instead (e.g. JWT, OAuth)
Check if all the endpoints are protected behind authentication to avoid broken authentication process
User own resource ID should be avoided. Use /me/orders instead of /user/654321/orders
Don't reinvent the wheel in Authentication, token generation, password storage. Use the standards.
Tokens must be transmitted using the Authorization header on every request
Authorization Code should be short-lived / Make token expiration (TTL, RTTL) as short as possible
Don't store sensitive data in the JWT payload, it can be decoded easily
Use Max Retry and jail features in Login
Consider using rate-limiting (throttling) to protect your resources from DDoS or brute force attacks
Code reviewer should always consider security guidelines during code reviews
Do not forget to turn the DEBUG mode OFF
Hide error details from clients, wherever applicable respond with generic error messages avoid revealing details of the failure unnecessarily
Do not pass technical details (e.g. call stacks or other internal hints) to the client
Never return sensitive data like credentials, Passwords, or security tokens
Avoid publishing secrets to the npm registry or Code repository
Inspect for outdated packages
Use a random complicated key (JWT Secret) to make brute-forcing the token very hard.
Support blacklisting JWTs or other authentication tokens
Do not use GET requests for state-changing operations
Run Node.js as a non-root user
Limit request payload size using a reverse-proxy or a middleware
The application must be programmed to defend against attacks from the OWASP TOP 10 (More details below)
Implement proper and regular security auditing of the application
Don't auto-increment IDs. Use UUID instead
Use a CDN for file uploads
Write audit logs before and after every security-related events
Consider logging token validation errors in order to detect attacks.
Take care of log injection attacks by sanitising log data beforehand.
Use HTTPOnly cookie flag
OAuth security
- While using oAuth, Always validate redirect_uri server-side to allow only whitelisted URLs
- Always try to exchange for code and not tokens (don't allow response_type=token)
- Define the default scope, and validate scope parameters for each application
Reject any non-TLS requests by not responding to any HTTP request to avoid any insecure data exchange. Respond to HTTP requests by 403 Forbidden
Use HTTPS on server side to avoid MITM (Man in the Middle Attack)
Setting HTTP headers appropriately can help to lock down and secure your web application. Consider using Helmet library if you are building API on node
Your API should convert the received data to their canonical form or reject them. Return 400 Bad Request with details about any errors from bad or missing data.
Use the proper HTTP method according to the operation: GET (read), POST (create), PUT/PATCH (replace/update), and DELETE (to delete a record), and respond with 405 Method Not Allowed if the requested method isn't appropriate for the requested resource
Don't use any sensitive data (credentials, Passwords, security tokens, or API keys) in the URL, but use standard Authorization header
All the data exchanged with the REST API must be validated by the API.
Serialize your JSON (to avoid arbitrary JavaScript remote code execution)
Validate content-type on request Accept header (Content Negotiation) to allow only your supported format (e.g. application/xml, application/json, etc.) and respond with 406 Not Acceptable response if not matched
Validate content-type of posted data as you accept (e.g. application/x-www-form-urlencoded, multipart/form-data, application/json, etc.)
Use an API Gateway service to enable caching, Rate Limit policies (e.g. Quota, Spike Arrest, or Concurrent Rate Limit) and deploy APIs resources dynamically
If you are dealing with huge amount of data, use Workers and Queues to process as much as possible in background and return response fast to avoid HTTP Blocking
Audit your application's design, implementation and security with unit/integration tests coverage frequently
Implement proper access control for resources and its access
Do not store sensitive information to localStorage
Measure and guard the memory usage
Discover errors and downtime using APM products
Use tools that automatically detect vulnerabilities
Embrace linter security rules
Prevent query injection vulnerabilities with ORM/ODM libraries
Prevent unsafe redirects
Avoid DOS attacks by explicitly setting when a process should crash
Protect the input values from being cached by the browser.
Text areas and input fields for PII (name, email, address, phone number) and login credentials (username, password) should be prevented from being stored in the browser. e.g.
Do NOT allow login with sensitive accounts (i.e. accounts that can be used internally within the solution such as to a back-end / middle-ware / DB) to any front-end user-interface
Implement Proper Password Strength Controls, Include password strength meter to help users create a more complex password and block common and previously breached passwords
Implement Secure Password Recovery Mechanism
Store Passwords in a Secure Fashion
Compare Password Hashes Using Safe Function and in a timely manner
Transmit Passwords Only Over TLS or Other Strong Transport
Require Re-authentication for Sensitive Features
Consider Strong Transaction Authentication and Authorization
Require JavaScript and Block Headless Browsers
Notify users about unusual security events
Regular review of user accounts and their permissions
Secure File uploads
- Whitelist allowed extensions. Only allow safe and critical extensions for business functionality
- Ensure that input validation is applied before validating the extensions.
- Validate the file type, don't trust the Content-Type header as it can be spoofed
- Change the filename to something generated by the application
- Set a filename length limit. Restrict the allowed characters if possible
- Set a file size limit
- Only allow authorised users to upload files
- Store the files on a different server. If that's not possible, store them outside of the webroot
- In the case of public access to the files, use a handler that gets mapped to filenames inside the application (someid -> file.ext)
- Run the file through antivirus or a sandbox if available to validate that it doesn't contain malicious data
- Ensure that any libraries used are securely configured and kept up to date
- Protect the file upload from CSRF attacks
In the event of authentication exception, Using any of the authentication mechanisms (login, password reset or password recovery), an application must respond with a generic error message regardless of whether:
- The user ID or password was incorrect.
- The account does not exist.
- The account is locked or disabled.
Implement a mechanism to protect the application against Automated Attacks
Implement Multi-Factor Authentication for optional but better security where advanced security is required
Enable logging and monitoring of authentication functions to detect attacks/failures on a real-time basis
- Ensure that all failures are logged and reviewed
- Ensure that all password failures are logged and reviewed
- Ensure that all account lockouts are logged and reviewed
Don't rely on client logic for security
Don't rely on client business logic
Don't perform encryption in client side code. Use TLS/SSL and encrypt on the server!
Use CSRF Protection
Be vigilant about the following:
- Injection: Almost any source of data can be an injection vector, environment variables, parameters, external and internal web services, and all types of users. Injection flaws occur when an attacker can send hostile data to an interpreter.
- Broken Authentication: Attackers have access to hundreds of millions of valid username and password combinations for credential stuffing, default administrative account lists, automated brute force, and dictionary attack tools. Session management attacks are well understood, particularly in relation to unexpired session tokens
- Sensitive Data Exposure: Rather than directly attacking crypto, attackers steal keys, execute man-in-the-middle attacks, or steal clear text data off the server, while in transit, or from the user's client, e.g. browser. A manual attack is generally required. Previously retrieved password databases could be brute-forced by Graphics Processing Units (GPUs). Secure Search Serves, do not log sensitive data
- Broken Access Control: Exploitation of access control is a core skill of attackers. Access control is detectable using manual means, or possibly through automation for the absence of access controls in certain frameworks.
- Security Misconfiguration: Attackers will often attempt to exploit unpatched flaws or access default accounts, unused pages, unprotected files and directories, etc to gain unauthorized access or knowledge of the system
- Cross-Site Scripting (XSS): XSS is the second most prevalent issue in the OWASP Top 10, and is found in around two-thirds of all applications
- Insecure Deserialization: Exploitation of deserialization is somewhat difficult, as off-the-shelf exploits rarely work without changes or tweaks to the underlying exploit code. Using Components with Known Vulnerabilities: While it is easy to find already-written exploits for many known vulnerabilities, other vulnerabilities require concentrated effort to develop a custom exploit.
- Insufficient Logging & Monitoring: Exploitation of insufficient logging and monitoring is the bedrock of nearly every major incident. Attackers rely on the lack of monitoring and timely response to achieve their goals without being detected
Forgot Password Service Guidelines
- Return a consistent message for both existent and non-existent accounts.
- Ensure that the time taken for the user response message is uniform.
- Use a side-channel to communicate the method to reset their password.
- Use URL tokens for the simplest and fastest implementation.
- Ensure that generated tokens or codes are:
  - Randomly generated using a cryptographically safe algorithm.
  - Sufficiently long to protect against brute-force attacks. Stored securely.
  - Single-use and expire after an appropriate period
Secure Management endpoints
- Avoid exposing management endpoints via the Internet.
- If management endpoints must be accessible via the Internet, make sure that users must use a strong authentication mechanism, e.g. multi-factor.
- Expose management endpoints via different HTTP ports or hosts preferably on a different NIC and restricted subnet.
- Restrict access to these endpoints by firewall rules or use of access control lists.
Where possible, always log:
- Input validation failures e.g. protocol violations, unacceptable encodings, invalid parameter names and values
- Output validation failures e.g. database recordset mismatch, invalid data encoding
- Authentication successes and failures
- Authorization (access control) failures
- Session management failures e.g. cookie session identification value modification
- Application errors and system events e.g. syntax and runtime errors, connectivity problems, performance issues, third party service error messages, file system errors, file upload virus detection, configuration changes
- Application and related systems start-ups and shut-downs, and logging initialization (starting, stopping or pausing)
- Use of higher-risk functionality e.g. network connections, addition or deletion of users, changes to privileges, assigning users to tokens, adding or deleting tokens, use of systems administrative privileges, access by application administrators,all actions by users with administrative privileges, access to payment cardholder data, use of data encrypting keys, key changes, creation and deletion of system-level objects, data import and export including screen-based reports, submission of user-generated content especially file uploads
- Legal and other opt-ins e.g. permissions for mobile phone capabilities, terms of use, terms & conditions, personal data usage consent, permission to receive marketing communications
In order to secure authorization-based transaction in such a system:
- Authorization should be performed and enforced server-side
- Transaction verification data should be generated server-side
- Application should prevent authorization credentials brute-forcing
- Transaction data should be protected against modification
- Confidentiality of the transaction data should be protected during any client / server communications
- When a transaction is executed, the system should check whether it was authorized
- Authorization credentials should be valid only by a limited period of time
- Authorization credentials should be unique for every operation
Protect Personally Identifiable Information (PII Data)
- Personally identifiable information (PII) is any data that can be used to identify a specific individual
- Protect Personally Identifiable Information in the Applications by encrypting them
- Follow the data privacy laws of the land

Some Important OWASP Guidelines

OWASP A2: Broken Authentication
- Require MFA/2FA for important services and accounts
- Rotate passwords and access keys frequently, including SSH keys
- Apply strong password policies, both for ops and in-application user management (link OWASP password recommendation)
- Do not ship or deploy your application with any default credentials, particularly for admin users or external services you depend on
- Use only standard authentication methods like OAuth, OpenID, etc. avoid basic authentication
- Auth rate-limiting: Disallow more than X login attempts (including password recovery, etc.) in a period of Y
- On login failure, don't let the user know whether the username or password verification failed, just return a common-auth error
- Consider using a centralized user management system to avoid managing multiple accounts per employee (e.g. GitHub, AWS, Jenkins, etc) and to benefit from a battle-tested user management system
OWASP A3: Sensitive Data Exposure
- Only accept SSL/TLS connections, enforce Strict-Transport-Security using headers
- Separate the network into segments (i.e. subnets) and ensure each node has the least necessary networking access permissions
- Group all services/instances that need no internet access and explicitly disallow any outgoing connection (a.k.a private subnet)
- Store all secrets in vault products like AWS KMS, HashiCorp Vault or Google Cloud KMS
- Lockdown sensitive instance metadata using metadata
- Encrypt data in transit when it leaves a physical boundary
- Don't include secrets in log statements
- Avoid showing plain passwords in the frontend, take necessary measures in the backend and never store sensitive information in plaintext
OWASP A5: Broken access control
- Respect the principle of least privilege - every component and DevOps person should only have access to the necessary information and resources
- Never work with the console/root (full-privilege) account except for account management
- Run all instances/containers on behalf of a role/service account
- Assign permissions to groups and not to users. This should make permission management easier and more transparent for most cases
OWASP A6: Security Misconfiguration
- Access to production environment internals is done through the internal network only, use SSH or other ways, but never expose internal services
- Restrict internal network access - explicitly set which resource can access other resources (e.g. network policy or subnets)
- If using cookies, configure it to "secured" mode where it's being sent over SSL only
- If using cookies, configure it for "same-site" only so only requests from same domain will get back the designated cookies
- If using cookies, prefer "HttpOnly" configuration that prevents client-side JavaScript code from accessing the cookies
- Protect each VPC with strict and restrictive access rules
- Prioritize threats using any standard security threat modelling like STRIDE or DREAD
- Protect against DDoS attacks using HTTP(S) and TCP load balancers
- Perform periodic penetration tests by specialized agencies
OWASP A7: Cross-Site-Scripting (XSS)
- Use templating engines or frameworks that automatically escape XSS by design, such as EJS, Pug, React, or Angular. - - Learn the limitations of each mechanisms XSS protection and appropriately handle the use cases which are not covered
- Escaping untrusted HTTP request data based on the context in the HTML output (body, attribute, JavaScript, CSS, or URL) will resolve Reflected and Stored XSS vulnerabilities
- Applying context-sensitive encoding when modifying the browser document on the client-side acts against DOM XSS
- Enabling a Content-Security Policy (CSP) as a defence-in-depth mitigating control against XSS
OWASP A9: Using Components With Known Security Vulnerabilities
- Scan docker images for known vulnerabilities (using Docker's and other vendors' scanning services)
- Enable automatic instance (machine) patching and upgrades to avoid running old OS versions that lack security patches
- Provide the user with both 'id', 'access' and 'refresh' token so the access token is short-lived and renewed with the refresh token
- Log and audit each API call to cloud and management services (e.g who deleted the S3 bucket?) using services like AWS CloudTrail
- Run the security checker of your cloud provider (e.g. AWS security trust advisor)
OWASP A10: Insufficient Logging & Monitoring
- Alert on remarkable or suspicious auditing events like user login, new user creation, permission change, etc
- Alert on irregular amount of login failures (or equivalent actions like forgot password)
- Include the time and username that initiated the update in each DB record

NOTE: If you want to update this list, please comment, I'll incorporate your changes.

PR/Code Review Best Practices

Niraj Kumar — Sat, 05 Sep 2020 17:30:14 +0000

My Other lists

I have collected some points and created this list for PR and Code reviews. I intend to refer to this post frequently to help me become a better reviewer.

Organization

Organizations with good code reviews ensure that all engineers take part in the code review process
Good code reviews use the same quality bar and approach for everyone, regardless of their job title, level, or when they joined the company.
Better code reviews pay additional attention to making the first few reviews for new joiners a great experience. Reviewers are empathetic to the fact that the recent joiner might not be aware of all the coding guidelines and might be unfamiliar with parts of the code. Provide relevant documentation (guidelines, style guides, etc) to help engineer improve their code.
Ensure no code makes it to production without a code review
PR and reviews should be time-bound. Ensure that PRs get reviewed, approved/rejected within a scrum cycle to avoid stale code
Foster a good code review culture in which finding defects is viewed positively, Give respectful and constructive feedback

Reviewers

Prioritize Review Over Coding
Integrate code review into your daily routine
Review at least part of the code, even if you can't do all of it
Review fewer than 400 lines of code at a time
Take your time while reviewing someone else's code, do not depend on others to catch errors
Do not bunch PRs for review. Instead, scatter reviews tasks throughout the working hours
Ensure the PRs comes with tests
Build and Test — Before Code Review
Ensure external documents if any (API, user manual, etc.) are updated
Give Feedback That Helps (Not Hurts)
Create and follow a code-review checklist.
While reviewing code, be mindful about the following:
- Security best practices
- Manageability (Readability, structure, style)
- Architecture
- Maintainability (Logic, Test Coverage)
- Correctness (Functionality)
- Invalid input/states
- Usability (Performance)
- Reusability
- Object-Oriented Analysis and Design (OOAD) Principles
Verify that the defects if any -are actually fixed
PR Rejections must always include a better explanation for why the PR got rejected

Author/Reviewee

There are various git workflow, get yourself acquainted with different git workflows: https://www.atlassian.com/git/tutorials/comparing-workflows
You should annotate PRs with self-explaining comments before the review
Create Small Change Lists, Change Lists Should Be Complete on Their Own
Never self-approve a PR, always ask your peers for approval after code review
Before opening a PR, make sure you squash all your commits into one single commit using git rebase (squash). Instead of having 50 commits that describe 1 feature implementation, there must be one commit that describes everything that has been done so far.
Prior to making PRs always check to see you have merged the most updated code from the master branch. Update all affected tests and or create new tests for files you changed.
Do not push any code with runtime errors thrown by dependencies or source code. When one appears, you should fix that bug ASAP—even if everything else seems to be running ok. There’s a reason that error was thrown, so fix it and don’t leave it for someone else to worry about it.
Remove all unnecessary console logs prior to making a PR

PR Checklist
☐ Unit Test provided
☐ Deprecated code removed
☐ Syntax & Formatting is correct (Linting should help with this)
☐ Is the approach to the problem appropriate?
☐ Can anything be simplified?
☐ Is the code too specific to the product and needs generalization?
☐ Do forms need to sanitized
☐ Any obvious security flaws or potential holes
☐ Are the naming conventions appropriate?
☐ Are there enough comments inline with the code?
☐ Is there documentation needed?
☐ Is there anything included that should not be?
☐ Are all dependencies declared?
☐ Are there any code smells? https://blog.codinghorror.com/code-smells/

NOTE: If you want to update this list, please comment, I'll incorporate your changes.

Ref.
https://www.ibm.com/developerworks/rational/library/11-proven-practices-for-peer-review/index.html
https://www.michaelagreiler.com/code-review-best-practices/
https://ardalis.com/github-pull-request-checklist/
https://gist.github.com/sherakama/0ba17601381e3adbe0cad566ad4d80a5

JavaScript/Node Best Practices

Niraj Kumar — Sat, 05 Sep 2020 15:56:13 +0000

My Other lists

I usually follow Airbnb's JavaScript Style Guide, still, I intend to keep this list as reference. Most of these rules will be enforced automatically if you integrate ESLint in your project.

Always use 'use strict' if you are still using es5
Don't use global variables
Always Prefer const over let. Ditch the var
Prefer writing pure functions over stateful functions which mutate data or produce side effects
Learn and use functional composition
Favor functional programming over imperative programming
Use method chaining
Prefer composition over inheritance
Use linters to ensure your code is consistent.
Use Airbnb JavaScript Style Guide for JavaScript (https://github.com/airbnb/javascript)
Avoid client-side console logs in production
Prefer '===' over '=='
Use default arguments instead of short-circuiting or conditionals
Function arguments (2 or fewer ideally)
Encapsulate conditionals in a separate function if possible
Avoid negative conditionals
Learn and practice implementation of SOLID Patterns
Use Promises, not callbacks
Async/Await is even cleaner than Promises, use it more
Use try/catch with async/await
Use Async-Await or promises for async error handling
Don't ignore rejected promises, log it to external logging service
Never use eval
Structure your solution by components
Wrap common utilities as npm packages
Separate Express 'app' and 'server'
Use environment aware, secure and hierarchical config
Distinguish operational vs programmer errors
Use only the built-in Error object
Handle errors centrally, not within a middleware
Exit the process gracefully when an unknown fatal error occurs
Use a mature logger to increase error visibility
Discover errors and downtime using APM products (sentry.io)
Catch unhandled promise rejections
Fail fast, validate arguments using a dedicated library
Use ESLint
Separate your statements properly
Prefer named function over anonymous. Name all functions, including closures and callbacks. Avoid anonymous functions, as it helps in profiling
Require modules by folders, opposed to the files directly
Require modules at the beginning of each file, before and outside of any functions
Detect code issues with a linter
Refactor regularly using static analysis tools
Avoid using the Node.js crypto library for handling passwords, use Bcrypt
Prevent evil RegEx from overloading your single thread execution
Don't block the event loop
Bootstrap using 'node' command, avoid npm start (In container environment)

NOTE: If you want to update this list, please comment, I'll incorporate your changes.

Ref.
https://github.com/goldbergyoni/javascript-testing-best-practices
https://github.com/ryanmcdermott/clean-code-javascript
https://github.com/goldbergyoni/nodebestpractices
https://github.com/RisingStack/node-style-guide
https://github.com/DrkSephy/es6-cheatsheet

Software Development Best Practices

Niraj Kumar — Sat, 05 Sep 2020 15:28:00 +0000

My Other lists

I had collected some points and created a list of some best practices for software development. I plan to refer to this list from time to time to help me become a better Engineer. Publishing the list so that others too may refer and get benefitted from this list.

Learn & practice Design Patterns
Practice KISS (Keep It Simple Stupid), DRY (Don't Repeat Yourself) design principles
Practice Occam's Razor problem-solving principles; Occam's razor simply states that of any given set of explanations for an event occurring, the simplest one is most likely the correct one
Always ask questions when in doubt. Do not assume anything, be explicit over being implicit.
Remember, Explicit is better than implicit
Don't be afraid to experiment, or to commit mistakes, however, you must learn from your mistakes quickly and ensure the same mistake does not happen again. Write test cases to protect yourself from common mistakes.
Test everything (Only your code, not other people's code which you are referring, like external libraries)
Keep learning, keep reading latest articles, books, watch tutorials, attend workshops, discuss with experts to update your knowledge and skillsets
Keep practising Algorithms and Puzzles solving to improve your problems solving skills
Document everything and don't forget to update those documents when information or relevant facts change
Never ignore errors or exceptions. Always log your errors with full details sans sensitive information
Log everything and store the logs a remote location for thorough analysis and for historical reference (Loggly/ELK Stack)
Assign a transaction id to each log statement
Increase transparency using smart logging
Continuously analyze Performance of your code and your application
Document your technical debts, and clear those debts as soon as the opportunity arrives
Mark unfinished/incomplete part of the code, //TODO: (Technical Debt)
Better yet, create a fresh ticket for these //TODO
Document your application's API by using Swagger
Periodically perform Security Audit of code/project (Very important)
Always perform even-sided validation Client and Server-side validation both
Comment code to explain intent, use doc comment to document the summary functionality of functions
Keep comments relevant as your code evolves
Don't use comments as an excuse for bad code. Keep your code clean
Don't use clean code as an excuse to not comment at all
Everyone in a team should use the same code formatting strategy (Same prettier configuration everywhere)
Always pull/update before commit/push
Put some extra effort to write a good commit message. You should describe your work briefly in the commit message. First line should be the work summary. Separate the subject from the body with a newline
Within your PR comment write a brief overview of the functionality of your code, or why you did it the way you did it, if possible, write assumptions as well if there are any
Update your Jira/Github Issue task item with actions/user stories, and development checklist
Always make time to review other's code. Add constructive, helpful comments or solutions instead of just pointing out errors or mistakes. Code reading/reviewing improves your thought process and overall understanding of the project
Always constructively criticize someone else's code if they ignore any coding best practices. Point out the specific "guideline" that is not being followed
Always calculate estimates for tasks, add a 20% buffer to the estimate. Try to complete the task within the given estimated time. However, refrain from overestimating, and adjust your estimating strategies continuously to become adept at estimation.
Always challenge yourself, keep updating your estimates, if your success rate is below 70%.
Always tackle the hardest tasks first. Solve the most complicated problems first before solving easy tasks. However, you can always use opportunities as and when it arrives to pick low hanging fruits.
Don't write code that you think you might need in future, but don't need yet
Always remove unnecessary code, keep cleaning dead codes
Fail fast and fail early. Check input and fail on nonsensical input or invalid state as early as possible, preferably with an exception or error response
When in doubt, throw exceptions
Write your code defensively. Always think about what can go wrong, what will happen on invalid input, and what might fail, which will help you catch many bugs before they happen
If a function or method goes past 30 lines of code, consider breaking it up. The good maximum module size is about 500 lines. Test files tend to be longer than this
Refactor whenever you see the need and have the opportunity to do so
Make code correct first and fast second. Make sure your code is working as expected and tests are green; in the refactoring phase, you may optimize your code. When working on performance issues, always profile before making fixes
Use meaningful and pronounceable variable names (Especially as Parameters)
Functions should do one thing and one thing only. Practice Single Responsibility Principle
Function names should say what they do
Always Unit Test your Code (even for simple/quick fixes and one-liners)
Writing code is easy, but reading it is hard, sometimes incomprehensible. So, write code as if you are writing poetry, your code should be easy to read
Still, don't be too verbose in your coding. Remember, compact but comprehensive coding gives less surface area for bugs to hide in
if you are using some reference code picked from some article or StackOverflow, always include the link beside the referenced code as a comment, so that reviewers can visit the link to understand why you did what you did
Organize your files around product features/pages/components, not roles. Also, place your test files next to their implementation.
Design a rollback solution for deployments
Don't over-optimize your code, keep it open for extension
Be stateless, kill your servers almost every day
Serve frontend content using dedicated middleware (Nginx, S3, CDN)
Good habits take some time to set in, so be vigilant and mindful about your habits; keep reading this list

NOTE: If you want to update this list, please comment, I'll incorporate your changes.

Ref.
https://kkovacs.eu/software-project-best-practices-checklist
https://opensource.com/article/17/5/30-best-practices-software-development-and-testing
https://github.com/elsewhencode/project-guidelines
https://github.com/dereknguyen269/programing-best-practices
https://blog.galaxyweblinks.com/software-design-tips-aligning-ideas-through-diagrams/

Building an Automated Equity Stock Trader - 1

Niraj Kumar — Thu, 27 Jun 2019 17:48:55 +0000

I had always wanted to learn about Share Market and Stock Trading, but kept on procrastinating. Then last year I read some articles on Financial Freedom, and the eagerness to better my finances rose again. But, my heart was never into Finance, or anything related to money. I am a programmer and I love programming so much so that I had practically ignored everything else - social life, money management, time management. Heck, sometimes I think, I get paid to do things I love and I might have done my work for a small bag of peanuts, except pizzas cost more.

Since, I am acutely aware of my procrastination issue, I decided to take a new year resolution to get better at financial matters, learn more about it and strive for financial freedom. I wanted to have a system where my money will keep growing even when I sleep.

So, I started learning about share market, by doing active trading and reading about it whenever I could find find time.

Pretty soon, I got hooked. Made some money, and lost more than I made. After a while, I realized these -

I was wasting time and money on penny stocks (which are mostly garbage) with the hope that with given enough time one of these will become multibagger
I have fat fingers (Bungled many trades)
Market is ruled by fast, disciplined, well informed, and well experienced traders
Technical Information about stocks are very important
It's better to do swing trading and keep rotating money to get better yield
I have poor memory, I cannot remember trades, stocks and their prices
I am poor at managing my portfolio
I am super lazy
I get scared easily
I take unnecessary unplanned risks

I had been approached by many portfolio management service providers, but I refused to accept their services - the reason being - I want to learn the system to become better at this. I want to build my own capability.

I had always wanted an automated system which could recommend me which stocks to buy/sell and give me alerts at the right time. Perhaps later, the system can do trading on its own. Trading can be learned, and it can be taught. Success depends on practice, and with better planning, better coding, better persistence. If I can build a system which will do my job for me - Earn me money while I sleep, then that would be an amazing achievement.

I am sure, there are many machines, such systems out there and I could use their service instead of trying to build my own. But, I had always been a doer, I want to use this opportunity to learn more and more. I am using this journey to learn new languages, new technologies - AI/ML, Big Data, Data Science, Algorithmic Trading, Read and Understand Financial Data, Realtime Data Processing. So many things to learn and do, it's all exciting.

Just recently I had completed the first step on this journey - Created a program to help me analyze my old trades and notify me of falling prices. It's a very important first step, and I will keep writing more and more about my journey.

Waste to Wealth

Niraj Kumar — Mon, 03 Jul 2017 06:34:53 +0000

We all produce waste. Some try to monetize it – get the value out of it, while some can’t. We try to keep our home clean and in the process we throw away our garbage without even putting a second thought to the value of it. I have seen my mother selling old newspaper, clothes, bottles, metal and with that savings would buy us treats. However, us city-dwellers lack patience, and space to store our garbage to be able to monetize it. Scrap dealers do not pay nicely unless there is good volume of valuable garbage for him.

Waste Bank!
But, what if there is a waste bank? What if a bank could convert our garbage to money? Imagine a scenario where a waste collector would come to our house daily or at a scheduled time to collect our garbage, and pay us for our garbage? I would use it. My family and my neighbors would use such services as soon as they see the value of their waste. This is a sound business strategy, albeit not a new one. Some tried, and most failed, because they got drowned in the operation phase and could not scale quickly. Such service can become profitable only at scale. Some of the waste can be quite valuable even in small quantity while some need grater volume to become valuable. Every piece of garbage has a value – biodegradables can be used to generate biogas and compost. Glass, plastic, clothes, metal, electronic items are recyclable.

Benefits!
Benefits are innumerable – however the most prominent one is clean environment with proper waste management. This sector needs another Jeff Bezos. Dedicated, intuitive, driven and fierce; Someone who values efficiency at every step; think and plan for long term. Because, with efficiency in place every single piece of garbage can be monetized and used properly instead of polluting our environment and putting humanity at risk. Value can be extracted from the low value garbage only with better efficient process and at larger scale.

How technology can help!
When I think about how technology can help this sector, I immediately think about how amazon used automation to put efficiency in its chaotic fullfillment centers. He constantly hammerd an already optimized and highly efficient process to make it more efficient. Similarly, automation and reiterative optimization can help in highly evolved garbage grader and sorter which will calculate and notify the value of waste in realtime. Garbage collection can be extremely inefficient. Data Science and machine intelligence can help in efficient route for garbage collection. Batch number based waste collection – automated grader and sorter would pick up batch number while sorting the garbage bag and immediately update the stats for the batch which customer can view on web-portal or can receive notification on their mobile app. Technology can help tremendously in putting efficiency in the operation and making the process profitable.

Note: I have witnessed some startup charging money to collect waste. What are they thinking? They should pay for collecting waste. But, I guess that will take some more time.

Startup Marketing Dilemma

Niraj Kumar — Sat, 03 Jun 2017 14:27:54 +0000

Many Tech-Startups start spending and focus heavily on Marketing and Sales activities early on, while their product/services platform suffers because of lack of features, and iterative improvements based on customers feedback. Some start marketing the product even when it has not reached the MVP level.

Marketing is necessary evil! And, it should be handled simultaneously. The argument here is – what if startups focus more on improving the product, making it more user-friendly, intuitive, and engaging - allowing it to grow naturally with minimal effort on marketing and sales activities.

I believe that ‘Word of Mouth’ publicity is more valuable in the early iterations; it attracts new customers to the platform and retains them for a longer period. In this highly competitive and quick evolving landscape; Customer engagement and retention is the key to success. If customers find a product/platform valuable, engaging, with superb UI/UX, and usable, they will most likely reuse it, pay for it and recommend to their connections of the world to use it.

Purpose of building a product/platform for a customer fails if they cannot use it due to complexities or bugs in the system. Our inherent trait is, we do not ask for help unless in dire situation. Similarly, If customers cannot figure out how to use the product for themselves, then they would be reluctant to ask for help. Few adventurous/experimental types may contact help-desk. Besides, you need to allocate more time and valuable resources on keeping a large team of customer support executives to provide help to customers, or to complete the process on behalf of them.

I think, instead of increasing the Marketing/Sales/BD budget and effort; startups should focus more on agile (fast-paced) improvement of their product to make it more usable than their competitors. They should hire more Tech Personnel (programmers, testers, and designers) to fast-track the iterative improvements of the product to make it more usable; instead of hiring more Marketing/Sales/BD people to sell vaporware.