DEV Community: anjireddy k

Redis vs Memcached — Which one to pick?

anjireddy k — Thu, 24 Sep 2020 15:31:52 +0000

Redis vs Memcached

Redis vs Memcached — Which one to pick?

When people talk about the Performance Improvement of an application, the one integral factor that everyone considers is server-side caching. Identifying the right cache provider that suits the requirement is an integral part of adopting the server-side caching.

Redis and Memcached are widely used open-source cache providers across the world. Most of the Cloud providers support Redis and Memcached out of the box.

In this article, I would like to share similarities and differences between the Redis and Memcached and when do we need to go for Redis or Memcached.

Similarities between Redis vs Memcached

Key-value pair data stores
Supports Data Partitioning
Sub-millisecond latency
NoSQL family
Open-source
Supported by the Majority of programming languages and Cloud providers

Redis vs Memcached — Feature Comparison

DataTypes Supported

Memcached: Supports only simple key-value pair structure

Redis: Supports data types like strings, lists, sets, sorted sets, hashes, bit arrays, geospatial, and hyper logs.

Redis allows you to access or change parts of a data object without having to load the entire object to an applicational level, modify it, and then re-store the updated version.

Memory Management

Memcached: Strictly in memory and extended to save key-value details into drive using an extension extstore

Redis: Can store the details to disk when the physical memory is fully occupied. Redis has the mechanism to swap the values that are least recently used to disk and the latest values into the physical memory.

Data Size Limits

Memcached: Can only store the data of size up to 1 MB

Redis: can store the data of size up to 512 MB (string values)

Data Persistence

Memcached: Doesn’t support data persistence

Redis: Supports data persistence using RDB snapshot and AOF Log persistence policies

Cluster Mode (Distributed caching)

Memcached: Memcached doesn’t support the distributed mechanism out of the box. This can be achieved on the client-side using a _consistent hashing _ strategy

Redis: Supports distributed cache (Clustering)

Multi-Threading

Memcached: Supports multithreading and hence can effectively use the multiple cores of the system

Redis: Doesn’t support multi-threading

Scaling

Memcached: can ve vertically scalable. Horizontal scalability is achieved from the client-side only (Using consistent hash algorithm)

Redis: Can be horizontally scalable

Data replication

Memcached: Doesn’t support data replication

Redis: Supports data replication out of the box. Redis Cluster introduces the master node and slave node to ensure data availability. Redis Cluster has two corresponding slave nodes for redundancy.

Supported Eviction Policies

Memcached:

Least Recently Used (LRU)

Redis:

No Eviction (Returns an error if the memory limit has been reached when trying to insert more data)
All keys LRU (Evicts the least recently used keys out of all keys)
Volatile LFU (Evicts the least frequently used keys out of all keys)
All keys random (Randomly evicts keys out of all keys)
Volatile random (Randomly evicts keys with an “expire” field set)
Volatile TTL (Evicts the shortest time-to-live and least recently used keys out of all keys with an “expire” field set.)
volatile LRU (Evicts the least recently used keys out of all keys with an “expire” field set)
volatile LFU (Evicts the least frequently used keys out of all keys with an “expire” field set)

Transaction Management

Memcached: Doesn’t support transactions

Redis: Support transactions

When to go for Memcached?

Memcached is recommended when dealing with smaller and static data. When dealing with larger data sets, Memcached has to serialize and deserialize the data while saving and retrieving from the cache and require more space to store it. When dealing with smaller projects, it is better to go with the Memcached due to its multi-threading nature and vertical scalability. Clustering requires a considerable amount of effort to configure the infrastructure.

When to go for Redis?

Redis supports various data types to handle various types of data. Its clustering and data persistence features make it a good choice for large applications. Additional features like Message queue and transactions allow Redis to perform beyond the cache-store.

In addition to the above-mentioned features, Redis supports the below features as well

Message queuing support (Pub/sub)
snapshots for data archiving/restoring purpose
Lua scripting

Conclusion: Redis and Memcached both can perform very well as a Cache store. Which one to pick varies from project to project.

It is wise to consider the pros and cons of the providers right from the inception phase to avoid changes and migrations during the project.

Hope you enjoyed the article. Please share your thoughts/ ideas in the comments box. Thank you for reading it.

References:

https://medium.com/@Alibaba_Cloud/redis-vs-memcached-in-memory-data-storage-systems-3395279b0941

12 Factor App Principles and Cloud-Native Microservices

anjireddy k — Thu, 10 Sep 2020 15:37:02 +0000

12-factor app is a methodology or set of principles for building the scalable and performant, independent, and most resilient enterprise applications. It establishes the general principles and guidelines for creating robust enterprise applications. 12-factor app principles got very popular as it aligns with Microservice principles.

Below are the 12-factor principles

Codebase (One codebase tracked in revision control, many deploys)
Dependencies (Explicitly declare and isolate the dependencies)
Config (Store configurations in an environment)
Backing Services (treat backing resources as attached resources)
Build, release, and Run (Strictly separate build and run stages)
Processes (execute the app as one or more stateless processes)
Port Binding (Export services via port binding)
Concurrency (Scale out via the process model)
Disposability (maximize the robustness with fast startup and graceful shutdown)
Dev/prod parity (Keep development, staging, and production as similar as possible)
Logs (Treat logs as event streams)
Admin processes (Run admin/management tasks as one-off processes)

Codebase (One codebase tracked in revision control, many deploys)

12-factor app advocates that every application should have its own codebase (repos). Multiple codebases for multiple versions must be avoided. Please do note that having branches would be fine. I.e. For all the deployment environments there should be only one repo but not multiple.

Multiple apps sharing the same code are a violation of the twelve-factor. Here you should opt-in for shared libraries.

From the 12-factor app perspective app, deploy meaning the running instance of an app like production, staging, QA, etc. Additionally, every developer has a copy of the app running in their local development environment, each of which also qualifies as a deploy.

Different versions (the version is like a code change that is available in one environment but not in other) may be active in multiple deploys.

Microservices: In Microservices, every service should have its own codebase. Having an independent codebase helps you to easy CI/CD process for your applications.

Twelve-factor app advocates of not sharing the code between the application. If you need to share you need to build a library and make it as a dependency and manage through package repository like maven.

Dependencies (Explicitly declare and isolate the dependencies)

It talks about managing the dependencies externally using dependency management tools instead of adding them to your codebase.

From the perspective of the java, you can think of Gradle as a dependency manager. You will mention all the dependencies in build.gradle file and your application will download all the mentioned dependencies from maven repository or various other repositories.

You also need to consider the dependencies from the operating system/ execution environment perspective as well.

Microservices: All the application packages will be managed through package managers like sbt, maven.

In non-containerized environments, you can go for configuration management tools like chef, ansible, etc. to install system-level dependencies.
For a containerized environment, you can go for dockerfile.

Config (Store configurations in an environment)

Anything that varies between the deployment environments is considered as configuration. This includes:

Database connections and credentials, system integration endpoints
Credentials to external services such as Amazon S3 or Twitter or any other external apps
Application-specific information like IP Addresses, ports, and hostnames, etc.

You should not hardcode any configuration values as constants in the codebase. This is a direct violation of 12-factor app principles.

12-factor app principles suggest saving the configuration values in the environment variables.

It advocates the strict separation between the code and configurations. The code must be the same irrespective of where the application being deployed.

As per “config”, what varies for the environment to the environment must be moved to configurations and managed via environment variables.

Microservices: Externalize the configurations from the application. In a microservice service environment, you can manage the configurations for your applications from a source control like git (spring-cloud-config) and use the environment variables to not to maintain the sensitive information in the source control.

Backing Services (treat backing resources as attached resources)

As per 12 factor app principles, a backing service is an application/service the app consumes over the network as part of its normal operation.

Database, Message Brokers, any other external systems that the app communicates is treated as Backing service.

12-factor app can automatically swap the application from one provider to another without making any further modifications to the code base. Let us say, you would like to change the database server from MySQL to Aurora. To do so, you should not make any code changes to your application. Only configuration change should be able to take care of it.

Microservices: In a microservice ecosystem, anything external to service is treated as attached resource. The resource can be swapped at any given point of time without impacting the service.

By following the interfaced based programming allow to swap the provider dynamically without impact on the system. Plug-in based implementation also helps you to support multiple providers.

Build, release, and Run (Strictly separate build and run stages)

The application must have a strict separation between the build, release, and run stages. Let us understand each stage in more detail.

Build stage: transform the code into an executable bundle/ build package.

Release stage: get the build package from the build stage and combines with the configurations of the deployment environment and make your application ready to run.

Run stage: It is like running your app in the execution environment.

Microservices:

You can use CI/CD tools to automate the builds and deployment process. Docker images make it easy to separate the build, release, and run stages more efficiently.

Processes (execute the app as one or more stateless processes)

The app is executed inside the execution environment as a process. An app can have one or more instances/processes to meet the user/customer demands.

As per 12-factor principles, the application should not store the data in in-memory and it must be saved to a store and use from there. As far as the state concern, your application should store the state in the database instead of in memory of the process.

Avoid using sticky sessions, using sticky sessions are a violation of 12-factor app principles. If you would store the session information, you can choose redis or memcached or any other cache provider based on your requirements.

By following these, your app can be highly scalable without any impact on the system

Microservices: By adopting the stateless nature of REST, your services can be horizontally scaled as per the needs with zero impact. If your system still requires to maintain the state use the attached resources (redis, Memcached, or datastore) to store the state instead of in-memory.

Port binding (Export services via port binding)

The twelve-factor app is completely self-contained and doesn’t rely on runtime injection of a webserver into the execution environment to create a web-facing service. The web app exports HTTP as a service by binding to a port, and listening to requests coming in on that port.

In short, this is all about having your application as a standalone instead of deploying them into any of the external web servers.

Microservices: Spring boot is one example of this one. Spring boot by default comes with embedded tomcat, jetty, or undertow.

Concurrency (Scale out via the process model)

This talks about scaling the application. Twelve-factor app principles suggest to consider running your application as multiple processes/instances instead of running in one large system. You can still opt-in for threads to improve the concurrent handling of the requests.

In a nutshell, twelve-factor app principles advocate to opt-in for horizontal scaling instead of vertical scaling.

(vertical scaling- Add additional hardware to the system

Horizontal scaling — Add additional instances of the application)

Microservices: By adopting the containerization, applications can be scaled horizontally as per the demands.

Disposability (maximize the robustness with fast startup and graceful shutdown)

The twelve-factor app’s processes are disposable, meaning they can be started or stopped at a moment’s notice. When the application is shutting down or starting, an instance should not impact the application state.

Graceful shutdowns are very important. The system must ensure the correct state.

The system should not get impacted when new instances are added or takedown the existing instances as per need. This is also known as system disposability.

Systems do crash due to various reasons. the system should ensure that the impact would be minimal and the application should be stored in a valid state.

Microservices: By adopting the containerization into the deployment process of microservices, your application implicitly follows this principle at a maximum extent. Docker containers can be started or stopped instantly. Storing request, state, or session data in queues or other backing services ensures that a request is handled seamlessly in the event of a container crash.

Dev/prod parity (Keep development, staging, and production as similar as possible)

The twelve-factor methodology suggests keeping the gap between development and production environment as minimal as possible. This reduces the risks of showing up bugs in a specific environment.

The twelve-factor developer resists the urge to use different backing services between development and production.

Microservices: This is an inherent feature of the Microservices that is run using the containerization techniques.

Logs (Treat logs as event streams)

Logs become paramount in troubleshooting the production issues or understanding the user behavior. Logs provide visibility into the behavior of a running application.

Twelve-factor app principles advocate separating the log generation and processing the log's information. From the application logs will be written as a standard output and the execution environment takes care of capture, storage, curation, and archival of such stream should be handled by the execution environment.

Microservices: In Microservices, observability is the first-class citizen. Observability can be achieved through using APM tools (ELK, Newrelic, and other tools) or log aggregations tools like Splunk, logs, etc.

By following the above-mentioned guidelines all you need is to debug an issue is to go to the central dashboard of your tool and search for it.

Admin processes (Run admin/management tasks as one-off processes)

There is a number of one-off processes as part of the application deployment like data migration, executing one-off scripts in a specific environment.

Twelve-factor principles advocates for keeping such administrative tasks as part of the application codebase in the repository. By doing so, one-off scripts follow the same process defined for your codebase.

Ensure one-off scripts are automated so that you don’t need to worry about executing them manually before releasing the build.

Twelve-factor principles also suggest using the built-in tool of the execution environment to run those scripts on production servers.

Microservices: Containerization also helps here to run the one-off processes as a task and shutdown automatically one done with the implementation.

That’s all for today. Hope you have enjoyed the article. Please share your thoughts or ideas or improvements in the below comments box.

References:

https://12factor.net/build-release-run https://www.nginx.com/blog/microservices-reference-architecture-nginx-twelve-factor-app/ https://blog.scottlogic.com/2017/07/17/successful-microservices-with-12factor-app.html

Originally published at http://www.techmonks.org on September 10, 2020.

Performance Optimization Considerations for an Enterprise Application

anjireddy k — Sat, 22 Aug 2020 10:35:30 +0000

Performance is an integral part of the Application design and plays a vital role in the success of your product/application. I would like to write a series of performance optimization techniques that help to design the best performant enterprise applications.

I would like to write a series of articles on performance. primarily there will be two parts. Part-1 talks about the performance considerations at the design / development level and Part-2 talks about the performance improvisation techniques.

To my understanding, performance optimization must be employed and evaluated at the below-mentioned stages/phases.

Application Design Level Considerations
Application Development Level Considerations
Application Testing and Deployment Considerations

Let us get started with the performance

Performance is the most important concern when designing applications. There are so many applications built around the same business domain. The clear winner would be the application that performs at the best and has excellent responsiveness and usability.

Before delving into the performance optimization techniques, let us understand what is caching and why it is a must for any application.

Adopting the cache into the application design helps to achieve the optimal performance of an application.

Below is the cache definition from the Wikipedia

In computing, a cache is a hardware or software component that stores data so that future requests for that data can be served faster; the data stored in a cache might be the result of an earlier computation or a copy of data stored elsewhere.

Wikipedia

In simple words, “Cache is a technique where data will get preprocessed/precomputed and stored somewhere (In-memory data store or distributed store) to provide the faster retrievals “

When a user requests the data, preprocessed data will get served instead of computing and serving the data on-demand. This improves the performance of the application at a larger scale.

Days are gone, where performance optimization of the application will be taken care of once done with the business implementation of the given requirement.

Performance optimization techniques/guidelines must be designed and employed from the project inception phase itself. As part of ‘Done’ definition of the user story, product owners must include the expectations from the responsiveness as well.

Below are a few policies recommended to be defined at the organization level and ensure these policies are considered as part of the application design/development.

Organization Level Policies

Capturing the operational requirements as mandatory for the project and designing your solution around it. You should consider the data or user growth per year/month based on your application
Employ the APM tools to measure/monitor performance of the applications (Adopting the observability from the inception phase of an application avoids surprises at the end) and define the SLAs
Layout the performance practices/principles and manage them as a framework. The framework must be evolving based on the requirements
Define the checklist for developers and testers aligned with your SLAs/ principles
Define a process that considers load and stress testing as a mandatory for every project.
Ensure application is evaluated with large volumes of the data

Following the defined practices leads to customer satisfaction and improves the quality of application at a greater level. Well defined strategies help you to optimize your application proactively before it impacts the customer business.

The Performance optimization framework (practices/guidelines) must be defined at the organization level and update it frequently with new findings and improvements. I believe, it is a good idea to have a performance inspector team, their whole and sole responsibility is identifying the performance issues in any given application.

Below are the few of the CORE Application Level Design considerations

Application Design level Considerations

Keep pages simple and lightweight. Home/Landing pages require special care
Define the SLAs at each module/page level (This must be reviewed at updated at regular intervals)
Identify the cache provider that meets your demands and design your cache framework around the chosen cache provider
Define framework/mechanism to load data on-demand as mandatory
Paging when dealing with large datasets
Identify the API modules/endpoints that require caching
Cache the data on the browser for applicable APIs using HTTP cache headers
Provide Multi-Channel interface (Desktop and mobile have different needs)
Adopt asynchronous calls when applicable
Minimize the service calls from the presentation layer by employing aggregated APIs
Ensure APM tools are enabled
As client-side applications are evolving, opt for client-side performance monitoring tools as well.
Evaluate SPA bundles before releasing them. Adding an external package/library must be evaluated from the size perspective as well.
Load sections/widgets based on user settings/preferences only on home page
Relational databases are very difficult to scale when dealing with huge concurrent requests. consider evaluating the No-SQL databases
ORMs always require special care. Avoid the N+1 issue by using projection queries/ named queries.
Explore the possibility of using read-only databases for ‘GET’ queries and master database for data modifications (Majority of Cloud providers support instant replication of data)
Build the cache as part of application warm-up instead of waiting for the user request

Application Development Level considerations

Design and optimize the database model using indexes and other optimizations techniques at the Database level
Apply the cache at the applicable layers (API, Business and DAO layers) using strict cache eviction policies
Employ HTTP cache headers for applicable endpoints/pages
Use projection queries or other alternatives to avoid N+1 issue when dealing with ORMs
Minify the CSS/JS using bundling techniques
Use compression algorithms to compress the data when transferring over the network
Load your JS or external packages on demand when possible
Consider using CSS sprites
Load the images when viewable in user viewport
Evaluate and explore the options to reduce your image sizes (you can employ techniques like lossy compression)

Testing Considerations

Defining and following the Automation, Load, and Stress Testing strategies help you to battle-test your application from day one itself.

Below are a few techniques

Automate your API testing
Measure the performance of each API to align with customer expectations or agreed on SLAs
Must prioritize the load testing and stress testing along with functional testing
Include strategies to validate the performance from geographical locations
performance must be considered as important as functional testing.

Deployment/Infrastructure strategies

This has a very important and high impact on the performance of an application. A simple network misconfiguration can cause much damage in terms of performance. An additional infra or network layer can delay your request from 10 milliseconds to 300 milliseconds.

Additional tools that you deploy over your application topology can negatively impact your application.

Make sure your infra configurations are similar across all the environments (Immutable Infrastructures)
Ensure your additional security/other layers are battle-tested before enabling them in the production environment (I have these surprises lot when I was working with Infra and DevOps team)
Having multiple Smart Load balancers to avoid the single-point-of-failure for your application. Load balancers can be employed the UI/client Application, API later and Database level
Configure all the required monitoring systems, log metrics, and other components of your application at the initial stage itself
Ensure your deployment architecture is reviewed by cloud provider team
DevOps team and Development team must collaboratively work towards of application instead of working on their own terms
Ensure everything is automated

References:

Image Credits:https://chelseatroy.com/2018/04/09/performance-optimization-an-example-implemented-in-ruby/

Thank you for reading the article and hope you have enjoyed it. If you have any suggestions or comments, please free to add in the comment box.

Please follow and like us:

Originally published at http://www.techmonks.org on August 22, 2020.

Performance Optimization Techniques

anjireddy k — Sat, 25 Jul 2020 11:36:45 +0000

Performance is an integral part of the Application design and plays a vital role in the success of your product/application. This is Part-2 of the performance optimization articles series. Performance optimization considerations for an enterprise-level application is discussed as part of Part-1.

Before delving into performance optimization techniques, let us discuss the factors that impact the performance of an application. below are the most critical factors that impact the performance from my view.

Factors that impact the performance of the application

Stuffing home/start page with lots of functionality
Making a huge number of API calls/ Loading large JS bundles as part of the initial load of the applications
Ineffective JavaScript modules loading strategies (Not effectively using lazy loading)
Using large size images without any image optimization/compression
Preloading all the data in the browser instead of considering user preferences/user actions
Not adopting CDN to deliver the content to the global audience (at least static content/files)
Invalid network/tool configurations
Using the same website/ APIs for Mobile and Desktop
Introducing additional tools only in production envronment

Performance Optimization techniques

This section gives you an overview of the various techniques that can be employed at various levels of your application. At a global level, below are few areas

User Interface

Below are the few of core best practices to consider as part of the user interface design and development

User Interface design best practices

Consider server-side rendering for your home/landing page if it stuffed with too much functionality
Load data on demand always
Enabling paging when dealing with large datasets

JavaScript best practices

Always keep your JS files bottom of your page
Use bundling and minification tools as part of your build process
Avoid using inefficient looping logic on client-side
Using async and defer while loading the JS files into the browse
use preload and prefetch
use DNS-Prefetch
Ensure you have enabled caching on your JS files (Every application comes with hashing the JS files.Ensure you are enabling them)
Remove unused packages or be attentive towards the size of the installed package
Use CDN to cache your static files like JavaScript, CSS, and images
Use compression tools to optimize the file sizes over the network

CSS best practices

Use preloading of CSS
Use CSS sprites
Avoid @import to include external stylesheets

Images best practices

define images based on the device
Avoid using large size images as part of the application (define max size and apply across the app)
Optimize your images without losing the quality (At Least a stage human eye can’t catch the difference). lossy compressions can help determine the optimal size.

API/Services

Caching can drastically improve the performance of the application if configured optimally. By employing caching, you can reduce the GET queries on the databases and reduce the network calls.

Object caching is used to store the processed data into a cache-store. The cached data will get served for the incoming requests.

Providers: Memcached, Redis cache, Ignite, EhCache, etc. These cache providers are available as part of the cloud provider as well.

Below diagram depicts the high-level flow involved in Object caching

By using Http cache headers we can cache the data in the browser. When the HTTP cache is enabled for a specific endpoint for 30 minutes, the browser will not make the request to the endpoint by storing the data in the cache itself.

By defining the below-mentioned HTTP directives, you can enable the HTTP caching.

Cache Control: contains one or more comma-separated directives. These directives determine whether a response is cacheable and if so, by whom, and for how long e.g. max-age or s-max-age directives.
Expires: specifies an absolute expiry time for a cached representation
ETag: an opaque string token that a server associates with a resource to uniquely identify the state of the resource over its lifetime. When the resource changes, the ETag changes accordingly.
Last-Modified: indicates when the response was generated, the Last-Modified header indicates when the associated resource last changed.

ORM has become an integral part of application development. ORMs allow you to encapsulate all your logic in your business layer instead of burying in a database.

Avoid N+1 problem by adopting projection queries
Load child tables only when required (consider lazy loading while writing the queries)
Avoid writing complex queries using ORM framework ( I personally choose named queries or native queries to avoid performance bottlenecks of ORM queries)

API CDN Providers

You can use the CDNs to provide optimal performance for your APIs across various geographical regions.

Providers: Fastly ( https://docs.fastly.com/en/guides/enabling-api-caching), Amazon Cloudfront, Key CDN, etc.

Database

Relational databases are difficult to scale when compared to NO-SQL. Consider exploring NO-SQL databases when evaluating databases for your application.

Below are the few of the techniques helpful while tuning the relational database

Created Optimized indexes
Determine the expected growth and design your database
Select fields instead of using ‘SELECT *’
Foreign keys can impact performance negatively. Please move your foreign key validation to the business layer if possible
Explore possibilities to use
Run your scheduled jobs in non-business hours
Explore on the possibility of using read-only databases for GET queries and master database for data modifications (Majority of Cloud providers support instant replication of data)
Indexes can negatively impact data modification operations. Ensure indexes are disabled when loading the huge amounts of data
Avoid distinct clause if possible
Always select limited data
One schema may not be suitable for all the requirements. If you have a reporting system, consider having a reporting database
Configure notifications when a query is executing on the server beyond defined threshold times.

Network

Network configuration is one other key area where the performance of an application depends on. We have mentioned a few of the practices as part of the Network considerations. These are just starting points and can be extended further.

DevOps / DevSecOps team should work collaboratively work with development team instead of working in their own terms
Configure network monitoring tools to identify when there is a sudden surge of requests
Enable notifications for your network failures and warnings
Ensure Autoscaling groups are configured and battle-tested before enabling them in production
Configure HTTP2 protocol for faster retrievals
Ensure your applications are tested from various geographical regions

Tools to monitor your UI performance

Chrome dev Tools (Lighthouse)

Google Page Insights

Please follow and like us:

Originally published at http://www.techmonks.org on July 25, 2020.

OAuth 2.0 Grant flows and Recommendations

anjireddy k — Fri, 17 Jul 2020 23:41:46 +0000

In this article, we would like to provide an overview of what is OAuth 2.0 and the concepts like scopes, Grant Types we must aware of before proceeding with OAuth 2.0.

What is OAuth 2.0?

OAuth 2.0 is an authorization framework that allows users to grant a third-party website or application to access the user’s protected resources without revealing their credentials or identity. For that purpose, an OAuth 2.0 server issues access tokens that the client applications can use to access protected resources on behalf of the resource owner.

OAuth Scopes

OAuth 2.0 scopes provide a way to limit the amount of access that is granted to an access token. When the app requests permission to access a resource through the Authorization server, it uses a scope parameter to specify what access it needs, and the authorization server uses the scope parameter to respond with the access that was actually granted.

OAuth 2.0 Terminology

Resource Owner : the entity that can grant access to a protected resource. Typically this is the end-user.

Resource Server (API Server): The server that is hosting the protected resources, capable of accepting and responding to protected resource requests using access tokens.

Client : An application making protected resource requests on behalf of the resource owner and with its authorization. The term client does not imply any particular implementation characteristics (e.g. whether the application executes on a server, a desktop, or other devices).

Authorization Server : The server issuing access tokens to the client after successfully authenticating the resource owner and obtaining authorization.

OAuth Grant Types

OAuth 2.0 provides below-mentioned grant types (“methods”) for a client application to acquire an access token that can be used to authenticate a request to API endpoints / other integrations.

Authorization code grant flow with PKCE ( P roof K ey for C ode E xchange)
Authorization code grant flow
Client credentials grant flow
Implicit grant flow
Resource owner credentials grant flow

Authorization code grant flow with PKCE (Proof Key for Code Exchange)

Please do note that the Authorization code grant flow is considered insecure without PKCE.

Before the introduction of PKCE, Authorization code grant flow is not recommended to use along with SPAs implemented using JavaScript frameworks. Traditionally the Authorization Code flow uses a client secret when exchanging the authorization code for an access token, but there is no way to include a client secret in a JavaScript app and have it remain a secret.

The above-mentioned issue is applicable to mobile native apps as well. In mobile apps, by decompiling the app, you can view the client secret. Thankfully OAuth team has solved the issue by extending the Authorization Code flow with PKCE extension.

The Authorization Code flow with PKCE adds an additional step which allows us to protect the authorization code so that even if it is stolen during the redirect it will be useless by itself.

The key difference between the PKCE flow and the standard Authorization Code flow is users aren’t required to provide a client_secret. PKCE reduces security risks for native apps, and SPAs as embedded secrets aren’t required in source code, which limits exposure to reverse engineering. Client_secret is used by the Authorization server to identify the client who is making the request.

In this approach, the client first generates a runtime secret called the code_verifier. The client hashes this secret and sends this value as code_challenge as part of the frontend request. The Authorization server saves this value. The client includes the code_verifier as part of the subsequent code exchange request. The Authorization server compares the hash of the code_verifier with the original code_challenge it received.

How does It work?

Authorization Code flow with PKCE is recommended for Mobile Apps, SPAs. If your application can secure the ‘Client_secret’, you can opt-in for Authorization code Flow

Authorization code grant flow

Authorization code grant flow is used in web apps are server-side apps where the source code is not publicly exposed. Your application must be server-side because, during this exchange, you must also pass along your application’s Client Secret, which must always be kept secure, and you will have to store it in your client.

How does It work?

Authorization code grant flow is recommended to use with the web apps that are server-side apps where the source code is not publicly exposed.

Client credentials grant flow

Client credentials grant flow allows a web service to use its own credentials, instead of impersonating a user, to authenticate when calling another web service. In the approach, the client (Daemon service, Cron job) sends the credentials to the Authorization server, on successful authorization; the authorization server sends back the access token to access the resources.

How does It work?

Client credentials grant flow is recommended to use to run the scheduled jobs or authenticating the external systems requests / Remote API calls.

Implicit grant flow

The Implicit grant type is used to requests the access tokens directly from the authorization server, without the use of the authorization code or client_secret. It is recommended to use as part of JavaScript applications like Angular or React JS applications. This grant type does not include client authentication because the client_secret cannot be stored safely on the public client. This grant type relies on the callback URL given when the client was registered, to validate the identity of the client.

How does It work?

Before the Introduction of Authorization code flow with PKCE, Implicit grant flow is recommended for SPA and Mobile applications where client secrets cannot be kept secret.

Resource Owner Credentials Grant flow

The Resource Owner Password Credentials flow allows exchanging the username and password of a user for an access token. In the grant flow, the client application accepts the user credentials (User Id and Password) in an interactive form (Login page) and sends over to the authorization server for validation. The authorization server validates the credentials and responds back with an access token.

The resource owner password credentials grant type is suitable in cases where the resource owner has a trust relationship with the client, such as the device operating system or a highly privileged application.

It is recommended only for first-party “official” applications released by the API provider.

How does It work?

This is not recommended to use with the client as it imposes additional security issues as the user entering the user id and password on an external login page. This is only recommended for the products from the Authorization Server.

Thank you for reading the article. Please share your thoughts in the comments box. If you like our article, please share it.

References:

https://auth0.com/docs/flows https://docs.wso2.com/display/IS530/Working+with+OAuth

Originally published at http://www.techmonks.org on July 17, 2020.

Difference between 401 (Unauthorized) and 403(Forbidden) status codes

anjireddy k — Wed, 15 Jul 2020 06:34:20 +0000

Difference between 401 (Unauthorized) and 403(Forbidden) status codes

When building a REST API there is always confusion when do we need to respond with unauthorized (401) and when do we need to respond with Forbidden (403). If the integration team doesn’t aware of the status codes, it would cause ambiguity when dealing with the REST APIs.

UnAuthorized (401) status code

This is recommended to use when the token is invalid or the API couldn’t able to identify/authenticate the user request. When REST API responded with a 401 status code, we need to verify whether the token is valid or expired.

Forbidden (403) status code

This is recommended to use when the token is valid but the user request doesn’t have the privilege to access the requested resource/endpoint.

Please follow and like us:

Originally published at http://www.techmonks.org on July 15, 2020.

Security Best Practices for REST APIs

anjireddy k — Sat, 11 Jul 2020 03:56:26 +0000

In the modern era, REST APIs become an integral part of the applications. By adopting the REST APIs, you can expose your services to web applications or mobile applications and all other digital platforms.

REST APIs must be built as a stateless service. REST API best practices deserve a separate article :). This article primarily focuses only on Security best practices for REST APIs.

below are the key concepts that should be considered while designing the REST APIS

Authentication / Authorization
Input Validations and Sanitization
Defining Content-Types
Output encoding
Rate limiters
Security for Data in Transit and Storage
Responding with Appropriate Status codes to avoid the ambiguity

Authentication and Authorization

Before delving into details let us first understand what is Authentication and Authorization.

Authentication: Authentication is the process of identifying whether the credentials passed along with the request are valid or not. here credentials can be passed as user id and password or a token assigned for the user session.

Authorization : Authorization is the process of identifying whether the received request is allowed to access the requested endpoint or method.

In the request processing pipeline, Authentication comes first and Authorization comes next. i.e. Authorization occurs only after successful authentication of the request.

Below are the most widely used Authentication types when dealing with Remote APIs (REST APIs / Web Services).

Basic Auth
Bearer Token
API Token
OAuth2.0
OIDC

Basic Auth

Basic Auth is the simplest way of dealing with Authentication when compared to other methodologies.

In the Basic Auth, the user has to send the user id and password in the format of userid:password encoded in base64 format. This method is preferred only over the https protocol only. This is highly discouraged to use over HTTP as your credentials are transferring in plain format.

Authorization: Basic base64(userid:password)

Bearer Token

Bearer Token Authentication is also known as Token-based Authentication. When the user logs into an application using the credentials, the Authorization server generates a cryptographic token to uniquely identifies the user. the applications can use the token to identify the user after a successful login. i.e. The application is required to send this token when accessing protected resources.

Similar to Basic Authentication, Bearer tokens are only recommended to send over HTTPS only.

Authorization Bearer

API Token

API Tokens are widely used in the web services / REST APIs security before the evaluation of Client-side frameworks. Still, many organizations use the API Tokens as a security measure for the APIs. This is the simplest way of implementing the security in REST APIs.

This is recommended when providing the communication between server to server requests. It is recommended to use the IP Address registration as well when using the API keys. i.e. API Token is uniquely identified along with the IP Address. This is not recommended to use as a methodology for end-user authentication and Authorization.

The API Key key can be sent as part of the query string or Authorization token or custom header or as part of the data.

OAuth2.0

OAuth2.0 is an authorization framework that allows users to grant a third-party website or application to access the user’s protected resources without revealing their credentials or identity. For that purpose, an OAuth 2.0 server issues access tokens that the client applications can use to access protected resources on behalf of the resource owner.

You probably see this option in the form of ‘Login using Google’, ‘Login using Facebook’, ‘Login using Github’ etc.

By default, OAuth generates the access tokens in the format of JWT (JSON web tokens). JWTs contain three parts: a header, a payload, and a signature.

Header: metadata about the token like cryptographic algorithms used to generate the token.

Payload : payload contains the Subject (usually identifier of the user), claims (also known as permissions or grants), and other information like audience and expiration time, etc.

Signature: used to validate the token is trustworthy and has not been tampered with.

Below are the OAuth roles you must aware of when dealing OAuth2.0

Resource Owner : the entity that can grant access to a protected resource. Typically this is the end-user.

Resource Server: The server that is hosting the protected resources

Client : the app requesting access to a protected resource on behalf of the Resource Owner.

Authorization Server : the server that authenticates the Resource Owner, and issues Access Tokens after getting proper authorization.

OAuth2.0 provides various flows or grant types suitable for different types of API clients. Grant Types are out of scope for this article.

OIDC is a simple identity layer built on top of the OAuth2.0. OIDC defines a sign-in flow that enables a client application to authenticate a user, and to obtain information (or “claims”) about that user, such as the user name, email, and so on. User identity information is encoded in a secure JSON Web Token (JWT), called ID token.

In the Open ID Connect, Request flow will happen as below.

user will be navigated to the Authorization server from the client app
The user enters the credentials to identify the user
Upon successful authentication, Server sends back the user to client along with authorization code
Client app requests the Authorization server for tokens (Access Token and Id Token) using the authorization code (we can use nonce here to incorporated additional security)
Authorization server responds back with tokens.

Input Validation

Input validation should be applied to both syntactical and semantic levels.

Syntactical: should enforce correct syntax of structured fields (e.g. SSN, date, currency symbol).

Semantic: should enforce correctness of their values in the specific business context (e.g. start date is before the end date, price is within expected range).

Basic Input validation guidelines

Define an implicit input validation by using strong types like numbers, booleans, dates, times, or fixed data ranges in API parameters.
Constrain string inputs with regular expressions.
Use whitelisting and blacklisting techniques
Define min and maximum lengths as a mandatory
Enforce Input validations on client-side and server-side
Reject unexpected/illegal content with valid error messages

Define Content-Types

we must define the allowed content-types explicitly. It is always good practice to define the valid content types and share them with the required shareholders. Upon receiving an unexpected or missing content-type header, API must respond with HTTP response status 406 Unacceptable or 415 Unsupported Media Type.

Output encoding

Content of given resources must be interpreted correctly by the browser, the server should always send the Content-Type header with the correct Content-Type, and preferably the Content-Type header should include a charset.

JSON encoders must be used when dealing with JSON Data

Rate Limiters

Rate limiters allow you to secure your APIs from the DDoS attacks. When exposing your API to publicly you must define the rate limiters. If you are opt-in for any cloud provider tools, they explicitly provide the rate-limiting capabilities to the public faced resources. you must adjust the configurations accordingly to your needs.

Security for Data in Transit and Storage

Ensure data is sent over HTTPS only. if any user tries to access over HTTP, you should upgrade it HTTPS and handle the request

Data in storage must be protected using best security practices. All the cloud providers provide you the inbuilt security (Encryption)for your backups.

Responding with Appropriate Status codes to avoid the ambiguity

Below are few common status codes used along with REST APIs

201 — Created
200 — OK
202 — Accepted and queued for processing
204 — No Content
304 — Not Modified
400 — Bad Request
401 — UnAuthorized
403 — Forbidden
404 — Not Found
405 — Method Not Allowed
406 — Not Acceptable (Used with Content Types)
415 — Unsupported Media Type
429 — Two Many requests

Please share your thoughts in the comments box to improve it further.

Before you go…

If you found this helpful please share it on Twitter, Facebook, LinkedIn, and your favorite forums. Big thanks for reading!

references:

https://auth0.com/docs/protocols/oauth2 https://swagger.io/docs/specification/authentication/oauth2/ https://cheatsheetseries.owasp.org/

Please follow and like us:

Originally published at http://www.techmonks.org on July 11, 2020.

Cross-Cutting concerns for an Enterprise Application

anjireddy k — Fri, 10 Jul 2020 09:51:19 +0000

Cross-cutting concerns for an Enterprise Application

Cross-cutting concerns

In this article, I would like to explain what is a Cross-cutting concern and what are cross-cutting concerns that must be considered as part of the system design. In this article, I would like to provide the core concerns that should be considered while designing the application. Please do note that additional cross-cutting concerns need to be applied based on the domain of the application.

Before delving into details, let us first understand what is a ‘ cross-cutting concern ‘ and why we must consider while building the enterprise applications.

Concern means “it is a behavior/functionality that we would like to implement in an application/ module”.

What is Cross-cutting-concern?

The cross-cutting concern is a concern that is applicable throughout the application and it affects the entire application.

Cross-Cutting concerns help you to manage the application level functionalities in a centralized location.

Below are the Cross-cutting concerns those are applicable for all the enterprise applications

Security
Performance
Request Tracing
Exception handling and Logging
Monitoring
Transaction Management (Distributed)
Audit Log (Does not require in every application)
Communication with External Systems (Does not require in every app)

Microservice Applications requires to take care of a few more additional concerns in addition to above-mentioned ones

Centralize Configuration Management
Distributed Request Tracing
Service Registration & Discovery
Service-to-service communication
Message Idempotency (When dealing with message queues)
Shared data / logic / libraries

Security

As part of the security, below are the few concepts you should look at

Authentication / Authorization
Data transport protocols/message handling
Identity and Access Management
Secure Accessibility of resources
Data Security (Encryption / Hashing considerations)
Security at REST and Transit

Performance

Below are the few areas/concepts you should look at

Server-side Cache
Http cache
Static resources cache / CDN
Browser cache

Also, please make sure you have the cache invalidation techniques in place to avoid serving the stale data to users.

Request Tracing

It is always good to trace the request from where it is originated and how it responded to the user. It is very helpful when debugging the application in production.

Exception handling and Logging

Centralized exception handling is one strategy to make sure our system is always responding in a positive note. Any unhandled exception can lead to crashing the entire application.

By centralizing all the exceptions into a datastore helps to identify the common errors that development needs to improve on and stability of a new feature when released to production.

Monitoring

The monitoring system helps you to identify the system behavior and issues that are occurring in the production. It helps you to detect the failures proactively and early to avoid system failures.

Transaction Management

Transaction management makes sure your application is following the ACID properties. By defining and adopting the transaction management strategy you can avoid data inconsistency issues.

Distributed Transaction Management is critical when dealing with Microservice systems.

Please let me know if you would like to add additional concerns in addition to the above-mentioned ones.

Hope you enjoyed the article. Please share your thoughts in the comments box below. Thank you for reading the article.

Angular Components And Data Binding Techniques

anjireddy k — Sat, 02 May 2020 21:44:22 +0000

In this article, we would like to discuss the Angular component and data binding techniques available in angular.

Angular components are building blocks of Angular application. An angular component represents a custom HTML element that represents a specific section on a page. Angular components always associated with a template.

To make Typescript class as a component, you need to decorate with ‘@component’ metadata decorator. Below is high-level responsibilities segregation in a component

@Component decorator options

selector: Defines the name of the HTML element that represents this component in a page
Template — Holds HTML of the component
TemplateUrl: Holds the HTML template path
Providers: Any additional services that a component want to access in addition to the global services
Encapsulation: Controls how the styling is applied to this component

Data Binding

Data binding is the process of connecting a UI element such as textbox or dropdown with the information that populates it. Using this data binding, the information will be passed from source to destination.

In angular terminology, Data binding responsible for coordinating the communication between the component’s class and its templates and often involves the passing the data

There are 4 types of data binding

Property Binding
Interpolation
Event Binding
Two-way binding

Property binding

Property Binding is a data binding technique that will help you to bind the properties of an HTML element with your component’s properties or methods. Property binding is a one-way binding.

<button [disabled]="btnDisabled"></button> 
// component.ts 
@Component({ 
selector: 'app-component', 
templateUrl: 'component.html' 
}) 
export class Component { 
btnDisabled = true; 
}

Property Binding

Interpolation

Interpolation is a one-way data-binding technique that allows you to bind the component’s class properties to UI elements. it uses double curly braces ( {{ your expression or property }} ) to display the data from component to view.

{{greeting}}

Event Binding

In any typical application, a user interacts with the application. As part of user interaction, the user needs to click the buttons or entering the details in text boxes, etc. All these actions come under events. In technical terms, we call them button events, keystrokes, change events, etc.

If you want to send the information from view to component’s class you need to use the event binding. This is also a one-way binding and exactly does the opposite of property binding.

To capture an event from the view, you need to wrap the event inside the parenthesis “()”

Save

Two-way binding

The two-way binding combines the property binding and event binding

That’s all for today’s topic. Thank you for reading. Please share your thoughts in the comments box.

Originally published at http://www.techmonks.org on May 2, 2020.

Angular Modules

anjireddy k — Wed, 25 Dec 2019 02:45:58 +0000

Angular modules help us to organize the application into cohesive blocks of related functionality. Angular modules are much similar to packages in java and namespaces in c#

Roles of an Angular Module

Imports Other Angular Modules
Identify components, pipes, and directives
Export Its features
Can be eagerly or lazily loaded

Types of An Angular Module

Root Module
CORE Module
Shared Module
Feature/Widget Module

Root Module

Every Angular application requires at least one module. Root Module is responsible for loading the root component and other parts of our app and any Angular-specific dependencies.

By convention, the root module is called as AppModule and get created under the ‘./src/app’ folder.

Core Module

Core modules should contain only the services, components and others that can be imported only once per application. Core Module must be only imported in AppModule and must prevent from loading from other modules.

This is especially important if you intend to lazy-load your feature modules. Since lazy-loaded modules are loaded on demand (when you access the route using the lazy-loaded feature), you could end up creating new instances of singleton services if you don’t put them in CoreModule.

Shared Module

Shared Module contains the code that can be used across the project. Shared Modules can be imported into the feature modules on a need basis. Shared Module cannot be imported into AppModule or CORE Module strictly.

These components don’t import and inject services from the core or other features in their constructors.

Feature Module

A featured module consists of a cohesive set of functionality focused on a specific application need such as a user workflow, routing, or forms. The main aim for feature modules is delimiting the functionality that focuses on particular internal business inside a dedicated module, in order to achieve modularity.

The featured module helps us to split the application into multiple modules which interns make the Root module thin so that the initial page loads quickly

Declarations: It is for things that you use your HTML templates/views. It includes components, directives, and pipes

Providers: for services

Imports: for importing external modules that the current module depends on

Exports: exports used to export the components, pipes, directives to use them in other modules within the application.

Frequently Used Modules

Angular Modules and scopes

The confusion starts with components and services not having the same scope/visibility

declarations / components are in local scope (private visibility),
providers/services are (generally) in the global scope (public visibility).

It means the components you declared are only usable in the current module. If you need to use them outside, in other modules, you’ll have to export them.

Originally published at http://www.techmonks.org on December 25, 2019.

Angular CLI and Commands

anjireddy k — Thu, 19 Dec 2019 20:53:46 +0000

Angular CLI stands for Angular Command Line Interface. It is a command-line tool to generate Angular applications. It will create all…

Continue reading on TechMonks »

Angular Architecture

anjireddy k — Wed, 18 Dec 2019 22:03:56 +0000

Angular is one of the widely used frameworks for building SPA applications. Angular made writing the client-side applications more ease by adopting the TypeScript.

Below are the key components in Angular.

Modules
Components
Templates
Metadata
Data Binding
Directives
Services
Dependency Injection

Module

In a general essence, the module is used to group the related classes together to achieve the functionality. The same definition is applicable for angular as well. In Angular, a module is a mechanism to group the related components, directives, pipes, services, etc together.

Components

Components play a key role in Angular. Every Angular app must have at least one component.

In Angular, a component represents a specific portion of the user interface (UI).

Each component will contain a selector that will represent the HTML associated with the component. To make a class a component, we need to add @Component({}), decorator.

Templates

The template contains the HTML that needs to be displayed on the user screen. In Angular, the template contains the HTML elements and Angular custom tags/expressions as part of the template. Before projecting the template content, angular transform expressions, and custom tags will be replaced by the associated HTML.

In short, template is a combination of HTML tags + Angular Expressions + Custom tags defined using Angular

Metadata

In general, metadata termed as data about data. In Angular, metadata used to specify how a class needs to be processed by the Angular framework. A class decorator is used to define the metadata about the class. For example, any class that has @Component class decorator attached to it known as a Component.

Metadata defines how a class needs to be processed by the angular framework

Data Binding

Data binding concepts are used to bind the data from your component class to the template that is associated with the component. Below are the various way to bind the data to the view/template

String Interpolation — represented as “{{ }}” also known as a mustache syntax. Angular process the expressions/variables insides the “{{}}” and insert the output into HTML.

Property Binding — Allows binding the properties of an HTML element/Angular custom element.

Event Binding — Allow the application to respond to user actions and inputs.

Directives

Directives are used to add additional behavior to your HTML.

Here additional behavior may be altering the layout page by adding/deleting the HTML elements or additional functionality to your HTML element.

There are two types of Directives

Structural directives can alter the layout of the page by adding/removing the HTML elements. Structural directives prefix with ‘*’

Attribute directives provide additional behavior or modify the appearance of your HTML elements.

Services

In Angular, Services are singleton objects which get instantiated only once during the application lifetime. It contains data that needs to be shared across the application.

The primary objective of service is to organize and share the Business logic, and data with different components of an angular application.

Dependency Injection

Dependency Injection is a process of injecting the dependent objects into a class from an external framework/ class. So that the class can focus on primary responsibility assigned to it. Dependency Injector will take of handling the lifetime of injected dependencies.

using dependency injection, we can externalize the injecting the dependencies to the classes and managing their lifetime.

Originally published at http://www.techmonks.org on December 18, 2019.