DEV Community: AlphaCodes

System Design 101-Everything to know about Distributed Systems…

AlphaCodes — Thu, 07 Oct 2021 17:16:53 +0000

In this Article, we are going to learn about Distributed Systems. Here we are focusing on these things,
.What is Distributed Systems
.How does Distributed Systems work
.Why Distributed Systems
.Characteristic of Distributed Systems
.Challenges of Distributed Systems.
.Risks using Distributed Systems
.Distributed System Architecture & Types
.Distributed Data Stores(Cassandra)
.CAP Theorem
.Distributed Messaging
.Data Copying
.Decentralized vs Distributed Systems
.Cloud vs Distributed Systems

What is a distributed system?

A distributed system in its most simplest definition is a group of computers working together as to appear as a single computer to the end-user.
These machines have a shared state, operate concurrently and can fail independently without affecting the whole system’s uptime.
A distributed system is any network structure that consists of autonomous computers that are connected using a distribution middleware. Distributed systems facilitate sharing different resources and capabilities, to provide users with a single and integrated coherent network.
The opposite of a distributed system is a centralized system. If all of the components of a computing system reside in one machine.
Unlike traditional databases, which are stored on a single machine, in a distributed system, a user must be able to communicate with any machine without knowing it is only one machine. Most applications today use some form of a distributed database and must account for their homogenous or heterogenous nature.
In a homogenous distributed database, each system shares a data model and database management system and data model. Generally, these are easier to manage by adding nodes. On the other hand, heterogeneous databases make it possible to have multiple data models or varied database management systems using gateways to translate data between nodes.

Distributed Information Systems: distribute information across different servers via multiple communication models.
Distributed Pervasive Systems: use embedded computer devices (i.e. ECG monitors, sensors, mobile devices).
Distributed Computing Systems: computers in a network communicate via message passing.

Distributed systems must have a shared network to connect its components, which could be connected using an IP address or even physical cables.

How does a distributed system work?

Distributed systems have evolved over time, but today’s most common implementations are largely designed to operate via the internet and, more specifically, the cloud. A distributed system begins with a task, such as rendering a video to create a finished product ready for release. The web application, or distributed applications, managing this task — like a video editor on a client computer — splits the job into pieces. In this simple example, the algorithm that gives one frame of the video to each of a dozen different computers (or nodes) to complete the rendering. Once the frame is complete, the managing application gives the node a new frame to work on. This process continues until the video is finished and all the pieces are put back together. A system like this doesn’t have to stop at just 12 nodes — the job may be distributed among hundreds or even thousands of nodes, turning a task that might have taken days for a single computer to complete into one that is finished in a matter of minutes.

There are many models and architectures of distributed systems in use today. Client-server systems, the most traditional and simple type of distributed system, involve a multitude of networked computers that interact with a central server for data storage, processing or other common goal. Cell phone networks are an advanced type of distributed system that share workloads among handsets, switching systems and internet-based devices. Peer-to-peer networks, in which workloads are distributed among hundreds or thousands of computers all running the same software, are another example of a distributed system architecture. The most common forms of distributed systems in the enterprise today are those that operate over the web, handing off workloads to dozens of cloud-based virtual server instances that are created as needed, then terminated when the task is complete.

Why distribute a system?

Systems are always distributed by necessity. The truth of the matter is — managing distributed systems is a complex topic chock-full of pitfalls and landmines. It is a headache to deploy, maintain and debug distributed systems, so why go there at all?
What a distributed system enables you to do is scale horizontally. Going back to our previous example of the single database server, the only way to handle more traffic would be to upgrade the hardware the database is running on. This is called scaling vertically.

Scaling vertically is all well and good while you can, but after a certain point you will see that even the best hardware is not sufficient for enough traffic, not to mention impractical to host.

Scaling horizontally simply means adding more computers rather than upgrading the hardware of a single one.

It is significantly cheaper than vertical scaling after a certain threshold but that is not its main case for preference.

Vertical scaling can only bump your performance up to the latest hardware’s capabilities. These capabilities prove to be insufficient for technological companies with moderate to big workloads.
The best thing about horizontal scaling is that you have no cap on how much you can scale whenever performance degrades you simply add another machine, up to infinity potentially.
Easy scaling is not the only benefit you get from distributed systems. Fault tolerance and low latency are also equally as important.

Fault Tolerance - a cluster of ten machines across two data centers is inherently more fault-tolerant than a single machine. Even if one data center catches on fire, your application would still work.

An important part of distributed systems is the CAP theorem, which states that a distributed data store cannot simultaneously be consistent, available, and partition tolerant.

Low Latency — The time for a network packet to travel the world is physically bounded by the speed of light. For example, the shortest possible time for a request‘s round-trip time (that is, go back and forth) in a fiber-optic cable between New York to Sydney is 160ms. Distributed systems allow you to have a node in both cities, allowing traffic to hit the node that is closest to it.

For a distributed system to work, though, you need the software running on those machines to be specifically designed for running on multiple computers at the same time and handling the problems that come along with it. This turns out to be no easy feat.

Key Characteristics of Distributed Systems

Scalability

Scalability is the capability of a system, process, or network to grow and manage increased demand. Any distributed system that can continuously evolve to support the growing amount of work is considered scalable.

A system may have to scale because of many reasons like increased data volume or increased work, e.g., number of transactions. A scalable system would like to achieve this scaling without performance loss.

Generally, although designed (or claimed) to be scalable, the performance of a system declines with the system size due to the management or environmental cost. For instance, network speed may become slower because machines tend to be far apart from one another. More generally, some tasks may not be distributed, either because of their inherent atomic nature or some flaw in the system design. At some point, such tasks would limit the speed-up obtained by distribution. A scalable architecture avoids this situation and attempts to balance the load on all the participating nodes evenly.

Horizontal vs. Vertical Scaling
Horizontal scaling means that you scale by adding more servers into your pool of resources, whereas Vertical scaling means that you scale by adding more power (CPU, RAM, Storage, etc.) to an existing server.

With horizontal scaling, it is often easier to scale dynamically by adding more machines into the existing pool; Vertical-scaling is usually limited to the capacity of a single server. Scaling beyond that capacity often involves downtime and comes with an upper limit.
Good examples of horizontal scaling are Cassandra **and **MongoDB, as they both provide an easy way to scale horizontally by adding more machines to meet growing needs. Similarly, a good example of vertical scaling is MySQL, as it allows for an easy way to scale vertically by switching from small to bigger machines. However, this process often involves downtime.

Reliability

By definition, reliability is the probability a system will fail in a given period. In simple terms, a distributed system is considered reliable if it keeps delivering its services even when one or several of its software or hardware components fail. Thus, reliability represents one of the main characteristics of any distributed system. Any failing machine can always be replaced by another healthy one in such systems, ensuring the completion of the requested task.

Take the example of a large electronic commerce store (like Amazon), where one of the primary requirements is that any user transaction should never be canceled due to a failure of the machine running that transaction. For instance, if a user has added an item to their shopping cart, the system is expected not to lose it. A reliable distributed system achieves this through the redundancy of both the software components and data. If the server carrying the user’s shopping cart fails, another server with the replica of the shopping cart should replace it.
Redundancy has a cost, and a reliable system has to pay to achieve such resilience for services by eliminating every single point of failure.

Availability

By definition, availability is when a system remains operational to perform its required function in a specific period. It is a simple measure of the percentage of time that a system, service, or machine remains operational under normal conditions. For example, an aircraft that can be flown for many hours a month without much downtime has high availability. Availability takes into account maintainability, repair time, spares availability, and other logistics considerations. If an aircraft is down for maintenance, it is considered not available during that time.

Reliability is availability over time, considering the full range of possible real-world conditions that can occur. For example, an aircraft that can make it through any possible weather safely is more reliable than one that has vulnerabilities to possible conditions.

Reliability Vs. Availability

If a system is reliable, it is available. However, if it is available, it is not necessarily reliable.
In other words, high reliability contributes to high availability. Still, it is possible to achieve a high availability even with an unreliable product by minimizing repair time and ensuring that spares are always available when they are needed.

Let’s take the example of an online retail store with 99.99% availability for the first two years after its launch. However, the system was launched without any information security testing. The customers were happy with the system, but they don’t realize that it isn’t very reliable as it is vulnerable to likely risks. In the third year, the system experiences a series of information security incidents that suddenly result in extremely low availability for extended periods. This results in reputational and financial damage to the customers.

Efficiency

To understand how to measure the efficiency of a distributed system, let’s assume an operation that runs in a distributed manner and delivers a set of items as a result.

Two standard measures of its efficiency are the response time (or latency) that denotes the delay to obtain the first item, and the throughput (or bandwidth), which denotes the number of items delivered in a given time unit (e.g., a second). The two measures correspond to the following unit costs:

The number of messages globally sent by the nodes of the system, regardless of the message size.
Size of messages representing the volume of data exchanges.

The complexity of operations supported by distributed data structures (e.g., searching for a specific key in a distributed index) can be characterized as a function of one of these cost units. Generally speaking, the analysis of a distributed structure in terms of the ‘number of messages’ is over-simplistic. It ignores the impact of many aspects, including the network topology, the network load, variation, the possible heterogeneity of the software and hardware components involved in data processing and routing, etc. However, it is quite difficult to develop a precise cost model that would accurately consider all these performance factors. Therefore, we’ve to live with rough but robust estimates of the system behavior.

Serviceability or Manageability

Another important consideration while designing a distributed system is how easy it is to operate and maintain. Serviceability or manageability is the simplicity and speed with which a system can be repaired or maintained

If the time to fix a failed system increases, then availability will decrease. Things to consider for manageability are the ease of diagnosing and understanding problems when they occur, ease of making updates or modifications, and how simple the system is to operate (i.e., does it routinely operate without failure or exceptions?).

Early detection of faults can decrease or avoid system downtime. For example, some enterprise systems can automatically call a service center (without human intervention) when the system experiences a system fault.

Challenges of distributed systems

Distributed systems are considerably more complex than monolithic computing environments, and raise a number of challenges around design, operations and maintenance. These include:

Increased opportunities for failure: The more systems added to a computing environment, the more opportunity there is for failure. If a system is not carefully designed and a single node crashes, the entire system can go down. While distributed systems are designed to be fault tolerant, that fault tolerance isn’t automatic or foolproof.

Synchronization process challenges: Distributed systems work without a global clock, requiring careful programming to ensure that processes are properly synchronized to avoid transmission delays that result in errors and data corruption. In a complex system — such as a multiplayer video game — synchronization can be challenging, especially on a public network that carries data traffic.

Imperfect scalability: Doubling the number of nodes in a distributed system doesn’t necessarily double performance. Architecting an effective distributed system that maximizes scalability is a complex undertaking that needs to take into account load balancing, bandwidth management and other issues.

More complex security: Managing a large number of nodes in a heterogeneous or globally distributed environment creates numerous security challenges. A single weak link in a file system or larger distributed system network can expose the entire system to attack.

Increased complexity: Distributed systems are more complex to design, manage and understand than traditional computing environments.

Risks of distributed systems

The challenges of distributed systems as outlined above create a number of correlating risks. These include:

Security: Distributed systems are as vulnerable to attack as any other system, but their distributed nature creates a much larger attack surface that exposes organizations to threats.

Risk of network failure: Distributed systems are beholden to public networks in order to transmit and receive data. If one segment of the internet becomes unavailable or overloaded, distributed system performance may decline.

Governance and control issues: Distributed systems lack the governability of monolithic, single-server-based systems, creating auditing and adherence issues around global privacy laws such as GDPR. Globally distributed environments can impose barriers to providing certain levels of assurance and impair visibility into where data resides.

Cost control: Unlike centralized systems, the scalability of distributed systems allows administrators to easily add additional capacity as needed, which can also increase costs. Pricing for cloud-based distributed computing systems are based on usage (such as the number of memory resources and CPU power consumed over time). If demand suddenly spikes, organizations.

Distributed System Architecture

Distributed systems must have a network that connects all components (machines, hardware, or software) together so they can transfer messages to communicate with each other.

That network could be connected with an IP address or use cables or even on a circuit board.
The messages passed between machines contain forms of data that the systems want to share like databases, objects, and files.
The way the messages are communicated reliably whether it’s sent, received, acknowledged or how a node retries on failure is an important feature of a distributed system.
Distributed systems were created out of necessity as services and applications needed to scale and new machines needed to be added and managed. In the design of distributed systems, the major trade-off to consider is complexity vs performance.

Types of Distributed System Architectures:

Distributed applications and processes typically use one of four architecture types below:

Client-server:
In the early days, distributed systems architecture consisted of a server as a shared resource like a printer, database, or a web server. It had multiple clients (for example, users behind computers) that decide when to use the shared resource, how to use and display it, change data, and send it back to the server. Code repositories like git is a good example where the intelligence is placed on the developers committing the changes to the code.

Today, distributed systems architecture has evolved with web applications into:

Three-tier: In this architecture, the clients no longer need to be intelligent and can rely on a middle tier to do the processing and decision making. Most of the first web applications fall under this category. The middle tier could be called an agent that receives requests from clients, that could be stateless, processes the data and then forwards it on to the servers.

Multi-tier: Enterprise web services first created n-tier or multi-tier systems architectures. This popularized the application servers that contain the business logic and interacts both with the data tiers and presentation tiers.

Peer-to-peer: There are no centralized or special machine that does the heavy lifting and intelligent work in this architecture. All the decision making and responsibilities are split up amongst the machines involved and each could take on client or server roles. Blockchain is a good example of this.

Distributed Data Stores

Distributed Data Stores are most widely used and recognized as Distributed Databases. Most distributed databases are **NoSQL **non-relational databases, limited to key-value semantics. They provide incredible performance and scalability at the cost of consistency or availability.

We cannot go into discussions of distributed data stores without first introducing the CAP Theorem.

CAP Theorem
The CAP theorem states that a distributed data store cannot simultaneously be consistent, available and partition tolerant.

Some quick definitions:

Consistency — What you read and write sequentially is what is expected
Availability — the whole system does not die — every non-failing node always returns a response.
Partition Tolerant — The system continues to function and uphold its consistency/availability guarantees in spite of network partitions

In reality, partition tolerance must be a given for any distributed data store. As mentioned in many places, one of which this great article, you cannot have consistency and availability without partition tolerance.

Think about it: if you have two nodes which accept information and their connection dies — how are they both going to be available and simultaneously provide you with consistency? They have no way of knowing what the other node is doing and as such have can either become offline (unavailable) or work with stale information (inconsistent).

In the end you’re left to choose if you want your system to be strongly consistent or highly available under a network partition.

Practice shows that most applications value availability more. You do not necessarily always need strong consistency. Even then, that trade-off is not necessarily made because you need the 100% availability guarantee, but rather because network latency can be an issue when having to synchronize machines to achieve strong consistency. These and more factors make applications typically opt for solutions which offer high availability.

Such databases settle with the weakest consistency model — eventual consistency (strong vs eventual consistency explanation). This model guarantees that if no new updates are made to a given item, eventually all accesses to that item will return the latest updated value.

Those systems provide BASE properties (as opposed to traditional databases’ ACID)

Basically Available — The system always returns a response
Soft state — The system could change over time, even during times of no input (due to eventual consistency)
Eventual consistency — In the absence of input, the data will spread to every node sooner or later — thus becoming consistent

Examples of such available distributed databases — Cassandra, Riak, Voldemort

Of course, there are other data stores which prefer stronger consistency — HBase, Couchbase, Redis, Zookeeper

The CAP theorem is worthy of multiple articles on its own — some regarding how you can tweak a system’s CAP properties depending on how the client behaves and others on how it is not understood properly.

Cassandra

Cassandra, as mentioned above, is a distributed No-SQL database which prefers the AP properties out of the CAP, settling with eventual consistency. I must admit this may be a bit misleading, as Cassandra is highly configurable — you can make it provide strong consistency at the expense of availability as well, but that is not its common use case.

Cassandra uses consistent hashing to determine which nodes out of your cluster must manage the data you are passing in. You set a replication factor, which basically states to how many nodes you want to replicate your data.

When reading, you will read from those nodes only.

Cassandra is massively scalable, providing absurdly high write throughput.

Even though this diagram might be biased and it looks like it compares Cassandra to databases set to provide strong consistency (otherwise I can’t see why MongoDB would drop performance when upgraded from 4 to 8 nodes), this should still show what a properly set up Cassandra cluster is capable of.

Regardless, in the distributed systems trade-off which enables horizontal scaling and incredibly high throughput, Cassandra does not provide some fundamental features of ACID databases — namely, transactions.

Distributed Messaging

Messaging systems provide a central place for storage and propagation of messages/events inside your overall system. They allow you to decouple your application logic from directly talking with your other systems.

Known Scale — LinkedIn’s Kafka cluster processed 1 trillion messages a day with peaks of 4.5 millions messages a second.

Simply put, a messaging platform works in the following way:
A message is broadcast from the application which potentially create it (called a producer), goes into the platform and is read by potentially multiple applications which are interested in it (called consumers).

If you need to save a certain event to a few places (e.g user creation to database, warehouse, email sending service and whatever else you can come up with) a messaging platform is the cleanest way to spread that message.

Consumers can either pull information out of the brokers (pull model) or have the brokers push information directly into the consumers (push model).

There are a couple of popular top-notch messaging platforms:
*RabbitMQ *— Message broker which allows you finer-grained control of message trajectories via routing rules and other easily configurable settings. Can be called a smart broker, as it has a lot of logic in it and tightly keeps track of messages that pass through it. Provides settings for both AP and CP from CAP. Uses a push model for notifying the consumers.

*Kafka *— Message broker (and all out platform) which is a bit lower level, as in it does not keep track of which messages have been read and does not allow for complex routing logic. This helps it achieve amazing performance. In my opinion, this is the biggest prospect in this space with active development from the open-source community and support from the Confluent team. Kafka arguably has the most widespread use from top tech companies. I wrote a thorough introduction to this, where I go into detail about all of its goodness.

Apache ActiveMQ — The oldest of the bunch, dating from 2004. Uses the JMS API, meaning it is geared towards Java EE applications. It got rewritten as ActiveMQ Artemis, which provides outstanding performance on par with Kafka.

Amazon SQS — A messaging service provided by AWS. Lets you quickly integrate it with existing applications and eliminates the need to handle your own infrastructure, which might be a big benefit, as systems like Kafka are notoriously tricky to set up. Amazon also offers two similar services — SNS and MQ, the latter of which is basically ActiveMQ but managed by Amazon.

Data Copying

Copying is another foundational concept in computing. In software design we use the concept of taking data copies to achieve speed and resilience in several different ways.

Replicas
A data replica is an exact copy of a database. Replicas are constantly, iteratively synced with one another so their contents are kept as identical as possible. They generally come in two flavours, active and passive. Active replicas support read and write access and play an (unsurprisingly) active role in serving clients. Active replicas can help with scale, resilience and location-base performance. Passive replicas are generally maintained for failover purposes; they don’t support read or write and are only synced in one direction (from the master data). The job of a passive replica is to be ready to take over from the main replica if it fails.

Caches
Unlike a replica, a cache is a not-necessarily-identical, read-only, copy of your data, usually maintained as a way to serve read requests faster and more cheaply than querying your main database. Caches may take advantage of

Location (keeping information physically close to the reading client).
Medium (keeping information in faster-to-access physical media like memory rather than disk).
Design (maintaining information in data structures optimised for fast read like key/value stores or graphs).

Backups
A backup is a copy of your data for emergency use in the case of catastrophic data loss. Usually a backup is written to a permanent storage medium like disk in multiple physical locations. Backups are slow to write to and restore from and usually they are not kept constantly in sync with the master data. Instead, they are periodically updated. They are useful but have severe limitations and can easily lull folk into a false sense of security. This false security becomes especially dangerous as the scale and complexity of a system increases.

Denormalization for Decoupling
Data duplication goes in and out of fashion as an architectural technique. It is often frowned upon because it’s a common cause of bugs. On the other hand, it is very useful for performance. Basically, like everything in life it has pros and cons and we need to make a judgment on whether to use it based on the current circumstances.

A decade ago we spent ages “normalizing” our databases, or removing data duplication. However in a distributed system we really need to have data available in more than one location. Every service cannot be calling back to the same database all the time or we lose all the benefits of distribution. Microservices usually maintain local equivalents of certain pieces of data that they can write and read without worrying about anyone else.

BUT the potential bugs have not mysteriously gone away. We have to somehow keep all those data copies in alignment (that doesn’t necessarily mean identical). By using data duplication we have to accept that we’ve just included an architectural technique that can be very bug prone in our design. It’s a tradeoff. It just means that we have to use development techniques that help resolve those problems such as Domain Driven Design, or operational techniques like managed stateful services that handle some of these issues for us.

Decentralized vs Distributed
Before we go any further I’d like to make a distinction between the two terms.

Even though the words sound similar and can be concluded to mean the same logically, their difference makes a significant technological and political impact.

Decentralized **is still **distributed in the technical sense, but the whole decentralized systems is not owned by one actor. No one company can own a decentralized system, otherwise it wouldn’t be decentralized anymore.

This means that most systems we will go over today can be thought of as distributed centralized systems — and that is what they’re made to be.

Decentralized is essentially distributed on a technical level, but usually a decentralized system is not owned by a single source.

If you think about it — it is harder to create a decentralized system because then you need to handle the case where some of the participants are malicious. This is not the case with normal distributed systems, as you know you own all the nodes.

This has been debated a lot and can be confused with others (peer-to-peer, federated). In early literature, it’s been defined differently as well. Regardless, what I gave you as a definition is what I feel is the most widely used now that blockchain and cryptocurrencies popularized the term.

Cloud vs distributed systems
Cloud computing and distributed systems are different, but they use similar concepts. Distributed computing uses distributed systems by spreading tasks across many machines. Cloud computing, on the other hand, uses network hosted servers for storage, process, data management.

Distributed computing aims to create collaborative resource sharing and provide size and geographical scalability. Cloud computing is about delivering an on demand environment using transparency, monitoring, and security.

Compared to distributed systems, cloud computing offers the following advantages:

Cost effective
Access to a global market
Encapsulated change management
Access storage, servers, and databases on the internet

However, cloud computing is arguably less flexible than distributed computing, as you rely on other services and technologies to build a system. This gives you less control overall.

Priorities like load-balancing, replication, auto-scaling, and automated back-ups can be made easy with cloud computing. Cloud building tools like Docker, Amazon Web Services (AWS), Google Cloud Services, or Azure make it possible to create such systems quickly, and many teams opt to build distributed systems alongside these technologies.

System Design- Tinder | Cost to develop | How to earn revenue

AlphaCodes — Fri, 10 Sep 2021 14:08:51 +0000

In this article, we’ll look at the design / system architecture of dating apps like Tinder / Bumble / Happen/ OkCupid / Hinge. How much does it cost to develop a dating app like Tinder . To get started, let’s understand the app and the features that we will talk about in this article.

This article mainly focuses on the architecture of Tinder.

Tinder’s technology may look simple but when a user left or Right swipe in the app, and a match is found. But Behind the scenes, however, a giant infrastructure consisting of thousands of services and terabytes of data supports each and every swipe and match on the platform.

What is Tinder?
As the modern generation of young people embrace the dating lifestyle, the online dating segment continues to grow. It requires a special technological spark to cultivate the feeling of using such dating apps and achieve long-term use. With the transformation of dating apps, Tinder is a good example of how dating apps can be transformed into a highly profitable business. The Tinder dating app is different from all other types of dating apps because it has a high matching conversion rate. Because of its user-friendly features, it is a hot topic among the younger generation. You can select your communication partner by swiping. When this sliding is mutual, communication opens a two-way channel. Hence, developing a dating app like Tinder is the same thing as developing a relationship, and it takes time and investment to make it unique.

The uniqueness of the tinder applies to the geographical proximity as the key aspect and makes it for individuals smoother to produce with other individuals of their preferences. You can simply view the possible matches and left(NO) or right (yes), initiates the process. The app analyzes user data with geographic locations, mutual friends or similar interests. The users must call their age and gender to the individual, with which they use a tinder to inform the other users with the same interests in their environment.

In Short Tinder is a social dating mobile app designed by UX / UI designers that allows individuals to search and find the right person to connect with the right person through assistance. Support for chat options is consolidated within the app. Sign In to Tinder app requires Facebook login so location and interests are used to find the right person. Users can make their profile look exciting by adding extra details and descriptions to their profile.

SCALE at which TINDER operates:
57+ million members
12+ million active users
55+ Billion matches
100+ Million downloads
1.8 billion swipes every day (left swipe, right swipe, super like)
40+ Language support

Design needs to be scalable to support 57+ million userbases. Tinder supports more than 40+ languages, which means that users are spread all over the world. Hence, this cannot be a simple application hosted on a single continent, but needs to be well distributed to provide the best performance to all users worldwide.

Tinder is entirely hosted in the AWS Cloud. There are no web applications other than IOS and Android. Tinder uses AWS amplify to develop and test mobile applications, MongoDB for DB and Redis for caching and in-memory databases.

Features:

Login using OAuth (Facebook)- The method of logging in via Facebook or phone number is very simple. The dating app algorithm extracts basic user information from Facebook, skipping the same old methods of filling out forms and creating personal profiles.

Swipes (Left, Right)- Swipe may be a distinctive sense resolution. This feature is designed to improve the method of finding dates. Swiping to the right means you like it, and swiping to the left means you don’t like it. The swipe option interface makes online dating clearer and more exciting.

Matching- The users would be obligated to acknowledge if they need some match for the appliance. This shall support them to approach different users whom they like. Also, pairs facilitate in monitoring that the two users have joined to each other with their agreement. Also, the feature includes un-matching that the user isn’t any longer fascinated by communicating. This supports system in avoiding negative spamming, stalking, etc.

chat- In order to talk to each other, candidates need to establish a means of communication with one another. Basics would be to possess a 1 to 1 chat messenger where they will send text messages, audio call or video call through the application.

push notification- The users are notified on a real-time basis just in case the person is found nearby their set criteria.

super likes- The user can swipe up or send a heart or a rose(different application provide different methods of super like) to other profile to prioritize them in the selection queue.

Recommendation Engine

An engine which provides several hundreds/thousands of profile when a person logs into the Tinder .

let’s talk about the features of the recommendation algorithm that tinder is using.

Tag Collecting: When a person performs OAuth using FB, Tinder collects a lot of important information like location, age, distance, gender preferences, places they’ve visited, likes, dislikes, etc. It also extracts a lot of information from photos and what we write in our profile to better match.

Cluster User Base: when a person enters / logs in to Tinder, they get a random point from Tinder and based on that point they fall into some basket, let’s say we have a basket from 1 to 10, this grouping helps to select these people. people in basket 1 prefer more / match people from buckets 1, 2 and 3. This is mainly due to the high probability of matching based on your likes and people who have similar tastes.

Active Use: Tinder’s main goal is to connect people, establish meaningful relationships, so if one of the parties is inactive, it doesn’t add up to Tinder’s main goal. Therefore, it is important to know how actively the person is using the app.

Your pickiness/Bad actors: If one is doing too much of right swipe, it’s bad, you may not be shown recommendation of other people. Also if one is not doing left swipe at all, still one is not gonna shown in the recommendation of others, as they are not contributing towards the objective of this dating application.

Do you reply? : How willingly a person is replying after a match. If the user don’t engage in longer conversation or messages are not exchanged than those profiles are penalized and not shown in recommendation of other people.

Progressive taxation: If one is getting too much of matches/attention, to make it fair for others, Tinder normalizes this by not showing that profile to many other users. At the same time, if someone is not getting much attention, tinder starts bringing that profile to other users.

Recommendation Engine properties:
This recommendation engine brings up the profile of other people based on the above-mentioned points.

Low latency: When a person logs in to the application, we need to load profiles/potential matches profiles real quickly. Therefore, our Recommendation Engine needs to have low latency(able to load profile faster).

Not real-time: It’s okay if it’s not real-time ie if someone newly joins tinder it’s okay if it takes some time to show this person’s profile on other accounts.

Easy to shard/distributed: Since we have tons of profiles from across the globe, this recommendation engine should be able to shard the data as we can’t keep it in one system.

Full-text search: we need to search through the whole profile of an individual considering different parameters ( location, age, distance, gender preferences)to provide better recommendations.

HTTP interface: or web socket to get the data and send it to the application.

Structure data: XML/JSON

What Tinder uses for storing and searching through data is “Elastic search” which is basically a search system.

Initially tinder was started with one cluster and couple of shards but after gaining popularity they did distributed system.
Elasticsearch is able to achieve fast search responses because, instead of searching the text directly, it searches an index instead. Additionally, it supports full-text search which is completely based on documents instead of tables or schemas.

Data are clustered for a given location. The whole point of dating apps is to meet people in real. If I am a user from location X, India, I will obviously like to get a match with someone who is from location X + (10 -50km) depends of users preference. So, how to achieve this?

How to shard data to make elastic search queries faster?

Shard the data by geographical location.!!

We here are dividing the whole world map into small boxes. We can place each server in these boxes to serve any requests originating from these boxes (ie particular lat-log within that box) will get served by servers in that location ( Ideally these servers can be at any physical location, but for each of these boxes/cells, there is one designated server). Now there are certain boxes where the population is high, there one server won’t be able to serve all the requests.

So how can we divide the world into boxes and distribute the load across our servers?

The size of the boxes in different areas is determined by Unique user count, active user count and query count from these regions. These points decides the size of the box/cell.

We have to find a balance score on the basis of the above factors to get the optimal size of the box/cell (for which we use Google s2 library to save these cells) and see the latency/performance for that area.

Whenever a person wants to open tinder, his phone makes a query to a system .This system is basically a mapper system which based on the lat-log of the user gives information to the application/user that all of your data is stored on which server. This server is the server where users information lies as well as this can be the server where user’s potential matches lies. As mentioned before servers can be in any physical location, but all the data belongs to that particular cell will reside on that one server.

Now consider this above map, let’s concentrate on cells 1,2,3,4 and 5. Information belongs to there cells will be store on ser1,ser2,ser3,ser4 and ser5.

So if a Tinder user is residing at cell 3 and has set range as 50 km i.e user want to know all potential matches within 50 km range from user’s location. The radius of 50 km includes all these cells from cell 1 to cell 5. Mapper will know to query data from all the cells which rely in 50 km range and gather recommendation

That’s how recommendation works on Geosharding.

1) Recommendation Engine:

As soon as the new user sign-in to the tinder app using FB OAuth, his profile details go to the ES feeder service using HTTP/ WebSocket. A copy will be store in DB also (by user creation service which adds it to the persistence) and another copy to the elastic search as we need a fast search for the recommendation. Kafka consumes these messages as need to index these data asynchronously.

ES workers pick up the message and send it to the location to the cell mapper which uses the s2 library and has lat-long information. It returns the shard to which this information was written. The ES Worker then notifies the ES, and uses the ES API to write the information to that particular shard.

User information is now saved in Elastic search and he is now ready to do left/right swipe. Then it calls the recommendation engine and which in turn call to the location to cell mapper again with lat log and it returns multiple shards to which it makes parallel calls to Shards and gets couples of documents/profile and send them via HTTP / web sockets .Now all the profiles are being rendered to the user and he’s ready for left/right swipe.

2) MATCHMAKING:
Let’s consider that there are two profiles X and Y

There can be three situations possible:

X and Y right-swipe each other at the same time.
X does right swipe to Y and Y doesn’t.
Y does right swipe X and X doesn’t until now.

There are millions of matches that occur every day. We can have one matching service one cell or We can group couple of cells together with one matchmaking service. so there will be couple of matchmaking service up and running (there will be lots of queries for recommendation queries so to balance out queries per location) and each matchmaking service belongs to couple of cells instead of just one cell as was in case of geosharding. Match also works in the same manner. Match won’t happen between countries, It will happen in the cell where a profile is recommended to a user.

For eg if we recommend 100 profiles to user, chances are there will be on an average 20/30 swipes, so we don’t need one matchmaking service per cell.

Whenever a user do the right swipe, a message is send to the matchmaking service preferably by web socket, where the location manager determines to which shard or matchmaking service this message will go, and redirects message to the gateway, which connects to Kafka. The message is now in the queue. Depending on the number of shards we have got as a result form location manager service, there will be one or many matchmaking services to which this information will be broadcasted to. Information captured here is who is right swiping whom, location, and other metadata. There can be parallel workers which keep reading message coming from the Kafka queue.

If X happens to right swipe Y , then an entry like “X_Y” enters into Redis and leaves it as it is. Now when Y right swipe X , then again the same process happens, match worker picks the message and checks in Redis weather “X has ever right-swiped Y’ i.e we will definitely find key “X_Y” and check for the metadata, which means a match has happened and message will enter in the matched queue which gets picked by match notification and through web socket sends it to both X and Y saying “It’s a match”.

If for some reason, X has never right swiped Y then what will happen? Then just a record “Y_X” will enter into Redis and that’s it. when X right swipe back Y then before adding the key it will check for the key.

3) Passport Feature:
When a user moves from one Region/location to another (could be travelling or moving to different places). This could be happening with in the city, state or country. When user open the app from new location a request is send to the server and with the help of the location mapper Data of the user from previous location cell’s shard if transferred to new Location cell’s shard.

User login + profile for tinder
We already know the ES stores user info, that is already geosharded .why don’t we just have one more API expose from ES to provide specific user profile info. The only optimization we can do is to have one more layer of cache in form of ES so that we can have better performance. We can store user-related info in a database as well. We can have RDBMS as we won’t have too many of records also it needs to be geosharded. so if geosharding is taken care of, we can have our details in RDBMS. We can also link order table info with the user table. We can also opt for NoSQL as it’s auto sharding, it automatically scales itself. We can go with MongoDB as well as it provides ACID property and sharding by geo.

How to enable user login? A user can log in using FB OAuth by registering our application in FB API. We can get lots of information like places user has ever visited, likes, dislikes, close friends ,etc. As Tinder wants to build relationship app, we need to have legitimate profile and decide should we really need to show this profile to other or not. We don’t need to implement sessions in here. Since we are trying to write an app in native android or apple SDK, we don’t need to have sessions all we need to maintain is authentication token.

MONITORING:
Without monitoring, we don’t know what’s happening with our system and also to check system performance and SLA compliance. One such tool is Prometheus which provides features like altering, write queries, and also stores time series data.

It can be used to monitor the application ,collect logs and monitor system’s performance. All the user events get forwarded to Kafka which then gets read by Prometheus where we write aggregators to identify latency in any geoshard(for eg: Suddenly our app will get trending by one tweet and lots of users start login in, traffic increase in that geo shard — ASG). All these information gets captured in dashboard.

Kafka is like an event sink where we can push any kind of data which internally has lots of topics and we can read it at Prometheus. The same system can leverage to consume other logs which generated by other application and these files get read by filebeat or logstash and get forwards to Kafka and can use the same system to track system performance.

Filebeat is a lightweight shipper for forwarding and centralizing log data. Installed as an agent on your servers, Filebeat monitors the log files or locations that you specify, collects log events, and forwards them either to Elasticsearch or Logstash for indexing.

Content moderation:
Constantly keeping eye on content. For eg : one can use celeb pictures or write bad status what if everyone is doing this and tinder is not suppressing this, then engagement goes down. Therefore, moderating content is important.
How to achieve this?

Every action performed by an user is an event, like user updates the picture, updates the status or does a left/right swipe, these event needs to get pushed in event sink and get stored in persistence. There we need to use some technology like map-reduce or Kafka streams or spark to get the useful info from event run Machine Learning algorithm on recent changes to check if the profile pic is user’s profile pic or is copied/using celeb pic, No swipe, only right swipe. We need to detect all these event, we also need to keep an eye on the rate at which the user is doing the right swipe, whether he’s really reading it, or blindly doing the right swipe.

How to make an app like Tinder?
When researching how to make a dating app like Tinder, you need to understand its exciting features. In addition to the simple registration and geolocation functions, there are also some exciting features, such as rewind options, super likes, unlimited rights swipes, and change location according to subscription plans. There are some other additional features, such as in-app purchases, social media integration, prevention of abuse and uncensored content, anonymous users, and matching suggestions using advanced matching formulas.

How much money do dating apps make?
If you are thinking — “how to make money on a dating app?”, then you have to look at the options below:

In-app purchases: In-app purchase is a regular way of earning profits employed by different types of apps by offering additional features. The same trend has been employed by tinder to offer additional features like icons, emoji, etc. This trend has been successful across mobile apps of different niches. Features can only be unlocked when users pay the fixed amount and subscribe to the plan.

Subscriptions: This is an option to give a scheduled trial period for users and charge a fixed subscription fee after the completion of the trial period for using the services. This is a highly used method for earning through the apps.

Tinder Plus: It is helpful for users who want to subscribe to the freemium model that offers extra options. Tinder app allows the users to start free for certain periods and then charges for using its extra features like a change of location, unlimited swipes, etc. The subscription plans are divided into two parts namely Tinder Plus and Tinder basic.

Tinder plus consists of all the exciting features that will make the users excited once they know the usefulness of the app. If you are looking to build a similar dating app like tinder, then it is wiser to follow the same trend and make all the exciting features chargeable after a scheduled trial period. Tinder has also differentiated the costs depending on the age of the user.

Tinder gold: This plan is exclusively for new members of the app and varies according to the age group of the users.

Sponsored content: Many business enthusiasts have partnered with Tinder to present their sponsored content to the users.

Profile Lift: Tinder asks for a fixed amount from the users to boost the profiles to the top that will enhance the profile views along with some additional features.

Ads: Ads are given to third party agencies for marketing on the apps. This will be useful for other business owners to show their products or services while giving you good profits.

How much does it cost to make a dating app?
The cost of developing a dating app significantly depends upon important factors like app size, app platform (Native or Cross-platform), design elements. level of functionality enclosed in it. Incorporating the fundamental above features may decrease your investment value. however, incorporating further functionalities could need additional funds and development costs

In this article we’ve discussed the design architecture of Tinder with Recommendation Engine , Geosharding , Matchmaking Engine User profile + login and content moderation. Factors to keep in mind while developing a dating app and ways to earn revenue…

Stay connected for more content like this

Is C# really dying.? Future of C#.!

AlphaCodes — Tue, 31 Aug 2021 19:25:11 +0000

C# is a general-purpose programming language. Technically speaking, you can use it for anything other than what has been developed. There are many coding styles suitable for the C# programming language (it can be used for both functional programming style and object-oriented programming style). It is compiled and strongly typed which means it is easier in doing development but if you are shipping your code into production , there’s little bit more to it.

C# was developed by Microsoft in 2000 in response to JAVA. They have some similarities, such as strongly typed,compiled, Object-Oriented ,working on a virtual machine and many more. C# provides support for Modern days functionalities for all kind of software development such as front-end ,back-end, mobile ,etc.

Lets talk about some pros and cons
PROS
Jobs- The question a person asks when choosing a programming language is “Can I use this to find a job?” No matter what career site you search , C# and .Net developers are most assault after programmers on the platform.There are a lot of jobs out there .

Tools- Visual studio is one of the best and most popular IDE in the market.

Frameworks- .NET Core is cross platform variant of .NET framework & Xamarin used for cross platform Mobile ddevelopment

Cross Platform- It means it allows our C# programs to run on multiple platform instead of only windows machine.

Speed- C# is fast compared to same Modern day interpreted programming languages such as Python and Javascript.

CONS
Learning Curve- Compared to some modern programming languages such as Python or Javscript .C# is quite had to learn.

Is C# Dying..?Should you learn C# in 2021?
My answer would be NO, C# is not dying and YES you should learn it in 2021 because of such new and upcoming features for every programming need.

If you are building a web application, you will consider where it should be hosted, and .Net allows you to host it on a host of your choice. In the past, you had to host to Windows & it would be more expensive compared to a linux host. For an enterprise this can be very important, because you can now host on AWS, Heroku, Windows, Azure, and Linux (in short, you can choose to host).

Let’s talk about some key points in C#
Web Application Development- There is a framework called ASP.Net, under this framework it brings MVC and Web assembly world using Blazor. Many Enterprise solutions are made using C# because of built in scaffolding ,OEM using entity framework ,security called Identity which allows you to build very fast, secure and scalable application.

Another approach to web development is Blazor, because Microsoft puts Blazor at the top of the MVC and Razor pages, because it is the future of web development. So, if u need to learn one thing it’ll be ASP.net but if you are already coding for a living and already doing things then take a look at Blazor because it has the best implementation so far in web Assembly till date & Another approach to web development is Blazor, because Microsoft puts Blazor at the top of the MVC and Razor pages, because it is the future of web development.

Mobile Development- Writing a single code base that supports Android and IOS devices gives us cross-platform mobile development. There are basically three ways to do this: React Native, Google’s Dart / Flutter, and Microsoft’s Xamarin.

Xamarin allows us to create a user interface that supports IOS and Android natively, and it also allows us to share business layer logic code between multiple platforms. Applications developed by Xamarin can run on Android devices, IOS, and Windows / Mac OS desktops. Since it is now open source, the community can jump in and build things that can extend Xamarin to infinity

With C #, you can use ASP.net and the Web API to create microservices or service-based sites, allowing you to run these functions sometimes without using infrastructure in your traditional web hosting or service functions. You can create these microservice patterns and host them on Azure, AWS, or a host of your choice.

Game Development- Build a game inside of C# in Unity that will allow you be break into Game Development.

There’s Embeded, Machine Learning as well as Internet of things(IOT) which are being supported by C# & Microsoft.

CLOUD- Microsoft provides you with one of the most powerful cloud platforms known on Azure. It provides you with cognitive services, enables you to perform speech recognition and text-to-speech services, and uses ML.Net for machine learning.

The great thing about C # and .Net is that it is now open source and the tools are free earlier it was not.