DEV Community: raviklog

How it all started: AI, ML & DS!

raviklog — Wed, 10 Jan 2024 06:21:09 +0000

Happy New Year 2024 to all! and Welcome to my first post this year and would like to present an introduction to AI, ML & DS.

I started with Data Science around 2019 and picked up a few basics (Hypothesis, Linear Algebra, Probability, Classification & Regression Techniques). With the help of R language did basic Data Analysis and simple ML Models. All these without even thinking about How this all started?, Why is it such useful in times to come?.

Modern-day Artificial Intelligence is somewhat a birthchild of great scientist "Alan Turing" whose initial question of Can Machines be intelligent? initiated a curiosity to design a Turing test. A Turing test is to imitate the human-like intelligence of a machine and make it indistinguishable from a human conversation and if humans cannot recognize if they are conversing with a machine then the machine passes the Turing test. The Turing test is a textual-based conversational approach and this has ignited the scientific community across the world. Improvements in that area by Academicians and Scientists gave rise to terming this human-like intelligence "Artificial Intelligence" by John McCarthy around 1956.

Humans have such remarkable intelligence why do we need AI?

This is the question that restricted me from starting on this subject quite late...The following reasons have made AI adoption faster across different sectors.

Efficient ways of handling and processing big data and making quicker non-partial decisions consistently.
Automating Repetitive tasks that consume human time which could be used for other creative and strategic aspects.
Pattern Recognition at large scale, Simulation, and creating virtual worlds that can aid in analyzing situations and scenarios for different industrial and real-world purposes.
Explorations at tough terrains like deep seas, mining operations, and other undiscovered places can be done through AI-controlled Robots
Accuracy in identifying good/bad scenarios, validating persons in huge crowds or important public gatherings through Computer vision technologies.
Medical diagnosis to predict future health issues, aids doctors in analyzing psychological issues, and also helps persons with disabilities to access real-life needs and communicate freely.

This is just a small list, and the possibilities are endless as wherever human intelligence could be applied AI can work along.

Ok, so where does Machine Learning fit in?

Machine Learning (ML) had its inception in 1959 when Arthur Samuels (of IBM) was fascinated by the idea of machines that could mimic human learning and decision-making. That enabled him to coin the term "Machine Learning". It is quite logical how the Interest and work in Artificial Intelligence in the early 1950s has given rise to the Machine Learning and Data Science fields.

Simply put, Machine Learning is a subset of AI. When we say be intelligent to a machine, we need to construct a certain framework through which the abstraction can be put to work. Machine Learning is a way in which we train the Machines to Learn about the General and Specific needs of humans.

Through (Knowledge Management System & Expert Systems) we impart Facts to the system, Domain knowledge, and also the rules to react to situations.
Through Statistical Models and ML Algorithms, we aid in training on data, analyzing patterns in Data, and Tuning attributes to make future predictions and aid in the decision-making process.
Through Big Data, we handle data characterized by high Volume, Velocity, and Variety, for storage, processing, and analytics) to build AI Systems. (There are so many Data pipelines, Data Analytics Tools, and AI applications, but at a foundational level the above 3 are what constitute systems building at a bigger scale)

How can we broadly classify Machine Learning?

Different technological and business needs mandate for classifying Machine Learning algorithms. Broadly classified as Supervised and Unsupervised learning. Other classifications are refined from this basic classification to meet specific business/technological needs.

In Supervised learning, the learning algorithm is trained on a labeled dataset, where each input is paired with its corresponding output. The algorithm learns to map input data to the correct output by generalizing patterns from the labeled dataset. Some of the common Algorithms that fall under this are (Logistic Regression, Naive Bayes, SVM, Neural Networks...etc.).

Alternatively in Unsupervised learning, the learning algorithm is given an unlabeled dataset and must find patterns or relationships within the data without explicit guidance. Some of the common algorithms that fall under this are (K-means, Hierarchical Clustering, DBSCAN..etc).

Algorithms like Neural Networks are tricky as they could be used as Supervised/Unsupervised formats, so it depends on the specific tasks we try to attain. Many of the AI applications today use an ensemble of different ML Algorithms to validate various statistical parameters and arrive at desired business outcomes.

How has Data Science as a field evolved?

Finally, the field of Data science is quite important for the success of both ML & AI with the Nature and scale of Data we are dealing with in this digital era.

The field of Data Science (DS) derives from various disciplines, but more importantly traced back to Statistics, Computer Science and domain expertise (from various industry verticals). Along with the above basic foundations, Information theory, Data Mining, and Business Intelligence, Bigdata & ML have given more prominence to Data Science to evolve rapidly.

Information theory, developed by Claude Shannon in the mid-20th century, has influenced data science. Information theory provides a framework for quantifying information and uncertainty.
Data mining involves discovering patterns and knowledge from large datasets. It played a role in the evolution of data science, especially in uncovering valuable information from data.
Business intelligence practices, which involve collecting, analyzing, and presenting business data, contribute to the foundations of data science.
The Explosion of digital data, coupled with advances in storage and processing power, led to the emergence of Big Data. Technologies like Apache Hadoop and distributed computing became essential for handling massive datasets.
The widespread adoption of Machine Learning techniques, fueled by improved algorithms and increased computing power, became a defining aspect of data science.

The domains of AI, ML, and DS are advancing swiftly, arguably outpacing the traditional trajectory of technological progress defined by Moore's Law. It's crucial for us all to continually learn and grow within these dynamic fields.

The Modern Ledger-Block Chain (Intro)

raviklog — Wed, 09 Aug 2023 12:29:29 +0000

The Classical Ledger has been there for many generations and would still be there until the complete Digitization across every walk of life takes over.
What is a Ledger?

Basically, Ledger is a Recording system to capture any type of transactions that involves 2 or more parties(individuals/organizations/countries/others). In Traditional Way it's been a Manual Ledger system storing transactions in Physical books/Documents. The transactions were entered manually and signed by the parties involved and InCharge to confirm the transaction. Then the Digitization of the Records in the Computer through Data Entry/Automation and an Centralized server to Store and manage the same helped to reduce the Paper-based approach and minimize errors and put in place an accountable system.

Some of the common applications of it can be observed in the following scenarios,

Financial Transactions (Credit,Debit,Transfer and others)
Legal Transactions (Such as Property/Infrastructure Contracts, Deeds and Mortgages/Loans)
Inventory Transactions (Keep Record of Stocks, Purchases/Sales...etc.)
Statistical Transactions (Statistics With respect to Countries Population, Economy, Infra. Projects, Education, Healthcare, Food Distribution and other welfare areas...This could also be done on Regional/Organization/Individual level as per the Scale)
Healthcare Transactions (Hospitals managing the medical conditions of patients for current and future purpose.)
Educational System (Track Student's Progress, Teacher's Performance/Rating, School's Academic & Placement results...etc.)

Where does the Digital Ledger fit in this environment?
Digitization has optimized the process of Ledger by reducing Manual errors, improved speed and reducing data loss but still there are some restrictions/issues that still prevail to have a better system. Some of the Concerns that arise from the current Digital system are,

Central Authority - The Ledger Managing Organization (Banks/Schools/Corporations/Governments) normally has the complete control over the Transaction, and this creates a chance of single point of failure and the security risk associated with it)
Transparency - The Need of the Transaction being completely transparent to end-user is only determined by the single entity and it's highly probable that the Rejection/Approval of the Transaction lies at the onus of that single entity. So, it's difficult for the end-user to verify the accuracy of the transaction and the data associated with it.
Security - With Single Entity managing the Transactions of multiple parties involved, there is higher chance of Fraud/Intrusion activities by hackers/miscreants and get sensitive information about the users (For ex: Bank handling millions of customers, Schools handling 100's of Students and Teachers information, Hospital handling 1000s of Patient Records).

Let's see the current Banking system that handles financial transactions online,

How Blockchain Provides its own solution?

As with the traditional Banking System, there is a Centralized Authority to approve every transaction that the customer initiates and there is a transactional/service fee associated and also within guidelines of the Central Authority. On the contrary, a Blockchain-based system is a decentralized network where no one party administers, and all the Nodes involved in the blockchain network verify and validate the transaction (Proof of Work) for it to be fully accepted. So, this brings down the transactional cost, aids faster payments, brings more transparency, provides better interoperability, and also could improve security. Since there are Private and Public banks and a long legacy of handling customer accounts and transactions, it is important to analyze all factors and constraints before fully adopting to Blockchain.
Some areas, where Blockchain technology is used/or currently in implementation as an alternative are around the following and much more.

Payments through digital currencies (bitcoins...etc) to facilitate faster payments with lower transactional fees
Loan verification & disbursement without the need for intermediaries and improve the security in borrowing money.
KYC handling to transfer customer records safely around financial institutions.
Securities markets (Stocks, bonds) to improve the interoperability and transparency among the trading parties.

this is not an exhaustive list and more such areas blockchain could aid partly/completely and provide a more robust banking experience in the future.

Do share your comments/feedback...In successive articles, we will discuss more about the blockchain process of transaction handling and its various categories as well.

PATTERN-THRU-PRACTICAL (PTP2)

raviklog — Mon, 27 Feb 2023 10:57:41 +0000

Talk of Services across all kinds of modern-day applications. Be it Application-Level/ Business-Level/Cloud-Level and many more. All of them provide Core functionalities that are consumed by end-users through different network channels. There are Systems/Organizations which architect their entire Product Offerings as Service Interfaces for Public/Private consumption. One such Architectural methodology is the "SOA (Service Oriented Architecture)".

SERVICE – ORIENTED – ARCHITECTURE
What is a Service Oriented Architecture?

Basic Building Blocks of SOA:
The below diagram depicts the basic building blocks of SOA Architecture. Different Systems/Organizations could have different variants of this and buzzwords that may be new, but overall, this captures the essence of the Architecture. Services now have transformed into Event-Driven, Micro-Services, SOAP, REST APIs, Containerized Services, Cloud Functions, and many others, but as new inclusion comes, the foundational concepts will remain the same and also gives that primary knowledge to understand and design better systems.

REAL WORLD ANALOGY: MEDIA SERVICES(PRINT,ONLINE...etc)

Hope you find it simple and easy to go through. Liked to hear your comments or suggestions!

PATTERN-THRU-PRACTICAL

raviklog — Sat, 18 Feb 2023 08:30:28 +0000

MODEL – VIEW – CONTROLLER(MVC)

What is a Model View Controller?

MODEL – DATA (BANK HOLDS FINANCIAL DATA IN ITS INFRASTRUCTURE)

VIEW – BANK INTERFACE FOR CUSTOMER(ATM,OFFICE, REPRESENTATIVE)

CONTROLLER – GETS INPUT FROM CUSTOMERS, OPERATES ON DATA

Driven by Data-Part2(Formats)

raviklog — Sat, 28 Jan 2023 07:22:33 +0000

In the previous article, we touched upon the basic properties of data, it's classification and the various schema types that are available for data organization. Pls checkout if you had missed it!

Driven by Data! (Part1)

raviklog ・ Jan 7

#database #beginners #datafact

Here in this post, let's discuss about the Data Formats.

What is a Data format?
It's a way of representing data for the purpose of storage and transmission across software systems. Let's say passing a request to an API service, transferring multimedia content across the internet, send/receive email or short messages in web/mobile and in all these cases, message format is required so that the sender and receiver could understand and communicate accordingly.

Is it not that Database also stores and helps for data transmission?
Yes, it is, but it behaves as a storage system as a whole, performing store, manage and transmit data. In addition, it also does Indexing, Querying and defining data relations. On the other hand, Data Format is more of the way the data is represented and does not go beyond that (for ex: we create config file in an XML/CSV/TXT format as a way of representing the configuration of system).
Similarly, MYSQL database can store data in tables using Flat format such as CSV/JSON/BLOB or in Structured Rows and Columns format. Database also allows you to query and index the data accordingly. So, a Data format is only a Representative part of data, but a Database is a management system that encompasses all entities related to data including the Data formats.

Is it really necessary to know about the data formats?
Yes, it's important when you decide to design a system, which deals with various internal & external components for its normal operations. Better analysis of formats helps to optimize data retrieval and transfer rates.
Data Formats also depends on the type of industry and the application that is linked to it. Many Industries have their own custom formats defined for security and proprietary purposes as well (for ex: Industries using different CAD/CAM systems have customized CAD data formats to talk to external suppliers/customers and other industrial partners)
Checkout the Wikipedia link for the Data exchanges in CAD format...CAD_DATA_EXCHANGE).

What are the different Format Categories?
On a broader sense, data is represented at the binary level (0s and 1s) for the computer to process it...But the Software systems define additional data formats for human readability and interoperability between systems. Some commonly used Classification as shown in the diagram below,

Binary data format is one where data is stored in sequence of 0s and 1s. It is used for efficient storage and faster retrieval of big data such as (Video files, Audio files, Images...etc).
Text data format is more of human readable form (with numbers, letters, symbols) where it is easily read and processed by humans. CSV, TXT, Json, XML and other documents, spreadsheets are all that type.
Vector formats are normally used for storing drawings and graphical images that can processed by software's for extensibility rather than static. SVG (Scalar Vector Graphics), EPS(Encapsulated Post Script), EMF(Enhanced Metafile) are all that formats.
Compressed formats are done by taking the original data and simplify them to smaller size using standard compression algorithms without affecting the original content.ZIP, RAR, TAR files are all Compressed formats that we use it quite regularly.
Custom Formats are kind of specific to certain products for proprietary and security reasons. It can also be a hybrid of the other formats described above. For ex: In case of Health records/Medical Imaging, custom formats are created for storing patient information that needs to be secured and needs the purview of the health company handling the same.

Cloud, Containers & Devices: With the Digital Explosion we have right now there is a need for lightweight data formats across Handheld, Embedded and IOT devices to handle quicker response and short message transfers. Also, the need for handling big data at the Cloud/Container ecosystems mandates for various new data formats as shown below.

What are the Factors to consider while selecting Data formats?
Selection of a Data format for IO operation/Data transmission hinges on the following basic factors as given in below diagram.

In addition to analyzing the factors above it is required to find and understand the trade-offs before arriving at specific data format. (Trade-off means whether we need to select a Text format against a Binary format for a simpler message transmission application or select a Binary format so that the data transmission handles huge data and is faster and more secure. It is advisable to check both the Advantages & Disadvantages of each format keeping the end-user experience as vital.)

Let's consider set of constraints
Most of the time, requirements and constraints drives us to go for a specific data format. Take the list of constraints based on the above factors.

Low Network bandwidth mandates for better compression, storage and transmission...Protocol Buffers, Avro are Binary format and have better compressions compared to Text based formats.
Low Network latency mandates for lesser data packet size, lesser time to synchronize between sender and receiver and reducing network hops...Binary encoding formats helps to achieve lesser packet size, Cloud based systems helps to access at central location at network for reducing hops and Avro, GZip, ORC, helps us to do the same.
Low Memory mandates for Efficient compression and minimalized storage requirements. Binary Formats and Compression methods helps in this and GZip, Avro, Protocol Buffers helps us do the same.
Low Processing power mandates for reduced complex data structures, efficient compression and simpler constructs. Json, CSV, Text has the simpler constructs and with Gzip compression can help to address the issue and with Protocol Buffers with inherent support for Binary format, Schema (which is scalable based on requirement) could also help in this regard.

As the constraints differ across application requirements, available Infrastructure and various other internal and external factors, we need to arrive at a format which is best for your application.

How to transmit the data faster and secure?
Clients & Servers are distributed across Public/Private networks and hence the need for encoding the data for faster transfer and encrypting the data for Secure transmission is a necessity.

The process of Encoding is used to convert the data from its original format into a format that can be transmitted or stored more efficiently. The Encoding process is typically used to compress the data and to make it more compact, so that it can be transmitted more quickly or stored more efficiently.

The process of decoding is used to convert the encoded data back into its original format so that it can be used by the system. The decoding process is typically used to decompress the data, so that it can be used by the system in its original format.

The Process Encryption is to secure the data through the network so that eavesdroppers cannot read the message easily. Both these processes are quite important in today's networked environment.

Similarly, The Process of Decryption is to decipher the encrypted message without any loss or inconsistency at the receiving end.

(Some common Encoding/Decoding methods are base64, UTF, ASCII, URL Encoding...etc... Some Common Encryption/Decryption methods are AES, DES,RSA....etc)...In my later posts, I will be covering the Encoding and Encryption concepts in more detail.

Below process diagram shows the concepts

(Encoding/Encryption can happen at different orders and stages and not necessarily as per above depiction. It all depends on the requirements and specific architecture the team is concerned about)

Here is the sample "Employee" object represented in different data formats.

For the above "Employee" data, Let me perform the base64 encoding format (with UTF-8). Below is the output for the same
As you can see the data is now modified from Json format to ASCII format useful for transmission across HTTP. With additional Encryption method like AES/RSA the same data will be more secure, and it will be difficult for hackers/intermediates to decipher it. Try out different sample data and perform Encoding/Decoding them using below URL
base64

For more in-depth understanding of the data formats discussed, please do check out the below links for specifications and standards globally referred and used.

Xml Specification ---> Xml_Spec
Json Specification ---> Json_Spec
Proto Buffer Spec ---> Proto_Buffer
CSV Specification ---> Csv_Spec
YAML Specification ---> Yaml_Spec
Avro Specification --->Avro_Spec

Pls do share your thoughts and list the data formats you have come across that would be helpful for other readers.

Types, DS & Algol in .NET

raviklog — Tue, 10 Jan 2023 12:02:12 +0000

Hi Folks! please check out my previous post, which deals with basic building blocks of .NET and how this current post followed from there

Building blocks of .NET

raviklog ・ Dec 13 '22

#beginners #programming #dotnet

Types are the building blocks of programming language, and it allows to define the variable type or an expression. It defines the set of values that a variable can take on and the operations that can be performed on those values. So, this aids the C# compiler to perform type checking, ensure type safety, and select the correct method overload at compile-time.

In .NET all of the types basically reach at the top of hierarchy "System. Object" but falls under either Value or Reference Types.

So still there is a need to arrange these types in a manner in which simple and composite level can be achieved. So that's where the classification of Primary/Primitive type and Secondary/Derived types comes into play

These Primitive types are basically Scalar (or single valued) in nature and its useful for handling simple values (numbers, characters, boolean and so on) ...

Integral types: sbyte, byte, short, ushort, int, uint, long, ulong
Floating-point types: float, double
Decimal type: decimal
Boolean type: bool
Character type: char

Just if we know only these types can we design an application...Let's say we create a calculator/ Tic-Tac-Toe game...so in this kind of scenarios...it's possible to use just the primitive types to create an application, but then it's limited and we can't create more intuitive and valuable applications by just using primitives...They can be basic foundation, but we need more elaborate types to handle extended applications.

Then comes the Secondary Data types/Composite types, set of value types that are built-in to the framework and are used to represent different types of data. Examples of secondary data types in .NET as given below,

An array is a collection of items of the same data type that are stored in contiguous memory locations.

A string is a sequence of characters and are immutable. This means that once a string is created, its value cannot be modified. Instead, any operation that would normally modify a string in-place will actually create a new string with the modified value.

An enumeration is a set of named constants. Enumerations are useful when you have a fixed set of values that a variable can take on, such as the days of the week, or the suit in a deck of cards.

A Structure is a value type that can contain a set of fields and methods. It is normally used to represent simple logic within an application. (For Example: below I have declared a default container with properties width and height and a method to calculate size of the container on its own. Simple data storage and logic could be done through struct.

public interface ISampleContainer
{
  public double GetContainerSize();
}
struct TestContainer: ISampleContainer
{
   private double width.
   private double height.
   private double magicNumber;
   public double GetContainerSize()
   {
     return (width * height * magicNumber);
   }
}

(Structures do perform encapsulation as above with members and functions, does an interface inheritance, but structures do not allow implementation inheritance from classes or other structures...So basically, the hierarchical system could not be formed through structures. So, we cannot have a parent structure/class for another structure.)

Class: A Class is a reference type that can contain a set of fields, methods, and events. It is also called a user defined type as it takes it existence based on the developer's necessity to handle the characteristics and behavior of objects of that type.
The True power of classes is it is the main way to exhibit OOPs concepts (Inheritance, Encapsulation, Abstraction & Polymorphism) ...Detailed post on OOPs will do it another post with samples.

class MyTestContainer: ISampleContainer
{
   public double width.
   public double height.
   public double containerFactor;

   public virtual double GetContainerSize()
   {
     return (width * height * containerFactor);
   }
}
class CircleContainer:MyclassContainer
{
   public double Radius {get;set;}
   public override double GetContainerSize()
   {
     return Math.PI * Radius * Radius;
   }
}

(Class "CircleContainer" does exhibit the Inheritance concept where it overrides the default base method GetContainerSize of Parent class MyTestContainer)

Do .NET provide some pre-defined structures and classes?
Yes, there are quite a number of them, some of most commonly used
Structures and Classes in FCL,

Data types define the kind of data that can be stored in a variable or expression, while data structures define how that data is organized and stored in memory. In case of large sets of data for an application, then we need efficient way to organize the data, so that certain operations like Analysis, Retrieval can be done quicker and optimal way.

Here are a few common data structures in .NET:

As a developer we have the types and data structures to store and organize data for managing the application-level information, but
Algorithms are used to solve problems and perform tasks in a systematic and efficient way.

Here are a few common algorithms that are frequently used in .NET development:

Sorting algorithms: Sorting algorithms are used to rearrange a list of items in a specific order, such as ascending or descending. Examples of sorting algorithms include bubble sort, selection sort, and quicksort.

Search algorithms: Search algorithms are used to find a specific item in a list or collection. Examples of search algorithms include linear search and binary search.

Graph algorithms: Graph algorithms are used to traverse and manipulate graphs, which are data structures that consist of vertices (nodes) and edges. Examples of graph algorithms include depth-first search and breadth-first search.

Recursion: Recursion is a technique in which a function calls itself to solve a problem. It is often used to solve problems that can be broken down into smaller, simpler subproblems.

Dynamic programming: Dynamic programming is a technique for solving problems by breaking them down into smaller subproblems and storing the results in a table to avoid redundant calculations.

By understanding and using these types, data structures and algorithms, we can write more efficient and effective code in .NET.

DS & Algos are quite an effective topic, I will try adding more posts to have clarity on the concepts and also do some code samples on the same.

Please do convey your comments/suggestions, it will help me to improve the content overall!

Driven by Data! (Part1)

raviklog — Sat, 07 Jan 2023 08:52:12 +0000

Everything we do (Write, Learn, Speak, Eat, Play, Travel, Work.... etc.) gets translated into Data and it's only getting multi-fold every single day...There is an estimate that every single day online we generate close to 2.5 quintillion bytes(around 2020 stats and this number would have crossed now in big way). If we consider a penny as a byte of data...then the whole earth laid out flat can be filled 5 times with that number...that's quite staggering!

1 Quintillion = 1,000,000,000,000,000,000(10^18)
2.5 Quintillion = 2.5 Billion Giga bytes.

According to Netflix_Data, you use 1 GB of data per hour when you stream a standard definition (SD) video. High definition (HD) videos, on the other hand, use 3 GB per hour. And 4K Ultra HD streams use up to 7 GB per hour of video...So you can calculate what's the 2.5 Quintillion comes to.

Some Stats online to check:

Checkout the links for more stats on Data usages online FinOnline
Wide variety of Data captures across the world checkout OurWorldinData

With this flow of Data, we need to categorize to find a meaningful information/insight out of it. Some common ways to categorize data as given below,

Structured vs Unstructured
Structured data is organized in a predefined format, such as a table with rows and columns. Unstructured data does not have a fixed format and may include things like text documents, emails, and social media posts.

Quantitative Data: Data that can be measured in numbers
Qualitative Data: Quality or Characteristic of a Data...No Measurement but core attributes/characteristics
Quantitative

Qualitative

Continuous vs Discrete:
Continuous data can take on any value within a certain range (e.g., Time Measurement).
Discrete data can only take on specific values/finite numbers (e.g., the number of children in a class).

Continuous

Discrete

Primary vs Secondary:
Primary data that is collected directly from the source (such as experimental data which is original and unbiased)
Secondary data that is already collected by other person or organization and external (such as Internet source, books, article and blogs)

Primary

Secondary

Public vs Private:
Public data is freely accessible by everyone (published online such as government data, social media posts)
Private data is secured and could be accessible only by a few authorized personnel for security and confidentiality(such as Individual’s financial records, trade secrets)

Public

Private

Some other categories

(Data originating from IOT devices, Video Telecasts called as Live/Streaming data)
Historical Data from books, Articles, recordings collected over time as offline/historical)
Based on Text/Image/Geometric/Video/Audio data collected.

Properties of Data: We need to organize this huge collection of data based on the properties, that helps us to access, search and retrieve through a defined system that we now call as "Databases", "Datastores", "DataLakes" and other modern terms.

Accuracy -> Free from errors/inconsistencies
Validity -> Data which meets the requirements of the system or application in which it is being used.
Timeliness -> Available when it is needed
Completeness -> Includes all the information that is required for a given purpose
Relevance -> Related to the problem at task at hand
Consistency -> Uniform and follows the rules and standards
Integrity -> Protected from unauthorized access or modification.
Accessibility -> Retrieved or Accessed by those who need it.
Understandability -> Easily interpreted and understood by those who need to use it. 10.Usability -> Data is usable if it can be effectively used for a given purpose.

So, all major databases are designed to take into consideration all the above properties and much more for the benefit of users,

What is the Purpose of the Database?

When you need to store and manage large amounts of data: A database is a good way to store and organize large amounts of data so that it can be easily accessed, managed, and updated.

When you need to support multiple users: A database can be used to store data that is shared among multiple users, allowing them to access and update the data concurrently.

When you need to store data for analysis: A database can be used to store data that needs to be analyzed or queried in various ways.

When you need to store data that needs to be backed up: A database can be used to store data that needs to be backed up regularly to prevent data loss.

When you need to enforce data integrity: A database can be used to enforce data integrity rules, ensuring that the data is consistent and accurate.
When you need to store data that needs to be secured: A database can be used to store data that needs to be protected from unauthorized access.

When you need to store data that needs to be accessed quickly: A database can be optimized to allow fast access to the data it stores.

That's a wrap on the Data basics and in the Next post, we will detail the Database concepts and latest trends. Please do give your suggestions/comments.

Not Just Programming!

raviklog — Thu, 29 Dec 2022 11:19:35 +0000

I was getting ready for my next article on .NET but then this thought of how it's always not programming/technical capability or know-how that helps to refine the Software development or any other piece of work.

It's Not an Advice! and just my observations and experiences that I get every day in Software space.

Read/Analyze what others have done:
Will be helpful for someone new to a project or completely new to Software Development or even other domains. It will be tough to read a past technical/domain documents or Historical Source codes, but I would say it improves certain things like "Analysis", "Patience", "Flexibility" and also "How different we can modify/innovate current requirements based on what was done before us" and also give the range of ideas to speak to your peers and also cross-teams.

Go Cross knowledge:
Sometimes solving mathematical or Logical puzzles or playing a sport or music or just a walk and chit-chat could help refine your thinking process which actually helps in your Software Development process and helps to handle bigger tasks or algorithms. After an early morning run or a fantastic badminton practice...suddenly there is a better clarity of how to design a system or solve a complex bug...Take up that excites you, but just do it as it is normal, don't force it.

Look for Alternatives:
Sometimes you might reach a solution immediately on top of the head and would be willing to complete that and push the source code and finish it off...hold on!...it might be great to get the solution just like that...you are already good in that, but it's always better to find multiple alternatives, which could be(an improvement in process/performance/quality...that alternative solution could help you at a right moment, where-in you might have to ship a Changeset in couple of hours/days...critical junction...so this pre-knowledge on alternatives will surely come handy)

Open to Review:
Most of the time it's difficult for Programmers/Designers to take things lightly if someone reviews our code and indicate mistakes (which might be trivial/complex/hybrid.)...Instead try to sit through the points one by one and analyze better and if you feel what you have done is fine...then try to convince in a friendly manner or check with other peers about the corrections suggested and accept them and keep the sync going. This will help to reduce the ramblings in meetings and improve our overall perspective about working in teams, change acceptance and negotiations.

Complex Work:
Let's say you are comfortable with existing project. Dev/Test Updates, Deployments and Client request resolution. So, this cycle could be good, but to prepare for bigger assignments, in between your regular tasks, take a complex algorithm to solve for another project in real-time...need to crunch extra hours...but in longer run, it will help to get noticed by many teams and increase your chance of getting complex and challenging projects.

Keeping Tabs on Trends:
As it's quite a competitive landscape, as technology is...It's a given that we need to embrace new trends on technology, those that interest us and practice them to continue the enthusiasm going on. Technology for an extended period is quite a challenging one. Relaxation, Less Motivation and Boring Regularity might haunt us, but then we need to diffuse and keep going forward and be ready for that fantastic opportunity.

I am pretty sure this is not go-to list or a complete list...Just the observations and learnings I got. Hope it at least have some convergence with your observations as well.

Please do Share your own comments/suggestions to improve on this.

Building blocks of .NET

raviklog — Tue, 13 Dec 2022 17:50:23 +0000

In the Previous post, tried to touch upon the .NET Ecosystem and its different variants and how it provides the core development support for the Developer community...pls check out if you have missed...

.NET Ecosystem

raviklog ・ Nov 22 '22

#dotnet #framework #beginners #core

In this article, presenting the basic building blocks that make up the .NET Framework in detail

As Humans we need language to communicate and that’s true for the world inside machines and there should be rules/specifications defined, so that developers can follow them and make the information flow in a meaningful way through applications.

CLI (Common Language Infrastructure) provides the base infrastructure with set of specification/standard that defines the basic rules for creating and managing the applications and also the runtime environment that supports application execution at multiple platforms. The core components that help to perform the same,

Common Type System (CTS)
Common Language Specification (CLS)
CLR (Common Language Runtime)
FCL (Framework Class Library)

The Common Type System (CTS) is a key component of the Microsoft .NET Framework, and it is used to define and enforce rules for the data types that can be used within .NET applications. CTS ensures that all .NET languages use a common set of data types and that they can interoperate and exchange data.
At the broader level, CTS differentiates data types as Value & Reference types, which indicates the way data is stored in memory.

Value types get stored in the stack part of the memory and Reference types gets stored in the Heap part of the memory and there are reasons to it... (More on stack & heap in my later article)

What are some of the common Rules that CTS mandates?

Any types declared in .NET languages need to fall under (either Value/Reference types). So, it gives the power to .NET to handle manage, store and execute according to its boundaries and make it strongly typed. All these types can be traced back to System.Object in .NET. (User defined classes are not inheriting directly from System.Object, but it is an implicit inheritance that the CTS handles)
CTS does not allow types to inherit more than one base type.
CTS also defines rules for operators such as (arithmetic and comparison operators), which can be overloaded and used with different types.
CTS defines rules for generics (to create type-safe collections), rules for interfaces (to define properties and methods), rules for enumerations, exceptions & nullable types

Below sample shows the usage of primitive types in 2 different languages (C# & VB.NET) and how CTS defines base types and organizes it in root System namespace.

The Common Language Specification (CLS) is a subset of CTS, and it ensures that all the.NET Framework languages can interoperate and exchange code between them.

CLS rules are more of how the languages implement specifications. Some of the common CLS rules are as below,

All languages must support the CTS, which defines a set of common data types.
All public methods and properties must use only CLS compliant data types, which are data types that are defined by the CTS and are supported by all languages.
All languages must support exception handling and must use the same mechanisms for throwing and catching exceptions.
All languages must support interoperability with other .NET languages and must provide mechanisms for calling methods and properties defined in other languages.

Does all .NET languages support CLS Rules?
Hmm...there are languages which are CLS Compliant (C#, VB.NET), and some are not (F#,IronPython), but since all these languages fall under.NET Framework, they can interact with each other. However in some cases, the Non-Compliance issues might arise during cross integrations and those issues need to be handled by the developer on case-by-case basis(like in some cases follow compliance rules, or opt for conversion route or hybrid of both)

*Does it seem CTS & CLS overlap? *
There is always this confusion on the overlapping of the CTS & CLS, but in broader view, the CTS defines a comprehensive set of rules for defining and using types in code, while the CLS defines a more limited set of rules that all.NET languages are required to support for better interoperability.

Then comes the FCL (Framework Class Library)- main staple for the day-to-day activities of a.NET developer, is a set of libraries and APIs that cover various domains and functionality, such as data access, networking, security, and user interface. some of the commonly used libraries

Check through the below URL for complete list of libraries supported by.NET
.NET FCL

Finally, the execution engine of the .NET Framework - CLR (Common Language Runtime).It is responsible for executing.NET applications and managing their runtime environment. The CLR provides services such as memory management, security, and exception handling, and it is responsible for enforcing the rules and conventions defined by the CTS to ensure that programs run safely and without causing unintended effects.
So once the developer generates the binaries intended for specific machine architecture (x86/x64) it can be used in the target machine for execution. So, in any client/target system, the .NET Runtime Framework needs to be installed to execute the binaries.

As more applications are migrating/already migrated to Cloud and Docker based systems, where-in compilation, deployment and execution are so modularized/containerized, this article here only tries to give an insight on how the.NET framework handles it for all standard.NET applications.

It might be that you have chance to develop Standard.NET applications or migrate them to newer designs or possibly creating compilers or more in-depth work on Framework, knowing the building blocks could help to drill down core specifications to build newer systems.

Let know of your comments/suggestions!!!

.NET Ecosystem

raviklog — Tue, 22 Nov 2022 12:53:53 +0000

.NET Ecosystem is quite vast, and it is difficult to comprehend certain times where all the systems fit in. In my first article here, I will attempt to simplify the details and hope you like it.

All of us either worked/heard/used applications of different variants (Mobile, Web, Cloud, Games, Desktop, Console, Services and so on in Microsoft or other Platforms over the years. .NET provides capabilities to develop and ship them to end users, which we could call it as the ecosystem)

.NET Framework is the Foundation of the Ecosystem and the first to be introduced by Microsoft around early 2002. It encompasses the following,

Common Language Infrastructure (CLI) which is basically a neutral platform for application development and execution irrespective of the Programming language that.NET supports (C#, VB.NET, F#...) …So basically, contains Language constructs that support this behavior.
Framework Class Library (FCL) that provides pre-defined set of Core Functions, Tools and Libraries which Developers can use to develop their applications.
Execution Engine of .NET Framework (“CLR or Common Language Runtime”) which executes the .NET applications at the target system. It happens in 2 steps

Language code is compiled into an Intermediate code (MSIL) during build/compile time.
During the execution of the application, the Just in Time (JIT) compiler turns the IL code into Machine readable code... (Sample hello world program is demonstrated below...here we can see how .NET source is compiled to Intermediate code by compilers and then at runtime machine specific code is generated through JIT)

(Refer the ildasm disassembler as part of .NET VS tools so see your IL code)
Do checkout the URL Link for more options to work on the tool

Ildasm.exe (IL Disassembler) - .NET Framework | Microsoft Learn

Use Ildasm.exe (IL Disassembler), which takes a portable executable (PE) file containing intermediate language (IL) code, and makes a text file for Ilasm.exe.

learn.microsoft.com

(.NET Framework 4.8 is the Latest version from this edition and planned for longer support.)

.NET Core is an improvement over the .NET Framework, where-in the idea behind is to provide cross platform support for other operating systems (for Linux and MacOS) and make it open source so that it reaches wider developer audience and allows developing modern applications. Basically, Microsoft has decoupled the .NET Framework from windows-based and created the Core as generic platform. Now the .NET apps can run in containers on Linux and other places. It encompasses the following

CoreCLR which is similar to what CLR does for .NET Framework with much bigger canvas of execution environment across platforms.
CoreFX which gets the roots of the FCL from .NET Framework and provide add-on libraries on top of that to support varied platforms and application categories.
Nuget the Package Manager introduced in Core and supports sharing of reusable code libraries…It also aids in getting latest updates/revisions of software. Developed as an SAS by .NET foundation and promoted as open source.

Better to check out the link core framework for various .NET versions currently active and in for LTS (Long term support)

.NET Standard at the basic level is a bridge for the libraries used in (.NET Framework, .NET Core, Unity, XAMARIN and so on...). So basically, a conformance path where-in the interoperability between applications built in.NET framework and other .NET variants could be maintained.
*.NET Standard (2.1) * is the Latest version.
Till .NET Standard (2.0) version, it is observed to be most supportive of.NET framework versions (till 4.8) but in.NET Standard (2.1) or later, the.NET takes its own deviation in terms of specifications for.NET 5 or later and it’s not part of Standard 2.1.

Tools Regular and repetitive tasks of developers such as development, configuration, analysis, testing and deploying is supported through the *VS (Visual Studio) * Code editor with range of embedded tools...This broadly reduces the time to look outwards and be more productive.

Code Editing
1. Intellisense that gives suggestions/recommendations for API access, helps to avoid syntactical errors and aid code completion.
2. Automatic code generations through shortcuts and support for macros.
3. Other interactive options such as (Indentations, Outlining, bookmarking, snippets and direct code run/debug operations and so on...)
Code Integration with GIT Repos (Developers can create local repositories to maintain code and also sync up with remote repository through VS... hugely helps to be consistent.
Exploring(VS provides Exploring/connecting to external databases in Remote servers or in Cloud, Solution explorers to maintain different Projects, Test Explorers to configure and maintain Test codes and other exploring options to enhance code productivity)
Shell/Command based (Executing build scripts or deployment batches could be done faster through PowerShell/command lines)

.NET through VS gives an integrated approach right from Code production to deployment and maintenance and just aids the whole lifecycle of Software Development.
(VS Code...the open-source version of Visual Studio is lightweight and also supports array of programming languages.
Do checkout this link for knowing more
VS Code

Finally, Extending .NET across other platforms

Unity is a Cross-platform gaming engine, that uses Mono (the open-source version of.NET framework based on ECMA standards) for their back-end scripting. Recently there seems to be a move-over to ILCPP by Unity. So, need to see how in coming times the adaptation of .NET in Unity/other Gaming platforms go through). To study more on Unity pls check out Unity
UWP (Universal windows platform) is a Windows only initiative that targets different devices (Windows PCs, Xbox, Surface, Holo Lens and others...developers can create apps and publish them on the windows store and get monetized as well...Check out the link for more details
UWP
Xamarin, Yet another Cross platform Mobile framework from Microsoft. Xamarin was introduced to create native apps for Android, iOS & Windows using C#.NET. To get started check out Xamarin
successor to Xamarin, Multi-Platform App UI with focus on building native desktop and mobile Apps for cross platform…Kind of unifying multiple.NET technologies. Check out the Link for learning more on this MAUI

Tried to converge different parts of this .NET ecosystem only at basic level and there is much more to explore.

hope it helps and do share your suggestions!