DEV Community: Aryama

NumPy & Data Manipulation

Aryama — Wed, 11 Mar 2026 17:12:31 +0000

By the end of this session you will learn:

• What NumPy is
• Why NumPy is faster than Python lists
• Core NumPy operations
• Data manipulation techniques
• A real-world example

1. Introduction

Python is widely used in data science, machine learning, and AI systems.

At the core of these systems lies a powerful library called NumPy.

NumPy stands for Numerical Python and provides:

High-performance arrays
Mathematical operations
Matrix computations
Fast vectorized operations

Many popular libraries are built on top of NumPy:

Pandas
Scikit-Learn
TensorFlow
PyTorch

Understanding NumPy helps us understand how modern AI systems process data efficiently.

2. Why NumPy? (Python List vs NumPy Array)

Python lists are flexible but not optimized for large numerical computations.

NumPy arrays are implemented in C, making them significantly faster.

Example: Performance Comparison

import numpy as np
import time

size = 1000000

list1 = list(range(size))
list2 = list(range(size))

start = time.time()
result = [x + y for x, y in zip(list1, list2)]
print("Python list time:", time.time() - start)

arr1 = np.arange(size)
arr2 = np.arange(size)

start = time.time()
result = arr1 + arr2
print("NumPy array time:", time.time() - start)

Explanation:

NumPy performs operations on entire arrays at once, called vectorization.

This removes the need for loops.

3. Creating NumPy Arrays

NumPy arrays are called ndarrays (N-dimensional arrays).

Basic Array Creation

import numpy as np

arr = np.array([1,2,3,4,5])
print(arr)

Creating Arrays of Zeros

zeros = np.zeros((3,3))
print(zeros)

Creates a 3×3 matrix of zeros.

Creating Arrays of Ones

ones = np.ones((2,4))
print(ones)

Creates a 2×4 matrix filled with ones.

Creating Ranges

numbers = np.arange(0,10,2)
print(numbers)

Output

[0 2 4 6 8]

4. Array Shapes and Dimensions

NumPy arrays can be multi-dimensional.

Example:

matrix = np.array([
    [1,2,3],
    [4,5,6],
    [7,8,9]
])

print(matrix.shape)

Output

(3,3)

Meaning:

3 rows
3 columns

5. Vectorized Operations

NumPy allows mathematical operations without loops.

Example:


arr = np.array([10,20,30,40])

print(arr + 5)
print(arr * 2)
print(arr / 10)

Output

[15 25 35 45]
[20 40 60 80]
[1. 2. 3. 4.]

Explanation:

The operation is automatically applied to each element of the array.

This is called broadcasting.

6. Indexing and Slicing

1D Array

data = np.array([10,20,30,40,50])

print(data[0])
print(data[1:4])

Output

10
[20 30 40]

2D Array

matrix = np.array([
    [1,2,3],
    [4,5,6],
    [7,8,9]
])

print(matrix[1,2])

Output

6
Extract Column

print(matrix[:,1])

Output

[2 5 8]

Explanation

: means all rows

7. Aggregation Functions

NumPy provides built-in functions for data analysis.

Example:


data = np.array([10,20,30,40])

print("Mean:", np.mean(data))
print("Sum:", np.sum(data))
print("Max:", np.max(data))
print("Min:", np.min(data))

Output

Mean: 25
Sum: 100
Max: 40
Min: 10

8. Boolean Filtering (Very Useful for Data Cleaning)

Example dataset:

scores = np.array([55,78,90,34,88,67])

Find students who passed:

passed = scores[scores > 60]
print(passed)

Output

[78 90 88 67]

Explanation:

NumPy allows filtering data without loops.

This is heavily used in data preprocessing pipelines.

10. Real-World Example – Netflix Recommendation System

Recommendation systems power platforms like Netflix.

Users and movies can be represented as vectors.

Movie List

movies = ["Interstellar", "Inception", "Titanic", "Avengers"]

User Ratings Matrix

ratings = np.array([
    [5,4,1,1],  # Alice
    [4,5,1,1],  # Bob
    [1,1,5,4]   # Charlie
])

print(ratings)

Rows represent users

Columns represent movies

Cosine Similarity

Recommendation systems often compute similarity between users.

Formula

similarity = (A · B) / (|A| |B|)
Implementing Similarity

from numpy.linalg import norm

alice = ratings[0]
bob = ratings[1]
charlie = ratings[2]

sim_alice_bob = np.dot(alice,bob) / (norm(alice)*norm(bob))
sim_alice_charlie = np.dot(alice,charlie) / (norm(alice)*norm(charlie))

print("Alice vs Bob:", sim_alice_bob)
print("Alice vs Charlie:", sim_alice_charlie)

Expected output

Alice vs Bob: ~0.98
Alice vs Charlie: ~0.32

Explanation

Alice and Bob have similar movie taste.

Charlie has different preferences.

Recommendation Logic

If Alice hasn't watched Avengers, we can recommend movies liked by similar users.

alice_ratings = np.array([5,4,1,0])
bob_ratings = ratings[1]

recommended_index = np.argmax(bob_ratings)

print("Recommended movie:", movies[recommended_index])

JS Concepts

Aryama — Tue, 28 May 2024 05:46:39 +0000

Topics:

TODO

template literals
How to create methods inside const
prompt()
Number()
Array methods: push, slice, indexOf , length , join , split
for(const it of arr){ }
{ result }
Word frequency
map , filter , reduce
Modify DOM elements

UX Design fundamentals

Aryama — Fri, 17 May 2024 00:19:49 +0000

User experience simply refers to the way a product behaves and is used in the real world. A positive user experience is the one in which the goals of both user and the organization that created the product are met.

It also refers to as a practice , a way to generate predictive designs considering human beings , how we are wired , how we interpret things , how we know how to use certain things or how we consider anything good or bad or otherwise. UX designs takes all these things into consideration and molds it to an experience that delivers value

Elements of User experience design:

- Strategy - Why was the product created , why people need it (user needs , business objectives , user research)

User research process:
Stakeholders interviews , competitor reviews , user interviews/research, exit product audit

Questions to ask stakeholders:

What does the product accomplish for the business?
How do you expect to differentiate this product?
What technology decisions have been made?
If users are using competitors product instead , what is the reason?

Questions to ask user:

How often do you use the product?
What step/task would you put off as long as you can?
Can you show me how to do that step/task?

- Scope - Feature and functions contained within the product (functional specification)

It helps reduce potential conflicts , to define what we are building , what we are not , and what we are not building right now. Its good process to deliver changes iteratively to the users.

- Structure - The number of places where you can go , organized contextually. A good structure organizes information in a way that provides intuitive access to content.

- Skeleton - The optimized organization and arragement of visual elements

What form will the product take?
How will content be manipulated and presented?
How will users move around and do things?

- Surface - UI screens , text , the user can interact with (visual design)

Basic principles of Visual Design:

- Alignment

- Proximity - Elements which are visually close to each other are perceived as a single group , related by the context of use.

- Contrast - It makes things the user wants to see, stand out

Gestalt Principles of UI Design:

People tend to focus on sum of all parts , rather than indivisual whole.

- Similarity: Items which share a visual characteristics are more related than items which are dissimilar

- Proximity: Elements placed closed to each other are perceived as related

- Connectedness: Connected Elements are perceived as related

- Continuation: The eye tends to follow visual paths and sequences in order to identify relationships between designed elements

- Figure/Ground:

User experience Laws:

- Aesthetic usability Effect: Things that look nice and aesthetic appeal more the the viewers/users

- Doherty threshold: Keep the wait time as short as you can

- Fitt's law: Make the Vital UI components large enough for users to accurately select them and give ample spacing between them

- Goal gradient effect: The closer you are to completing a goal or a task , the faster you work towards reaching it.

- Hick's Law: Too many choices often leads to a slower decision making process

- Jakob's law: Do not break the concept of similarity

- Millers law: Split tasks into logical , easy to process steps

- Zeigranik effect: Provide a clear indicator of progress to motivate user to complete the task

- The isolation effect: Unique and different items stand out.

- Teslers law: While some tasks seem unneccesarity complex, the complexity is integral to the function of the system and it can only be managed and not eliminated

Designing Tools:

Figma
Sketch
Adobe XD
Invision Studio

Advanced Tools for prototyping

Principle
Framer

Snowflake Introduction

Aryama — Thu, 11 Jan 2024 15:23:43 +0000

What is Snowflake?

It is an analytics database or Data warehouse as a service
It is a SaaS tool to load , analyze and report on massive data volumes
It provides data storage and data analytics solution
Data warehouse solution sitting on a cloud only
Snowflake cloud providers - Mircosoft Azure , Amazon Web services , Google cloud platform
Pay per second billing model. You only pay for what you store and running compute. When compute is not used, it is not charged.

Alternatives to Snowflake - Google BigQuery , Amazon Redshift , Databricks

Snowflake architecture

Architecture diagram:

It has a multi clustered shared architechture

Snowflake’s architecture is a hybrid of traditional shared-disk and shared-nothing database architectures. Similar to shared-disk architectures, Snowflake uses a central data repository for persisted data that is accessible from all compute nodes in the platform. But similar to shared-nothing architectures, Snowflake processes queries using MPP (massively parallel processing) compute clusters where each node in the cluster stores a portion of the entire data set locally. This approach offers the data management simplicity of a shared-disk architecture, but with the performance and scale-out benefits of a shared-nothing architecture.

It decouples both compute and storage.

It has 3 layers -> Storage layer , Compute layer ,Cloud Service layer

Storage layer -
At the centre , we have storage - stores table , views , data is stored in both structured and well as semi structured.

They are Compressed & Encrypted(AES 256) and then stored.

Snowflake converts them into optimised columnar compressed format (proprietary to snowflake)

Compute layer - Virtual warehouses are connected directly to the GCP compute instance/Amazon EC2 instances. VW are the place where the queries are executed. VWs can be scaled up or down on demand

Comes in Various Sizes
• X-Small - Single Node(DDL)
• Small - Two Nodes
• Medium - Four Nodes(Data load)
Large - Eight Nodes
• X Large - Sixteen(Data processing)

Storage and compute charged independently and only for usage

Eg: If you store TBs of data and no processing, you will be charged only for storage and not for processing.

Cloud Service layer

Authentication & Authorisation
User & Session Management
Query Compilation, Optimisation & Data caching
Virtual Warehouse Management , Coordinate Data Storage/Updates & Transaction
Metadata Management - Zero Copy Cloning , Time Travel, Data Sharing
Manage and Maintain the life cycle of a query

Features

Unlimited Storage & Compute - Advantages of infinite scalability, elasticity, & redundancy features and hence you can store more and scale up/down your compute as needed.
Supported by all the major cloud providers
Data Platform as Service - There is virtually no software to install, configure or manage.Ongoing maintenance, management, upgrades, and tuning are handled by Snowflake
Time Travel Feature & Fail Safe -
As part of continuous data protection lifecycle, snowflake allows you to access historical data (table, schema or database) at any point with in the defined period.
Clone or Zero Copy Clone - Clone or Zero Copy Clone creates a copy of database, schema or table without actually copying the data. It is a snapshot of the data to the source object.The clone object is writable object and independent of source object.Cloning is just an SQL statement and since it does not need any additional space, many data copy challenges can be easily solved.Cloning feature is also used to build environments like Prod to QA or QA to Dev or visa-versa without any extra storage cost
Support for semi structured data
On demand pricing

Micro partitions

Common Server Setups for Web Applications

Aryama — Fri, 18 Aug 2023 13:34:33 +0000

While going through the basic System design courses , one of the most common term which I came across is SERVERS. Now we know that these are one of the most important components in systems , here is a list of different ways to setup a web service.

Single Server: This technology is straightforward but could suffer from slow performance due to resource competition and lacks the ability to scale horizontally.
Separate Database Server:
In the architecture, an application server and a database server operate with separate resources, allowing individual vertical scaling, but the network latency between them increases due to the database being a network hop away.
Load Balancer: By distributing work among multiple servers, the system achieves native horizontal scaling and defends against DDOS attacks with rule-based protection, yet introduces complexity that could potentially lead to performance bottlenecks and complicate aspects like SSL termination and stick sessions.
HTTP Accelerator: By caching web responses in memory to enhance rapid serving, alleviate the web server's CPU load, and utilizing compression for reduced bandwidth consumption, the system requires tuning and could experience decreased performance with a low cache-hit rate.
Master-Slave Database Replication: Can improve read and write performance. Adds a lot of complexity and failure modes.
Load Balancer + Cache + Replication: The approach involves integrating load balancing for caching and application servers, along with database replication.
Database-as-a-Service: Delegate database management to external providers, such as Amazon's RDS and other hosted versions of popular databases, allowing someone else to handle the operation.
Backend as a Service: When developing a mobile app and aiming to avoid backend intricacies, outsourcing the backend to third-party services like Parse or Firebase lets us focus on the challenging mobile platform aspects, given their popularity and convenience.

9.Platform as a Service: Opt for a partially managed backend solution that offers greater flexibility compared to Backend as a Service (BaaS), leveraging options like Google App Engine, Heroku, Salesforce, and various alternatives, allowing us to build our application while delegating most backend responsibilities.

Ref: https://www.digitalocean.com/community/tutorials/5-common-server-setups-for-your-web-application

Low Level Design fundamentals

Aryama — Wed, 19 Jul 2023 18:14:16 +0000

Prerequisites — OOPs fundamentals

Solid Principles:

S — Single Responsibility Principle

O — Open/Closed Principle

L — Liskov Substitution Principle

I — Interface Segmented Principle

D — Dependency Inversion Principle

Single Responsibility Principle : A Class should have only one reason to change.

Open/Closed Principle : Open for extension but closed for modification.

Liskov Substitution Principle : If Class B is the subtype of Class A , then we should be able to replace object of A with B without breaking the behaviour of the program.

Interface Segmented Principle : Interfaces should be such , that client should not implement unnecessary functions they do not need.

Dependency Inversion Principle : Class should not depend on interfaces , rather than concrete classes.

Advantages of following these principles:

Avoid Duplicate Code.
Easy to maintain.
Easy to understand.
Flexible software.
Reduces complexity.