DEV Community: Ramakrishnan83

PySpark & Apache Spark - Overview

Ramakrishnan83 — Fri, 02 Feb 2024 22:00:38 +0000

PySpark is Python API for Apache Spark. It enables us to perform real-time large-scale data processing in a distributed environment using python. It combines the power of python programming with power of Apache Spark to enable data processing for everyone who are familiar with python.

Spark SQL and DataFrames:
Spark SQL is Apache Spark’s module for working with structured data. It allows you to seamlessly mix SQL queries with Spark programs. With PySpark DataFrames you can efficiently read, write, transform, and analyze data using Python and SQL. Whether you use Python or SQL, the same underlying execution engine is used so you will always leverage the full power of Spark.

I will be discussing more the Spark SQL and Data frames in my blog.

Spark Core and RDD
Spark Core is the foundation of the platform. It is responsible for memory management, fault recovery, scheduling, distributing & monitoring jobs, and interacting with storage systems. Spark Core is exposed through an application programming interface (APIs) built for Java, Scala, Python and R. These APIs hide the complexity of distributed processing behind simple, high-level operators.

Apache Spark recommends using Data Frames instead of RDDs as it allows you to express what you want more easily and lets Spark automatically construct the most efficient query for you.

Apache Spark can run on single-node machines or multi-node machines(Cluster). It was created to address the limitations of MapReduce, by doing in-memory processing. Spark reuses data by using an in-memory cache to speed up machine learning algorithms that repeatedly call a function on the same dataset.

How Apache Spark Works:
Spark was created to address the limitations to MapReduce, by doing processing in-memory, reducing the number of steps in a job, and by reusing data across multiple parallel operations. With Spark, only one-step is needed where data is read into memory, operations performed, and the results written back—resulting in a much faster execution. Spark also reuses data by using an in-memory cache to greatly speed up machine learning algorithms that repeatedly call a function on the same dataset. Data re-use is accomplished through the creation of DataFrames, an abstraction over Resilient Distributed Dataset (RDD), which is a collection of objects that is cached in memory, and reused in multiple Spark operations. This dramatically lowers the latency making Spark multiple times faster than MapReduce, especially when doing machine learning, and interactive analytics.
The Spark framework includes:

Spark Core as the foundation for the platform
Spark SQL for interactive queries
Spark Streaming for real-time analytics
Spark MLlib for machine learning
Spark GraphX for graph processing

One of the key benefits for Apache spark:
Fast
Through in-memory caching, and optimized query execution, Spark can run fast analytic queries against data of any size.

Architecture
PySpark works on master-slave model. Master refers to the "driver" and the slaves are referred as "Workers". Application creates a spark context and sends the information to Driver Program. The driver program interacts with the workers to distribute the work.

In the next post, we will start with simple examples on Pyspark Dataframes.

Introduction to Python Data Types

Ramakrishnan83 — Sun, 24 Sep 2023 13:30:28 +0000

Myself Ram started my Data Science course and completed the below concepts in last 4 weeks. I am planning to write on monthly basis what I have learned in this journey. Let's get started

In this post , I will be covering example for Python Data Types.

Variables
Variables are names given to data items that may take on one or more values during a program’s runtime.

Rules:

It cannot begin with a number.
It must be a single word.
It must consist of letters and _ symbols only.

val = 10       # val is of type int
name = "Sally" # name is now of type str
name = 10      # name is now of type int

Data Types:
Python has built-in data types that allows us to store different type of values. In Python, the data type is set when you assign a value to a variable:

Integers are whole numbers like 1,2,3,0,-1,-2,-3. They can be positive, negative or 0. Integers are immutable in Python

x = 10    # positive integer
y = -5    # negative integer 
print(type(x)) # <class 'int'>
print (x + y) # 5
print (x - y) # 15
print (x * y) # -50

Floats represent real numbers like -1.5, -0.4, 0.0, 1.25, 9.8 etc. float() adds a decimal point to the number. Floating point numbers always have at least one decimal place of precision.

x = 3.14
y = 5.0

Boolean Type Boolean represents logical values True and False. Useful for conditional testing and logic. For example: x = True y = False Boolean operators like and, or, not can be used to compose logical expressions and conditions.

String
String represent sequences of unicode characters like letters, digits, spaces etc. They are immutable in Python. String objects can be accessed through Index position . Index starts from 0.

String support operations like concatenation, slicing, length etc. Format specifiers like %s can be used for formatting for custom needs

str1 = 'Welcome'
print(str1[0]) # W
Str1 = 'Welcome to '
str2 = 'Python'
print(Str1 + str2) # Welcome to Python
Str3 = 'NewYork'
print(I Live in  %s' % name) # I Live in NewYork

List

Lists are the most basic data structure available in python that can hold multiple variables/objects together for ease of use.

2.Lists are mutable - you have the ability to change the values
You can access the values in the list through index.

3.When you specify the index range [ from : to] , the "to" index position is not included in the output.

L = ["Chemistry", "Biology", [1989, 2004], ("Oreily", "Pearson")]
L[0] # Chemistry
print(len(L)) #4
L[0:2] # "Chemistry", "Biology"

Extend -> Takes each item in the list and adds one by one
L.extend([10,20])
Append -> Takes the whole list that needs to be added and adds as single item
L.append([11,21])
Sort -> L.Sort()
Reverse Sort -> L.Sort(Reverse = True)

Sort vs Sorted

sort(): Sorts the elements of a list in place
sorted(): Assign the sorted elements to a new list and the original list is left as is.

Shadow Copy:

A=[10,11,12]
B=A -> Shadow Copying. Any update we make in A will be reflected in B
B=A[:] -> It copies all the elements from A to B. Updates in A is not reflected in B

How to remove duplicate values in List

Couple of Options
1. Convert to set.  s = set(list) . stores the unique variable in the set.
2. result = []
for i in test1:
    if i not in result:
        result.append(i)
print (result)

Tuple

Tuples are ordered collections of values that are immutable. Allows storing different data types.
We can access elements using an index but cannot modify tuples.
Tuples support operations like concatenation, slicing, length etc.

thistuple = ("apple", "cherry","banana",123,[1,2,3])
print(len(thistuple)) # 4

Range
A range represents an immutable sequence of numbers. It is commonly used to loop over sequences of numbers. Ranges are often used in for loops:

nums = range(3) # 0 to 2 
print(list(nums)) # [0, 1, 2]

for i in range(3):
   print(i)
# Output:
# 0
# 1  
# 2

create ranges with a start, stop, and step size.

nums = range(3, 8, 2) 
print(list(nums)) # [3, 5, 7]

Set
Sets are unordered collections of unique values. They support operations like membership testing, set math etc.
Sets contain unique values only. Elements can be added and removed.

thisset = {"apple", "banana", "cherry"}
thisset.add("orange")
thisset.remove("banana")
thisset.discard("banana")

Set math operations like union intersection work between sets.
set1 = {1, 2, 3}
set2 = {3, 4, 5}
print(set1 | set2) # {1, 2, 3, 4, 5}
print(set1 & set2) # {3}

Frozenset
Frozenset is an immutable variant of a Python set. Elements cannot be added or removed.
colors = frozenset(['red', 'blue', 'green'])

Dictionary

Dictionary items are ordered, changeable, and does not allow duplicates.
Dictionary items are presented in key:value pairs, and can be referred to by using the key name.

student = {
  'name': 'Ram',
  'age': 30, 
  'courses': ['CSE', 'ECE']
}
student['name'] = 'Krish' # update value
print(student['courses']) # ['CSE', 'ECE']

# Add new Keys
thisdict = {
  "brand": "Ford",
  "model": "Mustang",
  "year": 1964
}
print(thisdict)
thisdict.update({"color": "red"})
print(thisdict)

{'brand': 'Ford', 'model': 'Mustang', 'year': 1964}
{'brand': 'Ford', 'model': 'Mustang', 'year': 1964, 'color': 'red'}

x = thisdict.keys()       # List all the keys in the dictionary

# Access data in the dictionary
for key in student:
    print(key, student[key]) # print each items

# To remove items
del thisdict["model"] # Remove data by Keys
thisdict.popitem() #Removes last added item or random item
thisdict.pop("model") # Removes item by key
thisdict.clear() #Clears all the values in the dictionary
del thisdict # Deletes the dictionary

None
The None keyword is used to define a null value, or no value at all.
None is not the same as 0, False, or an empty string. None is a data type of its own (NoneType) and only None can be None.

x = None

if x:
  print("Do you think None is True?")
elif x is False:
  print ("Do you think None is False?")
else:
  print("None is not True, or False, None is just None...")

"None is not True, or False, None is just None..."

Summary of Data Types:

Conclusion
Please review the contents and post your comments. if there is areas of improvement, feel free to comment. I need to keep improving on my content quality

Enhancing Stored Procedure Efficiency: A Journey of Problem Solving and Best Practices

Ramakrishnan83 — Mon, 28 Aug 2023 13:56:16 +0000

Introduction:
In the world of database development, addressing performance challenges and designing efficient solutions is paramount. This blog delves into an insightful review of a stored procedure that revealed areas of improvement and design flaws. The scenario involves identifying a specified number of records from both a parent and a child table in a 1:Many relationship, and marking them for integration. The developer encountered performance issues with the child table, prompting the need for optimization. Let's explore the challenges, the reimagined approach, and valuable lessons learned along the way.

Problem Context:
The initial developer's attempt to optimize the stored procedure showcased gaps in understanding and design. The primary requirement was to select a specific number of records from both parent and child tables for integration purposes. However, the developer overlooked setting the incorrect identifier between the two tables. The child table had identifiers across all records instead of the required subset.

A Holistic Approach:
The key to overcoming such challenges lies in adopting a comprehensive approach that encompasses thorough questioning, data analysis, testing, and efficient coding practices.

Questioning for Clarity:

The journey starts with asking vital questions to grasp the complete scope of the requirement.

Frequency of data pulls from the source system.
Volume of records to be pulled.
Post-pull identifier changes in both tables.
Real-time vs. batch processing considerations.
Impact of inserts during read and update performance.

Data Analysis:

Don't rely solely on provided information by other teams; delve into the data patterns.

Identify key columns and uniqueness.
Scrutinize unique identifiers for parent-child linkage.
Seek alignment with business teams for data interpretation.

Test with Precision:

Effective testing minimizes errors and boosts troubleshooting efficiency.
Start with a small subset of data.
Test all use cases comprehensively.
Capture data lifecycle and patterns.
Align testing scenarios with stakeholders' expectations.
Gradually increase data volume for testing

Performance Tuning:

Optimizing the solution is an integral part of the process, and it requires careful planning and coding strategies.

Develop requirements and optimized coding concurrently to maintain alignment.
Choose the Right Approach
Select the appropriate methodology: temporary tables, CTEs, or joins.
Analyze data volume and access patterns to inform the design.
Ensure consistency in the design approach for each use case.

Solidify Logging and Monitoring:

Implement logging mechanisms from the development phase.
Enable comprehensive monitoring for future maintenance.

The Role of Solution Architects and Project Managers:
In the modern landscape, Solution Architects and Project Managers play pivotal roles in ensuring effective solutions.

Solution Architects bridge the gap between requirements and efficient design.
Mentoring new team members fosters skill development and knowledge transfer.
Writing SQL is one aspect; writing efficient SQL is the key to success. Conclusion: The journey of optimizing a stored procedure is a blend of strategic thinking, meticulous analysis, testing prowess, and efficient coding. By embracing a holistic approach, learning from challenges, and leveraging the expertise of Solution Architects and Project Managers, developers can create solutions that not only meet the immediate requirements but also set the foundation for robust, high-performance systems.