DEV Community: Aditya47x

Introducing VecSpark

Aditya47x — Sat, 07 Dec 2024 21:50:39 +0000

VecSpark is a library I designed to leverage the power of PySpark for handling vector embeddings at scale. It provides efficient methods for:

Calculating similarity scores using various metrics.
Chunking large text for database storage and further processing.

Built on top of PySpark, VecSpark enables seamless distributed computation of vector operations, making it ideal for large-scale NLP and analytics applications.

Features

Similarity Calculations:
- Supports multiple metrics, including Cosine, Euclidean, Manhattan, Minkowski, Pearson, Hamming, Bhattacharyya, and Chebyshev distances.
Text Chunking:
- Breaks down text into manageable chunks, ready for database insertion or vector processing.

Installation

pip install vecspark

Usage

1. Initialize PySpark

from pyspark.sql import SparkSession

spark = SparkSession.builder \
    .appName("VecSpark Application") \
    .config("spark.master", "local[*]") \
    .getOrCreate()

2. Chunk Text from Files

from vecspark import load_and_chunk_txt

# Chunk a text file into segments of 200 tokens
chunks = load_and_chunk_txt("sample.txt", max_tokens=200)

# Display chunks
for idx, chunk in enumerate(chunks):
    print(f"Chunk {idx + 1}: {chunk}")

3. Calculate Similarities

spark = SparkSession.builder.appName("TopNSimilar").getOrCreate()

# Example DataFrame
data = [
    ("record1", [1.0, 2.0, 3.0]),
    ("record2", [4.0, 2.0, 1.0]),
    ("record3", [3.0, 5.0, 6.0])
]

schema = StructType([
    StructField("data", StringType(), True),
    StructField("features", ArrayType(FloatType()), True)
])

df = spark.createDataFrame(data, schema)

df.show()

vector = [2.0,3.0,6.0]

newdf = df.withColumn('Cosine', cosine(F.col('features'),F.lit(vector))).withColumn('Dot', dot(F.col('features'),F.lit(vector))).withColumn('Eucledian', euclidean(F.col('features'),F.lit(vector))).withColumn('Manhatten', manhattan(F.col('features'),F.lit(vector))).withColumn('Minkowski', minkowski(F.col('features'),F.lit(vector),F.lit(1))).withColumn('Pearson', pearson(F.col('features'),F.lit(vector))).withColumn('Hamming', hamming(F.col('features'),F.lit(vector))).withColumn('Bhattacharya', bhattacharyya(F.col('features'),F.lit(vector))).withColumn('Chebyshev', chebyshev(F.col('features'),F.lit(vector)))

newdf.show()

Output:

Spark version: 3.5.3
+-------+---------------+
|   data|       features|
+-------+---------------+
|record1|[1.0, 2.0, 3.0]|
|record2|[4.0, 2.0, 1.0]|
|record3|[3.0, 5.0, 6.0]|
+-------+---------------+

+-------+---------------+----------+----+---------+---------+---------+----------+-------+------------+---------+
|   data|       features|    Cosine| Dot|Eucledian|Manhatten|Minkowski|   Pearson|Hamming|Bhattacharya|Chebyshev|
+-------+---------------+----------+----+---------+---------+---------+----------+-------+------------+---------+
|record1|[1.0, 2.0, 3.0]| 0.9926846|26.0|3.3166249|      5.0|      5.0|0.96076894|    3.0|   -2.092647|      3.0|
|record2|[4.0, 2.0, 1.0]|0.62347966|20.0| 5.477226|      8.0|      8.0|-0.8910421|    3.0|  -2.0447733|      5.0|
|record3|[3.0, 5.0, 6.0]| 0.9732576|57.0| 2.236068|      3.0|      3.0| 0.8910421|    2.0|  -2.5114248|      2.0|
+-------+---------------+----------+----+---------+---------+---------+----------+-------+------------+---------+

Example

Using in a GenAi fucntion with ollama

I am repeating the code from Ollama All-MiniLm Docs with just using PySpark and Vecspark instead of ChromaDB

import ollama
documents = [
  "Llamas are members of the camelid family meaning they're pretty closely related to vicuñas and camels",
  "Llamas were first domesticated and used as pack animals 4,000 to 5,000 years ago in the Peruvian highlands",
  "Llamas can grow as much as 6 feet tall though the average llama between 5 feet 6 inches and 5 feet 9 inches tall",
  "Llamas weigh between 280 and 450 pounds and can carry 25 to 30 percent of their body weight",
  "Llamas are vegetarians and have very efficient digestive systems",
  "Llamas live to be about 20 years old, though some only live for 15 years and others live to be 30 years old",
]

def embedding(doc):
    return ollama.embeddings(model="all-minilm",prompt=doc)["embedding"]

rows = [
    Row(document=[doc], embedding=embedding(doc)) for doc in documents
]

schema = StructType([
    StructField("document", StringType(), nullable=False),
    StructField("embedding", ArrayType(FloatType()), nullable=False)
])

sc = SparkSession.builder.appName("Example").config("spark.driver.memory", "4g").config("spark.executor.memory", "4g").getOrCreate()

df = sc.createDataFrame(rows, schema=schema)

## Data Stored in Vector DB
df.show()

prompt = "What animals are llamas related to?"

response = ollama.embeddings(
  prompt=prompt,
  model="all-minilm"
)["embedding"]


finalDB = df.withColumn("Cosine", cosine(F.col('embedding'), F.lit(response)))

finalDB.show()

data = finalDB.orderBy(F.col("Cosine").desc()).select("document").first()

print(data)

output = ollama.generate(
  model="nemotron-mini",
  prompt=f"Using this data: {data}. Respond to this prompt: {prompt}"
)

print(output['response'])

Output:

+--------------------+--------------------+
|            document|           embedding|
+--------------------+--------------------+
|[Llamas are membe...|[0.048099015, 0.1...|
|[Llamas were firs...|[0.17195219, 0.16...|
|[Llamas can grow ...|[0.009301755, 0.1...|
|[Llamas weigh bet...|[0.24956073, 0.17...|
|[Llamas are veget...|[0.3719531, -0.03...|
|[Llamas live to b...|[0.11673717, 0.16...|
+--------------------+--------------------+

+--------------------+--------------------+----------+
|            document|           embedding|    Cosine|
+--------------------+--------------------+----------+
|[Llamas are membe...|[0.048099015, 0.1...| 0.7713934|
|[Llamas were firs...|[0.17195219, 0.16...| 0.6688401|
|[Llamas can grow ...|[0.009301755, 0.1...| 0.5910876|
|[Llamas weigh bet...|[0.24956073, 0.17...|  0.654605|
|[Llamas are veget...|[0.3719531, -0.03...| 0.7110807|
|[Llamas live to b...|[0.11673717, 0.16...|0.66238075|
+--------------------+--------------------+----------+

Row(document="[Llamas are members of the camelid family meaning they're pretty closely related to vicuñas and camels]")
 Llamas are closely related to vicuñas and camels, as they belong to the same family.

You can find the code on https://github.com/Adi-ayro/vecspark

ChromaDB for the SQL Mind

Aditya47x — Wed, 04 Dec 2024 20:47:02 +0000

Hello, Chroma DB is a vector database which is useful for working with GenAI applications. In this article I will explore how can we run queries on Chroma DB by looking at similar relations in MySQL.

Schema

Unlike SQL you cannot define your own schema. In Chroma You get fixed Columns each with its own purpose:

import chromadb

#setiing up the client
client = chromadb.Client() 
collection = client.create_collection(name="name")

collection.add(
    documents = ["str1","str2","str3",...]
    ids = [1,2,3,....]
    metadatas=[{"chapter": "3", "verse": "16"},{"chapter":"3", "verse":"5"}, ..]           
    embeddings = [[1,2,3], [3,4,5], [5,6,7]]
)

Ids: They are unique ids. Note that you need to supply them yourself unlike sql there is no auto increment
Documents: It is used to insert the text data that is used to generate the embeddings. You can supply the text and it will automatically create the embeddings. or you can just supply embeddings directly and store text else where.
Embeddings: They are in my opinion the most important part of the database as they are used to perform similarity search.
Metadata: this is used to associate any additional data you might want to add to your database for any extra context.

Now that the basics of a collection are clear lets move on to CRUD operations and the we will see how we can query the database.

CRUD Operations

Note: Collections are like Tables in Chroma

To create a collection we can use create_collection() and perform our operations as needed but if the collection is already made and we need to refrence it again we have to use get_collection() or else we will get a error.

Create Table tablename

#Create a collection
collection = client.create_collection(name="name")

#If a collection is already made and you need to use it again the use
collection = client.get_collection(name="name")

Insert into tablename
Values(... , ..., ...)

collection.add(
    ids = [1]
    documents = ["some text"]
    metadatas = [{"key":"value"}]
    embeddings = [[1,2,3]]
)

To Update the inserted data or to Delete the data we can use following commands

collection.update(
    ids = [2]
    documents = ["some text"]
    metadatas = [{"key":"value"}]
    embeddings = [[1,2,3]]            
)

# If the id does not exist update will do nothing. to add data if id does not exist use
collection.upsert(
    ids = [2]
    documents = ["some text"]
    metadatas = [{"key":"value"}]
    embeddings = [[1,2,3]]            
)

# To delete data use delete and refrence the document or id or the feild
collection.delete(
    documents = ["some text"]         
)

# Or you can delete from a bunch of ids using where that will apply filter on metadata
collection.delete(
    ids=["id1", "id2", "id3",...],
    where={"chapter": "20"}
)

Queries

Now we will look at how certain queries look like

Select * from tablename

Select * from tablename limit value

Select Documents, Metadata from tablename

collection.get()

collection.get(limit = val)

collection.get(include = ["documents","metadata"])

While get() is there for fetching a large set of tables for more advanced queries you need to use query method

Select A,B from table
limit val

collection.query(
    n_results = val #limit
    includes = [A,B] 
)

Now we get 3 possible ways to filter the data: Similarity Search (what vector databases are mainly used for), Metadata filters and Document filters

Similarity Search

We can search based on text or embeddings and get the most similar outputs

collection.query(query_texts=["string"])

collection.query(query_embeddings=[[1,2,3]])

In ChromaDB, where and where_document parameters are used to filter results during a query. These filters allow you to refine your similarity search based on metadata or specific document content.

Filter by Metadata

The where parameter lets you filter documents based on their associated metadata. Metadata is usually a dictionary of key-value pairs you provide during document insertion.

Filter documents by metadata like category, author, or date.

# Insert documents with metadata
collection.add(
    documents=["Document about AI", "Another document on AI", "General science content"],
    metadatas=[
        {"category": "AI", "author": "John"},
        {"category": "AI", "author": "Doe"},
        {"category": "Science", "author": "Alice"},
    ],
    ids=["doc1", "doc2", "doc3"]
)

# Query with a metadata filter
results = collection.query(
    query_texts=["Artificial Intelligence"],
    n_results=5,
    where={"category": "AI"}  # Only retrieve documents with category = "AI"
)

# Output
print("Filtered documents:", results['documents'])

# You can filter using multiple conditions:  
where={"category": "AI", "author": "John"}

# Supports operators like `$gt`, `$lt`, `$in`, etc. For example:
where={"date": {"$gt": "2024-01-01"}}

Filter by Document Content

The where_document parameter allows filtering directly based on the content of documents.

Retrieve only documents containing specific keywords.

# Insert documents
collection.add(
    documents=["AI is transforming the world", "AI and machine learning", "Physics is fascinating"],
    ids=["doc1", "doc2", "doc3"]
)

# Query with a document content filter
results = collection.query(
    query_texts=["What is AI?"],
    n_results=5,
    where_document={"$contains": "AI"}  # Only retrieve documents containing "AI"
)

# Output
print("Filtered documents:", results['documents'])

Key Notes:

Use operators like $contains, $startsWith, or $endsWith.
- $contains: Match documents containing a substring.
- $startsWith: Match documents starting with a substring.
- $endsWith: Match documents ending with a substring.
For example:
```
where_document={"$startsWith": "AI"}
```

Common Use Cases:

We can combine all three filters like this:

Search Within a Specific Category:

collection.query(query_texts=["Machine learning"], n_results=5, where={"category": "ML"})

Search Documents Containing a Specific Term:

collection.query(query_texts=["Physics"], n_results=5, where_document={"$contains": "gravity"})

Combine Metadata and Document Content Filters:

collection.query(
    query_texts=["AI"], 
    n_results=5, 
    where={"author": "John"}, 
    where_document={"$startsWith": "AI"}
)

These filters enhance the precision of your similarity searches, making ChromaDB a powerful tool for targeted document retrieval.

Conclusion

I wrote this article because I felt that the document leaves much to desire when trying to make my own program, I hope this helps!

Thanks for reading if you liked the article pls like and share. Also if you are new to software architecture and would like to know more I am starting a group based cohort where I will personally work with you and a small group to teach you everything about Software Architecture and Design principals. You can fill the form below if you are interested . https://forms.gle/SUAxrzRyvbnV8uCGA

You probably don't need to build large scale microservices. Here is what you can do instead

Aditya47x — Wed, 20 Nov 2024 15:41:46 +0000

So, you are done with building your small projects that are only used by you (or maybe a few friends). Now you want to go big!! you want to make something for the public. Or you want to start your own startup, your first online business and you really don't want to screw this up you want your app to have zero downtime you want it run perfectly to not crash and you are right. There is nothing more problematic than having your servers crash at production and not take the load of your incoming users

So you explore all the options. You look at all the big websites ( Amazon, Netflix, Uber ) and they all use microservices architecture. Netflix very famously uses 1000s of microservices to run its website. Ok so maybe that's what you need to handle your massive chunks of users... Right... Right???

Well No Not Really.. The reason I am saying this is because microservices are harder to make, test, deploy and monitor. They require more servers and more compute and storage at the beginning stage that what would be taken by your single server monolith. But monoliths may become a problem later on so what should I do? well here are some key points and steps that you should consider before you start building you app.

By the way, if you are very new to this and don't even know what are monoliths and microservices. I would recommend you should check out this article https://www.freecodecamp.org/news/microservices-vs-monoliths-explained/

Pro Tip: Don't Optimize Prematurely

If tomorrow isn't the due date today is not the do date. In software you should avoid premature optimizations because it will slow you down be a technical debt and it might not even work because as you grow you will need to change your directions and goals multiple times. the features that you think works now might not be so needed in the future. you cannot anticipate what your users might need so don't try to bother just build deploy and then see the reaction. As the silicon valley motto says Move Fast and Break Things. it might work better. Also just remember that microservices are peak optimizations they are used by companies which have millions of users and billions if not trillions of api calls per month. They also have teams of tens of thousands of engineers working and contributing continuously and lets be honest that might be your dream but that is not your day. It will happen but it is not happening tomorrow unless you really made something that just reshaped humanity forever (if that happens congrats now people will gladly wait a few weeks till you sort this situation out) but yeah this is the entire point of microservices

Pro Tip 2: Pls understand what microservices really represent

As I said before microservices are peak optimizations they are part of what we call distributed system design. It is built for scale but not just any scale Ultra Massive Scale. Billions and Trillions of API Request per month (Yes Trillions that is 1,000,000,000,000) and maybe even 10 20 or even 1000 times more than that. That is not something you need from very start no matter how fast you are growing this scale will give you at least a couple of years to manage and make decisions around your website. Big companies need to think at that scale but that doesn't mean that you should. (I do wish all of you folks reading that you grow that big but that is not happening in a single day so don't worry about it before you launch or are at a nascent stage )

Now that we have gotten that out in the clear here is a step by step on how you can optimize

Now I will be mentioning a lot of GCP services but almost all cloud companies have similar offerings just with a different name.
Note: For the sake of simplicity I don't add the role of security and monitoring services from the start but In a real world app (which you are planning to make) they are just as important as the main app itself. That is also the reason that I suggest you to start small so that you can give ample time to that as well

Step 0 :MVP

For your minimal viable product I will suggest that you keep it very simple. You can use just a single Compute Engine Instance for your entire stack. You can also use App Engine as well with a containerized app. This is ideal for around 1000 to 2000 users. This reduces costs (Might even be zero with free tier on App Engine) and as you can probably guess is very very easy to setup. It allows you to make it quick and release and test it quick.
But Please note the reason that I call this is a step zero is because even I don't think this is reasonable as if you have a bigger audience or as soon as you make a more formal launch and try to grow with it it will collapse (Here once again App Engine is more scalable it can grow but latency and costs can prove to be a issue). It is something that should only be used for a beta launch.

Step 1: 3-Tier Architecture with CDN

Now, This is where we start getting into a more real launch. This is the standard 3 Tier Architecture. Database and Servers are decoupled. For Servers I will suggest a bigger Compute engine Instance. If you are building containerized apps we are talking about GKE (Google Kubernetes Engine) or App Engine and a database of your choice (Relational or Non Relational is choice that you need to make based on your app). Also every static file (Images, Videos, PDFs, CSS, JS, even some HTML files) should only be served via a CDN this reduces the load on your main servers and also makes your app much faster.

Now for Compute engine it is best to use a Managed Instance Group they will manage autoscaling for you
Similarly this can be achieved with GKE. Now for a GKE vs App Engine the thing is this is something that you need to determine. If you think App engine saves you cost or it might even cost a bit more but that is more important than trying on setting up GKE it mostly matters in that sense.

This design works well for 10000 to 50000 users and is resilient enough where it should not break at any amount of users. The reason that we move on to higher forms of Architecture is due to the costs or latency advantages not purely because your app will break

Step 2: Add Load Balancers

Now that you have so many users. I think its time to be in more than one place. This is the start of distributed design. We will add a Global HTTP Load Balancer and our Infrastructure and Database systems are present in more than one cloud region. This provides your Users with a faster and experience.

The choice of regions have to be based on where your users are ideally you would like to place it closest to where you maximum users are.
Note: I have removed App engine from the choices of Infrastructure System because honestly I don't believe it is very cost optimized at this stage. You could use either GCE or GKE based on your app design.
At this point if all pieces are setup correctly you should easily be able to handle even millions of customers with right no of zones, clusters and machines. Now moving further its all about how can we optimize costs, add new features and make it faster.

Step 3: Adding Caching

As I said before, Now it all about increasing efficiency and going on that point you should setup a cache.
A cache server allows faster retrieval and quicker response times, especially when dealing with high volumes of traffic or in situations where the primary source has limited resources or capacity.
GCP's Memory store is a managed service of Redis.
You can also use bigger databases like Spanner or Big Table if you think you database systems can benifit from this.

Step 4: Adding Cloud Run for Small Steps

Now, Your app is already very big maybe you have some services that don't need that much compute or use time to warrant a full time running compute instance. Could be just a image detection service or a few batch jobs. This is where you should add cloud run and just let them run as separate services rather than trying to fit them into your regular app servers.
Also for the databases you will most likely need PB scale DB's like Cloud Spanner and Cloud Big Table.
While not mentioned here you will also most likely need to build ETL and Big Data analysis pipelines to analyze customer data. Depending on your app use case you will also need stricter security and compliance requirements (Data Retention Laws, Data Security Laws, Audi log Sinks. Disaster Recovery plans etc.) So while the diagram itself does not show much changes there is a lot more intricacies than just having a running app that doesn't crash.

Step 5: At this point. You might as well try microservices.

Yeah!! Now you are big. You most likely have multiple users, performing multiple interactions per second. We are looking at multiple api endpoints, backwards compatibility, separate responses for Mobile and Web Apps maybe even a few enterprise customer services. Essentially your servers are getting anywhere from more than 100000 API requests per second to potentially tens of millions. At this scale (while exceptions do exist) there tends to be a lot of people involved in planning , design and development of whatever it is that you do. So now while monoliths can scale to this point (remember Google, Wikipedia, Stack Overflow are monoliths to this day). The amount of complexity it requires to manage makes sense from a cost and management standpoint to decouple your services down to a bunch of microservices.

Conclusion

Well here are my opinions on why you most likely don't need to start building your app as microservices. I outlined a way on how you can start small and gradually build your app as your demands scale. I think it is a better approach because it allows developers to move fast from public feedback and don't have to hold back, I recommend this approach because as a app starts to see public feedback it tends to change a lot in terms of directions and features So it is better from a design perspective to keep it simple and small from the get go.

Self Promotion:

Pro Tip 3: Don't Optimize Prematurely but it doesn't hurt to think about it

While I absolutely don't recommend that you spend your time making all these complex design choices when your app has like 100 users. Here are some tips that you should try to incorporate from the start

Have a Clean Documentation

Document Everything, You will really don't know when you need it. It makes onboarding new people easier and also it makes it easier for you to get back at your code after working for a few months (You will think that you know everything about your code you wrote it, but trust me few moths down the project you will not remember what you did)

Set some standards for the code you write

Standardize Language Versions, Databases, Libraries and everything that you use and do down to the variable names. You don't want to run into the It works on my machine problem. Your software that you use should have fixed versions. Code should have a standardized naming conventions.

Embrace Modularity

Make your code modular, this may sound simple but a lot of people don't do that which later causes problems in collaboration working. Also it will make it easier to decouple into cloud runs or microservices when you actually decide to make the transition.

DEV Community: Aditya47x

Introducing VecSpark

Features

Installation

Usage

1. Initialize PySpark

2. Chunk Text from Files

3. Calculate Similarities

Example

Using in a GenAi fucntion with ollama

ChromaDB for the SQL Mind

Schema

CRUD Operations

Queries

Similarity Search

Filter by Metadata

Filter by Document Content

Key Notes:

Common Use Cases:

Conclusion

You probably don't need to build large scale microservices. Here is what you can do instead

By the way, if you are very new to this and don't even know what are monoliths and microservices. I would recommend you should check out this article https://www.freecodecamp.org/news/microservices-vs-monoliths-explained/

Pro Tip: Don't Optimize Prematurely

Pro Tip 2: Pls understand what microservices really represent

Now that we have gotten that out in the clear here is a step by step on how you can optimize

Step 0 :MVP

Step 1: 3-Tier Architecture with CDN

Step 2: Add Load Balancers

Step 3: Adding Caching

Step 4: Adding Cloud Run for Small Steps

Step 5: At this point. You might as well try microservices.

Conclusion

Self Promotion:

Pro Tip 3: Don't Optimize Prematurely but it doesn't hurt to think about it

Have a Clean Documentation

Set some standards for the code you write

Embrace Modularity

This is it here was all my tips and techniques. Thanks a lot for reading