DEV Community: Rabea Abdelwahab

Transforming Business Operations with LLMs: A Path to a Production Ecosystem

Rabea Abdelwahab — Tue, 11 Jun 2024 14:06:01 +0000

All of us techies, have experimented with Large language models (LLMs) like GPT at some shape or form, and the promise is it will help businesses work smarter and more efficiently. While there's been plenty of experimentation, we're now at an exciting point where these AI applications and tools, could really become integrated into companies' core operations in a scalable, reliable way.

But to make that happen, easier said than done, we can't just drop an LLM in the middle of our systems, as some companies sell that dream. There is a need for an ecosystem, that assures the longevity and scalable growth of AI into your enterprise. Like others, I have been experimenting and finding the best places where LLMs and LLM applications can have impact, and this is part of a series talking about the components of what I think the near future AI platform would look like.

What is the next step for an AI Platform

HITL - Human in The Loop Subsystem
Feedback Subsystem
Evaluation Subsystem
Decision Executor Subsystem

HITL - Human In The Loop as a subsystem

LLMs can't always make perfect decisions, especially when first deployed, a lot of fine tuning, prompt engineering and context testing is needed. Humans in the loop, are essential, to review and approve/reject decisions the LLMs are unsure about.

Over time, as the system proves itself, more decisions can be fully automated. But that initial human oversight builds trust into your AI platform.

For any LLM app to function effectively within a business context, it requires this subsystem, that can handle decision-making with varying levels of confidence.

Low-confidence decisions should be routed to this subsystem, where a human can evaluate and approve or decline the decisions. This ensures that while LLM applications learn and improve, human oversight maintains decision accuracy and reliability.

Even high-confidence decisions, may initially require human auditing, until the business is confident in the LLM app's performance. This process builds trust and allows for the gradual transition of decision-making responsibilities.

Even high-confidence decisions, may initially require human auditing, until the business is confident in the LLM app's performance. This process builds trust and allows for the gradual transition of decision-making responsibilities.

Feedback as a Subsystem

False positives, i.e. The LLM incorrectly flags a legitimate transaction as fraud.
False negatives, i.e. The LLM fails to flag a fraudulent transaction.

Humans involved in the decision-making process, or advanced LLMs auditing a process, must provide actionable feedback, whether through simple upvotes or downvotes, textual feedback, or discrepancies between human and LLM decisions.

This feedback helps identify whether issues stem from input quality, inference problems, or contextual misunderstandings, allowing for targeted improvements.

Evaluation as a Subsystem

LLMs like any other model, drift over time, and system prompts could change in way that will effect your LLM Apps from functioning as expected. At this stage, you have a good amount of data from your LLM Apps, Feedback from your customers, HITL system and the decisions made to take it to the next level.

Evaluation is a key subsystem in our platform, because many of the outputs of our Customers, System prompts, HITL and Feedback will pour into it as parameters to weigh in the efficiency of our LLM Apps, the components of an evaluation subsystem can vary, you can start simple, by introducing metrics to measure the quality of your prompts, context and output, and then for each LLM app, this can grow in different directions, you might even have custom built models to evaluate certain scenarios and applications.

With all that feedback collected, you can't just let it sit there. It has to flow back into actually retraining and fine-tuning the LLM model itself through reinforcement learning techniques. This closes the loop so the AI keeps getting smarter.

Decision Executor as a subsystem

Once an LLM makes a decision (issuing a refund, flagging fraud, etc.), you need a component to implement that decision across your company's systems.
It's the final piece in the path to production ecosystem, the action-taker subsystem, responsible for executing decisions across your business.
Acting as a proxy to integrate seamlessly with existing systems. Whether issuing refunds, canceling orders, or reporting fraud, the action-taker ensures that decisions are implemented efficiently and accurately.

So far, I see this ecosystem being built as a separate platform that integrates with but is distinct from your core business systems. That way you get modularity and observability, as LLM apps move fast and change frequently.

Putting it all together

Integrating this platform into existing business infrastructure requires a standalone architecture that encapsulates LLM apps and their supporting systems. Communication with others can be managed through event-driven or request-response patterns, with robust observability to monitor and maintain the platform's stability.
I want to emphasize that such ecosystem can grow with time, we are still finding and figuring out how LLMs will have the biggest impact to our businesses.

Ethical Considerations

Ethical considerations also come into play, necessitating moderation agents and policies to ensure responsible usage of LLMs.
There are certainly challenges around trust, and responsible AI obstacles to overcome. But with the right supporting infrastructure, LLMs could soon graduate from experiments and SaaS to become powerful Enterprise AI assistants woven into your day-to-day operations. Exciting times!

Know your engineers to lead them to success

Rabea Abdelwahab — Wed, 08 Mar 2023 19:08:28 +0000

There are many engineers out there, and their daily routine would be something like; having coffee, checking that everything is running fine on production, hacker news, a couple of subreddits, and jumping to the biggest part of the day, where you get plugged in and build something that will make a customer or a user more efficient, fast, useful, or unfortunately get laid off (Yes, we do not mean to do that, most of the time) .

The engineers you hire at your company are of different mindsets even technically, they come from various work environments, which rules how your product or services you are offering will shape eventually, and how willing is this person to handle the tasks you give.

This is not; a bad/good engineer post and this is also not to say that an engineer can’t have all of the below, but each one of us can surely relate to one of them:

Soldiers

Steady performers, they will turn words into action, story cards into features, they have dedicated ownership of the service they are working on, and they believe they are making it better with each change they add to it. As simple as the word ownership can be, it is not found in every engineer, but the good news is; its teachable.

Gatekeepers

Their abilities shine when making a situation turn from red to green as soon as possible, with a customer first mentality, they fix now, and question later. whether it is a person or a team, this unit is the gate keeper of your product and the safety net from down times.

They code review religiously, and they participate in standardizing the way everybody contributes to the platform.

Communicators

They will question a feature, and debate how it will effect the customer/user , they will come up with better solutions and evidence to back their case. one is a gem, a product owners’ best friend, since this will be the access for understanding what is happening on the technical side, and how the team can make a better product/service.

Architects

They shine when it comes to designing a solution, taking into consideration the current state of the product and what are the next steps to implement for it to grow in the right direction and power future changes. — Be careful of over architecting.

Explorers

They are the drivers of proofs of concept, the non-stop seekers of whats out there that can make our products better, a technology, a design pattern, or as simple as a new standard in the way we work. They believe that even if its not broken, it can be much better.

I take this as a small guide to know the engineer I am talking to, to help me understand what can be their favorite part of the day and what challenges his mind.

A CEO, product manager, or any leader, will be lucky to have all these types of engineers, every one of them is a key component of a solid productive team, and what will always be common between them is the curiosity and openness to learn more everyday.

Learnings in Engineering Leadership - From Battles Won and Others Lost.

Rabea Abdelwahab — Wed, 08 Mar 2023 19:06:08 +0000

My background is mainly in startups, as an individual contributor, or a lead for a small team, I always enjoyed working with a group and I decided to move to more corporate environment, to start building that company’s engineering team and rebuilding one of their core systems.

A best case scenario for a team, is having leadership and management work together to accomplish great positive change that is measurable.

This is not one of these best case scenarios, this is my personal experience being put into a position where I needed to be both and what I found beneficial for my team.

Instill confidence in your team with your decisions and transparent communication

As a leader, you will be taking many decisions for your team, other decisions could be falling into your lap. Decisions you can control and the thought process behind them should be easily explainable to your team members, the decisions that fall from others, the “why” and how it was taken, should be a story you are able to tell if needed, if you can’t, your answer should be “I don’t know yet, but I will get us the answer”.

Even as a leader, there will be decisions that you can’t control, and even if its a battle fought and lost, your team just wants to know that you stood up for them.

Work comes in waves, an opportunity for making mistakes

While strategically, any business is trying to hit their goals continuously, having different teams in a company, means there will be calmer times than others, these little gaps are an amazing opportunity. As a tech leader, these times are the best to sharpen the tools for the next wave, whether its tackling technical debt, or preparing for a new project, or documenting an existing feature. Always have a set of technical goals to achieve in the calm times, to make the team’s life better.

Your team will also have their ideas on what to improve, this is a great time to let them run with theirs and learn from mistakes they make, they will get better technically, or improve in scoping work, or passing knowledge to others.

Specialization is OK, and most of the time its great!

You are leading a set of personalities with different interests, as much as Nirvana for any business would be that everyone is equal in every skill and interest, that is not the case. You have the potential leaders, the unit testing gate keepers, the DevOps fans, and tooling explorers, etc. Make the best out of these qualities, those team members are willing to own and run with an initiative because they are passionate about it, don’t stop a streak of productivity in the sake of making every team member know everything. You would ask, what about if I lose someone that is specialized in a certain area? I am with training people and help achieve sustainable skills in teams, is it the highest priority? I would argue that it’s not.

Make the best out of each member’s qualities, they are willing to own and run with an initiative because they are passionate about it

How to be a step ahead of the business

While transparent communication should be the de facto in any hierarchy, unfortunately, it is not the case every time, your business owners could decide that the change can be communicated in a shorter timeframe, not giving much buffer time. The way to go around this is to give the business the confidence to throw raw ideas of where they want the direction to go, making you part of the brainstorming process, because once an idea becomes a decision, you lose some valuable time to prepare for it, this is to assure that you are a step ahead, not to steer your ship depending on it, but to have a plan in your head if needed.

Moreover, communicating to your team a future direction changing scenarios, mentally, the team is also aware of the possibility of a change, you will build a team of agile forward thinkers.

Yes, there is strategic planning, and goals that are visible that we need achieve, but there is also projects and initiatives that come in between, and those are the direction changing decisions I am mentioning here.

When an idea becomes a decision, you lose some valuable time to prepare the team for it.

Engineers need to be good at communication, don’t accept less

When hiring or coaching team members, good communication should be the number one skill to make your engineers master, lone-wolves can’t be an option, and lack of communication between team members and the business will cause a lot of conflict.

The effect of a team that lacks communication skills is more layers of management to translate to the business, and project managers and facilitators to connect between team members, its a recipe for failure.

Good communication should be the number one skill to make your engineers master

All the above, if I were to summarize it; if you happen to be a leader and a manager, which is not uncommon, your mission is to learn from your environment quickly, and value time (your time and your team’s ) so you will be the effective leader they need. Make your mistakes visible to your team, and believe that there is no written in stone guidelines to leadership, no one way of becoming a good one, it’s about trial and error and improving continuously.

DynamoDB — Features that made it the database of choice— Use Case

Rabea Abdelwahab — Wed, 08 Mar 2023 18:50:40 +0000

Since DynamoDB’s release in 2012, I have used it in simple scenarios, as a persistent key-value store, but never got into the best practices and design patterns that could let me maximize the benefits of it. As most of AWS technologies, Dynamo evolved, and now it offers more, such as transactions, range-based partitioning, and more native integrations with other AWS services.

Well finally, I had a good use case, a business capability (service) that needs a NO-SQL type of storage that can scale, but as an AWS customer, I had more options to choose from to be fair. As a note, the service I am building had some unknowns; expected high spikes on certain days of the week, with extensibility in mind for the future if it grows as expected. with that in mind, the choice went as below:

DynamoDB from our stoplight chart above, will be the best candidate to move forward, The next step is to explore “the how”, using Single Table Design.

Single-Table is a concept for utilizing DynamoDB’s design under the hood to connect all data needed in one table for a business capability, making use of well-thought of access patterns, for maximum performance and efficiency. you can learn more here.

To understand more if my use-case fits the bill, I needed to understand how DynamoDB stores the data. For it to be highly available, scalable, with a single digit latency, my service has to abide by the way Dynamo is designed.

Quick Overview — Basics of Dynamo

DynamoDB keeps your data in partitions (SSD under the hood), with Partition Key being the identifier of a record, the decision in which partition it goes, happens through a hash function. Optionally you can add a Sort Key, which is helpful to introduce to your records, as DynamoDB is able to order your records by it in ascending order in the same partition, makes the trip to grab your data much shorter.

Read/Write Patterns

In our use-case, I needed a breakdown of my application’s read/write patterns, this way I am able to translate it to what will be the records in my Dynamo Table.

Customer needs to be able to view invoice details.
Customer needs to be able to create/view transactions (charges/refunds) on an invoice.
Customer needs to be able to view each transaction individually.
Customer needs to be able to create/view comments on an invoice.

From my user stories, it is obvious that invoice number is the best candidate to be the Partition Key in our table, solving for other scenarios utilizing the Sort Key for more ways to access the data as shown above.

To give an example of how a combination of partition key and a sort key brings us back the data we need, I wanted to visualize it as below depending on AWS docs.

The benefits of DynamoDB as mentioned before are many, but one of the most important ones, is the flexibility it gives Tech and Engineering to respond to the business direction change or any demand, scenarios such as:

I need to send the data to our data warehouse to analytics (Dynamo -> Streams -> RedShift) or (Dynamo -> S3 -> Data Warehouse)
I need to be able to send emails on updates/deletes/inserts (Dynamo -> Dynamo Streams -> Lambda -> SeS) or (Dynamo -> TTL Field -> Lambda -> SES)

With this single table design, we were able to implement a service that is performant, price efficient, and extensible in the future with all the integrations that Dynamo comes with.