DEV Community: Kedar Supekar

Supercharge compound AI with Amazon Bedrock and Karini AI

Kedar Supekar — Mon, 13 May 2024 07:10:25 +0000

Introduction

Generative AI has become a shared C-Level priority with many enterprises setting goals in their annual statement and numerous press releases. As Generative AI is gaining traction, there is much anticipation around their evolving model performance capabilities. However, as developers increasingly move beyond Generative AI pilots, the trend is shifting to compound systems. The SOTA results often come from compound systems incorporating multiple components rather than relying solely on standalone models. A recent study by MIT Research has observed that 60% of LLM deployments in businesses incorporate some form of retrieval-augmented generation (RAG), with 30% utilizing multi-step chains or compound systems.

Rise of Compound Systems

A Compound AI System addresses AI tasks through multiple interconnected components, including several calls to different models, retrievers, or external tools. AI models are constantly improving, with scalability seemingly limitless. However, complex, multifaceted compound systems increasingly achieve the most advanced results. Combining the models with other components allows businesses to build dynamic systems that can address complex scenarios based on user queries at runtime, reduce model hallucinations, and increase user control and trust. Enterprises can design their compound systems based on their performance goals. E.g. In some applications, even the largest model may need to be more performant or too expensive. Still, an ensemble of smaller fine-tuned models augmented with optimized search and retrieve capabilities can give the best results. Github Copilot is an excellent example of this approach. While enterprises are making a shift in compounding AI systems, the emerging challenges are how to design, optimize & operate these systems. The compound systems consist of a data processing loop, query optimization loop, and operations management capabilities, and they can be independently optimized for better performance.

*Karini AI Platform powered by AWS Gen AI for Compound AI Systems
*
AWS provides a broad set of Gen AI managed services such as Amazon Bedrock, Amazon SageMaker, and OpenSearch to build scalable generative AI applications. Amazon Bedrock is the most trusted and scalable fully managed service that offers a choice of high-performing foundation models from leading AI model providers and Amazon via a single API, along with a broad set of capabilities to build generative AI applications with security, privacy, and responsible AI.

Karini AI is a no-code Generative AI platform with a broad set of capabilities to build Compound AI systems purposefully built using AWS services to speed up production-grade application development. AWS customers can use best-of-breed capabilities to build production-grade RAG in a matter of minutes.

Data Processing Loop: Karini AI utilizes Amazon Textract and proprietary technologies to create LLM-ready data and provides built-in chunking algorithms. Customers can choose Amazon Bedrock hosted models or custom models hosted via Amazon SageMaker for chunking. Amazon OpenSearch delivers a secure and scalable vector store.

Query Optimization Loop: Karini AI employs the easy-to-use Prompt Playground to author, test, and compare the model performance of Bedrock-hosted models or custom models using Amazon SageMaker. Enterprises can leverage one of the many built-in chains, such as Q&A, summarization, classification, or Agentic workflows. Multiple ways are available to optimize retrieval using techniques such as query rewrite, query expansion, and context generation. Customers can also customize LLM-driven responses for greetings and follow-up questions.

Operations and Visibility: Karini AI provides built-in observability for tracing RAG chains and understanding low performing conversations. Copilot supports fine-grained feedback collection to gather user preferences and create instruction fine-tuning datasets. The built-in dashboards provide system performance and cost monitoring across model endpoints for Amazon Bedrock and SageMaker-hosted models. Karini AI provides enterprise connectors for significant number of data sources such as Amazon S3, Websites, Google Storage, Azure Storage, and Dropbox to unify data silos into a single vector store and also respects source system role-based access controls during serving.

Here is a quick end-2-end Karini AI Generative AI recipe powered by Amazon Bedrock models.

Conclusion:
Compound AI systems mark a significant advancement in AI technology by integrating various components to solve complex challenges that were once out of reach for traditional AI models. These systems are highly flexible, allowing for tailored responses and greater control over outputs. Karini AI’s advanced platform, coupled with Amazon Bedrock, enables the creation of sophisticated compound AI systems for any use case. By adopting these systems, businesses can enhance innovation, increase the quality and reliability of their AI solutions, and build stronger trust with their customers.

GenAIOps: Navigating AI Deployment for Enterprises

Kedar Supekar — Fri, 22 Mar 2024 05:35:10 +0000

Enterprises are adopting Generative AI to help solve many complex use cases with natural language instructions. Building a Gen AI application involves multiple components such as an LLM, data sources, vector store, prompt engineering, and RAG. GenAIOps defines operational best practices for the holistic management of DataOps (Data Operations), LLMOps (Large Language Model Life cycle management), and DevOps (Development and Operations) for building, testing, and deploying generative AI applications.

Challenges in GenAIOps automation
While pilot projects using Generative AI can start effortlessly, most enterprises need help progressing beyond this phase. According to Everest Research, a staggering 50%+ projects do not move beyond the pilots as they face hurdles due to the absence of established GenAIOps practices. Each step presents unique challenges, from connecting to enterprise data to navigating the complexities of embedding algorithms and managing query phases. These include:

Access to Enterprise Data: This involves creating connectors to various storage solutions and databases, considering different ingestion formats like files, tabular data, or API responses. Unlike traditional ETL, extraction, cleaning, masking, and chunking techniques require special attention, especially when dealing with complex structures like tables in PDFs or removing unwanted HTML tags from web crawls.
Embedding Algorithms: The constantly evolving nature of embedding algorithms (Refer MTEB Leaderboard) means it's crucial to experiment with the top models to select the most effective one for your needs. Failure to do so can adversely impact the search process.
Query Phase Management: This phase can be vulnerable to adversarial actors who may try to 'jailbreak' (refer to jailbreakchat) the prompts or overwhelm the system, impacting other users and potentially causing a cost spike.
Chunk Retrieval Process: For the chunk retrieval process, the similarity search may not retrieve adequate information or be unable to retrieve matching chunks, leading to insufficient context for comprehensive and relevant answers. Advanced retrieval chains are required to augment prompts with personalized context. (e.g., What are claims exclusions for “my” insurance plan? )
Prompt Efficiency: Open source LLMs are catching up fast with proprietary LLMs in language understanding, as evident in the open LLMs leaderboard. Hence, writing efficient prompts is very important to get a relevant and comprehensive answer. Bad prompts can either confuse the LLMs or lead to inadequate responses.
Understanding the Enterprise Domain: While Generative AI effectively addresses numerous inquisitive challenges within enterprises, Large Language Models (LLMs) often struggle to grasp the specific nuances of individual enterprise domains. LLMs are trained on publicly available datasets by crawling the world wide web, but enterprise data is behind firewalls; hence, LLMs may not understand a specific internal term used within a business, leading to an “I don't know” response or a response related to a similar term in Wikipedia dictionary leading to hallucination.
Content Safety: LLMs may spout toxic or unsafe content without proper guardrails, leading to brand reputation issues. The brand reputation concern is genuine, as reported by Chevrolet’s public AI chatbot(MSN), which produced results touting Ford's products. Imagine building these AI chatbots for children or other uninformed or vulnerable populations that may be led astray with misinformation.

User Experience: Most Gen AI systems do not focus on end-user experience. Chat GPT has set the standard for user experience, but OpenAI has control of the end-to-end pipeline, including the model. Lack of good experiences, such as streaming responses, A/B testing framework, lack of exhaustive user feedback mechanism, adequate seeding questions, or lack of follow-up questions, may diminish user engagement.

GenAIOps Best practices for enterprises
Effective GenAIOps operationalization requires skills such as AI engineers, safety and security experts, and domain experts. The diagram below provides best practices for a typical RAG workflow depicted in the challenges section. Let's dive into the best practices below,

GenAIOps best practices

1.Data Management: Utilize standard storage, database, and SaaS application interfaces to minimize bulk distributed data replication and incremental ingestion. To make it LLM-ready, utilize distributed runtimes for extraction, cleaning, masking, and chunking data. Maintain a copy of source metadata to the vector store to ensure downstream querying systems can use it for pre-filtering for more relevant answers.

Model Selection: Depending on your dataset, use the most appropriate embedding model for your use case. Try at least the top 2 embedding model techniques (Refer MTEB Leaderboard) during the experimentation phase to understand search relevance based on human-generated standard questions and answer pairs. Utilize synthetic questions generated by LLMs if you don't have human-generated question-answer pairs.
Query Phase Management: To prevent intentional or unintentional adverse behavior, use a suitable classification model to block questions and provide canned responses. Monitor adverse prompts for trends and take appropriate action to improve classification methods iteratively. To safeguard against spam attacks, enable user- and token-based throttling to limit attack vectors.
Retrieval Optimization: Use user metadata for pre-filtering to produce a narrower set for semantic search for optimal retrieval. Many vector databases, such as OpenSearch, MongoDB, and Pinecone, provide hybrid search capabilities. Depending on your source datasets, use additional retrieval chains to retrieve the entire or partial document to provide adequate context for your LLM query. For example, in an R&D chatbot, if the user asks to summarize a particular science paper, your retrieval chain must retrieve the entire science paper based on matching chunks.
Building Efficient System Prompts: Building system prompts is the most critical task to get the most optimal response. Due to the lack of a universal framework for prompts, ensure you follow the standards most appropriate based on LLM or your task (e.g., conversation, summarization, or classification). Maintain a library of best practice prompts for enterprise-specific use cases to benefit others. Including and enabling domain experts to design system prompts is essential as they are intimately familiar with datasets and expected outputs. Provide a prompt playground so domain experts can intuitively write system prompts, including examples, “Do not” rules, and expected response format. Provide a playground to quickly compare against authorized models for your enterprise. Maintain versions of the prompts so you can promote the best version to production.
Model Experimentation: Many enterprises start with SaaS model providers such as Azure OpenAI or Amazon Bedrock. Open-source models such as Llama2, Mistral, and MPT and their variants are catching up fast. Try out your application against at least 2-3 leading SOTA models to understand response time, domain understanding, and quality of response. Typical enterprise applications may not need the bells and whistles of multi-headed SaaS models, so using open-source models may be as effective as you scale out and offer a better price per performance. For the rapid testing, build an evaluation script to utilize the”LLM as a judge” approach to compare the responses' relevance, comprehensiveness, and accuracy. If the general purpose model does not provide relevant and comprehensive responses, resort to domain-specific fine-tuning or instruction fine-tuning techniques and employ the fine-tuned model in your RAG.
Content Safety: To prevent harmful, toxic responses, augment system prompts to instruct LLMs to redact harmful content from the response. Employ additional controls using other classifiers to block harmful responses entirely to ensure trust and safety. Use a standard set of questions for automated testing to ensure RAGs are regression tested to account for any changes in LLM, system prompts, or changes in data.
Enhancing User Experience: Ultimately, user experience is essential to increase engagement and attract new users. Add streaming if you are building a conversational system, provide appropriate feedback options so users can rate responses, and volunteer to provide correct responses to build the knowledge base. Provide custom instructions, seeding questions to start the conversation, and follow-up questions. Generative AI is rapidly evolving, so it is vital to continue to monitor user feedback and incorporate additional capabilities such as multi-modal (image and text).

Experts in AI engineering, cloud computing, security, data engineering, and UX engineering built Karini’s Generative AI platform. The combined expertise and platform design provide built-in GenAIOps best practices. These best practices enable enterprises to execute rapid prototyping, production deployment, and continuous monitoring. The Generative AI application's observability capabilities, evaluation, and central performance monitoring allow continuous quality and enterprise governance improvement.

Conclusion
Staying at the forefront of scientific advancement and the evolving landscape of models, Karini AI eliminates technical debt. Our no-code approach to Generative AI application deployment ensures you don’t compromise on quality or speed in bringing products to market. Karini AI is adaptable and perfect for various applications, including virtual assistants, text generation, summarization, Q&A, semantic search, classification, and image creation.

Generative AI in Enterprises

Kedar Supekar — Thu, 21 Mar 2024 04:17:45 +0000

Hype of Generative AI

Generative AI is not just a fleeting trend; it's atransformative force that's been captivating global interest. Comparable in significance to the dawn of the internet, its influence extends across various domains, altering the way we search, communicate, and leverage data. From enhancing business processes to serving as an academic guide or a tool for crafting articulate emails, its applications are vast. Developers have even begun to favor it over traditional resources for coding assistance. The term Retrieval Augmented Generation (RAG), introduced by Meta in 2020 (1), is now familiar in the corporate world. However, the deployment of such technologies at an enterprise level often encounters hurdles like task-specificity, accuracy, and the need for robust controls.

Why enterprises struggle with Industrializing Generative AI
Despite the enthusiasm, enterprises are grappling with the practicalities of adopting Generative AI.

According to survey by MLInsider,

62% of AI professionals continue to say it is difficult to execute successful AI projects. The larger the company, the more difficult it is to execute a successful AI project.
Lack of expertise, budget, and finding AI talent are the top challenges organizations are facing when it comes to executing ML programs.
Only 25% of organizations have deployed Generative AI models to production in the past year.
Of those who have deployed Generative AI models in the past year, several benefits have been realized. About half said they have seen improved customer experiences (58%) and improved efficiency (53%).

In summary, Generative AI offers massive opportunities to enterprise but due to skills, requirements for enterprise security and governance, they are still behind in the adoption curve.

Industrialization of Generative AI applications
The quest for enterprise-grade Generative AI applications is now easier, thanks to SaaS-based model APIs and packages like Langchain and Llama Index. Yet, scaling these initiatives across an enterprise remains challenging. Historical trends show that companies thrive when utilizing a centralized platform that promotes reusability and governance, a practice seen in the formation of AI and ML platform teams.

Enterprises should think about Gen AI platforms with the above four layered cake,

Infrastructure - Most companies have a primary cloud infrastructure and typically utilize Gen AI building blocks offered by the cloud.
Capabilities - These are set of foundational building block services offered by cloud native (e.g. Opensearch, Azure OpenAI) or 3rd party SAAS products(e.g. Milvus Vector search)
Reusable services - Central Gen AI teams typically have to build a RAG (Retrieval Augmented Generation), Fine Tuning or Model Hub Services that can be readily consumed with enterprise guard-rails
Use cases - Using the reusable services, use cases can be deployed and integrated with a variety of applications such as Customer support bot, summarizing customer reviews and more.

Many Data, ML and AI vendors are snapping these capabilities on top of their existing platform. ML Platforms that start with supervised labels and depend on model building & deployment aspect of MLOps, Generative AI platforms begin with a pre-trained Open source model(e.g. Llama2) or proprietary SAAS model(GPT4), focuses on capabilities to contextualize Large Language models and deploy capabilities to enable smarts in applications such as Copilots or Agents. Hence we propose a radically different approach to fulfill the promise of industrialized Gen AI that focuses on LLMOps development loop ( Connect to Model Hub -> Contextualize Model for Data -> Human Evaluation )

Introducing Generative AI Platform for all
Karini AI presents "Generative AI platform", designed to revolutionize enterprise operations by integrating proprietary data with advanced language models, effectively creating a digital co-pilot for every user. Karini simplifies the process, offering intuitive Gen AI templates that allow rapid application development. The platform offers an array of data processing tools and adheres to LLMOps practices for deploying Models, Data, and Copilots. It also provides customization options and incorporates continuous feedback mechanisms to enhance the quality of RAG implementations.

Conclusion
Karini AI accelerates experimentation, expedite market delivery, and bridge the generative AI adoption gap, enabling businesses to harness the full potential of this groundbreaking technology.

Generative AI: Strategic Approch for enterprises

Kedar Supekar — Mon, 18 Mar 2024 07:13:05 +0000

In the past twelve months, the corporate landscape has been abuzz with the potential of generative AI as a groundbreaking innovation. Despite broad recognition of its transformative power, many firms have adopted a tentative stance, cautiously navigating the implementation of this technology.

Is a cautious approach prudent, or does it inadvertently place companies at risk of lagging in a rapidly evolving technological landscape?

Recent investigations forecast the staggering benefits of generative AI, suggesting potential productivity gains in trillions of dollars per annum by 2030 if harnessed effectively.

The rewards surpass the apprehensions, provided the adoption of this technology is executed with strategic foresight. It's not about restricting generative AI but about sculpting its usage within well-defined parameters to mitigate potential challenges, including uncontrolled expenses, security breaches, compliance issues, and employee engagement.

Below, we outline ten strategic approaches for enterprises to capitalize on generative AI effectively and securely.

1. Adopt a Streamlined Approach to Business Case Development: Generative AI, an emerging technology, demands a departure from traditional business case development. Enterprises should prioritize rapid experimentation and learning to pinpoint practical technology applications swiftly. Discover and Explore

Action Points:

a. Accelerate pilot projects and proof-of-concept initiatives to cultivate knowledge and skills.
b. Discover and Explore and Test on repeat

Avoid:

a. Postponing initiatives due to the need for more absolute clarity.
b. Over-reliance on cumbersome business case development processes.

Initiate with Straightforward Applications: Before venturing into more complex applications, begin by unlocking value within existing business processes.

Action Points:

a. Concentrate on internal applications as foundational steps.
b. Prioritize data readiness for customized solutions.

Avoid:

a. Early deployment of customer-facing applications due to higher associated risks.
b. Use case lock where you’re working to solve a specific problem in one particular way.

Streamline Technology Evaluation: Most generative AI tools offer similar capabilities, rendering extensive evaluation unnecessary.

Action Points:

a. Collaborate with firms like Karini.ai for initial use cases whose platform provides immediate access to no-code tools for operationalizing Gen AI smartly.
b. Focus on trust and integration capabilities that open your LLMs, Models, and Data to all available options.

Avoid:

a. Elaborate and potentially outdated analysis of technology providers.
b. Vendor lock on a single platform that will cause crippling limitations.

Harness External Expertise: The scarcity of AI expertise necessitates partnerships for successful implementation and integration.

Action Points:

a. Assess internal expertise gaps, seek external support accordingly, and embrace a low-code/no-code platform, i.e., Karini.ai, which will keep the journey quick and safe.
b. Facilitate technology assimilation into the enterprise.

Avoid:

a. Isolated attempts at implementation.
b. Restrictive partnerships limit future technological choices.

Design a Flexible System Architecture: Architectures must be dynamic to accommodate evolving technologies, use cases, and regulatory landscapes.

Action Points:

a. Foster innovative and forward-thinking architectural design.
b. Anticipate and plan for future architectural adjustments.

Avoid:

a. Rigid architectures based on present-day technology functioning.
b. Over-reliance on existing processes for future technology support.

Implement Robust Security Protocols: Addressing generative AI's unique security challenges through custom policies and robust partnerships.

Action Points:

a. Develop tailored policies and procedures.
b. Partner with platforms that are active protectors of your data security.

Avoid:

a. Dependence on outdated security frameworks.
b. Technology adoption paralysis due to fear of risk.

Establish Innovative KPIs: New KPIs should reflect generative AI's unique value and impact on business operations.

Action Points:

a. Develop KPIs centered around long-term value creation.
b. Learn from both successes and failures.

Avoid:

a. Ignoring the learning opportunities presented by unsuccessful initiatives.

Foster Open Communication: Ensure continuous feedback and open communication channels for iterative improvement and employee engagement.

Action Points:

a. Integrate feedback mechanisms into all AI systems, like Karini uses in our CoPilot. 👍👎💬
b. Maintain transparent communication about AI's impact on the workforce.

Avoid:

a. Relying solely on conventional feedback methods.

Promote Comprehensive Learning and Development: Equip employees with the necessary skills and understanding to leverage AI tools effectively.

Action Points:

a. Provide extensive learning opportunities; Gen AI is empowering.
b. Align learning initiatives with broader change management strategies.

Avoid:

a. Limiting learning opportunities to direct users of AI tools AI needs to be democratized.

Embrace Iterative Learning: Cultivate a learning and continuous improvement culture to maximize the value derived from generative AI.

Action Points:

a. Prioritize learning and skill enhancement.
b. Engage in iterative development to refine use cases and technology applications.

Avoid:

a. Pursuing overly ambitious initial use cases.
b. Disregarding the evolving nature of AI technologies.

As enterprises stand at the cusp of this generative AI revolution, adopting a 'wait-and-see' approach may inadvertently place them at a competitive disadvantage.

The promise of generative AI far overshadows the perceived risks, demanding proactive engagement rather than cautious observation. Now is the opportune moment for enterprises to embrace generative AI, navigating its introduction with calculated measures to offset potential risks.