Trend Detection and Analysis with AiConfig

Introduction

In this blog post, you will be guided with the well planned use-case on Trend Detection, and its analysis.

The "Trend" data extraction or mining itself is part of the data mining process, where one can adapt to understand the trends within the data. When it comes to the trend analysis, we are going to consider the upward, downward or moderate trends etc. All these trends will be really useful for the business to make the informed decisions.

Background

Trend Detection and its analysis is one of the most interesting and excited topic in the area of machine learning. Until recently with the introduction of the large language models, it has been the major challenge in the area of machine learning as it required the model to have the domain knowledge or understanding.

In addition, there was a heavy NLP work involved in performing the trend data extraction. Among which the detailed analysis or a plan is a far reach.

However, the good news with the large language models are, it's really straightforward for one to accomplish. That's the beauty of the crafted prompts for trend detection, and it's analysis and formation of the detailed plan.

Hands-on

For the demonstration purposes, You will be shown the end to end execution of the Trend detection, its analysis or plan formation by making use of the AiConfig.

One of the primary reasons for using the AiConfig is its ease to use and highly configurable nature of prompts and its configurations, including the configurable LLM usages etc.

Also, the other biggest advantage that you will be seeing is with the prompt chaining. Please refer to this blog post to understand more about the Prompt Chaining

Here's the plan. At high-level, we are going to have a two-step based prompt instruction for performing the trend detection, and it's analysis with the detailed plan.

Please find the Trend Detection and Analysis/Plan AiConfig. You can copy and save it on your local.

{
  "name": "Trend Detection",
  "schema_version": "latest",
  "metadata": {
    "models": {
      "gpt-3.5-turbo": {
        "model": "gpt-3.5-turbo",
        "top_p": 1,
        "temperature": 0,
        "system_prompt": "Detect and analyze emerging trends or patterns",
        "presence_penalty": 0,
        "frequency_penalty": 0
      }
    },
    "parameters": {}
  },
  "prompts": [
    {
      "name": "trend_detection_prompt",
      "input": "{{content}}\n\n1. Identify and list any emerging trends, patterns, or significant changes in the data. These trends could be related to market trends, consumer behavior, social media trends, or any other relevant domain-specific trends.\n\n2. Describe the nature of each detected trend, including whether it is an upward or downward trend, its potential impact, and any contributing factors or drivers.\n\n3. If the data is time-series data, provide a historical perspective by highlighting how the trend has evolved over time and when it started.\n\n4. Consider the context and potential implications of each detected trend. What might these trends mean for businesses, industries, or society as a whole?\n\n5. If applicable, suggest actions or recommendations based on the identified trends. What steps can organizations or individuals take to respond to these trends?\n\nEnsure that your trend detection response is insightful, well-supported by the data, and provides a clear understanding of the detected trends.\n\n**Example Data:**\n\n{\n    \"TrendDetection\": [\n        {\n            \"Trend\": \"\",\n            \"Nature\": \"\",\n            \"Impact\": \"\",\n            \"Context\": \"\",\n            \"Recommendation\": \"\"\n        }\n    ]\n}\n\n\nDo not respond with your own suggestions or recommendations or feedback.\n",
      "metadata": {
        "model": {
          "name": "gpt-3.5-turbo",
          "settings": {}
        },
        "parameters": {
          "content": "I recently purchased this smartphone, and I'm blown away by its performance and camera quality. The battery life is excellent, and the sleek design adds a touch of elegance. Definitely worth the investment"
        },
        "remember_chat_context": true
      }
    },
    {
      "name": "detailed_analysis",
      "input": "Based on the trend detection response {{trend_detection_prompt.output}} Consider all the trends with the Significant Impact, prepare a summary with the details on why that trend is significant. Mention with the pros and cons with the recommendations as specified in the JSON output.",
      "metadata": {
        "model": {
          "name": "gpt-3.5-turbo",
          "settings": {}
        },
        "remember_chat_context": true
      }
    },
    {
      "name": "detailed_plan",
      "input": "Based on the trend detection response {{trend_detection_prompt.output}} Consider all the trends with the Significant Impact, prepare a nice detailed plan and layout on how to tackle the demanding trends. Provide some solutions with the examples.",
      "metadata": {
        "model": {
          "name": "gpt-3.5-turbo",
          "settings": {}
        },
        "remember_chat_context": true
      }
    }
  ]
}

You need to do the following steps and tweak the prompts based on your requirements. Especially the context or the content for which you wish to perform the trend analysis. There are two ways with which you could execute the AiConfig. Here is the first option of using the config and load it up to LastMileAI and perform the cell by cell execution from top to bottom.

1. Use the above Trend Detection AiConfig.
2. Login to LastMileAI
3. Navigate to the Workbooks and then do the following to upload the aiconfig file.

Let's go one step at a time to understand the over all workflow.

Following is the initial "Trend" detection CORE prompt logic, which is responsible for analysis and extracting the trends in JSON format. We shall call this cell as trend_detection_prompt. The output of this cell will be utilized further for more detailed trend analysis and planning purposes.

You will see how to configure the content of your analysis. Please use the parameterized cell for your trend detection and analysis.

Here's the output of the first cell.

Let's see how the "Trend" detailed analysis cell, where the more detailed analysis is being performed. Notice below the usage of trend_detection_prompt cell output, which is considered as an input for this purpose.

Let's see how the "Trend" detailed planning is performed. Notice below the usage of trend_detection_prompt cell output, which is considered as an input for this purpose.

If you are wondering how to programmatically execute the AiConfig, then you are on the right track. Let me walk you through the process on how to accomplish. First, please make sure to copy the above-mentioned Trend Detection Aiconfig.

Here's the code snippet for initializing the AiConfig. In here, we are going to load the config, set the parameter, save it for further analysis. Alternatively, you could also resolve it on the fly.

%pip install python-aiconfig

"""**Note - Please make sure to restart the session on Google Colab**"""

from aiconfig import AIConfigRuntime

# Load the aiconfig.
config = AIConfigRuntime.load('Trend Detection_aiconfig.json')

# Set a global parameter
config.set_parameter("content", "I recently purchased this smartphone, and I'm blown away by its performance and camera quality. The battery life is excellent, and the sleek design adds a touch of elegance. Definitely worth the investment")

config.save()

Set the OpenAI Key if you are making use of the OpenAI LLM. Here I am considering the Google Colab usages. However, it can be easily tweaked as per your needs.

import os
import openai
from google.colab import userdata

openai.api_key = userdata.get('openai_key')

Now comes the most interesting part of the AiConfig execution for "Trend" detailed analysis. Here's the code snippet.

First run the initial prompt

from aiconfig import AIConfigRuntime
config = AIConfigRuntime.load('/content/Trend Detection_aiconfig.json')
result = await config.run("trend_detection_prompt")

Next, we shall execute the trend detailed analysis prompt.

"""# **Trend Detailed Analysis**"""
result = await config.run("detailed_analysis")
# Show output
print(config.get_output_text("detailed_analysis"))

Now comes the yet another interesting part of the "Trend" detailed planning. Here's the code snippet.

"""# **Trend Deatiled Plan**"""
result = await config.run("detailed_plan")
# Show output
print(config.get_output_text("detailed_plan"))

Conclusion

Hope you have learned the art of prompt engineering for the "Trend" data mining. Also, the usage of AiConfig for configuring and executing the prompt based Apps with ease.

Comments

[tanya rai]

Amazing post!! I think LLMs supporting ML workflows is a really interesting use case. Unrelated but you should check out Nomic AI as well they do cool things with data/cluster labeling using generative AI.

[Ranjan Dailata]

Perfect, thanks a lot for your feedback @tanyarai. Here are a few more interesting ones for you :)

neuml.github.io/txtai/workflow/ - Txtai has built an amazing open-source library with a config-driven workflow.

clarifai.com/ - They got the visual workflow builder too. Clarifai is the leading full-stack AI, LLM, and computer vision production platform for modeling unstructured image, video, text, and audio data.