DEV Community: shashank agarwal

🚀 Building the Enterprise-Grade AI Evaluation Platform the Industry Needs

shashank agarwal — Mon, 14 Jul 2025 09:15:03 +0000

# The dream: Evaluate any AI model with 3 lines of code
from novaeval import Evaluator
evaluator = Evaluator.from_config("evaluation.yaml")
results = evaluator.run()

The Technical Challenge

As AI models proliferate, developers face a critical problem: How do you systematically compare GPT-4 vs Claude vs Bedrock for your specific use case?

Most teams resort to manual testing or build custom evaluation scripts that break every time APIs change. We needed something better.

Enter NovaEval

NovaEval is an open source, enterprise-grade evaluation framework that standardizes AI model comparison across providers.

Technical Architecture:

Unified Model Interface: Abstract away provider differences
Pluggable Scorers: Accuracy, semantic similarity, custom metrics
Dataset Integration: MMLU, HuggingFace, custom datasets
Production Ready: Docker, Kubernetes, CI/CD integration

Code Example:

# evaluation.yaml
dataset:
  type: "mmlu"
  subset: "abstract_algebra"
  num_samples: 500

models:
  - type: "openai"
    model_name: "gpt-4"
  - type: "anthropic"
    model_name: "claude-3-opus"

scorers:
  - type: "accuracy"
  - type: "semantic_similarity"

CLI Power:

# Quick evaluation
novaeval quick -d mmlu -m gpt-4 -s accuracy

# Production evaluation
novaeval run production-config.yaml

# List available options
novaeval list-models

Contribution Opportunities

We're actively seeking contributors in:

🧪 Testing: Improve our 62% test coverage
📊 Metrics: Build RAG and agent evaluation frameworks
🔧 Integrations: Add new model providers and datasets
📚 Documentation: Create tutorials and examples

Getting Started:

pip install novaeval
Check out: https://github.com/Noveum/NovaEval
Look for good first issue labels
Join our GitHub Discussions

Discussion Questions:

What evaluation metrics matter most for your AI applications?
Which model providers would you like to see supported?
What's your current AI evaluation workflow?

Announcing the MagicAPI AI Gateway: The Fastest AI Proxy for Developers!

shashank agarwal — Thu, 07 Nov 2024 13:40:39 +0000

🚀 Announcing MagicAPI AI Gateway: The Fastest AI Proxy for Developers!

We are thrilled to introduce the MagicAPI AI Gateway—the world's fastest AI Gateway proxy, meticulously crafted in Rust for unparalleled performance. Whether you're a developer seeking reliable AI API access or a tech enthusiast exploring the latest in AI infrastructure, MagicAPI is your go-to solution.

Why MagicAPI AI Gateway?

🌟 Blazing Fast Performance

Built with Rust, MagicAPI leverages zero-cost abstractions to deliver lightning-fast request handling. Experience low latency and high throughput like never before.

🔄 Unified API Interface

Seamlessly route requests to multiple AI providers, including OpenAI and GROQ, through a single, unified interface. Simplify your AI integrations with smart, provider-specific request routing.

📡 Real-time Streaming Support

Enjoy real-time streaming capabilities with minimal overhead, ensuring smooth and efficient data flow for your applications.

🔍 Robust Features

Built-in Health Checking: Keep your services running smoothly with automatic health checks.
Configurable CORS: Tailor cross-origin resource sharing to fit your security needs.
Efficient Proxying: Optimize request and response handling for maximum efficiency.

Get Started Quickly

Installation

Install via Cargo:

cargo install magicapi-ai-gateway

Or use our pre-built Docker image:

docker pull magicapi1/magicapi-ai-gateway:latest
docker run -p 3000:3000 -e RUST_LOG=info magicapi1/magicapi-ai-gateway:latest

Docker Compose

Choose between building from source or using the prebuilt image with Docker Compose for easy deployment.

Built for Developers

(https://magicapi.com/) AI Gateway is designed with developers in mind, offering:

Asynchronous I/O with Tokio for optimal resource utilization.
Connection Pooling via Reqwest for efficient HTTP connections.
Memory-efficient request/response proxying.

Join Our Community

We welcome contributions and feedback! Check out our GitHub repository to get involved. Our community is active and ready to support you.

Explore More

GitHub: MagicAPI AI Gateway
Cargo: MagicAPI on crates.io

License

MagicAPI AI Gateway is dual-licensed under the MIT License and the Apache License (Version 2.0), giving you the flexibility to choose the license that best suits your needs.

We can't wait to see what you'll build with MagicAPI AI Gateway. Dive into the future of AI integration today!

Most affordable Whisper API

shashank agarwal — Tue, 22 Oct 2024 07:09:17 +0000

🎙️ Whisper Speech-to-Text API by NeevCloud

Try it here -> https://api.market/store/neevcloud/whisper

The Whisper Speech-to-Text API is the most affordable speech-to-text solution available online. With up to 6x cheaper pricing compared to OpenAI’s Whisper model and 3x more affordable than other providers, it delivers exceptional value while being super scalable. Whether you need to transcribe individual audio files or process vast amounts of multimedia data, this API can handle it all with high accuracy and performance.

🔑 Key Features:

💸 Most Affordable Solution: Up to 6x cheaper than OpenAI’s Whisper and 3x cheaper than other providers.
📝 High-Accuracy Transcriptions: Leverages advanced machine learning to provide accurate speech-to-text conversion.
🗣️ Speaker Diarization: Identify and separate different speakers in an audio file.
🌍 Multilingual Support: Transcribe audio in multiple languages or translate spoken language into English.
⚡ Real-Time or Async Processing: Choose between real-time or asynchronous processing for flexibility.
📂 Flexible Input: Submit audio files via URLs or upload files directly.

💡 About Whisper:

Whisper is a state-of-the-art automatic speech recognition (ASR) system trained on 680,000 hours of multilingual data from the web. It supports transcribing audio in multiple languages and translating spoken content into English. Designed by OpenAI, Whisper is particularly useful in scenarios such as:

🎧 Transcription Services: Easily transcribe meetings, interviews, lectures, and other spoken content.
📝 Subtitling and Closed Captioning: Generate subtitles for videos, improving accessibility for deaf or hard-of-hearing viewers.
🌐 Language Learning and Translation: Use Whisper for language learning, pronunciation practice, and cross-lingual communication.
📱 Accessibility Tools: Integrate Whisper into assistive technologies for those with speech impairments or disabilities.
🔍 Content Searchability: Transcribe multimedia content into text to allow efficient search and analysis.
🎙️ Voice-Controlled Applications: Use Whisper to build voice-driven applications and interact naturally with technology.
📞 Customer Support Automation: Transcribe and analyze calls in real-time to automate customer support.
🎙️ Podcasting and Journalism: Transcribe interviews and podcasts quickly for faster content creation.

💰 API Pricing:

The Whisper Speech-to-Text API charges per second of audio processed, with 1 second = 1 API unit. This pricing is highly competitive for developers working with audio and video transcription at scale.

Audio Length	Conversion to Seconds	API Units Consumed
12.9 minutes of audio	12.9 * 60 = 774 seconds	774 API units
5 minutes of audio	5 * 60 = 300 seconds	300 API units
30 seconds of audio	30 seconds	30 API units

Important:

✅ No charges for checking task status.
❌ No charges if the task fails (only charged for 200 HTTP responses).

📤 API Endpoints:

1. Process Audio via URL 🌐

Submit an audio file by URL for transcription or other tasks like speaker diarization.

Endpoint: POST /neevcloud/whisper/process_url/

Parameter	Description	Example
`url`	The URL of the audio file.	`"https://example.com/audio.wav"`
`task`	The task to perform: "transcribe", "translate".	`"transcribe"`
`language`	Language of the audio (or set to "None" for no detection).	`"None"`
`batch_size`	Size of the audio chunks to process (in seconds).	`64`
`timestamp`	Whether to generate timestamps ("none" or "chunk").	`"chunk"`
`diarise_audio`	Whether to separate speakers in the audio.	`false`
`is_async`	Choose asynchronous processing (true or false).	`false`

Request Example:

curl -X 'POST' \
  'https://api.magicapi.dev/api/v1/neevcloud/whisper/process_url/' \
  -H 'accept: application/json' \
  -H 'x-magicapi-key: <your-api-key>' \
  -H 'Content-Type: application/json' \
  -d '{
        "url": "https://example.com/audio.wav",
        "task": "transcribe",
        "language": "None",
        "batch_size": 64,
        "timestamp": "chunk",
        "diarise_audio": false,
        "is_async": false
    }'

2. Process Audio via File Upload 📂

Upload an audio file directly for transcription and other tasks.

Endpoint: POST /neevcloud/whisper/process_file/

Parameter	Description	Example
`file`	The audio file to upload.	`@your-audio-file.mp3`
`task`	The task to perform: "transcribe", "translate".	`"transcribe"`
`language`	The language of the audio file.	`"None"`
`batch_size`	Size of the audio chunks to process (in seconds).	`64`
`timestamp`	Generate timestamps for the chunks ("none" or "chunk").	`"chunk"`
`diarise_audio`	Whether to separate speakers in the audio.	`false`
`is_async`	Choose asynchronous processing (true or false).	`false`

Request Example:

curl -X 'POST' \
  'https://api.magicapi.dev/api/v1/neevcloud/whisper/process_file/' \
  -H 'accept: application/json' \
  -H 'x-magicapi-key: <your-api-key>' \
  -H 'Content-Type: multipart/form-data' \
  -F 'file=@your-audio-file.mp3' \
  -F 'task=transcribe' \
  -F 'language=None' \
  -F 'batch_size=64' \
  -F 'timestamp=chunk' \
  -F 'diarise_audio=false' \
  -F 'is_async=false'

3. Check Task Status 📊

Retrieve the status and result of a specific transcription task using its task ID.

Endpoint: GET /neevcloud/whisper/status/{task_id}

Parameter	Description	Example
`task_id`	The unique ID for the task you wish to check.	`"ad371472-e6e9-4ecf-b20f-10884230a09e"`

Request Example:

curl -X 'GET' \
  'https://api.magicapi.dev/api/v1/neevcloud/whisper/status/ad371472-e6e9-4ecf-b20f-10884230a09e' \
  -H 'accept: application/json' \
  -H 'x-magicapi-key: <your-api-key>'

Response Example:

{
  "status": "completed",
  "output": {
    "text": "Many people think that the best way to escape war is to dwell upon its horrors...",
    "chunks": [
      {
        "timestamp": [0, 7],
        "text": "Many people think that the best way to escape war..."
      },
      {
        "timestamp": [9.84, 16.5],
        "text": "Them vividly upon the minds of the younger generation..."
      }
    ]
  },
  "task_id": "ad371472-e6e9-4ecf-b20f-10884230a09e",
  "audio_duration_seconds": 30.589
}

📢 Error Handling:

In the event of an error, the API will return standard HTTP error codes:

Error Code	Description
400 Bad Request	Invalid parameters or missing fields.
401 Unauthorized	Invalid or missing API key.
500 Server Error	An issue occurred on the server while processing.

Important: No API units will be charged in case of task failures or errors.

Try it here -> https://api.market/store/neevcloud/whisper

Sell APIs? I made something that can help you!

shashank agarwal — Wed, 11 Oct 2023 21:27:34 +0000

Do you sell APIs or AI models? Or want to sell APIs and want to create an API store?

Well, I had some APIs from a failed startup that I put on RapidAPI. It actually worked well and I was making over $2K/month. So naturally, I wanted to grow my business.
Well, for that I needed a proper website. And a self-service platform that I could control. I know how to do sales and run ads.

I started making a platform for my APIs and soon realized it was too much work and was sort of overkill for just one API seller i.e.me.

So converted this to a platform that anyone can use to sell APIs. It just takes 10 minutes to create an API store.
You can sign up here -> https://MagicAPI.app.

If you want to create an API Store, just let me know. I can onboard you.

Redis is redefining itself

shashank agarwal — Fri, 10 Jun 2022 05:16:45 +0000

Redis is an amazing in memory #database. But that's all it used to be, a really amazing distributed hash table to get or set key value pairs. But recently while consulting for a startup I came across Redis Enterprise and it totally blew my mind away.

I have never seen a big database company totally revamp their offerings. For example, MongoDb is still trying to find a way to monetize their DB by changing the license to Server Side Public License. ElasticSearch did something similar and that's why Amazon created OpenSearch(a fork of ElasticSearch).

*What all can Redis Enterprise do?
*

They introduced bloom filter and cuckoo filter as first class citizens. Bloom filter is probably one of the most underrated yet one of the most widely used algorithm I know of. Google Chrome uses Bloom filter to check if a URL exists in their malicious URLs Dataset.

Besides that, Redis also introduced a new JSON module that let's users save any JSON objects and then allows the users to run queries on any fields.

Redis Enterprise has a lot more modules such as TimeSeries, Streams, AI and more.

The entire ecosystem is really new but it totally has the capability to replace a whole suite of document databases, #caches and indexed search systems.

The only "con" would be the pricing. It's a bit expensive and also it's not #opensource but has Redis source available License(RSAL).

I am pretty sure some companies are already working on an open sourced alternative to Redis Enterprise and various modules.

It's just amazing to see how databases are evolving. I would have never imagined someone using Redis as their main database but it's happening.

What do you think? Would you trust #Redis with your Production load?

[Tutorial] Generate Embeddable HTML code for any URL using Pipfeed’s Extract API

shashank agarwal — Tue, 10 Nov 2020 17:33:56 +0000

Embeddable Cards provide a clean, responsive, and shareable card for any content on the web. Cards are the easiest way to leverage Pipfeed’s extract API for any media, Cards provide a responsive embed. 40% of Users will click, hover, or view Cards with videos, images, and rich media. Cards are responsive and adapt to automatically fit any site they are placed in.

But a lot of these embed APIs aren’t very customizable and usually results in a longer load time. Using Pipfeed’s extract API, you can generate a pure HTML code in the framework and style of your choice. For this example, we will be using bootstrap cards to style the generated cards.

Read the full article at-> https://pipfeed.com/2020/11/10/generate-embeddable-html-code-for-any-url-using-pipfeeds-extract-api/

How to parse Google Search result in Java?

shashank agarwal — Mon, 05 Oct 2020 05:56:13 +0000

Google is an amazing resource but there are no APIs to really parse google search results. This is the JAVA code that I wrote that will help you parse any google search results.

How does Google Search work?

For example, if you are searching for "How to parse Google Search result in Java" then this is the URL that you would want to hit: https://www.google.com/search?q=How+to+parse+Google+Search+result+in+Java&num=10

The part after "q" is used to pass the query and "num": This tells google how many results to return.

Getting HTML search results from Google

This is a code that will search Google and return the HTML of the page.

/**
   * The method will return the search page result in a {@link String} object
   *
   * @param googleSearchQuery the google search query
   * @return the content as {@link String} object
   * @throws Exception
   */
  public static String getSearchContent(String googleSearchQuery) throws Exception {
    //URL encode string in JAVA to use with google search
    System.out.println("Searching for: " + googleSearchQuery);
    googleSearchQuery = googleSearchQuery.trim();
    googleSearchQuery = URLEncoder
        .encode(googleSearchQuery, StandardCharsets.UTF_8.toString());
    String queryUrl = "https://www.google.com/search?q=" + googleSearchQuery + "&num=10";
    System.out.println(queryUrl);
    final String agent = "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)";
    URL url = new URL(queryUrl);
    final URLConnection connection = url.openConnection();
    /**
     * User-Agent is mandatory otherwise Google will return HTTP response
     * code: 403
     */
    connection.setRequestProperty("User-Agent", agent);
    final InputStream stream = connection.getInputStream();
    return getString(stream);
  }

The above code will "URL" encode the given search term string and then call google with the given search string and return the results in JAVA using URLConnection class. You can change the headers as well to prevent google from blocking you.

Parsing results from google search result HTML

We want to only find the real results from Google Search and for this we can use this simple Jsoup based HTML parser in JAVA:

/**
   * Parse all links
   *
   * @param html the page
   * @return the list with all URLSs
   * @throws Exception
   */
  public static List<String> parseLinks(final String html) throws Exception {
    List<String> result = new ArrayList<String>();
    Document doc = Jsoup.parse(html);
    Elements results = doc.select("a > h3");
    for (Element link : results) {
      Elements parent = link.parent().getAllElements();
      String relHref = parent.attr("href");
      if (relHref.startsWith("/url?q=")) {
        relHref = relHref.replace("/url?q=", "");
      }
      String[] splittedString = relHref.split("&sa=");
      if (splittedString.length > 1) {
        relHref = splittedString[0];
      }
      //System.out.println(relHref);
      result.add(relHref);
    }
    return result;
  }

The above code is a bit tricky. So it is first finding "h3" elements inside "a". Then it looks at the parent element of the current node. From the parent node it then finds the URL.

Google search result URL starts with "/url?q=", so we use regex to remove this String. Also, the URL is followed by "&sa=", so we split the string at this character and use the first part of the URL.

The full code is available at https://system.camp/searching-google-results-and-parsing-in-java/

A.I. powered Article Extractor as an API

shashank agarwal — Wed, 01 Jul 2020 06:11:16 +0000

Check out the article extractor API here: https://rapidapi.com/imshashank/api/a-i-powered-url-article-extractor/

This is the same extractor that powers https://pipfeed.com

We use a combination of various A.I. models & services to create the most advanced Article extractor out there.

Some of the fields returned from the API:

contentURL: Html of the URL extracted and hosted on AWS CloudFront
mainImage: The featured image of the article
Category: A.I. model is used to predict the category of the article from over 70 categories.
Summary: Summary generated using A.I. to highlight the most important sentenced of the article
Images: List of all images relevant to the article.
SourceLogoUrl: Logo of the blog
SourceName: Name of the blog
Title
Authors
Language
Description
Keywords
Tags

Let me know what do you guys think?

I will help your blog reach 5000+ readers

shashank agarwal — Fri, 26 Jun 2020 10:14:16 +0000

So I am looking to help 5 bloggers. I am the founder of pipfeed.

My goal is to help 5 bloggers get over 5000 followers on pipfeed. Send me a message if you are interested. Anyone who gets 500+ followers will be eligible for sponsored articles on pipfeed. Which will be the first target.

So I create a forum.

Can you guys please add your blog here:
https://forum.pipfeed.com/d/2-please-introduce-yourself

I will review each blog and personally reply to what you can do to improve.

You can also send me a message on this discord server: https://discord.gg/PJDfqp8

8 lessons we learned building PipFeed mobile app

shashank agarwal — Mon, 11 May 2020 04:35:23 +0000

So I have spent my last 3 months building PipFeed.com. PipFeed is an A.I. powered curated reading app. Think of it as when Pocket met Medium. I am an ex-AWS engineer and my main language of choice is JAVA but I can code in PHP, python, js etc.

The backend for PipFeed is in JAVA with a few services in python, node.js & PHP. In January 2020 I learned Flutter and built the mobile app for PipFeed using it. This is my first experience building a mobile app and here are some of the learnings that I would like to share with you all.

1) Have a CI/CD

For mobile apps, it is extremely important to have a CI/CD. We use codemagic, it ties up nicely with our Flutter ecosystem. Mobile apps need a lot more work to release like signing, bundle, etc etc.

P.S. I still don’t have a CI/CD for my backends and I deploy my AWS backend from my laptop using Cloudformation templates.

2) Mobile apps crash a lot

As compared to websites, backend code, and stand-alone software, mobile apps fail and throw exceptions a lot more. There are all sorts of errors like network errors, some image fails to load etc.

To make the app stable, we had to add a lot of exception handling, null checks, and mostly retries in almost all network calls.

3) Distributed logs

We use Sentry & Crashlytics to manage and catch all our exceptions & logs. We really love the service and it has a nice pricing model. We have now integrated our backend in JAVA with Sentry.

4) Analytics is everything

We use a mix of firebase, mixpanel & AWS cloudwatch for our analytics. Analytics on mobile is a bit harder as everything you want to track requires a code change. Also, that would mean a lot of network calls for each event.

We found a way and moved the analytics to our backend system. So on an action like read, like, comment, etc when there is a database update, we trigger lambdas using AWS DDB stream and send metrics from there to other services. This helps us reduce the overall number of network requests and makes it easier to have the logic out of the mobile app.

5) It’s so hard to A/B test

As compared to any other software A/B testing is really hard on mobile. With websites you can have a version that people will see when they visit your URL with mobile your users will have a whole mix of versions, devices, screen-sizes and a lot more variables. So we are still figuring out how to do this.

Implementing A/B testing is harder in mobile as we need to create two separate code blocks and use firebase remote config or something similar to run once code block for some users and another for some other.

6) Login/SignUp is the most important screen in your app

The most important piece of code in your mobile app is your login/signup page. We learned this the hard way. We never paid much attention to our sigun/login page but when we analyzed the analytics we saw that from all the people who “installed” the app only 60% were actually signing up.

Hence we now do rigorous testing of our signup page. After so much work we still keep finding a lot of bugs/error in our login & signup flow.

7) Updates are harder because of backward compatibility

With AWS, I am used to pushing to Production almost every other day. But with mobile, you need to take care of backward compatibility a lot more.

Like we have a bug in our app where a blog will show as being “unfollowed” even when the user is following it. This was caused by two bugs with one being in the mobile app and another in our backend. Now if I fix the backend bug the current mobile app will break. So what we do? We fixed the bug in the mobile app and after around 15 days or when 90% of users have updated to the latest app, we will push fix to the backend.

8) Have a clear 2-way communication channel with users

It is very important to have some way in which users can reach out to you. At first, we thought just asking our users to email us would be enough but I think around 5% of all users ever emailed us. That’s when we found Instabug. This is a really amazing tool and they provide a 2-minute installation. I think in all the tools mentioned here, this was the easiest to setup. We run all sorts of surveys including Net promoter Score surveys, survey to nudge users to upgrade etc. Also, Instabug lets users submit bugs reports and even vote on feature requests.

If you have a mobile app, it is very important to have a NPS survey. This is probably the one metrics VCs really care about.