DEV Community

Pneumetron
Pneumetron

Posted on Originally published at pneumetron.com

AWS launches SageMaker model leaderboard; IBM brings time-series models to Confluent

AWS releases a standardized leaderboard for 70+ LLMs inside SageMaker Studio, while IBM and Confluent deploy time-series foundation models directly into streaming pipelines for real-time forecasting and anomaly detection.

Read the full article on Pneumetron →


AWS has released a standardized model comparison view directly inside Amazon SageMaker Studio. The new JumpStart Model Leaderboard ranks 70+ large language models from providers including Meta, Google, Qwen, IBM, and Arcee on reasoning benchmarks, inference speed, and estimated instance cost. The tool is designed to replace the current practice of stitching together benchmark scores and pricing from disparate model cards and third-party sites.

The leaderboard displays quality scores on three reasoning benchmarks—BIG-Bench Hard, GPQA Diamond, and MMLU-Pro—using evaluation harnesses and settings that AWS runs identically across every model. It pairs these scores with performance and cost metrics measured by sending a standardized test workload to a SageMaker endpoint deployed with each model's default configuration. Users can filter the list by license, provider, and modality, then select a use case profile and an optimization goal to view the specific deployment configuration that matches their workload. The interface uses a blue heatmap to highlight top performers in each column, and users can pin the model name column while scrolling through additional metrics.

AWS defines four optimization goals that dictate which configuration metrics appear in the table. Balanced shows the setup with the lowest estimated cost per token. Cost optimized targets the lowest hourly instance price, accepting higher latency. Latency optimized minimizes time to first token, while Throughput optimized maximizes tokens per second, often using speculative decoding. The performance metrics reported are time to first token, throughput per request, and instance cost. The cost figures are strictly SageMaker on-demand instance rates. AWS explicitly notes that these estimates exclude data transfer, storage, inference component costs, and discounts like Savings Plans or Reserved Instances.

To illustrate the trade-offs, the announcement provides an example for an internal help-desk assistant filtering for a commercially licensed model optimized for latency. The comparison shows how a larger model with a 95.9 score on BIG-Bench Hard and 87.2 on MMLU-Pro incurs a $37.66 hourly instance cost with a 2.33-second time to first token. A medium-sized alternative scores 94.2 and 84.9 respectively, drops the cost to $5.75 per hour, and halves the latency to 0.98 seconds. A smaller model further reduces latency to 0.27 seconds and cost to $2.61 per hour, but the reasoning scores drop into the high 70s and low 60s.

Illustrative photo: circuit board, computer, chip, data processing
Illustrative image: blickpixel / Pixabay

Model BBH MMLU-Pro TTFT (s) Throughput (tokens/s) Instance cost ($/hr)
Model A (large) 95.9 87.2 2.33 37 $37.66
Model B (medium) 94.2 84.9 0.98 44 $5.75
Model C (small) 77.4 66.1 0.27 61 $2.61

The methodology aims to solve the comparability problem inherent in published benchmark scores. Different teams often use different prompts, generation settings, and hardware, making direct comparisons unreliable. AWS runs each benchmark with the same evaluation harness and settings across all models. Performance and cost come from sending a standardized test workload to an endpoint deployed with each JumpStart configuration, the exact configurations users can launch. AWS advises treating small score differences as ties, since they often fall within normal run-to-run variation. Teams should use the leaderboard to build a shortlist, then validate top candidates on their own data and prompts before production deployment.

At launch, the leaderboard covers LLM text generation. AWS plans to extend it to private hubs, other task types like object detection and text embeddings, and additional benchmarks as new models are added to JumpStart.

In a separate release, IBM and Confluent have integrated time-series foundation models directly into the Confluent data streaming platform. The integration brings IBM's time-series foundation models—part of a family that has accumulated over 50 million downloads—into Confluent Cloud and Confluent Platform. The models run natively as built-in functions within Apache Flink, allowing teams to apply forecasting and anomaly detection to streaming data without deploying separate GPU infrastructure. The models are designed to run on standard CPUs alongside the streaming pipeline, both in the cloud and on-premises.

Illustrative photo: laboratory, analysis, chemistry, research
Illustrative image: jarmoluk / Pixabay

Confluent describes the architecture as a "Streamhouse," a system that continuously captures, processes, and serves current business context. The platform brings together connect, stream, process, govern, and query capabilities to make current context available to applications, analytics, and agents. By invoking the time-series models inside Flink, data can be transformed and enriched as it flows, making predictions available to applications and agents in real time. The models provide a pretrained starting point that learns general patterns from diverse historical data, reducing the need to train a separate model for every individual time series. Depending on the task, teams can fine-tune the models or combine their outputs with other data and business logic.

The integration supports several specific model functions. AI_FORECAST generates demand or volume predictions with a P10–P90 confidence range, which can be refreshed as new observations arrive. AI_DETECT_ANOMALIES establishes an expected range for each series based on recent history, flagging readings that fall outside a chosen confidence level. PatchTST-FM forecasts trading volume to help execution algorithms pace large orders, while TSPulse produces embeddings for similarity search across historical sensor patterns.

Organizations are using the models to automate operational decisions. In manufacturing, a pretrained TTM model reads SCADA telemetry to forecast product quality hours ahead, allowing optimization processes to score control variable adjustments before they hit the production line. In retail and supply chain, the models forecast store-level demand to trigger automated purchase orders. Financial teams use the signals to compare individual card transactions against personalized spending ranges for fraud detection, or to forecast market volume for trade execution.

The models do not replace existing rules or specialized systems. Instead, they act as an intelligence layer that routes signals to operational applications, where business logic and human judgment make the final call. The integration requires setting up the Confluent streaming pipeline and configuring the Flink functions, but it removes the operational overhead of managing separate inference endpoints for time-series workloads.

Sources


Originally published on Pneumetron.

Top comments (0)