DEV Community: Bart Czernicki

Forecasting The NFL GOAT Quarterback with Artificial Intelligence

Bart Czernicki — Tue, 11 Mar 2025 14:32:01 +0000

Intro

We naturally enjoy comparing, ranking, and even labeling things as "The Greatest." This tendency spans virtually every area, including products, industries, and especially sports. Athletes are frequently ranked, but only a select few earn the prestigious distinction of being part of the "Greatest of All Time" (GOAT) conversation. For instance, NHL legend Wayne Gretzky famously carries the nickname "The Great One."

In the NFL, Tom Brady is widely regarded as the GOAT quarterback due to his unrivaled achievements across numerous statistical categories. However, many believe Patrick Mahomes' career trajectory is positioning him to potentially challenge Brady's status. Mahomes has already secured three Super Bowl victories and recently aimed to achieve something unprecedented: to win three consecutive Super Bowls. About a month ago, during Super Bowl LIX, Mahomes and the Kansas City Chiefs were handily defeated by the Philadelphia Eagles. Consequently, following Super Bowl LIX, the debate about the NFL's quarterback GOAT was once again a hot topic across sports media pundits.

Video examples from recent sports talk shows comparing Mahomes and Brady:

‘Mahomes will never be Tom Brady’, Did Super Bowl LIX end the GOAT debate?: https://www.youtube.com/watch?v=bt4kMkDq-7o
Can Patrick Mahomes still catch Tom Brady in the GOAT-bate?: https://www.youtube.com/watch?v=6zAgYtXFOCA
Tom Brady Is Still The GOAT, Not Patrick Mahomes: https://www.youtube.com/watch?v=Hv3PbMk9FDM
Mahomes never had ‘better team’ in Super Bowl, unlike Tom Brady | Pro Football Talk: https://www.youtube.com/watch?v=1E8cP52ymz8

🗒️ Note: The videos selected above are illustrative of the recent NFL quarterback GOAT debate. These debate videos were recorded after the most recent Super Bowl LIX, which does give us additional information (Mahomes Super Bowl loss performance). However, these videos have recency bias that needs to be taken into consideration. In a few months, even if there aren't new NFL statistics to consider, the recency bias will have somewhat worn off.

We have seen what the sports media has to say about the Brady vs Mahomes debate. I thought it would be interesting to have Generative AI provide an analysis, forecast, and opinion of its own! In the next few sections, we will see how this can easily be approached with AI, potential optimizations and similar advanced implementations that can be done.

Background on Generative AI Approach

Artificial Intelligence will be used to demonstrate the research process and final probabilistic forecast. All the tools that will be used to generate this comparison can be found in OpenAI's ChatGPT platform (ChatGPT.com). Specifically, the following features were used:

OpenAI GPT 4.5 Research Preview Model
Basic Prompt Engineering (kept to a minimum to give AI the most autonomy)
Deep Research (ChatGPT Pro Subscription) used 26 different web sources to gather the required intelligence

The key highlight among the tools used is OpenAI's Deep Research feature. OpenAI Deep Research is essentially an opinionated AI automation workflow. It operates by clearly defining the goal, investing significant time in robust internet-based research, and delivering a comprehensive, detailed report to the user. One caveat of using this automation is the duration it takes to complete; research tasks can sometimes require several minutes, occasionally up to 25 minutes. The additional processing time is essentially the trade-off for obtaining high-quality, accurate results.

The OpenAI Deep Research process, equipped with multiple automation steps and several tool integrations, significantly enhances outcome accuracy. This improvement is evident in the benchmark results shown below, using the Humanity's Last Exam dataset. This benchmark comprises 2,700 highly curated expert-level questions from various domains and is intended to be particularly challenging. The results clearly indicate that the Deep Research process delivers superior performance compared to individual AI models.

More information on Humanity's Last Exam can be found here with example questions and updated results: https://agi.safe.ai/

The prompts used for this research can be found in this chat by following the ChatGPT link below. Note the iterative response and locating the web sources used: https://chatgpt.com/share/67c25ceb-a834-8006-8ce6-c5147d40da92

Summary Analysis Table: Project Mahomes Career and Compare to Tom Brady

🤖 AI Generated

The table below was AI Generated and copied from the Deep Research response: https://chatgpt.com/share/67c25ceb-a834-8006-8ce6-c5147d40da92

I cleaned the table up, statistics were clarified for consistency and some minor statistics were removed.

Career Metric	Tom Brady (Final Career)	Patrick Mahomes (Projected Career)
Super Bowl Wins	7 ✅ (Holds NFL Record)	5 (projected) Assumes two more championships to finish with 5 total
Super Bowl MVP Awards	5 (Holds NFL Record)	5 (projected) Likely earns MVP in most future SB wins
NFL MVP Awards	3	4 ✅ (projected) On pace to add 2 more to current 2 (2018, 2022)
Pro Bowl Selections	15 ✅ (Holds NFL Record)	14 (projected) Consistent Pro Bowler (6 selections by 2023)
First-Team All-Pro	3	5 ✅ (projected) Could lead NFL in QB play a few more times (2 so far)
Career Passing Yards	89,214 ✅ (Holds NFL Record)	≈80,000+ (projected) Maintains ~4,000+ yards/year into late 30s
Career Passing TDs	649 ✅ (Holds NFL Record)	≈600+ (projected) Continues 30+ TDs/year; potential to approach Brady’s total
Playoff Record	35–13 ✅ (Holds NFL Record)	30+ postseason wins (projected) 17–4 (as of 2025)
Playoff TD Passes	88	100± ✅ (projected) On track to break Brady’s record (46 TD in 21 games already)
Playoff Passer Rating	89.8	~105 ✅ (projected) Likely remains around 100+ (currently 105.4, highest ever)

Probability Assessment: Will Mahomes Become the GOAT?

🤖 AI Generated

This section was AI Generated and copied directly from the prompt Deep Research prompt: https://chatgpt.com/share/67c25ceb-a834-8006-8ce6-c5147d40da92

Taking all the above into account – championships, stats, awards, and the projected trajectory – how likely is it that Patrick Mahomes surpasses Tom Brady as the consensus GOAT quarterback? It is certainly possible, but far from guaranteed. Brady’s combination of 7 titles and longevity records is an enormous benchmark for Mahomes or anyone to eclipse. Mahomes would likely need to win at least 5+ Super Bowls total (i.e. 2 or more additional championships in the coming years) and maintain elite play for another decade to build the statistical resume to rival Brady. Given Mahomes’ talent and early success, he has a legitimate shot to approach those milestones. However, the NFL’s unpredictability (injuries, team changes, strong competitors) means we must temper expectations.

Probability Estimate: We assess the probability of Mahomes eventually overtaking Brady as the GOAT to be around ✶30%. This represents a moderate chance – reflecting that Mahomes is on a Hall-of-Fame trajectory and is arguably ahead of where Brady was at the same point in his career, but also acknowledging that beating a legend’s legacy is exceedingly difficult. In other words, there’s roughly a one in three chance (in our estimation) that when Mahomes’ career is over, the majority of fans and analysts will rank him above Brady.

Explanation: A ~30% chance accounts for both the upside and the hurdles. On one hand, Mahomes could continue to pile up MVPs and possibly reach 5 or 6 Super Bowl titles, which would put him in GOAT contention. For example, if Mahomes adds even two more rings (to reach 5) while also finishing top-3 all-time in yards and TDs, many would argue that his era dominance (and superior efficiency stats) make him the GOAT even if he falls short of 7 rings. On the other hand, there’s a strong possibility that Mahomes ends up with an extraordinary career that still falls just short of Brady’s legacy – perhaps ending with on the order of 3–4 titles and top-5 passing totals. In that case, he’d likely be remembered as #2 all-time behind Brady. Indeed, after Mahomes’ loss in Super Bowl LIX, commentators noted that Mahomes “remains distantly behind Brady” in the GOAT race – he’s only halfway up the mountain in terms of rings (3 vs. 7).

Overview of AI Deep Research Analysis

📜 "All models are wrong, but some are useful."

-- George E. P. Box (British statistician and thought leader recognized for his foundational contributions to statistical modeling)

I consider the AI-generated "model" from OpenAI's Deep Research to be very powerful. Why? With simple prompt engineering, Deep Research was able to successfully:

Identify the key criteria to effectively compare two NFL quarterback careers.
Gather Patrick Mahomes' career statistics and project realistic final career statistics.
Cite the various internet knowledge stores used.
Summarize the key metrics, rank them, communicate and present them in an easy to undertand table.
Take all of the information into account and provided a final quantitative probabilistic estimate (30%). The estimate (while not statistically verified) is quite reasonable.

Now, let's compare this to the analysis we typically see from major sports networks. When these sports shows are produced, multiple analysts and producers are involved in creating the segment. Yet, as demonstrated by the sample videos provided, many sports experts don't deliver even half the depth or thoroughness offered by this AI-generated analysis. Human-generated sports analyses often rely heavily on subjective projections and frequently overlook crucial comparison metrics.

This doesn't imply the AI model is flawless. For instance, it did not account for head-to-head matchups between Tom Brady and Patrick Mahomes, nor did it consider how Mahomes' reliance on mobility might lead to a steeper performance decline over time. These are two key areas that human analysts did cover. Still, this AI-generated model closely matches or even surpasses the quality of analysis commonly presented on major sports networks.

We must also consider cost efficiency in such analyses. Producing a sports segment on a major network that analyzes the Brady vs. Mahomes GOAT debate typically costs tens of thousands of dollars. This AI-driven analysis, however, achieved comparable or better results at a mere fraction of that cost and time, requiring only a handful of well-crafted prompts!

Possible Model Optimizations paired with Artificial Intelligence

While the Deep Research performed by OpenAI is impressive, it would be unreasonable to suggest there aren't significant opportunities for optimization. Optimizations can enhance both the analytical approach and the accuracy of the final probabilistic forecast (such as the likelihood of Mahomes surpassing Tom Brady). By structuring optimizations into a clear workflow or process, AI can be positioned to excel particularly in analytical tasks.

Currently, statistical metrics (e.g., passing yards, Super Bowl wins) are treated as independent events. While this method is acceptable because certain errors can average out, optimal accuracy demands treating these as dependent, correlated events. For instance, Mahomes winning a Super Bowl inherently requires winning regular-season and playoff games (with rare exceptions). An improved model would explicitly account for these dependencies.
It is possible to break the problem down into smaller components. Perform the Gathering of Intelligence (statistics), Simulating/Projecting of Patrick Mahomes' stats, metric comparison and final analysis communication could be done in several separate steps. This would ensure that each step's quality is validated before passing on any intelligence and moving onto the next step.
Monte Carlo Simulations can be used to simulate Patrick Mahomes' remaining career statistics and achievements. This could create a range of probabilistic outcomes that includes a variety of constraints and rare events. For example, you could simulate Mahomes' career anchored on a typical quarterback aging curve. You could introduce "black swan" (rare) events like major injuries (Robert Griffin), suspensions (Michael Vick) or early retirement (Andrew Luck).
Most of these comparison metrics are single-point estimations. For example, the AI generated model projects Patrick Mahomes to pass for 600 touchdowns (TDs). An estimate that provides a confidence interval range is a better estimate of uncertainty. As an example, a sophisticated simulation could show a 95% Confidence Interval of Patrick Mahomes ending with a range of 525-650 touchdowns to end his career.
Tom Brady's career data is static and should be sourced directly from historical records rather than relying on repeated web scraping. Similarly, Brady and Mahomes’s five head-to-head matchups (unless Brady returns from retirement) are historical events. Brady won three of these matchups, including the critical AFC Championship game and the Super Bowl. These outcomes are established facts and should be easily accessible within a dedicated knowledge base for accurate comparative analysis.

Other Approaches

The comparison between Tom Brady and Patrick Mahomes can also be approached using decision-making. This would optimize and frame the problem as a decision. This can be implemented using various components of the Decision Intelligence Framework.

More information on the Decision Intelligence Framework: https://github.com/bartczernicki/DecisionIntelligence.GenAI.Workshop

You may also be asking if OpenAI Deep Research can be replicated on other platforms. The answer is: Yes! For example, below are two AI automation pipelines that are similar to OpenAI Deep Research implemented on Azure OpenAI.

Risk Analysis with Generative AI Reasoning (SEC 10-K) https://github.com/bartczernicki/RiskAnalysisWithGenerativeAIReasoning
Document Intelligence With Advanced AI Reasoning https://github.com/bartczernicki/DocIntelligenceWithAdvancedAIReasoning

Conclusion

📜 "The difference between smart and stupid is half an hour."

-- Old Proverb, Anonymous

This proverb may sound harsh, but it emphasizes how significantly our understanding can improve with just a small investment of time. Surprisingly, even thirty extra minutes spent gaining knowledge can be a differentiator in enhancing your approach, recommendations, or decisions. Therefore, at a minimum I highly recommend trying OpenAI Deep Research to gain that critical knowledge advantage. For solutions requiring more customized approaches, Microsoft AI services can help craft almost any AI automations. It truly is a game-changer, enabling you to tackle complex knowledge problems by having AI "spend the 30 minutes" on your behalf!

While there's room for further refinement, the current AI research automation capabilities are already transformative, significantly expanding access to insights traditionally reserved for expert analysis or costly management consulting engagements. Though not yet flawless, this technology undoubtedly represents a substantial advancement in research quality, efficiency, and cost-effectiveness.

Ultimately, products like OpenAI Deep Research (and other AI knowledge automations) not only enhance debates like the Mahomes-Brady discussion but also signal a broader shift toward democratizing high-quality analysis and insights. As these AI-driven methodologies continue to develop, we can expect widespread disruption across various sectors. Use this to your advantage.

Microsoft and the Generative AI Surge: What is Coming in 2024?

Bart Czernicki — Tue, 28 Nov 2023 01:49:05 +0000

Intro

2023 is the year of "Generative AI"!! Many companies including Microsoft are all-in in crafting new platforms, services, solutions, offers hoping it is the next big "growth engine" for years to come. Specifically, for Microsoft there is a tsunami of new innovations hitting the market and not just from the "Microsoft" brand either. To get a complete grasp of Microsoft's current AI strategy and upcoming innovations with Generative AI for early 2024, consider consuming the content from the conferences listed below. In these conferences, Microsoft either owns or co-owns AI intellectual property. Also, AI was either the main theme or a sub-theme in these four recent conferences:

OpenAI DevDay 2023 – November 6th, 2023
GitHub Universe 2023 – November 8th – 9th 2023
Microsoft Ignite 2023 – November 14th – 17th 2023
.NET Conf 2023 – November 14th – 16th 2023

OpenAI DevDay 2023

OpenAI DevDay 2023 was a developer conference hosted by OpenAI on November 6, 2023, in San Francisco. OpenAI is a separate company from Microsoft, however Microsoft does own a significant stake in the "for profit" component of OpenAI. Therefore, the innovations and advancements announced from OpenAI usually will make it to Microsoft Azure. So, this conference is an excellent event to gauge what is coming to Microsoft Azure AI in the next few weeks/months.

A full summary of announcements from OpenAI: https://openai.com/blog/new-models-and-developer-products-announced-at-devday

Some of the major announcements made at the event were:

GitHub Universe 2023

GitHub is a cloud-based company with services tailoring to software developers. Like LinkedIn, GitHub is owned by Microsoft (100%), but it operates separately under the GitHub brand. GitHub Universe 2023 took place, which primarily focused on new AI products & services. This conference is particularly interesting to developer personas interested in AI innovations with Microsoft.

GitHub Universe 2023 summary & key takeaways: Universe's key takeaway: Innovate better with AI-powered workflows on a single, unified platform - The GitHub Blog

Here is a summary of the GitHub 2023 Universe Conference, focusing on AI and developer productivity:

GitHub announced the general availability of GitHub Copilot Chat , a natural language interface that allows developers to write and understand code using their own language. Copilot Chat is powered by OpenAI's GPT-4 model and can generate code, explain concepts, detect vulnerabilities, and help debug code.
GitHub also introduced previews of GitHub Copilot Enterprise , a new offering that enables organizations to customize and train their own Copilot models using their own data and codebases.
GitHub launched the GitHub Copilot Partner Program , a network of partners that provide integrations and support for Copilot across various languages, frameworks, and platforms.
GitHub unveiled new AI-powered security features , such as Copilot Security Advisor, which scans code for potential security issues and suggests fixes, and Copilot Vulnerability Detector, which alerts developers of known vulnerabilities in their dependencies.
GitHub shared the latest Open-Source Trends Report , which revealed the growth and impact of open source and generative AI projects on GitHub. The report showed that generative AI projects increased by 253% in 2023, and that more developers are using pre-trained AI models in their workflows.

Microsoft Ignite 2023

Microsoft Ignite 2023 was a four-day event that showcased the latest innovations and advancements in AI, data, and cloud technologies. For a complete list of everything that Microsoft announced, please review the Ignite 2023 Book of News which includes areas to drill into: videos, blog posts, demos etc.: https://news.microsoft.com/ignite-2023-book-of-news/

Recorded keynotes & breakouts are available on Microsoft's Ignite channel (also some sessions can be found on YouTube): Session catalog (microsoft.com)

Below are some of the main ignite highlights related to AI:

Microsoft announced its first AI chip, code-named Athena, which is designed to compete with Nvidia's GPUs for AI acceleration in data centers. The chip will power cloud-based AI workloads such as OpenAI models, Bing, GitHub Copilot, and ChatGPT.
Microsoft expanded its partnerships with AMD and NVIDIA to provide more AI infrastructure options for customers.
Microsoft launched Windows AI Studio, a toolkit that brings together AI tools and a catalog of generative AI models that developers can fine-tune, customize, and deploy for local, offline use in their Windows apps.
Microsoft updated its Azure AI, data, and application services to help customers turn their AI ambitions into reality. These include modern data solutions, beloved developer tools, and a suite of application services that can run any language, any platform, and at enterprise-grade quality.
Microsoft showcased how Copilot for Microsoft 365, an AI-driven assistant, can reduce digital debt and increase productivity for users. Copilot can help users search, write, summarize, catch up on missed meetings, and more.

.NETConf 2023

.NETConf 2023 was a three-day virtual developer event that focused on the release of .NET 8, the latest version of the .NET platform offered with long-term support. The conference featured sessions on various topics related to .NET development, with a special emphasis on performance, AI, machine learning, and general development features in .NET 8.

All the sessions were recorded and available to watch on the "DotNet" YouTube channel: https://www.youtube.com/@dotnet/videos

Some of the highlights of the conference were:

Performance : .NET 8 has seen many improvements in performance across the entire stack, such as faster startup, lower memory usage, better code generation, and more. In fact, over 1,500 commits have been made to make .NET 8 the fastest .NET. Obviously, you will want these new AI applications to perform as optimized they can be! To highlight one AI specific example .NET 8 includes CPU accelerations to perform vector math much faster. The conference showcased some of the benchmarks and best practices for optimizing .NET applications for performance.
AI and Machine Learning : .NET 8 introduces several new features and enhancements for building intelligent applications with AI and machine learning, such as ChatGPT integration, Stable Diffusion support, prompt engineering, custom GPTs, and more. The Semantic Kernel library is close to v1.0 release and Machine Learning v3.0 was released as well. The conference demonstrated how to use these features with the Azure OpenAI service, ML.NET, and ONNX Runtime.
General Development Features : .NET 8 also brings many new and improved features for general development, such as C# 10, F# 6, Visual Studio 2022, .NET MAUI, Blazor Full Stack, Copilot and more. The conference explored how to use these features to create modern, cross-platform, and full-stack applications with .NET.

An Innovative OpenAI Approach to Crafting a Vector Database in SQL Server using Semantic Kernel

Bart Czernicki — Mon, 31 Jul 2023 14:07:14 +0000

Background

Large language models (LLMs) have signaled a transformative shift in artificial intelligence (AI), providing a doorway to tasks rooted in natural language processing. While they are a crucial component of traditional AI, it's important to acknowledge that these models also come with their share of limitations. Their versatility and wide-ranging utility have positioned them as fundamental building blocks for numerous applications, leading some to re-christen them as "foundation models". Despite their value, current LLMs (including GPT-4)
have inherent limitations:

Lack of access to current information. For example, several current OpenAI models have been trained on data up to September 2021
Lack of access to private document stores. For example, most OpenAI models were not trained on private corporate contracts
Lack of cognitive reasoning. LLMs (either OpenAI or open-source) can provide nonsense answers sometimes called "hallucinations"
Limited amount of context. Most LLMs have small token contexts (memory) that help it remember what topic the chat was about or details of the previous chat exchange

Good news is that there are approaches to overcome these limitations and improving the utility of large language models (including OpenAI). One of these approaches is "adding more memory" using a vector database for storing and retrieving embeddings, which are mathematical representations of features or attributes of data. Vector databases allow for fast and accurate similarity search and retrieval of data based on their vector distance or similarity, making it possible to find the most similar or relevant data based on their semantic or contextual meaning. This kind of "grounding" the LLM with context-specific data can provide more accurate and relevant outputs to answer specific questions.

However, there are challenges associated with this approach. Maintaining, storing, and searching embeddings through vector databases can be challenging in production settings. Embeddings are heavy objects and the storage requirements can grow quickly, especially with larger datasets. There are many new open-source and startups trying to solve for this issue. In this article, it will be illustrated how SQL Server or Azure SQL can be used to solve these issues by building an end-to-end vector database, efficiently store vectors in columnstore indexes, perform search and be able to answer questions with Semantic Kernel reasoning!

Case for SQL as a Vector Database

Vector databases are not an new concept. In fact, there has been years of research on this subject and vector database have been used in specific scenarios (i.e. Information Retrieval, Bing Search). If you launch a search engine and look up old articles for vector databases, you will find articles from over a decade ago! However, there has been a lot more interest in vector database correlated with the explosion of OpenAI and new powerful embeddings models. As any technology space going through disruption, many startups and open-source offerings have appeared: Qdrant, Pinecone, Chroma, Faiss and integrations into other enterprise products: Azure Cognitive Search, Redis, ElasticSearch etc. One product to consider for vector search is Microsoft's SQL database engine.

Features of Microsoft SQL for Vector (Indexes) Databases

Proven enterprise-grade product line with pedigree since late 1990s. It has been around for quite some time, and likely continue to be around for years
SQL Server engine runs almost anywhere you would need a vector database: on-premises (local), IoT, Docker container, cross-cloud, Linux/Windows, x64 and ARM architectures etc.
No-cost (free) development environment!! SQL Server offers a development SKU ideal for development of AI and vector-based solutions offered for free to test out the engine.
The power of SQL language. SQL has been around in existence for quite some time and it is very easy to perform complex analytical, mathematical, filtering etc. using it. Compare this with some vector database startups, that are just adding basic filtering (where clause) or lack ability to perform a simple join!
Columnstore indexes, partitioning, distributing indexes in tables allow Microsoft SQL to scale to tens of millions of documents in a Vector Index. For example, Columnstore indexes provide massive performance gains by reducing I/O through data compression and batch processing, which allows more data to be stored in memory and entire "areas" of data to be eliminated during query processing
Enterprise features like security, business continuity, query intelligence optimizations, integrations with hundreds of other systems & software are first-class citizens. Many open-source vector database alternatives can't match this functionality posture.
You probably already have used SQL Server/Azure SQL or are using it in your enterprise. You are probably NOT using one of the open-source or new startup vector stores. In the former example, it is clearly easier to "integrate" a vector store into an existing database system.

Introducing the "Vector Database SQL Search" Demo Console Application

Main GitHub location and developer instructions to get started:

https://github.com/bartczernicki/MachineIntelligence-TextAnalytics-TPLDataFlows

Features:

The console application uses 50 selected books (with over 33 million text characters) from the Project Gutenberg site from various authors: Oscar Wilde, Bram Stoker, Edgar Allen Poe, Alexandre Dumas and performs enrichment using multiple AI enrichment steps
Downloads book text, processes text analytics & embeddings, creates a vector database in SQL, demonstrates vector search and finally using Semantic Kernel answers sample questions using semantic meaning from OpenAI embeddings
Stores all enrichment output for each book in a separate JSON file, which can then be used in a Data Lake (Lakehouse, Delta Lake) for big data analytics & AI
Rather than processing text analytics enrichment in single synchronous steps, it uses .NET 8 TPL Dataflow model to create efficient pipelines that can saturate multiple logical CPU cores
Illustrates that the SQL Server engine or Azure SQL can be used as a valid Vector Store at scale, can perform hybrid vector search and provide Semantic Kernel Q&A over the database
The console application is cross-platform .NET 8.x. It can run on macOS, Linux, Windows 10/11 x64, Windows 11 ARM

Three Pipeline Execution Options showing the power of Vector Embeddings Search using SQL

Option 1 processes the entire Data Enrichment Pipeline (builds the OpenAI embeddings Vector Database in SQL)
Option 2 processes the Q&A pipeline using Semantic Kernel over the Vector Database (Note: Option #1 must have been run beforehand at some point and the application needs access to the current vector store)
Option 3 processes the Q&A pipeline with reasoning using Semantic Kernel over the Vector Database. It also includes a rudimentary confidence score illustrating how "sure" the LLM is in its answer. (Note: Option #1 must have been run beforehand at some point and the application needs access to the current vector store)

Data Enrichment Pipeline in Action

Essentially performs all of the text analytics, AI (OpenAI) work to materialize the SQL Server Vector Database. The start of the pipeline is a list of 50 books and their URL locations. At the completion of the pipeline a full AI & ML enriched set of JSON documents and a vector embeddings database is built.

Q&A Pipeline in Action

Uses both the SQL Vector Search Database & Semantic Kernel to answer question(s) from the 50 processed novels. The sample questions show how the power of the SQL Vector DB and Semantic Kernel can work together on a simple vector search, more complex hybrid search to enhance the accuracy and filter out any disambiguation between context(s) of novels.

Learn More - Upcoming Articles

Main GitHub location: source code, developer instructions to get started and more details on the application:

https://github.com/bartczernicki/MachineIntelligence-TextAnalytics-TPLDataFlows

After introducing this solution, in the next upcoming three parts of this series I will cover the three core components that make up the solution:

Part 1) How the SQL Server engine can be effectively used as a vector database, facilitate hybrid search capabilities and provide massive scale to millions of documents
Part 2) .NET 8.x TPL Dataflows Pipelines - How .NET is uniquely positioned to run anywhere with performance and leverage advanced concepts like saturating all of the available the CPU environment to execute a data-enrichment AI pipeline
Part 3) Semantic Kernel configuration and how Semantic Kernel can be configured to adapt to context from almost any vector store and work with vector stores on advanced features. Finally, I will cover additional concepts that can improve recall of Q&A in sophisticated context environments.

Make Great Decisions Using Ben Franklin's Decision Making Method

Bart Czernicki — Mon, 01 May 2023 12:21:46 +0000

Decision-making ability is regarded by executives as one of the most highly desired business skills! However, making decisions can be a daunting task, particularly when the stakes are high. As with the training of top-tier athletes, decision-makers can employ techniques, plans, and instruments to improve and achieve more. One such tool is the weighted pros and cons list, which was popularized by none other than Benjamin Franklin. This article will explore how a systematic approach to decision-making, utilizing Franklin's simple pros and cons tool, can lead to better outcomes. Furthermore, this write-up will examine the steps involved in this approach, highlight the benefits of this method, and provide an example of how it can be applied in an individual decision. By the end of this article, readers will have a greater understanding of how to make great decisions using a systematic approach, and the confidence to apply this knowledge in their own decision-making processes.

Note: All the images in the article were generated by Artificial Intelligence using **Stable Diffusion 2.x¹. The pictures depicting "salespeople & teams" were generated by artificial intelligence (AI) and any likeness to anyone is a coincidence.

Background

Benjamin Franklin is one of the most well-known figures in American history, with his life and achievements having a significant impact on both the United States and the world. Born in Boston in 1706, Franklin was a man of many talents, excelling in areas such as printing, writing, science, and diplomacy. He is best known for his role in American politics and his contributions to the development of American society, such as his invention of the lightning rod and the bifocal lens.

However, Franklin's contributions extend beyond his inventions and political leadership. He was also a keen observer of human behavior, and his writings on decision making and problem-solving have had a lasting impact on the field of decision theory. Franklin developed a method for making decisions known as the pros and cons list, in which he would weigh the potential advantages and disadvantages of a particular choice. This method has become a staple in decision-making processes, and its effectiveness has been proven time and again with many research studies.

Ben Franklin's Pros & Cons Decision Making Method

Ben Franklin was a close friend of Joseph Priestley in the 1700s, who was himself a chemist, political theorist, educator amongst other things. As trusted friends, Ben Franklin and Joseph Priestley exchanged ideas throughout the years on many ideas. In the summer of 1772, via a letter Joseph Priestley sought advice from Ben Franklin about a high-stakes decision he had to make. Priestley had a decision to make between relinquishing his position as the minister of the fame Unitarian church Mill Hill Chapel in Leeds or accepting a very desirable position as the general assistant to the Earl of Shelburne. Rather than providing an opinion on which option to select, Ben Franklin responded to Priestley by teaching him how to make a decision using a systematic process!

Below is the actual letter Ben Franklin wrote in 1772 detailing the decision process to his friend Joseph Priestley:

When these difficult Cases occur, they are difficult chiefly because while we have them under Consideration all the Reasons pro and con are not present to the Mind at the same time; but sometimes one Set present themselves, and at other times another, the first being out of Sight. Hence the various Purposes or Inclinations that alternately prevail, and the Uncertainty that perplexes us.

To get over this, my Way is, to divide half a Sheet of Paper by a Line into two Columns, writing over the one Pro, and over the other Con. Then during three or four Days Consideration I put down under the different Heads short Hints of the different Motives that at different Times occur to me for or against the Measure. When I have thus got them all together in one View, I endeavour to estimate their respective Weights; and where I find two, one on each side, that seem equal, I strike them both out: If I find a Reason pro equal to some two Reasons con, I strike out the three. If I judge some two Reasons con equal to some three Reasons pro, I strike out the five; and thus proceeding I find at length where the Ballance lies; and if after a Day or two of farther Consideration nothing new that is of Importance occurs on either side, I come to a Determination accordingly.

And tho’ the Weight of Reasons cannot be taken with the Precision of Algebraic Quantities, yet when each is thus considered separately and comparatively, and the whole lies before me, I think I can judge better, and am less likely to take a rash Step; and in fact I have found great Advantage from this kind of Equation, in what may be called Moral or Prudential Algebra.

In bullet summary, these are the steps Ben Franklin recommends when making a decision, which he called his "Decision Making Method of Moral Algebra":

Frame a decision that has two options (Yes or a No)
Divide an area into two competing halves: a "Pro" side and "Con" side
Label the top of one side "Pro" (for) and the other "Con" (against)
Under each respective side, over a period of time (Ben Franklin recommended days, this could be minutes) write down various reasons/arguments that support (Pro) or are against (Con) the decision
After spending some time thinking exhaustively and writing down the reasons, weight the different Pro and Con reasons/arguments
Determine the relative importance of each reason or argument. This is done by taking reasons/arguments that are of similar value (weight) and crossing them off of the other competing half. Multiple reasons can be combined from one side to form a "subjective" value (weight) to balance out the other half. (For example, two medium "Pro" reasons might add up to an equal value of a single important "Con" reason)
The side with the most remaining reasons is the option one should select for the decision in question

Example - Deciding on a Luxury Family Vacation

Let's assume a family is planning a luxury family vacation next year to the Maldives. How would the family approach this important decision using this decision-making framework? Taking the first 4 steps from the section above (frame the decision, provide for/con reasons in two halves), the family may write down the supporting reasons in the table below:

After making the initial list, the next steps are to spend some time thinking about the decision. For a luxury vacation, the family could consider spending a few days before making the final conclusion. During that thought-process, the family should rank the reasons and cross off the ones that are equal weights. In the table below, the reasons are now re-ranked and those of equal weights are crossed off. For example, Memories and Overspending are both weighted as #1 and equal each other out. However, there are two values in the Pro (For) column weighted as #3 that equal to one #3 Con (Against); those would cancel each other out as well. After crossing off the equal weights, the side with the most remaining reasons is the one that should be selected. Therefore, based on this decision-making process, the family is going on the luxury vacation!!!

Why This Simple 250 Year Old Decision Making Method Matters

Can you really make decisions using a simple process like this? In enterprise organizations, sophisticated decision support systems or management systems are powerful tools that use big data, artificial intelligence, statistical probabilities, or confidence interval ranges to make complex decisions. These complex systems are highly effective and can provide valuable insights that can be used to guide business strategies and improve overall decision-making. However, despite the effectiveness of these advanced decision systems, Ben Franklin's decision-making method is still relevant and necessary! For example, when Google bought YouTube, the Google executives had only one hour to accept the final term sheets!! In this high-stakes decision with a serious time crunch, there was simply no time to create a sophisticated decision model with AI/statistics! This is where simple systematic decision making processes demonstrate their true value.

One of the reasons why Ben Franklin's decision-making method is still important is that many people, both in their personal lives and in business, do not use a systematic or organized reasoning process when making important decisions. Instead, most individuals rely on their past experiences or subjective gut feelings when making decisions. This approach can be heavily biased and can lead to inconsistent decision-making. Humans have selective recall, are influenced by recent events, and are likely to make different decisions based on their current mood or personal preferences. These biases and personal preferences can impact the quality and objectivity of decisions made, and as a result, it is essential to have a structured approach to decision-making. The table below highlights the value simple decision making models can provide over simple intuition (experience), while illuminating that there are more advanced options available.

Ben Franklin's decision-making method provides a simple yet effective technique to overcome biases and personal preferences. This method encourages decision-makers to consider all factors involved in the decision-making process, including the potential benefits and drawbacks of each option. By examining both positive and negative factors, decision-makers can gain a clearer understanding of the implications of their choices, leading to more informed and balanced decisions.²

For those who are making crucial high-stakes decisions without any organized process, using Ben Franklin's decision-making method is a recommended starting point. This technique allows for a balanced review of the decision-making process, ensuring that all factors are considered before arriving at a final decision. By utilizing this method, decision-makers can minimize their biases and personal preferences, resulting in decisions made based on objective analysis and thoughtful consideration.

Footnotes

Stable Diffusion on Hugging Face: https://huggingface.co/spaces/stabilityai/stable-diffusion ↩
Business Analytics - Decision Making Process: https://www.techtarget.com/searchbusinessanalytics/definition/decision-making-process ↩

Measuring Sales Performance using Simple Statistical Models

Bart Czernicki — Wed, 01 Feb 2023 16:25:45 +0000

Measuring sales performance is a crucial aspect of running a successful business. Accurately tracking and analyzing sales data helps companies understand their strengths and weaknesses, perform forecasts, identify trends, and make informed decisions that drive growth. In this article, I will illuminate how some simple statistical models can be used for measuring sales performance. Whether it is a small or enterprise sales team, simple quantitative techniques can be used to provide valuable sales insights or draw attention to areas of need. After reading this article, you will see various examples how simple models are applied in real life scenarios.

Price's Law - Top Sales Performers Account for 50% of Sales Output

Price's Law² is a statistical principle that describes the distribution of productivity in organizations. It states that a small proportion of employees will produce a disproportionately large amount of output. In the context of sales performance, this law suggests that a small number of salespeople will be responsible for a significant portion of total sales. According to Price's Law², approximately 50% of the total output of a sales team will be generated by the square root of the total number of employees in the team. For example, if a sales team has 100 members, approximately 10 of them will generate 50% of the total sales. Conversely, if a sales team is large 10,000 about 100 salespeople will account for 50% of the sales outcomes. This means that Price's Law estimates overall top performance drops as the size of the sales team increases! To state this another way:

In a sales team consisting of 100 sales members, 10 top performers will account for 50% of outcome; 10% of the sale force are top performers.
In a sales team consisting of 1,000 sales members, 32 top performers will account for 50% of outcome; 3.2% of the sale force are top performers.
In a sales team consisting of 10,000 sales members, 100 top performers will account for 50% of outcome; 1% of the sale force are top performers!

Basically, with a larger the sales team structure, less of them are contributing to the overall success. Price's Law is one of the core reasons enterprise organizations split up their sales teams into various sales areas. For example, you may have a sales team focused on various technology domains, geographical regions or even industry verticals. This allows multiple top-performing teams to emerge across various sub-sales areas.

Another quick way to operationalize Price's Law² for sales performance is to use it to understand that a sales organization will naturally skew sales outcomes to the top performers. This is attributed to the nature of how sales accounts are allocated, the manifestation of "luck versus skill" and other conventions that govern sales. Furthermore, this principle has important implications for sales performance management. By understanding Price's Law, sales managers can focus their efforts on identifying and supporting the top-performing salespeople, who are likely to have the greatest impact on sales results. Additionally, Price's Law can be used to identify areas for improvement in sales performance. For example, if a large portion of sales are generated by a small number of salespeople, the manager may focus on developing and implementing training programs to improve the performance of the rest of the team.

It is important to note that while Price's Law² can provide useful insights into sales performance, it's a statistical principle and should not be used as a rigid rule or the sole measure of sales performance. Other factors, such as market conditions, customer behavior, and product features, can also impact sales results and should be considered in sales performance evaluations. For more information, please consider watching this video presentation on Price's Law³.

Regression to the Mean - That new Rockstar Salesperson will likely perform closer to the Historical Sales Average

Regression to the mean⁴ is a statistical phenomenon where the average of a population is calculated as the expected value of the population, and outliers (values that are much higher or lower than the average) tend to return towards the average in the long run.

In the context of hiring a new salesperson, regression to the mean can come into play when a new hire has a history of exceptional sales performance. If the new hire's sales quota is based solely on this past performance, it's possible that their future performance will not match their previous success. This is because their past performance was likely influenced by various factors such as luck, temporary market conditions, and other external circumstances. For example, if a new salesperson has previously exceeded their sales quota by 100%, it's unlikely that they will always consistently perform at this level going forward. Instead, their future performance is likely to return closer to the average of the sales team, and possibly even below the average. This is regression to the mean at work.

There is a super simple statistical regression formula (illustrated below) that can be used. Many social sciences studies show that cognitive correlation regarding performance is usually in the 20% (0.2) - 30% (0.3) range. This means that using a job interview performance as the key driver of future sales outcomes will only likely explain 20%-30% of future sales outcomes. In fact, only 3% of 708 cognitive studies showed correlations of 50% or more on future performance from a single cognitive driver (like a job interview)⁵. Basically, you need more factors to judge a correlated future performance.

In the job interview scenario shown above:

A new "rockstar salesperson" interviews for a sales position. A sales leader estimates that they will produce triple than the current team
If for example, historically on average a salesperson performs at $1 million/year, tripling this new "rockstar salesperson's" performance would lead to a sales performance expectation of $3 million/year
The regression to the mean formula states that it is more likely this new "rockstar salesperson" will be closer to the average and not optimistic outlier. Their sales performance should be estimated closer to the average: $1 million/year + $2 million/year * 30% = $1.6 million/year
The sales leader can certainly forecast for the salesperson to perform considerably better. However, using the regression to the mean calibrates the estimate conservatively. Note that using the regression to the mean, the larger the initial estimate ($2 million/year vs $1 million/year), will result in a bigger the regression to the mean as a percentage change.
For decisions & judgements based on a single driving factor, you want to use the cognitive correlation coefficient of between 0.2 - 0.3. If you are very optimistic and confident, you could perhaps use something slightly higher (0.4). What if you don't want to rely on cognitive intuition/cognitive expertise? The good news is that if you have actual data, the correlation coefficient (r) can be easily calculated⁶.

To avoid making an overly optimistic sales forecast for a new salesperson, it's important to consider multiple factors when assigning a performance quota, such as the salesperson's skills, experience, and the market conditions that they will be operating in. However, when considering one main factor (job interview/resume) it is best to calibrate the performance expectations closer to the average. This can help to ensure that the new hire's sales performance is sustainable and in line with expectations.

Law of Large Numbers - Consistent Patterns & Trends will emerge with enough Sales Engagements

The law of large numbers⁷ is a statistical concept that states that as the sample size increases, the average of the sample approaches the population mean (ground truth). For example, to test the performance of a system of flipping a coin heads or tails, a few coin flips could be performed. After enough sample flips the "ground truth" of 50% emerges (50% probability of a coin landing on a head or tail). In the beginning of the chart below, note there are some wild swings in the estimate, but the trend converges to a 50% probability eventually with a large enough sample. Law of large numbers can apply to even large complex systems if similar processes are used, and the events are more or less independent.

In the context of sales processes, the law of large numbers can be used to find trends and make more accurate predictions about future sales performance or the "ground truth" performance of sales. For example, if a company wants to predict the average daily sales for a new product, it can collect data on the sales of that product over a period of time and use that data to calculate the average daily sales. The longer the period of time for which data is collected, the larger the sample size and the more reliable the average will be as an indicator of future sales performance. This allows for much better forecasting.

In addition to predicting future sales performance, the law of large numbers can also be used to identify trends in sales processes. For example, if a company notices that sales are consistently higher on certain days of the week or during certain months of the year, it can use this information to make strategic decisions about product promotions, staffing levels, and other factors that affect sales.

It's important to keep in mind that the law of large numbers⁷ only works if the metrics you're tracking don't change a lot from one time to the next. For example, if the number of sales changes a lot from month to month, it will be harder to see a clear trend. That is why it's important to keep track of your sales over a long time, and to be careful when interpreting the results. These trends can manifest themselves as consistent historical probabilities that can be used as input into other forecast models.

Statistical Distributions - How Negative Binomial Can be used to Measure Call Center Sales Goals

There are various statistical distributions that have a small set of parameters that are very effective tools in estimating processes. One such distribution is the negative binomial distribution. A negative binomial distribution⁸ is a way to understand what happens when you're counting a certain number of events, but the number of events you need to observe is unknown (and you need to predict it). In the context of sales performance, you can use a negative binomial distribution to make predictions about the number of sales you'll make and given the number of sales attempts it will take, given consistent performance number.

To use a negative binomial distribution⁸ in a sales scenario, you need to know two key inputs: the number of desired successful sales and the consistent probability of making a sale. You can use past sales data to find the average number of successful sales (trends from law of large numbers), and use market research or other information to estimate the probability of making a sale. With this information, you can make predictions about the number of sales you'll make. Assume the following call center sales scenario:

There is a sales bonus given out every month if a salesperson closes 20 deals in a month in a call center
The probability of closing a sales deal is consistently shows a 10% probability from a sales call; every sales call there is a 1 in 10 chance of closing business
How many sales calls does the salesperson need to make to earn the monthly sales bonus?

Using an online negative binomial calculator⁹, it can probabilistically estimated that to close 20 deals in a month from a 10% sales conversion:

1.6% probability it will take up to 100 sales calls to achieve the monthly sales bonus
25.5% probability it will take up to 150 sales calls to achieve the monthly sales bonus
70.6% probability it will take up to 200 sales calls to achieve the monthly sales bonus

It is probably common sense that more sales calls (attempts) will help achieve the desired sales conversion number. However, the negative binomial distribution helps illuminate a numerical forecast and put probabilities on the forecast. This provides an incredible amount of information to the sales leadership or the sales individual to plan their monthly success accordingly.

Summary

Simple statistical models can play a significant role in sales performance management, as they allow companies to make data-driven decisions that can help improve their overall sales results. One of the key benefits of using simple statistical models is that they can provide a more objective view of sales performance, as opposed to relying solely on human intuition or ad-hoc methods. This can help to eliminate bias and minimize the impact of personal opinions, leading to more accurate and reliable results.

Moreover, these models are also relatively straightforward to implement, and require only basic statistical knowledge and a minimal investment in terms of time and resources. This makes them a cost-effective solution for companies looking to improve their sales performance, especially for those with limited budgets or resources.

However, it is important to note that these simple statistical models should not be used as the sole measure of sales performance. While they provide valuable insights into key drivers of sales, they are not designed to provide sophisticated sales forecasting, which requires more complex models and algorithms. Therefore, companies should complement these models with other methods, such as market analysis and customer insights, to get a more comprehensive view of their sales performance and make more informed decisions.

Footnotes

Stable Diffusion on Hugging Face: https://huggingface.co/spaces/stabilityai/stable-diffusion ↩
Price's Law: https://salesmetry.com/blog/2021/01/22/prices-law-competence-is-linear-incompetence-is-exponential/ ↩
Price's Law - Jordan Peterson Video: https://www.youtube.com/watch?v=8z3OZ7QuJE0 ↩
Regression to the Mean: https://www.statisticshowto.com/regression-mean/ ↩
"Noise: A Flaw in Human Judgement" Daniel Kahneman (page 151): https://www.amazon.com/Noise-Human-Judgment-Daniel-Kahneman/dp/0316451401 ↩
Calculate the Correlation Coefficient: https://www.statisticshowto.com/probability-and-statistics/correlation-coefficient-formula/ ↩
Law of Large Numbers: https://www.investopedia.com/terms/l/lawoflargenumbers.asp ↩
Negative Binomial Distribution Details: https://dlsun.github.io/probability/negative-binomial.html ↩
Online Negative Binomial Calculator: https://homepage.divms.uiowa.edu/~mbognar/applets/nb1.html ↩

My Mom's Predictive Dream about my family before she passed. How does statistics interpret these events?

Bart Czernicki — Mon, 23 Jan 2023 15:30:33 +0000

In 2017, a few months before my mother passed away, my mom had a dream that in the near future my wife and I would have a family consisting of three daughters. I thought this dream prediction is an interesting study on "predictive life events", so I decided to write about it and synthesize this with basic math & statistical modeling concepts. Basically, after reading this article you will learn how to craft simple statistical models to assign probabilities of life events!

In this article, I will cover the following concepts:

Most people think that measuring the environment is impossible, hard or don't know where to start. This article will show the reader how to craft simple statistical models using basic statistical rules to quantify (put math labels on) life events. You will see how easily these models can be crafted (even in your head!) and applied to forecasts, predictions and future decision making
One doesn't need sophisticated Artificial Intelligence, Machine Learning or Neural Nets for performing accurate forecasts; classic "old school" statistical foundations still apply to many use cases
Creating statistical models is an exercise in extracting knowledge and applying information about the environment. Secondly, one will see that predictive modeling is almost an "art" as much as it is about the mathematics and statistics
How "predictive" my mom's dream forecast was and what causal conclusions one can draw from these types of life events. For example, did my mom predict afterlife's existance?

Note: All the images in the article were generated by Artificial Intelligence using Stable Diffusion 2.x¹. The pictures depicting "daughters" were completely generated by artificial intelligence and any likeness to anyone is a coincidence.

Background Story

Going back several years to one day in early 2017; I received a call from my parents that my mom had a dream the night prior, that I would have a family with my wife consisting of three daughters. At that time of my mom's predictive dream, my wife and I had been married for 2 years and we had no kids together. We were just starting family planning. My mom had not been well, so I thought this was just one of those "comfort" dreams where my mom was connecting with my "future family" via dreams. Having said that, that day in 2017 I didn't think much of my mom's dream. Fast forward seven years to 2023...I am blessed with a bigger family and YES, we have three young daughters!!! Our first daughter was born in 2018, second in 2020 and our third in 2022. Immediately when we found out the gender of our third baby in 2022, I started to think back to my mom's dream prediction of three daughters. How strange was this? Was this in fact a prediction from the "beyond"?

Calculating the Family Probability of Three Daughters - Part 1 - Naive Approach

Let's tackle the first question. How strange was my mom's predictive dream? Can a number be placed on it and in effect measure the probability of my mom being correct? Therefore, what is the probability for having a family of three daughters? One of the foundational rules of statistics can be used to calculate this: "The Statistical Multiplication Rule"². The statistical multiplication rule, also known as the "rule of product," is a principle in probability that helps to calculate the probability of several events occurring together. It states that the probability of several independent events occurring simultaneously is the product of the probabilities of each event occurring individually.

For example, consider the probability of rolling a "4" on a six-sided die and also flipping a coin that lands on heads. The probability of rolling a 4 is 1/6 (one of the six sides of the dice) and the probability of flipping a coin to show heads is 1/2 (50%). Using the statistical multiplication rule, the calculated probability of both events occurring together is 1/6 x 1/2 = 1/12 (8.3%). Simple, right? The statistical multiplication rule can be used to calculate the probability of complex events that are composed of multiple independent events.

The multiplication rule can be applied to my mom's predictive dream of three daughters to assign a numeric probability. Based on global birth data, the actual probability of having a daughter is less than 49%³. To keep it simple, assume the probability of having a daughter [P(Daughter)] is 1/2 (50%). Furthermore, assume having each specific baby gender is an independent event, which means having a prior daughter has no impact on the gender of the future baby. This means that the total probability of having three daughters can simply be calculated by multiplying the independent probability of having a daughter three times. This equates to 1/2 * 1/2 * 1/2 = 1/8 (12.5%).

So, is this it? Well, this is a very naive model of the prediction and only part of the solution, but a good start. There are several key problems with this simple model, to highlight a couple issues:

The model is assuming there will be three children born. The correct probability statement of the model above should be "Given there are three children in a family, what is the probability that all three would be girls". This is called a conditional probability⁴, as the probability is "conditioned" on the given event occurring.
The model is missing a great deal of other various independent information that can influence the model of starting a family with children. Many (biological/environmental/family/economic) events must "perfectly align" to have a baby; it clearly isn't just as simple as forecasting of the gender.

Calculating the family Probability of Three Daughters - Part 2 - Improved Model with Selected Independent Events

It turns out the model that was calculated above is quite naive, overly optimistic, and frankly not exactly correct. While it is "roughly true" the probability of a baby daughter is roughly 1/2 (50%), life is much more complex. There is much more information to be added for the ability to maintain a family to birth three kids. To simplify, the statistical model will have two key components:

Calculated probability of three daughters, given a family of three children (calculated in Part 1 as %12.5%)
Calculated probability of maintaining a family with partner and being able to have three children with a spouse

To calculate the probability of maintaining a family and being able to have children, the following could be considered:

What if either my wife or I couldn't have kids?
What if our marriage failed (before or during the family process) and ended in divorce?
What if one of us lost our job and we couldn't afford to have more children?
Going more out of the box...What if WW3 started or an asteroid was going to destroy the planet?

There are almost infinite scenarios to list! This is where the process of crafting statistical models becomes an exercise in understanding our environment and somewhat of an "art"! Which events should be considered in the model to make proper inferences, claims or forecasts? Highly unlikely events such as WW3 happening or a black hole swallowing up earth will absolutely impact family planning, however, probably not enough to impact to the statistical model. More importantly, which of these events are independent? Which ones can be used in the model using the earlier introduced "the statistical multiplication rule"?

Some of the events that can be considered are more likely to happen on average (i.e. a marriage ends in divorce, infertility) versus a catastrophic asteroid/comet impact. All these events have a non-zero (>0%) probability of happening even if the probability is super low (nothing in forecasting the future has a 0% probability). The good news is that many of these events have been studied with many years of empirical data and can simply be looked up at no cost using a search engine. For example, scientists estimate the chance of an asteroid/comet hitting Earth and causing catastrophic impact in a given year is 1 in 300,000 (or 0.0003% probability)⁵. Since a catastrophic impact either happens or it doesn't (binary event), it can be re-stated another way: there is a 299,999 in 300,000 (or 99.9967%) chance of an asteroid/comet NOT having catastrophic impact in a given year. It could be argued a great deal of these events probably shouldn't be taken into consideration into the statistical model of "a family with three daughters". It is generally accepted to exclude highly improbable events from models (i.e. large asteroid/comet hitting Earth) as they do not change the overall "spirit" of the model nor do they change the overall probability meaningfully. Excluding these events does potentially leave the statistical model unable to predict "black swan" events. For example, how many business/sales/inventory supply chain/stock market forecasts predicted a global pandemic (black swan event) in 2020? As the global pandemic has played out over 3 years so far, it turns out not many forecasts accounted for "black swan" events. Therefore, it is up to the statistician to decide which events to include.

After adding a set of selected independent events, what does the model architecture look like visually? The diagram below displays the overall statistical structure. The foundation is still the calculation of the calculated probability of having three daughters (given a family of three kids) multiplied by the probabilities of selected set of independent events. Per the diagram below, the statistical model has now become a synthesis of math & subjectively selected independent events. Note even with just some basic & plausible events added, the model's probability output has been cut in half to 6.5%! The "simple" statistical model has now become quite complex, but arguably much more accurate. This is where you can clearly see it is up to the statistician to select the events to add into the model and compose the potential story/narrative that this will ultimately convey; where statistical modeling becomes "art".

The model structure above is meant to be more instructive in nature, but it is much better than the model in Part 1. An external observer could argue to add more high impact probabilistic impact events, such as: chance of getting a serious disease, chance of a serious accident, chance of having a special needs baby etc. In fact, some thought leaders state that because the statistician performing the modeling is applying "structured game" modeling concepts (dice, flipping coin, cards) and determining a set of fixed unknowns (set of independent events), that it really is impossible to perform this measurement. This misuse of using simple statistical models to model complex life unknowns is called the Lucid fallacy⁶. The next part will demonstrate how all these various events can be potentially "collapsed" and replaced using very accurate empirical data.

Calculating the Family Probability of Three Daughters - Part 3 - Consolidate & Collapse Information

Clearly adding many independent events adds inherent complexity to the statistical model. As mentioned earlier, which actual events are added to the model is directly influenced by the statistician's diligence and can quickly fall into the Lucid fallacy⁶. This variability could greatly affect the predictive power of the statistical model. The second key issue with adding a many independent events is that almost no events are truly independent. Independent events are ones where one event doesn't convey information about another event. For example, an asteroid hitting earth probably is truly indepdent of a family having three daughters. However, there could be inter-dependence and dependence on multiple events together. This is what statisticians need to be aware of. For example, the probability of divorce can be influenced if one partner in the relationship can't have children or if one partner can't keep a job. Basically, some partners simply won't stick around through tough times and that is a dependence. Another example: what if someone falls seriously ill causing them to lose their job, lose their fertility or lose their partner thus not being able to ever achieve a family of 3 children. Therefore, a great deal of these events are called "naively" independent in the real world. Finally, unless one has the true intersections of the data, it is hard to calculate the how much dependence there is on each of the events.

Rather than coming up with a selected set of independent events that are "just right" for the model, using empirical data these events can be "collapsed" into an average probability with fewer assumptions. Luckily, we are living in the era of big data and these available data sets are very accessible for research. There are families that have gone through family planning, have tried to have three children and potentially failed. I decided to use the USA Census Bureau data as it has USA family structure that on average resembles the economic, fertility, capability to have a family of both parents with three or more children. In my opinion, the most recent USA census data is much more predictive in nature versus global data for my model as I reside in the USA, and I am tied closer to the economic and environment in the USA. There are two key pieces of data from tens of thousands of USA families used in the updated model:

Percentage of families having three or more children in a family - 12.6% (can be interpreted as a frequentist probability from historical data)⁷
Percentage of families with both parents present - 72.9% (can be interpreted as a frequentist probability from historical data)⁷

Note: I am making the assumption that these two events are independent (for the most part). The structure of a family (i.e. single father, single mother or both parents) does not convey much information about how many children are present. Conversely, how many children are in a family doesn't convey much information on the structure of the family either. A good test of independence is asking simple questions of "knowing". For example, if someone told you there is a couple that lives several blocks from your residence and that is the only piece of information you have. Then that person asked you the following: "How many kids does that couple have?". Without other pieces of information, it is unlikely to determine a confident and accurate answer. Therefore, these two events from the Census Bureau could be considered independent and multiplied together using the statistical multiplication rule.²

These two data sets can be interpreted as average probabilities and "collapse" all those possible independent events (divorce, economic, fertility, fear of catastrophe) on average into a single probability. The model is much simpler and backed by government certified data! Furthermore, it also can be argued that empiricla data eliminates the lucid fallacy⁶ to a large degree, as a great deal of the subjectivity has been replaced with census data. The statistical model with the updated derived family census probabilities is now forecasting a 1.1% probability of my mom's predictive dream being possible due to random chance. Basically, this is a 1 in a 100 chance.

Why my mom didn't Predict Supernatural Existence

If this had been set up as a proper frequentist inference hypothesis experiment, then one could conclude something called statistical significance⁸. This is because my mom's predictive dream was so unlikely to happen by chance (1.1% calculated probability) and it was under a commonly used statistical threshold (5%); 1.1% < 5% thus statistical significance. The term statistical significance⁸ sounds impressive linguistically, but pragmatically, it does not allow someone to claim anything about the environment; nor does it prove things like the afterlife or supernatural existence. You probably have heard the term "correlation does not imply causation". While this model did not perform a correlation exercise with this statistical model the same spirit applies. So, how does one interpret or use a statistical model? One can use the model created to ask the question, "Is this weird or likely to happen just by chance?" and if the probability is low, one should investigate further. Based on the model probability, in my opinion one can claim that this predictive dream was highly unlikely due to chance, and a further investigation is warranted. So, let's investigate further.

John Allen Paulos is a mathematician and author who has written about the role of probability and statistics in everyday life, including the topic of dreams and their potential relationship to the future. In his book "A Mathematician Plays the Stock Market", Paulos discusses the idea that some people believe dreams can be predictive, offering an example of someone who dreams about a specific stock and then sees that stock rise in value the next day. Paulos notes that while this may seem like a coincidence, it is an example of the "law of truly large numbers," which states that with a large enough sample size, any outcome is possible. Paulos goes on to explain that it is natural for people to look for patterns and connections in the world around them, including in their dreams, and that this can lead to the illusion of causality when there is none. In other words, while it may seem like a dream was predictive of a future event, the reality is that the event was likely to happen anyway, and the dream was simply a coincidence.

Highly recommended book Innumeracy for basic statistical understanding

In his other book "Innumeracy", Paulos derives a simple statistical model (much like that was done above) for predictive dreams showing that they are potentially not as rare as people assume. He estimates that on a given night, the probability of a dream having some predictive nature just by random chance is 1/10,000 (0.01% probability). Highly unlikely, right? However, during a year you have 365 days or chances to have a 0.01% of a predictive dream. This is a little tricky to calculate if you are not versed in some statistics. It is NOT calculated as 365 days * 0.01% = 3.65%. There are several ways to-do this, but one simple way to think about this is using the Statistics Multiplication Rule²:

First, set up the solution as a binary event problem. This means that there is a possibility of just two outcomes 1) having a predictive dream in a day result in an accurate outcome or 2) NOT having a predictive dream or predictive outcome in a given day. There are two possible outcomes: a predictive dream or no predictive dream in this system and no other possible outcomes for simplicity.
Second, assign probabilities. Since, the system has only two possible event outcomes the probabilities must add up to 100%. Therefore, the probability of having a predictive dream per day is = 0.01% and the probability of NOT having a predictive dream per day must be 99.99% (100% - 0.01% = 99.99%).
Third, use the multiplication rule to calculate the probability over the time period. The probability of NOT having a predictive dream in 365 days is simply calculated 99.99% * 99.99% * 99.99% (repeated 365 multiplication times) or 99.99%^365 = 96.4%. This should make sense as time goes on and the time period of days becomes larger, the chances of eventually NOT having a predictive dream eventually goes lower.
Fourth, now that there is a calculated probability of NOT having a predictive dream in a year (365 days) as 96.4%, the probability of having a predictive dream in a year can be calculated. Using the same compliment rule, if the probability of NOT having even a single predictive dream in a year (365 days) is 96.4%, then the probability of having at least one predictive dream in a year must be the compliment (100% - 96.4% = 3.6%) which adds to 100%; 3.6%.

Therefore, Paulos claims that in a given random year, a random human has a 3.6% chance of having at least one predictive dream. Still a pretty low chance. Now consider this probability over multiple years, all the people you know or all the people in the USA and the probability of a predictive dream actually becomes quite likely and almost certain to happen. Some examples of long periods of time with calculations:

In a 20-year time period and assuming the probability of having a predictive dream per year is consistently 3.6%; there is a ~52% chance (better than half) that a person will have a predictive dream in a 20-year period. (Note: this is calculated using the technique above or other methods)
If you know 100 people/acquaintances and assuming the probability of having a predictive dream per year is consistently 3.6%; there is a 97.4% chance that at least one of those 100 people will have had a predictive dream in a given year! (Note: this is calculated using the technique above or other methods)

These predictive dream probabilities become highly un-impressive by the law of large numbers even if the probability is very small (i.e. Earth will eventually be hit by a catastrophic asteroid/comet given enough chances over time). However, there are several huge environment assumptions and let's consider the devil's advocate predictive dream scenario:

Assume the person having predictive dreams is keeping track of their dreams and noting the details. If someone's aunt or mom is calling them every day telling them about their various dreams and she hits on an occasional detail that probably isn't very impressive. However, if someone gets a call out of the blue one day of a predictive dream that happens to be highly detailed and correct; that is a little more "stranger" and much less explained by the law of large numbers as it is essentially a sample of one.
Predictive dreams are usually relayed in a past sense where hindsight bias⁹ can become relevant. For example, someone gets that big promotion for a job, and they call their parents to share the good news; then their mom responds "I dreamt that last week that you would get a promotion! I just knew it!". A lot less impressive for someone to say they knew it was going to happen after an event has happened.
Finally, most predictive dreams have little substantiative detail and are loosely interpreted as predictive. A great example is Nostradamus, astrology, or a fortune cookie where the details are quite squishy and could apply to many things in one's life. Astrology and fortune cookies are phrased in amorphous ways, so that almost anyone can interpret them as specific meaning for them individually.

What is interesting in my mom's dream scenario is that my mom's predictive dream was highly specific: having three daughters with my wife. It wasn't a nebulous astrology/fortune teller prediction of "you will have a successful family". Most importantly, I don't ever remember my mom having predictive dreams nor telling me about them. This is probably my personal bias speaking and I am sure my mom did have predictive dreams she told me about, but honestly can't name another case. If my mom had even a handful predictive dreams and she had been calling me about then, I would be in fundamental agreement with John Allen Paulos's interpretation. To be clear, I do not think any afterlife or supernatural was confirmed with the statistical model. However, given the specificity of the predictive dream and how it was basically a single prediction event; I do believe there was "something" special here that wasn't explainable by just random chance.

Summary and Conclusion

My mom's unique dream prediction came true without her even knowing our family plans and all the challanges my family could face. There are many things that I can't explain and can't answer. I want to believe that there was some kind of supernatural connection, but I am a logical person and can't ignore the math and science in this world. I think about this a lot, what if my mom had never dreamed this dream? What if this never happened? I am a strong believer that everything happens for a reason and that there is a higher purpose for all of us. While my mom's predictive dream was the vehicle for the article, the core concept was to show an approach how to craft simple statistical models. In this article:

Introduced an approach to quantify life events using simple statisical models
Showed how many probabilities have been studied and can simply be looked up, which can then be used as inputs into the model
Illuminated how to keep the model approach, techniques and data as transparent as possible. This way the reader can draw their own conclusions and challenge the approach accordingly. Basically, don't "lie with statistics" and paint your own biased narratives with statistics.
Demonstrated howto document personal biases and potentially include a devil's advocate position to try to a "middle ground" narrative
Exhibited that statistical models should be used as tools that can illuminate "rare" events. Good statistical models are basically indicators of "strange" and "is it weird". However, they should NOT be used as tools to draw causal conclusions. Usually much further thought and investigation should be done

Footnotes

Stable Diffusion on Hugging Face: https://huggingface.co/spaces/stabilityai/stable-diffusion ↩
General Multiplication Rule (Explanation & Examples): https://www.statology.org/general-multiplication-rule/ ↩
Birth rates for male to female are skewed to male across the world. On average 51%+ of new baby births are male: https://ourworldindata.org/sex-ratio-at-birth ↩
Conditional Probability foundations: https://www.investopedia.com/terms/c/conditional_probability.asp ↩
Chance of catastrophic impact from asteroid or comet: https://stardate.org/astro-guide/faqs/what-chance-earth-being-hit-comet-or-asteroid ↩
Lucid fallacy (Explanation & Examples): https://en.wikipedia.org/wiki/Ludic_fallacy ↩
US Census America's Families and Living Arrangements: https://www.census.gov/data/tables/2022/demo/families/cps-2022.html ↩
Statistical Significance Overview: https://www.investopedia.com/terms/s/statistical-significance.asp ↩
Hindsight Bias (Explanation & Examples): https://en.wikipedia.org/wiki/Hindsight_bias ↩

Automated Machine Learning (AutoML) - 9 Different Ways with Microsoft AI

Bart Czernicki — Mon, 04 Oct 2021 17:32:24 +0000

Intro

Microsoft's AI strategy is to "Innovate and accelerate with powerful tools and services that bring artificial intelligence to every developer". One of those core foundational patterns is AutoML (Automated Machine Learning), which allows end-to-end automation and application of common data science best practices to deliver production-ready ML pipelines and operationalized models with ease.

The Microsoft AI platform is vast made up of many services, infrastructure, software and solutions; including AutoML implementations. In this article, I will uncover various techniques to perform AutoML (Automated Machine Learning) on the Microsoft AI Platform. Some of these techniques re-use similar backend technology, however they deviate enough in their integrations, purpose and user interfaces to call them "different". Did you know that you can perform AutoML 9+ different ways using Microsoft AI?! In each AutoML "way"; I will describe the nuances, their superpowers and when to use one "way vs other".

Why

Why have many different ways to execute a data science design pattern; such as AutoML? The idea is that an organization may have made multiple investments across different technologies, architectures, programming languages or platforms and Microsoft AI activates AutoML across a wide breadth of these services!

1) Azure Machine Learning - Studio & Web Portal

Azure Machine Learning is a portfolio of enterprise cloud services that aim to build and deploy Machine Learning models faster. AML studio is a web portal that provides a web UI to interface with Azure Machine Learning. You can get started with AutoML using the web portal easily

When to use: Use the Studio portal to get started easily without writing any code by leveraging the web UI in the Azure cloud. It is a great way to learn about the AutoML capabilities before diving in deeper. The studio portal is ideal for crafting models for a POC, hackfest or in a time crunch when you do not need to automate the process. Do note that there is some functionality not completely exposed inside the portal, where the SDK might be a better fit. This is a cloud service, so to perform anything sophisticated you will need an Azure subscription (at least FREE trial).

For a complete tutorial, navigate to: https://docs.microsoft.com/en-us/azure/machine-learning/tutorial-first-experiment-automated-ml

2) Azure Machine Learning - SDK & APIs

The Azure ML SDK & APIs provide a code-first approach to performing various tasks like AutoML.

When to use: The SDK & APIs are great to use when you need a code-first approach to AutoML, integrate with data science tools (Jupyter, VSCode) or need to automate some of the AutoML tasks. The SDK exposed is written in Python friendly to most data scientists. All of the backend APIs are REST-based. Therefore, with some additional work you can use these APIs with your favorite programming language. The SDK & APIs work across different Azure services and can also be used locally. For example, you can perform AutoML tasks and have them run on source files and local compute. Therefore, depending on how you use the APIs this can be very cost effective or even no-cost.

...
ensemble_settings = {
                     "ensemble_download_models_timeout_sec": 600
                     "stack_meta_learner_type": "LogisticRegressionCV",
                     "stack_meta_learner_train_percentage": 0.3,
                     "stack_meta_learner_kwargs": {
                                                    "refit": True,
                                                    "fit_intercept": False,
                                                    "class_weight": "balanced",
                                                    "multi_class": "auto",
                                                    "n_jobs": -1
                                                  }
                    }

automl_classifier = AutoMLConfig(
                                 task='classification',
                                 primary_metric='AUC_weighted',
                                 experiment_timeout_minutes=30,
                                 training_data=train_data,
                                 label_column_name=label,
                                 n_cross_validations=5,
                                 **ensemble_settings
                                )
...

For a complete tutorial, navigate to: https://docs.microsoft.com/en-us/azure/machine-learning/how-to-configure-auto-train

3) Visual Studio 2019/2022 - Model Builder

Model Builder is a free extension to Visual Studio (proper) based on ML.NET and extensibility into the cloud that offers code-generation, integrates with full .NET and source services.

When to use: Model Builder targets the "developer data scientist" persona. Therefore, if your team is using Visual Studio, have .NET investments (DevOps, APIs), build C# solutions, require complete offline AutoML support; Model Builder is a great option. Model Builder has wizard-like UI that walks the user through AutoML. It is based on the ML.NET (Machine Learning .NET open-source framework, but it does extend into the Azure cloud. It does not have all of the capabilities of the Azure ML AutoML service, however it provides a solid foundation. It is provided at no-cost and works with the free (Community Edition) of Visual Studio as well. A perfect scenario is a developer who has an existing .NET solution and wants to explore if adding Machine Learning (with AutoML) is worth it. They can provide some data to Model Builder, run the experiments and have a C# set of training and inference code written in seconds!

For a complete tutorial, navigate to: https://dotnet.microsoft.com/learn/ml-dotnet/get-started-tutorial/intro

Model Builder recent AutoML updates and roadmap: https://devblogs.microsoft.com/dotnet/ml-net-june-updates-model-builder/#new-and-improved-automl

4) Machine Learning .NET - AutoML with C# APIs

This is a code-first approach to AutoML with ML.NET & C#. The developer now is writing C# code to interface with the Machine Learning .NET SDK & APIs.

When to use: Use the ML.NET AutoML APIs when you need to have explicit, automated, configurable control over the AutoML process using .NET. If your team has developers that understand the core AutoML pattern, this is a great way to integrate AutoML capabilities with your solutions. All your existing .NET architecture does not have to change, nor do you have to take a dependency on a cloud service! For example, if you have a solution that processes some sort of data sets and need to automate Machine Learning tasks, ML.NET will fit right in.

using Microsoft.ML;
using Microsoft.ML.AutoML;
    // ...
    MLContext mlContext = new MLContext();
    IDataView trainDataView = mlContext.Data.LoadFromTextFile<SentimentIssue>("my-data-file.csv", hasHeader: true);

For more information, navigate to: https://docs.microsoft.com/en-us/dotnet/machine-learning/how-to-guides/how-to-use-the-automl-api

5) Machine Learning .NET - AutoML with CLI

Very similar to the previous scenario. However, this is a script-first approach to AutoML with ML.NET & CLI (Command Line Interface). The developer now is writing CLI code to automate AutoML tasks.

When to use: You would consider using the ML.NET - Auto ML CLI when you need to automate tasks using scripting techniques. If you have existing pipelines scripted out or are using another programming language/conventions for automation, then the CLI AutoML interface is a great option to look at.

For a complete tutorial, navigate to: https://dotnet.microsoft.com/learn/ml-dotnet/get-started-tutorial/intro

6) Synapse Analytics & Studio - Spark Tables with AutoML

Synapse Analytics and Synapse Studio allow you to take existing Synapse investments & technology and enrich it with AutoML capabilities.

When to use: If you have Spark tables inside Synapse, want to train and operationalize ML models with Synapse or have heavy SQL code investments; consider AutoML with Synapse. Therefore, you can take your data lake or logical data warehouse and layer the AutoML pattern over it. Synapse Studio provides an AutoML wizard user interface to guide the user through the process, so it is super easy to get started. Furthermore, once your model is built with AutoML you can then use scoring wizard for dedicated SQL pools to operationalize your model directly in SQL! Note that Synapse leverages the Azure ML - AutoML platform, so you will need to have that service live. Finally, both AzureML (AutoML) and Synapse are cloud services; so you are going to be committing to the cloud. This is a commercial cloud service that will cost money to use.

For a complete tutorial, navigate to: https://docs.microsoft.com/en-us/azure/synapse-analytics/machine-learning/tutorial-automl

7) Power BI - AutoML for dataflows

Automated machine learning (AutoML) for dataflows enables business analysts to train, validate, and invoke Machine Learning (ML) models directly in Power BI. It includes a simple experience for creating a new ML model where analysts can use their dataflows to specify the input data for training the model.

When to use: You should consider using AutoML for dataflows, when you have or plan to have investments in the Power BI platform. This basically allows you to enrich Power BI dashboards/reports/data models across multiple sources with AutoML. The AutoML integration with Power BI provides an easy to use wizard interface. One of the nice integration pieces is that after an ML model is trained, AutoML automatically generates a Power BI report that explains the likely performance of your ML model. As the dataflows are re-run and the data is refreshed, the crafted AutoML models are also refreshed. Similar to the Synapse integration, Power BI uses automated ML from Azure Machine Learning to create your ML models. However, you don't need an Azure subscription to use AutoML in Power BI. The process of training and hosting the ML models is managed entirely by the Power BI service. This is a commercial cloud service that will cost money to use.

For a complete tutorial, navigate to: https://docs.microsoft.com/en-us/power-bi/transform-model/dataflows/dataflows-machine-learning-integration

8) Microsoft Research & Open-Source - FLAML

FLAML (Fast and Lightweight AutoML) is a lightweight Python library that finds accurate machine learning models automatically, efficiently and economically. It frees users from selecting learners and hyperparameters for each learner. It is fast and economical. The simple and lightweight design makes it easy to extend, such as adding customized learners or metrics. FLAML is powered by a new, cost-effective hyperparameter optimization and learner selection method invented by Microsoft Research.

When to use: FLAML is open-source, free, exposed as a Python API (code-first) and you can track progress publicly on GitHub. Microsoft Research currently owns this project and it will be integrated in other Microsoft AI platform services. Consider using FLAML, when you want to use cutting edge AutoML techniques before they appear in Microsoft products at no-cost. For example, if you use Model Builder .NET, some of the capabilities of FLAML exist there. This means you can use FLAML locally & not have to take a dependency on the cloud. FLAML focuses on core (classical) Machine Learning algorithms. This means the Scikit-learn family of algorithms can be automated using FLAML. For deep learning (neural nets, PyTorch, Tensorflow) consider the Neural Network Intelligence implementation. FLAML is meant to be barebones and lightweight, so simple AutoML use cases are a great fit.

from flaml import AutoML
from sklearn.datasets import load_iris
# Initialize an AutoML instance
automl = AutoML()
# Specify automl goal and constraint
automl_settings = {
    "time_budget": 10,  # in seconds
    "metric": 'accuracy',
    "task": 'classification',
    "log_file_name": "test/iris.log",
}
X_train, y_train = load_iris(return_X_y=True)
# Train with labeled input data
automl.fit(X_train=X_train, y_train=y_train,
           **automl_settings)
# Predict
print(automl.predict_proba(X_train))
# Export the best model
print(automl.model)

For a complete tutorial, navigate to this Jupyter Notebook: https://github.com/microsoft/FLAML/blob/main/notebook/flaml_automl.ipynb

Link to FLAML GitHub repository: https://github.com/microsoft/FLAML

9) Microsoft Research & Open-Source - Neural Network Intelligence**

NNI (Neural Network Intelligence) is a lightweight but powerful toolkit to help users automate Feature Engineering, Neural Architecture Search, Hyperparameter Tuning and Model Compression.

When to use: (Very similar to FLAML) NNI is open-source, free, exposed as a Python API, command-line, visual web UI and you can track progress publicly on GitHub. MConsider using NNI, when you want to use cutting edge AutoML techniques before they appear in Microsoft products at no-cost. The core difference between NNI and FLAML is that NNI focuses on deep learning and supports frameworks such as PyTorch, Keras, Tensorflow etc. but also supports traditional machine learning packages (Scikit-learn). NNI is much more of a complete framework than FLAML with a monthly release cadence; whereas FLAML is meant to be lightweight. The NNI framework can run locally with no cloud dependencies at no-cost!

For a complete tutorial, navigate to this Jupyter Notebook: https://github.com/microsoft/nni/blob/master/examples/notebooks/tabular_data_classification_in_AML.ipynb

Link to NNI GitHub repository: https://github.com/microsoft/nni

.NET 6 Preview - Major Blazor Performance Improvements

Bart Czernicki — Thu, 18 Feb 2021 15:14:05 +0000

.NET 6 Preview 1 bits were released yesterday (info below)...
https://devblogs.microsoft.com/dotnet/announcing-net-6-preview-1/
https://devblogs.microsoft.com/aspnet/asp-net-core-updates-in-net-6-preview-1/ (ASP.NET)

.NET 6 Preview 1 for Blazor WebAssembly (WASM) includes many performance enhancements planned. For example, in Preview 1 the interpreter has been improved greatly. This gives Blazor WASM a VERY nice performance bump in .NET 6 for "FREE" by just using the new .NET 6 runtime.

I ran some performance tests comparing .NET Core 3.x, .NET 5 and .NET 6 Preview 1...and you can see .NET 6 is 22%-32% faster across various compute devices/browsers.

Run the performance tests yourselves and compare!
.NET 6 - https://statisticsandblazordotnet6.azureedge.net/
.NET 5 - https://statisticsandblazordotnet5.azureedge.net/
.NET 3 (Core) - https://statisticsandblazordotnetcore3.azureedge.net/

Source Code: https://github.com/bartczernicki/Test-Blazor-WebAssembly-StatisticsAndML-DotNet6

Blazor & Machine Intelligence Demos

Bart Czernicki — Fri, 12 Feb 2021 21:29:44 +0000

Below is a sample of my Blazor apps with Machine Intelligence (Statistics, Analytics, Machine Learning, AI, Information Retrieval etc.). This coalesces all of my Machine Intelligence repos in a single area and point to their respective: live demo, source code, video & presentation locations.

Live Demos

•Baseball ML Workbench (.NET Core 3.x, Blazor Server): https://aka.ms/BaseballMLWorkbench

•Statistical Simulations & Visuals (.NET Core 3.x, Blazor WASM): https://statisticsandblazordotnetcore3.azureedge.net

•Statistical Simulations & Visuals (.NET 5, Blazor WASM): https://statisticsandblazordotnet5.azureedge.net

•ML.NET Baseball Predictions & Lucene Information Retrieval (.NET 5, Blazor WASM, Azure Static Website): https://mlnetandblazor.azureedge.net

•ML.NET Baseball Predictions & Lucene Information Retrieval (.NET 5, Blazor WASM, Azure App Service): https://mlnetandblazor.azurefd.net

Source Code

•Baseball ML Workbench: https://github.com/bartczernicki/MachineLearning-BaseballPrediction-BlazorApp

•Statistical Simulations & Visuals (.NET Core 3.x, Blazor WASM): https://github.com/bartczernicki/Test-Blazor-WebAssembly-StatisticsAndML

•Statistical Simulations & Visuals (.NET 5, Blazor WASM): https://github.com/bartczernicki/Test-Blazor-WebAssembly-StatisticsAndML-DotNet5

•ML.NET Baseball Predictions & Lucene Information Retrieval (.NET 5, Blazor WASM, Self-Host): https://github.com/bartczernicki/Test-Blazor-MLNet

Presentations

•Adding Machine Intelligence to your Blazor Apps: https://github.com/bartczernicki/ArtificialIntelligence-Presentations/blob/master/AddingMachineIntelligenceToYourBlazorApps.pptx

•Machine Learning Community Standup - Blazor WASM & ML.NET with .NET 5: https://github.com/bartczernicki/ArtificialIntelligence-Presentations/blob/master/DotNetCommunityStandup-MLNetBlazorDotNet5.pptx

•Virtual ML.NET Conference - Decision Analysis with ML.NET: https://github.com/bartczernicki/ArtificialIntelligence-Presentations/blob/master/VirtualMLNet-DecisionAnalysisWithMLNet.pptx