During the European Football Championship 2024, we had a football betting game in my company. Since GenAI is useful for many use cases, I decided to predict the results using GenAI technologies. This blog post shows my experience and which AWS services are suitable for this use case.
Getting started with Amazon PartyRock
Amazon PartyRock is an Amazon Bedrock playground that's easy to use and free. So I decided to build a PartyRock app to predict football results.
GenAI models also have some basic knowledge of football. As I wanted to increase the accuracy of the predictions, I created a csv-file with all the results of the qualification round. I extracted the results from the German Wikipedia as it was in a tabular format that I could easily convert to a csv-file. The English Wikipedia page has more information. Since I only wanted to have the results, I decided to continue with the German Wikipedia page.
I ended up with a simple PartyRock application. It contains an upload field for the csv-file and two text fields to enter the teams playing in the next game:
In the prompt I told the model to predict the results. At first it didn't predict any results because the csv-file with the previous results didn't contain enough information. After changing the prompt again and telling the model to use it's own knowledge, I got real results. The last part of that sentence was important: "Use the data from [Previous results] and also your knowledge."
No more settings were needed - PartyRock started predicting the results.
Choosing the the best GenAI model
It's easy to compare the results of different GenAI models in PartyRock. You can simply add two text generation widgets with the same prompts and choose different models. I chose Claude 3 Sonnet and Jurassic-2 Ultra. PartyRock recognized that the Jurassic-2 model wasn't suitable for the complex task, so it chose Claude 3 Haiku instead, which is also displayed in the PartyRock app.
Predicting football results is a game of chance, so there's no model that's always right. The fact that PartyRock chose to use a different model instead of Jurassic-2 Ultra shows that they have different advantages, so it is always a good idea to test different models.
Bedrock
PartyRock is based on Bedrock, so it should be possible to get the same results directly in Bedrock. I entered a similar prompt in the Bedrock Playground available in the AWS console. Instead of uploading the csv-file to PartyRock, I copied the content into the prompt for Bedrock. As expected, Bedrock is able to predict football results.
If you want to implement a custom application for this use case, Bedrock can be the technical foundation as it provides the required functionality.
Bedrock Knowledge Bases
Knowledge Bases allow you to add custom documents to Bedrock. Instead of copying the previous result csv-file into the prompt, you can upload it to an S3 bucket. The content of the S3 bucket is then used to answer questions.
As a recommended option, Bedrock suggests creating a vector store based on Amazon OpenSearch Serverless. In my opinion, OpenSearch is very expensive, so I wanted to choose a cheaper option. First, I tested MongoDB Atlas, which has a free tier. However, the free tier doesn't include the features needed to use it as a vector store in Bedrock. Since Pinecone offers a free tier that can be used as a Bedrock vector store, I used it for this test.
Did Bedrock also predict football results with Knowlege Bases? No, unfortunately not. It tried to find information about matches between the two teams. However, it was unable to predict results because it didn't find enough specific information in the source files. I think Bedrock Knowledge Bases is very useful for retrieving information about documents - but it's not suitable for this use case.
Amazon Q Business
Amazon Q Business offers similar functionality. I uploaded the qualification results to a Confluence page with is configured as a data source in Q Business. The result was the same as with Bedrock Knowledge Bases. Q Business was very content centric. Since there was no information about the specific match, it didn't predict any results.
Prompt engineering in Amazon PartyRock
Prompt engineering was a task I had to pay attention to all the time. I had to change the prompt several times. After the first round I had to take into account the results of the previous round.
In the final rounds I had to tell the model that she also had to take into account additional time and the penalty shoot-out. This made the prompt more complex than before.
Evaluation
For the first week, I was on position one of the betting game for the European Football Championship 2024. At the end, I ended up at position 5 - probably better then predicting the results on my own as I'm not an sports expert.
The GenAI models often predict results such as 1:2 or 2:1 which is a common result - I also could use this strategy without GenAI on my own.
Different models resulted in different results. As I used two models, I had to decided which result I used in the game. So I always chose the result that made more sense to me.
As I was on the first position of the prediction game during the first days, I assume that it's easier to predict results when a better team plays against the weaker team. In the final rounds, both teams had similar skills, wo it was harder to predict the results.
Sometimes the model wasn't sure which result would be better, so it answered with two ideas. Also, I think some results after penalties are not realistic, like in this screenshot.
Summary
Predicting the football results for Euro 2024 was fun. Of course, if you are a sports expert, you will probably have better predictions. However, I don't think it was a bad idea to use GenAI as I got better results than other participants. I think this is not the primary use case for GenAI - but I was able to use it for predictions as well.
The important part for me: I learned a lot about GenAI in general and in AWS, and how to use prompt engineering for specific use cases.
Top comments (0)