AI prototyping projects are short-lived and will typically have one of a few different conclusions:
- A finding that the project is viable and can be converted to a real system with additional engineering effort.
- Determining that the project is not currently viable with the data available to the organization and the currently available technology.
- A demonstration of partial progress, obstacles overcome, obstacles identified, and a request for additional time and resources to determine project viability.
Note that none of the expected outputs of an AI prototyping project is a deployable AI product - your goal is to determine viability of a project with the organization’s data and technical team.
AI prototyping projects will typically end in a formal report and/or presentation with recommendations to senior leadership on what the next steps of the project should be. These steps will usually result in a direction to suspend or continue the project - potentially converting the prototype to a real AI application that is deployed to real users.
In this fifth and final article in the AI Prototyping Projects series let’s talk through the details of these steps as AI prototyping projects invariably conclude.
Presenting your Findings
In the final phases of an AI prototyping project, the team should take some time to document the results of their project in a short executive summary.
This summary acts to preserve the high-level information about the prototyping project and should include:
- A list of functional aspects of the prototype that are currently operational
- Accuracy metrics, with one or two key metrics highlighted and explained in plain english
- A review of the initial goals defined when launching the AI Prototyping project
- Areas of concern such as difficulties with certain types of requests, system limitations, or known deficiencies
- High-level summary of the remaining work needed to either make the project a viable AI product that can be released or reach a viable AI prototype.
- Recommended next steps
This document will serve as a very brief digest for executives and project managers looking at how the project impacts the planned portfolio of work for the coming quarter or year. It also will help the organization remember the project as time goes on and the details grow hazy.
However, the main value of this summary is to facilitate conversations organizational leadership will have on if this project should continue or be tabled. These conversations may or may not include the project team, so having a concise and effective document is important.
Some organizations may also want a presentation. The presentation should have roughly the same content as the document, but is typically told in a more interactive and engaging way - often highlighting a key use case or two and showing interactive demos of those capabilities if leadership has not yet seen it in action.
A key factor for both a written report and a verbal presentation is that these artifacts should be accurate and truthful. It is your responsibility to highlight where your system excels, its current areas of deficiency, and any known risks you feel the project still has. It’s acceptable to talk about plans to remedy these deficiencies and address the risks, but it is important to present your project in as unbiased a manner as possible.
This approach is critical in a very “hyped” industry like artificial intelligence where vendors and developers routinely make unachievable promises of their technologies while glossing over costs, performance gaps, and scalability issues.
Remember: your goal on an AI prototyping project is to give the business an accurate look into what a technology looks like - along with its strengths, weaknesses, and risks.
Preserving your Prototype
Another thing the team should do as an AI prototyping project winds down is to take steps to preserve the project by documenting key information like:
- The structure of the project
- How to build and launch the project
- What external resources are required to run the project
- How to configure the project and where to find credentials that are too secure to be stored in documentation
- How the data source for the project was assembled (and how it can be updated)
- Any additional data cleaning / preparation steps
- Why the team took the approach they did
- Other approaches that were considered
By documenting these pieces of information as the project is winding down, the team reduces the risk of critical pieces of information being lost to turnover over time or simply being forgotten as team members move on to other projects.
Keep in mind that AI prototyping projects may not immediately be converted to full projects or may be revisited later down the line once data collection practices, technological developments, or additional time becomes available.
When the team documents the high-level approach and requirements of their project they set the organization up for success in the future as new people join the team or when the project is reactivated after months of being dormant. These actions ultimately increase the impact of the project on the organization by reducing the risk of that effort being wasted or unable to be replicated later.
Promoting your AI Prototype to an AI Project
Let’s say you have an AI prototyping project that just wows your leadership team. You love its performance and capabilities, you feel the technical questions have been answered and have assurances that the system can scale to the level you need it to.
It’s likely in this scenario that you and your organization are really excited to see people get real-world usage out of your new system.
So what’s next?
When you “promote” an AI prototyping project to an AI project, your focus changes from exploring feasibility, demonstrating a concept, and identifying risks to a focus typically reserved for software projects: building something reliable and scalable and deploying it to production in a repeatable manner.
It’s unlikely that the approach your team took in a short prototyping project is the same approach that should be used to serve thousands of daily users.
In many prototyping projects the team takes shortcuts simply to prototype out a concept in a short amount of time. This means that your software might handle only the “happy path” where no errors occur, users interact with it in an expected manner, and all resources are available.
In an AI project you need to care more about these factors. Expect to spend engineering time on concerns like:
- Scalability - how many parallel requests or sessions the system can support
- Error handling - responding to offline resources, networking issues, and other problems
- Observability - detecting issues that occur in production
- Security - detecting and preventing attacks on software or AI systems
- Authentication & authorization - restricting access to known users with valid permissions
- Fairness - ensuring your AI system operates without bias towards genders, ethnicities, or ages
- Escalation - allowing users to report issues with your AI system and get human intervention to help alleviate issues
- Testability - verifying that the system continues to perform well now and in the future
- Data Management - the collection and ongoing maintenance of any training or fine-tuning data, or any curated data sources referenced as RAG data sources.
Some of these factors might be things that you addressed in the prototyping phase, but even if that’s true they should likely be revisited. After all, the focus of a prototyping project should be to determine the viability of a system and any potential issues that might arise, not write efficient, secure, and reliable software code.
Additionally, the development pace on AI Prototyping projects can be rapid, which can make it harder to spot edge cases or evaluate all possibilities. By deliberately revisiting these concerns later on, you allow you to throw away your “demo code” in favor of more reliable, secure, efficient, and testable production code.
With AI systems it is not uncommon to start working using one dataset or model and switch to another as time goes on. You may have used a certain large language model during prototyping but wish to change to another one for production use cases. This type of work could require code changes, but absolutely will require a comprehensive retesting of the application under the new model.
In some cases your team may have used an AI as a Service offering like Azure AI Services’ capabilities around speech, language, and vision. Your team may choose to move away from this service in favor of their own custom-built models for increased control, performance, or improved operational costs. In these cases you likely have a significant amount of training to do to train and evaluate your own models - which may benefit from an AI prototyping of their own - just scoped at developing a custom model for that specific purpose.
Ultimately, your AI system should be flexible enough to change as your organization’s needs inevitably change over time. It’s normal to move from model to model or upgrade API versions as a project goes on or to retrain machine learning models on new data.
AI projects are software projects and, just like software projects, you’ll need to be able to update your deployed systems as issues are found. However, unlike traditional software projects, your systems will likely require periodic updates to keep up with the world your models represent. For this reason, your AI project will need to think through factors related to machine learning operations (MLOps) involving updating, re-evaluating, and deploying new models as data changes.
As you can see, AI projects are similar to software projects in some respects, but in many ways they’re uniquely different and their own creatures entirely with different concerns.
However, AI projects bring new capabilities to the table in terms of assisting users in new ways, automating processes that were previously difficult to automate, and ultimately delivering new value and new experiences to others in ways that help accomplish your organizational goals.
If you’re interested in learning more about how your organization can succeed in AI prototyping projects or converting an AI prototyping project to an AI project, please get in touch as we’d love to talk.
Top comments (0)