Our team has been using generative AI tools such as Github Copilot and Codium in their everyday coding tasks. This has greatly enhanced the software development experience and boosted productivity.
To further improve efficiency, we have also developed a custom UI for ChatGPT, now accessible across the company. This tool has proven to be a valuable time-saver for both developers and designers.
We are also exploring the use of vision models and have started incorporating them in web front-end development. One concept we are working on is employing a vision model to generate a functional web app from images.
This concept differs from Copilot or Codium, which provide micro-level assistance, such as writing a function, refactoring a class, or adding tests. Meanwhile, other tools in the market aim for pixel-perfect pages but often produce unusable code due to incorrect page layout plans, inline CSS, an overload of HTML tags, or hard-coded data in the layout. As a result, their output is essentially unusable for developers.
Our tool, on the other hand, scaffolds the entire application and offers a comprehensive app generation process with usable code that developers can work with.
Simply input images and receive a fully functioning NextJS app in return!
How is this approach different?
The AI agent applies a range of prompting techniques to mirror a developer's thought process. Initially, it generalizes the layout from several images, before subsequently devising unique content for each page.
The produced code is reusable.
This tool is capable of generalizing layouts from various images, thus enabling the creation of shareable components. Additionally, it separates data or app state from the layout, facilitating seamless integration with APIs for real data.
Figure 1: Input UI image with sidebar on the right-hand side and a list of transactions
Figure 2: The tool is able to detect reusable components, such as sidebar and topbar, and put them in shared folder /app/_components
Figure 3: The tool is able to separate transactions data from the layout code
How much development time can it save?
The tool performs well on certain apps, creating pages that mirror the UI images. In some instances, it can create input forms with a 70% match, requiring only minor CSS adjustments to align with the UI design.
We utilized the tool for a client project, and it reduced development time by approximately 20%.
We are enhancing our prompt engineering to tackle challenges with certain layout types. We're also considering a feedback loop, where the apps created by the model are fed back into it for self-improvement.
Figure 4a: UI design of contact form
Figure 4b: Generated page of contact form
Figure 5: Generated ReactJS & TailwindCSS code for the contact form
Can I try this tool?
Indeed, a Proof of Concept (PoC) is available for you to try out. Refer to the instructional video for guidance on how to use the app.
This is currently a PoC, and due to the unpredictable nature of generative A, its effectiveness may vary depending on your application. We appreciate any feedback and encourage you to email us at teamcodest@codelink.io.
Here are some examples of applications generated entirely by the tool without any manual intervention.
Demo link
codest.ai
Top comments (1)
Wow!! I have been playing with Azure's Vision model a bit, but this post really blew my mind! Very cool! Thanks for sharing this.