The AI tools panel · Chapter 3 of 12 · AI Engineering · new chapter every Sunday afternoon
By the end of this chapter: Understand what app generators do well and where the ceiling is.
The problem
Sofia, a senior backend engineer, has been tasked with prototyping an internal tool to parse incoming support tickets, suggest a short response, and optionally generate a code snippet to reproduce a reported bug. She has access to several AI code assistants, including GitHub Copilot (v1.84), Claude 3.5 Sonnet (2024-09), OpenAI GPT-4-Turbo (2024-10), and the open-source Code Llama 2-7B-Chat (2024-07). However, she is unsure which tool to use for this specific task and how to integrate it into her workflow. The prototype needs to be completed quickly, and the team is counting on her to deliver a working solution.
Sofia's initial attempt to use GitHub Copilot to generate the entire application resulted in a promising start, but she soon realized that the generated code lacked the nuance and customization required for their specific use case. The tool had created a basic CRUD application, but it did not account for the complex business logic and integrations needed for the support ticket parser. Sofia spent several hours trying to modify the generated code, but it became clear that she was fighting the tool rather than working with it.
The experience left Sofia wondering what app generators like GitHub Copilot are truly capable of and where their limitations lie. She needs to understand how to effectively utilize these tools to accelerate her development process without sacrificing the quality and customization required for her project. The clock is ticking, and Sofia must find a way to leverage the strengths of app generators while avoiding their weaknesses.
Before you start
To follow along with this chapter, you should have a basic understanding of the AI code assistants mentioned earlier, such as GitHub Copilot, Claude, OpenAI GPT-4-Turbo, and Code Llama. You should also have a Python environment set up with the necessary dependencies installed, including the transformers library (version 4.24.0 or later) and the pytorch library (version 2.0.0 or later). You can verify your setup by running the following command:
pip install transformers pytorch
And then checking the versions:
import torch
from transformers import AutoModelForSeq2SeqLM
print(torch.__version__)
print(AutoModelForSeq2SeqLM.__module__)
This should output the versions of the installed libraries.
Understanding App Generators
App generators like GitHub Copilot are designed to accelerate the development process by generating boilerplate code, suggesting completions, and even creating entire applications. However, they are not a replacement for human judgment and expertise. To effectively utilize these tools, you need to understand their strengths and weaknesses.
App generators excel at generating repetitive, straightforward code, such as CRUD applications, API clients, or data models. They can also help with code completion, suggesting the next line of code based on the context. However, they struggle with complex business logic, custom integrations, and nuanced requirements. In Sofia's case, the generated code lacked the necessary customization and nuance for their specific use case.
Evaluating App Generator Output
When working with app generators, it's essential to evaluate the output critically. You should check the generated code for correctness, completeness, and maintainability. Ask yourself:
- Does the generated code meet the requirements?
- Is the code readable, maintainable, and efficient?
- Are there any potential security vulnerabilities or performance issues?
To evaluate the output, you can use tools like linters, code formatters, and security scanners. You can also use code review tools to collaborate with your team and ensure the generated code meets your standards.
Build it
To demonstrate the capabilities and limitations of app generators, let's create a simple support ticket parser using GitHub Copilot. We'll use the transformers library to fine-tune a pre-trained model for our specific task.
- Create a new Python file called
support_ticket_parser.pyand add the following code:
import torch
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
# Load the pre-trained model and tokenizer
model = AutoModelForSeq2SeqLM.from_pretrained("t5-base")
tokenizer = AutoTokenizer.from_pretrained("t5-base")
# Define a function to parse support tickets
def parse_support_ticket(ticket):
# Preprocess the ticket text
inputs = tokenizer.encode_plus(
ticket,
add_special_tokens=True,
max_length=512,
return_attention_mask=True,
return_tensors="pt",
)
# Generate a response using the model
outputs = model.generate(
inputs["input_ids"],
attention_mask=inputs["attention_mask"],
max_length=128,
)
# Postprocess the response
response = tokenizer.decode(outputs[0], skip_special_tokens=True)
return response
# Test the function
ticket = "The user is experiencing issues with the login feature."
print(parse_support_ticket(ticket))
This code uses the t5-base model to generate a response to a support ticket. You can modify the code to fine-tune the model for your specific use case.
- Run the code and evaluate the output. Check if the generated response is accurate, complete, and maintainable.
When this breaks
App generators can break in several ways, including:
- Overfitting: The generated code is too specialized for the training data and fails to generalize to new, unseen inputs.
- Underfitting: The generated code is too simplistic and fails to capture the complexity of the problem.
- Security vulnerabilities: The generated code contains security vulnerabilities, such as SQL injection or cross-site scripting (XSS) attacks.
To fix these issues, you can try the following:
- Fine-tune the model: Adjust the hyperparameters, such as the learning rate or batch size, to improve the model's performance.
- Add more training data: Increase the size and diversity of the training dataset to improve the model's generalizability.
- Use security scanners: Integrate security scanners, such as OWASP ZAP or Burp Suite, to identify and fix security vulnerabilities.
What it costs
Using app generators comes with several costs, including:
- Time: Evaluating and refining the generated code can take significant time and effort.
- Quality: The generated code may not meet the required standards, leading to maintenance and debugging issues.
- Security: The generated code may contain security vulnerabilities, which can compromise the entire system.
To mitigate these costs, you should:
- Establish clear requirements: Define clear, concise requirements for the generated code to ensure it meets the necessary standards.
- Use code review tools: Collaborate with your team to review and refine the generated code, ensuring it meets the required quality and security standards.
- Monitor and maintain: Continuously monitor and maintain the generated code, updating it as necessary to ensure it remains secure and efficient.
In the interview
When discussing app generators in an interview, you may be asked questions like:
- "What are the benefits and limitations of using app generators?"
- "How do you evaluate the output of an app generator?"
- "What are some common issues that can arise when using app generators, and how do you address them?"
A weak answer might focus solely on the benefits of app generators, without acknowledging their limitations. A strong answer would discuss the trade-offs, such as the potential for overfitting or underfitting, and explain how to mitigate these issues.
For example, a strong answer might say:
"I've used app generators like GitHub Copilot to accelerate my development process, but I'm also aware of their limitations. One of the main issues is overfitting, where the generated code is too specialized for the training data. To address this, I fine-tune the model using additional training data and evaluate the output critically to ensure it meets the required standards."
As you progress in your career, you may be asked more senior-level questions, such as:
- "How do you integrate app generators into your development workflow, and what are the benefits and drawbacks of this approach?"
- "What are some best practices for evaluating and refining the output of app generators, and how do you ensure the generated code meets the required quality and security standards?"
Your tasks
- Create a simple CRUD application using GitHub Copilot and evaluate the generated code for correctness, completeness, and maintainability.
- Fine-tune a pre-trained model for a specific task, such as sentiment analysis or text classification, and evaluate the performance of the model.
- Integrate an app generator into your development workflow and discuss the benefits and drawbacks of this approach.
Done looks like:
- A working CRUD application with a clear, concise evaluation of the generated code.
- A fine-tuned model with a clear, concise evaluation of its performance.
- A written discussion of the benefits and drawbacks of integrating an app generator into your development workflow.
Your tasks this week
Do the exercises above before the next chapter. Reading a tutorial and doing
one are different activities and only one of them changes what you can build.
Stuck on any of them? Say so — describe what you tried and what happened:
tell me where you got stuck. I read every one, and the questions
that come back more than twice get answered in the next chapter.
The AI tools panel
Chapter 3 of 12. New chapter every Sunday afternoon.
Next: Editor-native assistance.
· The full syllabus and every chapter so far
· Subscribers also get the condensed notes for this chapter, the running
recap of everything the series has covered, and the extended guidance:
subscribe
Written by Amit Chakraborty — founding engineer and senior architect: React Native, AI and RAG systems, production architecture. Portfolio · LinkedIn · GitHub.
Need this built, reviewed or taught to your team? Get in touch or email amit@devamit.co.in. Available for senior and founding engineering roles, consulting and training, remote worldwide.
Top comments (0)