I believe AI will be another service like the internet or a cell phone, and it's important to use it correctly by adding the right context, being aware of token usage, and following your own process.
For this reason some months ago I finished different courses about how to use Claude:
- A course with Ivan Davidov and a small contribution from Debbie O'Brien, on setting up agents with Playwright.
- The anthropic Claude courses
I checked the Addy Osmani Agent skills repo and checked his courses on linkedin.
And I am taking the Mosh Hamedani course Claude Code for Professional Developers and finished other claude skills course.
Also, in one of the jobs, I used skills developed by other QAs. I initially struggled with complex queries and generating API automation test cases due to the complexity of the user stories. But after some feedback from the agents and the user stories were clearer and with more context, like including the legacy stored procedure or checking the PR code, I got better results using the skills with GitHub copilot.
It's better to create your own agents with your rules and process. You need a framework with concrete coding rules and conventions, for your test cases.
For example, for test cases, I prefer critical user journeys with detailed steps and assertions in bullet points, rather than 10 tests that test a small part of the real user flow.
For automation frameworks, I like to follow these rules:
- Create components such as grid, combo, and calendar instead of helpers with that functions.
- The Page Object class only contains the elements with the components and general functions.
- On spec file I access the elements of the component like loginPage.loginButton.click() instead of create a LoginClick on the Page class.
- For the selectors I prefer getByRole because I think it is better for accessibility, and the user sees buttons and text instead of complex xPaths or data-test-ids.
- Add assertions that I can reuse in several tests on the page object or in the component.
- Wrap all actions into steps so the Playwright report can be read and understood by anyone on the team.
- Generate any data with FakerJS
How to setup Claude
Once you have the structure of your framework, it is important to generate CLAUDE.md using the /init command. This command will analyze the structure of your repo and create a file with the main structure and commands to run and execute the tests of your project, and each time you request something, Claude will read this file. You need to check the generated file and ensure that includes all the info.
With Claude, you can define
- Hooks: that will be executed before any action or after any action. For example, you can run ESLint after the file is saved.
- Skills: to reuse the same instructions, for example, to create manual test cases with some format. Before you can create commands but now are integrated into skills.
Create a hook to prevent access to .env file
The .env file, usually contains all the passwords. For security, it's better that the agent can't access it. The course Claude Code in Action includes an example that restricts access with JavaScript, but for practice I also created a hook to prevent access with a Python script.
To create the hook:
1) Create the scripts with the rule or command that you want to execute. I created a hooks folder with:
- read_hook.js: This JS script checks if the file is .env and restricts access, but allows access to .env.example because this is the example for others; qa can create the .env file.
- bash_env_check.py: This is a Python file with the same restriction, and I decided to check the same restriction with Python, but now to restrict reading the .env file with commands.
2) Add the hook in the .claude/settings.json file.
- Add a hooks key
- Add the event that triggers the hook. For example, the PreToolUse event is executed before each tool execution and can prevent the tool from executing any action.
- Add the matcher; in this case, it will be "Read|Edit" to prevent Claude from reading or editing the .env file
- Add the hooks key as:
type: command
command: "node ./hooks/read_hook.js"
statusMessage: "Checking file access..."
{
"permissions": {
"allow": ["mcp__playwright", "Bash(playwright-cli*)", "Bash(npx tsc --noEmit*)"],
"deny": []
},
"hooks": {
"PreToolUse": [
{
"matcher": "Read|Edit",
"hooks": [
{
"type": "command",
"command": "node ./hooks/read_hook.js",
"statusMessage": "Checking file access..."
}
]
},
"PostToolUse": []
}
}
If you request access you will see an error
Create a skill to create the manual test case from an Azure DevOps User story
I created the user story and a Figma design. I added only one screenshot without any interaction.
But usually Figma designs are like this, which include the web and mobile versions also other Figma designs include interaction
I am not a fan of BDD, so I didn't follow the BDD syntax for the user story. I've worked with and without BDD.
Description:
The admin user needs to register the different servers used to connect to the ERP to see some reports from the Firebird database.
Server Fields
Id - Server ID (numeric auto-increment)
Key – Server key (unique identifier)
Name – Server name
URL – URL to connect to the ERP API
Active – Whether the server is enabled
Scope: view the registered servers, search by name or key, add, edit, delete, and mark a server active/inactive.
In the attached file, you can find translations for English, Japanese, Spanish, and German.
Acceptance Criteria
Servers grid
- Admin can see a list of registered servers showing Key, Name, URL, and Active status.
- Admin can search the list; results filter as text is typed. The list supports pagination with page sizes of 10, 20, 50, and 100 rows.
- The list can be exported to Excel and PDF.
- When there are no servers, the list shows a "There are no servers registered yet. Click \"Add New\" to register the first one." message.
Add / Edit
- An admin can add a new server, edit an existing one, or delete it.
- Key, Name, and URL are all required; Save is disabled while the form is invalid. Key: number between 1 and 999999. Name: text, maximum 150 characters. URL: must start with http:// or https://, maximum 200 characters. -Key must be unique; uniqueness is validated server-side. Saving a duplicate Key returns a server error message, and the record is not saved. -A new server defaults to Active.
Delete / Active state
- Deleting a server requires confirmation; canceling the confirmation leaves the server unchanged.
- An admin can mark a server as active or inactive; inactive servers remain visible in the list with Active = No.
Navigation
- Clicking a server name opens that server's companies (another user story) Permissions
- Only authenticated admin users can access the servers screen and its actions.
- I attached a CSV file with the translations to the different languages.
While I was testing, I found some gaps in my user story and Figma design, like the missing confirmation before deleting a server and the missing empty state. So I requested Claude to add a comment.
It's important to review the user stories and create test cases at the start of the sprint to prevent gaps before the developer starts coding so the developer can generate unit tests and know the test cases can prevent bugs.
I prefer creating small skills for specific tasks rather than a general skill.
For manual test cases I created these skills:
- read-testcase: Read Azure DevOps test case by ID
- manual-test: To create the UI and API test cases, read the user story, ask for any gaps in the user story, and include a preview of the test before creating.
First you need to add the Azure DevOps MCP on Claude with an Entra Id account. And add your Azure DevOps Project
I created an entra Id user mcp_azure@abi1521hotmail.onmicrosoft.com, because you need to access with your Entra ID account instead of your personal email.
claude mcp add azure-devops \
-- npx -y @azure-devops/mcp your_organization \
--authentication interactive \
--tenant your_tenant_id
How to create a skill
1) Inside the .claude folder, create a skills folder
2) Add a folder with the name of the skill, e.g., read-testcase
3) Add a SKILL.md file
4) Add a references folder (optional) if you want to include a file with some references; for example, for the manual test skill, I created one file for API tests and another for E2E tests
5) Add a scripts folder (optional): Executable actions for a specific task.
The structure of the SKILLmd file is:
- name (required): the name of the skill should be the same as the folder
- description (required): Claude will read the skill's description to use automatically. The description should describe what the skill does and when it should be used.
- allowed-tools (optional): To restrict the tools Claude can use when the skill is active
I usually add some description about the role of the skill for example I want to create update the test cases for a user story or for playwright code, sometimes the code already exists but the manual tests are outdated.
---
name: manual-tests
description: Create and update manual Test Cases in Azure DevOps from a user story — API, E2E/UI, and performance types and wire them into a Test Plan/Suite. Use to write, generate, revise, or refresh manual test cases, e.g. "create test cases for story 123", "turn this Playwright spec into ADO test cases", "update test case 456 steps", "create test cases from this Figma design".
allowed-tools: >-
Read,
mcp__azure-devops__core_list_projects,
mcp__azure-devops__wit_get_work_item,
mcp__azure-devops__wit_update_work_item,
mcp__azure-devops__testplan_list_test_plans,
mcp__azure-devops__testplan_create_test_plan,
mcp__azure-devops__testplan_list_test_suites,
mcp__azure-devops__testplan_create_test_suite,
mcp__azure-devops__testplan_create_test_case,
mcp__azure-devops__testplan_add_test_cases_to_suite,
mcp__azure-devops__testplan_update_test_case_steps,
mcp__claude-in-chrome__tabs_context_mcp,
mcp__claude-in-chrome__tabs_create_mcp,
mcp__claude-in-chrome__navigate,
mcp__claude-in-chrome__computer,
mcp__claude-in-chrome__read_page
---
Act as a manual tester: turn a user story (or an automated test / feature description) into well-structured **manual** Test Case work items in Azure DevOps, and keep existing ones current when the story changes. Test cases come in three flavours — API, E2E/UI, and Performance — each with its own step template (see `references/`).
For the steps that I usually follow:
- Provide some error handling like confirm the Azure DevOps is connected.
- Include concrete code on the same step or as reference file.
- Minimize verbosity I prefer skills with less than 300 lines.
- Include a checklist to verify that follow all the rules that I setup.
- I usually request first the planned changes or the planned test cases
- Ask any doubt instead of guessing.
# Step 1: Verify the MCP is available and Test Plan Id
Before anything else, confirm the Azure DevOps MCP is connected (e.g. attempt a small read like `mcp__azure-devops__testplan_list_test_plans`). If it is not available, stop and show:
Azure DevOps MCP is not set up. Configure it in .mcp.json (set your org) and set
AZURE_DEVOPS_TOKEN in your shell, then restart Claude Code and try again.
**Determine the target project** — never hardcode it: use the project the user named; otherwise propose the `.mcp.json` default (`--project` arg) and confirm before writing; if neither is clear, list projects with `mcp__azure-devops__core_list_projects` and ask.
`testplan_*` and `wit_*` tools require an explicit `project` arg on every call, so resolve it once up front and reuse it. If a single request spans projects (e.g. a story in one project, test cases destined for another), confirm each.
**Determine the destination Plan/Suite** — ask the user whether cases go into an **existing** Plan/Suite or a **new** one, if they haven't said:
- **Existing** — `mcp__azure-devops__testplan_list_test_plans` (project) to find the named plan; `mcp__azure-devops__testplan_list_test_suites` (project, planId) to find the named suite, or use the root suite (`parentSuite` empty/itself) if they don't want a sub-suite.
- **New** — create the plan with `mcp__azure-devops__testplan_create_test_plan` (project, name, iteration — default to the project root iteration unless they name a sprint), then the suite with `mcp__azure-devops__testplan_create_test_suite` (project, planId, `parentSuiteId` = root suite id, name). Confirm both names before creating.
Capture `planId` and `suiteId` — needed later when the cases are created and added to the suite.
**Find the test case (update mode)** — if the user gave a test case ID, use it directly; if they gave a story ID and asked to refresh its cases, fetch the story with `mcp__azure-devops__wit_get_work_item` (`expand: "all"`) and follow its `Microsoft.VSTS.Common.TestedBy` links to the linked test cases.
Checklist Example
## Checklist
Run through this silently before presenting the draft/diff for approval, and once more before any write call — it's a final gate, not documentation.
- [ ] Mode (create vs. update) chosen; for updates, current steps read fresh before drafting changes
- [ ] Test type chosen and the matching `references/*.md` template followed
- [ ] Steps derived from real source (story fields / actual spec file / stated facts) — zero invented endpoints, fields, or thresholds
- [ ] Figma link in the story spotted and the design opened and inspected before drafting (E2E create mode); design-quoted text marked High confidence
- [ ] Open doubts (unspecified texts, behaviors, edge cases) batched and asked before drafting; assumptions only after the user deferred or couldn't answer
- [ ] No extra `|` inside any action/expected-result text — `|` is the tool-input delimiter (Step 5), not part of the preview's `→` format
- [ ] No assertion-only "Observe/Inspect/Verify" steps; multi-assertion expected results written as bullets (steps XML via `wit_update_work_item`)
- [ ] Journeys are positive-scenario only, negative scenarios split into their own (parameterized) cases; cleanup postcondition present when the case creates data
- [ ] Destination plan + suite resolved (plan/suite created on request when none exists) — create mode
- [ ] Draft (create) or before→after diff (update) presented and user approved before any write
I created other skills to create the automated playwright tests some of the skills are:
- read-testcase: To read the test case on Azure DevOps
- check-auth: Check if the file that stores the token is created on the .auth folder. This is created on the global setup to allow Claude check the website.
- page-snapshot: To get the current html to the website to get the locators the file instead of guessing.
- playwright-component: To create or update any new component for example if the website to test includes a DatePicker can create a DatePicker component
- playwright-page: To create the page(s) required for the test.
- playwright-automation: To create the test case with playwright.
- playwright-cli: This is the official playwright skill that explains how to automate the browser.
- update-packages: To update the npm packages to the latest version and check any breaking changes.
I'm still working on that but you can see the repo with the skills: https://github.com/apis3445/TestingDojo
Create claude actions .yml files to tag claude on issues or review your PR.
You can include a .yml file to allow Claude to tag your GitHub issues or to review your PR automatically.
You need to
- Give Claude access to your GitHub Repo
- Upload a .yml file with the action that you want Claude to complete. You can see the different yml files on their Claude Code action repo.
One disadvantage I encountered is that when I was trying to use the .yml to review the PR, if you change the .yml for the code review, you will get an error that the file needs to be the same as the main branch, but you can add a comment @claude review to get the PR review comment.
The PR review is part of your Claude usage. In my experience at one of my jobs, some suggestions weren't good. The review suggested changing code that will be fixed in other user stories, or that flagged code as incorrect due to a lack of context, even though the context was in the PR description. You can reply with suggestions, for example: @claude, I fixed it, and Claude will check again.
But I prefer Qodo, which is focused on PR review. You can see both PR reviews here: https://github.com/apis3445/TestingDojo/pull/15
Now the disadvantage that I found:
- LLM context memory is limited and needs to load all the info every time because if you restore one session, there is a limit and it needs to compress the context.
- The Claude Skilljar free courses are outdated; for example, one course explains commands, but commands were integrated into skills, or the PR review changed.
- LLMs don't say 'I don't know' — they guess and generate code like the correct solution, and if it's wrong, they try the same idea instead of asking. Sometimes I get better results after doing manual research.
- And with large files at least Copilot deleted all the code trying to add one code change, so with larges files around 2,300 lines struggle to update content.
You can use the mcp context7 to get up-to-date docs for AI agents and for example for angular, google added claude skills but there is still a gap to LLM don't write deprecated code.
Create skill or automate your flows require continuous feedback so it's important that improve the skills and be aware of the token usage.
Thanks for reading so don't hesitate to ask questions or share your feedback. If this helped you, feel free to share it with the community.
Top comments (3)
Using an agent for test generation becomes much safer when the generated tests are classified by purpose: regression, boundary, mutation-killing, or contract coverage. I would keep the failing seed and the repository revision with every generated case, so an apparently useful test can be reproduced after the implementation changes.
Hi! Thanks for your feedback. I will add the tag to the next version of the skills. Another of my ideas was also tag for unit, integration, e2e, performance test.
I believe agents and skills are iterative processes, and I will continue to improve the generated test cases. You always need to check the test cases and the generated code by AI. For tracking I'm linking the test cases to the user story, but I didn't log them with the skill, so I think it's a good idea that a skill change can change the output and track that changes.
Another useful skill/agent could be to check the current test cases against the software's current version to identify gaps or confirm whether tests are failing due to bugs or code changes.
Thank you for sharing such an excellent post. I really enjoyed reading it.
I’m a Python Full-Stack Engineer with over 10 years of experience designing and building scalable software solutions for clients across a variety of industries. Along the way, I’ve learned that successful projects depend not only on strong technical execution but also on creating real business value.
With my recent contract completed, I’m exploring new opportunities to collaborate with professionals who value innovation, practical problem-solving, and long-term partnerships. I enjoy discussing ideas that combine technical excellence with sound business strategy, creating outcomes that benefit everyone involved.
I believe every connection has the potential to become something meaningful. If you're interested in exchanging ideas, exploring opportunities, or simply connecting with someone who enjoys building impactful technology, I'd be happy to hear from you.
Wishing you success in your future endeavors, and I look forward to connecting.