DEV Community: artshllaku

Your Tests Are Passing. That Doesn't Mean They're Good

artshllaku — Fri, 13 Mar 2026 23:21:36 +0000

I've worked on projects with 85% test coverage where bugs still made it to production every week. The team would look at the coverage report, see green, and feel safe. But the tests were lying.

Here's what was actually happening:

test('should process user data', async () => { const result = await processUser(mockData); expect(result).toBeTruthy(); });

This test "covers" the processUser function. Coverage goes up. CI is green. But what does it actually prove? That the function returns something that isn't null, undefined, 0, or false. That's it. The function could return completely wrong data and this test would still pass.

Or this one — common in E2E tests:

test('user signup flow', async ({ page }) => { await page.goto('/signup'); await page.fill('#email', 'test@test.com'); await page.fill('#password', '12345678'); await page.click('button[type="submit"]'); await page.waitForURL('/dashboard'); });

It walks through the whole signup flow but never checks if the user was actually created, if the right page content loaded, or if the form shows errors for bad input. It checks that a redirect happened. That's all.

Coverage tools can't catch this. Coverage measures which lines of code ran during your test. It doesn't care if you actually verified anything.

What's the difference between a good test and a bad one?

A bad test runs your code and checks almost nothing. A good test runs your code and verifies that specific things happened correctly.

Bad:

expect(response).toBeTruthy(); expect(user).toBeDefined();

Good:

expect(response.status).toBe(200); expect(user.email).toBe('test@test.com'); expect(user.role).toBe('admin'); expect(user.createdAt).toBeInstanceOf(Date);

Bad tests give you confidence that isn't real. You think your code works because tests pass. Then a customer reports a bug that your test suite should have caught — but didn't, because the assertions were too weak.

The real metrics that matter

Instead of asking "how much of my code is tested?", start asking:

How many assertions does each test have? A test with one weak assertion is barely a test.
What kind of matchers am I using? toBeTruthy() is almost always the wrong choice. toBe(), toEqual(), toContain() actually verify values.
Am I testing edge cases? The happy path working doesn't mean your code handles errors, empty inputs, or boundary values.
Am I testing behavior or implementation? Tests that break when you refactor are testing the wrong thing.

I built a tool to automate this

I got tired of reviewing PRs and manually pointing out weak tests, so I built gapix — a CLI tool that analyzes your test quality automatically.

It parses your test files using AST analysis, looks at every single assertion, categorizes your matchers, and gives each test file a quality score from 0 to 100.

npx @artshllaku/gapix analyze ./src

What it checks:

Assertion density — tests with zero or only one assertion get flagged
Matcher strength — using toBeTruthy() where you could check actual values
Edge case coverage — whether you're testing more than just the happy path
Assertion categories — equality checks, mock verifications, error handling, DOM assertions
Overall quality grade — Poor, Fair, Good, or Excellent for each file

It generates an interactive HTML report you can open in the browser:

npx @artshllaku/gapix show-report

You get a dark-themed dashboard showing every test file, its score, individual findings, and suggestions for improvement.

It works with any framework

Jest, Vitest, Playwright, Cypress - doesn't matter. If your tests are written in TypeScript or JavaScript, gapix can analyze them. It reads the AST, not framework-specific APIs.

Optional AI analysis

If you want deeper analysis, you can connect an AI provider (OpenAI or Ollama) and gapix will give you context-aware suggestions — not generic advice, but specific findings based on your actual code and what your tests are checking.

npx @artshllaku/gapix set-provider openai npx @artshllaku/gapix set-key sk-your-key npx @artshllaku/gapix analyze ./src

Without AI, it still runs full rule-based analysis using AST parsing. The AI just adds an extra layer.

Why this matters more than coverage

I've seen this pattern too many times:

Team sets a coverage threshold (80%)
Developers write tests to hit the number
Tests become a checkbox, not a safety net
Bugs get through because tests don't verify real behavior
Team loses trust in the test suite
People stop writing tests or start skipping them

The fix isn't more tests. It's better tests. One well-written test with strong assertions is worth more than ten tests that just call functions and check toBeTruthy().

Get started

`# Run it once without installing
npx @artshllaku/gapix analyze ./src

Or install globally

npm i -g @artshllaku/gapix
gapix analyze ./src
It's free, open source, and takes about 30 seconds to get your first report.`

GitHub: https://github.com/artshllk/gapix

I'd love your feedback. If you try it on your codebase and something doesn't work right or the suggestions aren't helpful, open an issue. I'm actively working on this and want it to be useful for real-world projects.

Master Mobile Web Testing with Playwright: A Beginner’s Guide

artshllaku — Wed, 19 Mar 2025 01:36:23 +0000

Playwright is a powerful tool for automating browser testing, and it’s not just limited to desktop browsers. With Playwright, you can also test your web applications on mobile devices. Let me explain.

What Can Playwright Do for Mobile Testing?
Playwright allows you to emulate real mobile devices like smartphones and tablets. This means you can test how your website or web app behaves on different devices without needing the physical hardware. You can configure Playwright to simulate various device-specific behaviors, such as:

Screen size and viewport: Test how your site looks on different screen resolutions.

User agent: Match the browser details of specific devices.

Touch support: Simulate touch interactions for mobile devices.

Geolocation, locale, and timezone: Test location-based features or language-specific content.

Permissions: Check how your app handles permissions like notifications or camera access.

Color scheme: Test dark mode or light mode compatibility.

While Playwright is a very flexible and powerful tool, it’s important to note that it only works for mobile websites, not native mobile apps. If you’re looking to test native apps, you’ll need tools like Appium. But for responsive web design and mobile web testing, Playwright is a fantastic choice.

How to Emulate Mobile Devices in Playwright

Playwright comes with a wide range of pre-configured device profiles that you can use right out of the box. These profiles include popular devices like iPhones, iPads, and Android phones. You can find the full list of supported devices in Playwright’s official GitHub repository:
Device Descriptors Source.

Here’s a quick example of how to set up mobile emulation in your Playwright configuration:

const { chromium, devices } = require(‘playwright’);
const iPhone = devices[‘iPhone 16 Pro’];

(async () => {
 const browser = await chromium.launch();
 const context = await browser.newContext({
 …iPhone, // Emulates iPhone 16 Pro
 });
 const page = await context.newPage();
 await page.goto('https://yourwebsite.com');
 // Test code
 await browser.close();
})();

You can also specify the viewport and other settings manually if you don’t want to use a pre-configured device profile. For example:

const context = await browser.newContext({
 viewport: { width: 402, height: 874 }, // iPhone 16 Pro dimensions
 userAgent: 'Mozilla/5.0 (iPhone; CPU iPhone OS 17_0 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.0 Mobile/15E148 Safari/604.1',
 hasTouch: true,
});

Challenges and Limitations

While Playwright is a great tool, there are a few challenges you might face:

No Native App Support: As mentioned earlier, Playwright can’t test native mobile apps. It’s strictly for web applications.

Real Device Behavior: Emulation is great, but it’s not the same as testing on a real device. Some behaviors, like performance or specific hardware interactions, might differ.

Dynamic Content: If your website has dynamic content (e.g., ads or animations), it might behave differently on emulated devices compared to real ones.

Why Use Playwright for Mobile Testing?

Despite its limitations, Playwright is a powerful tool for developers and testers who want to:

Ensure their website is responsive and works well on all screen sizes.

Test cross-browser compatibility on mobile devices.

Automate repetitive testing tasks and save time.

Simulate complex user interactions, such as touch gestures or geolocation-based features.

Final Thoughts

Playwright is a powerful and easy-to-use tool for testing mobile websites. While it doesn’t replace testing on real devices completely, it’s a great way to find and fix responsiveness problems and make sure your website looks good on different screen sizes. If you’re already using Playwright for desktop testing, adding mobile emulation to your tests is a simple and smart choice

Give it a try, and let me know how it works for you! If you’ve faced any challenges or have tips to share, feel free to leave a comment below.

Best E2E Automation Testing Practices

artshllaku — Wed, 27 Nov 2024 12:39:45 +0000

This article shares insights and best practices for end-to-end (E2E) testing based on my hands-on experience. I started with minimal knowledge in this field, but over time, I learned the importance of building robust, reliable tests. Facing challenges like flaky tests and unstable pipelines taught me valuable lessons. My goal here is to go beyond the basics and offer strategies that reduce test maintenance, improve stability, and enhance readability in complex projects.

Instead of reiterating what's already covered in official documentation, this guide focuses on practical techniques I've applied successfully in real-world projects. If you're new to E2E testing or want to deepen your understanding, I recommend exploring these resources alongside my experiences:

Official Cypress Best Practices Guide
Official Playwright Best Practices Guide

Defining Test Purpose and Scope

One of the first lessons I learned was the importance of clarity when starting a test. Ask yourself:

What functionality am I testing?
What is the expected outcome?
What are the test's boundaries?

For example, when verifying an e-commerce application's checkout flow, define whether you're testing the ability to complete a purchase, inventory updates, or order confirmation emails. Narrowing your scope prevents unnecessary interactions and keeps tests focused.

Examples
Well-Defined Test Purpose: Test login functionality using valid credentials and verify successful redirection.

Scope Control: Skip database checks if the goal is purely to validate UI behavior.

Using TypeScript for Stronger Tests

Early on, I worked with JavaScript in my tests, but as my projects grew, I realized the benefits of TypeScript. Its type safety and IDE support significantly improve test maintainability by catching errors during development and enhancing code readability.

Here’s a simple example:

interface UserCredentials {
  username: string;
  password: string;
}

const login = ({ username, password }: UserCredentials) => {
  cy.get('[data-testid="username"]').type(username);
  cy.get('[data-testid="password"]').type(password);
  cy.get('[data-testid="login-button"]').click();
};

Using TypeScript ensures that my test inputs are always valid, especially in complex flows involving API responses or structured data. This consistency has saved me hours of debugging.

Writing Readable Tests

Another lesson I learned the hard way is that tests need to be clear and intuitive for anyone on the team, not just developers. Avoid embedding unnecessary logic and focus on leveraging framework-specific syntax for simplicity.

Examples
❌ Complex Logic:

cy.get('.items').then(($items) => {
  Array.from($items).forEach(item => {
    if (item.innerText.includes('Special')) {
      cy.wrap(item).click();
    }
  });
});

✅ Framework Features:

cy.get('.items')
  .contains('Special')
  .click();

The second approach is not only cleaner but also leverages Cypress features, reducing the chances of flakiness due to minor UI changes.

Integrating E2E Tests with GitHub Actions

One of my most impactful contributions was automating E2E tests in the CI/CD pipeline using GitHub Actions. This ensures tests run with every push or pull request, catching issues early.

Here’s an example of a workflow I’ve used:

name: CI

on:
  push:
    branches:
      - main
  pull_request:
    branches:
      - main

jobs:
  e2e:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v2

      - name: Set up Node.js
        uses: actions/setup-node@v2
        with:
          node-version: '20'

      - name: Install dependencies
        run: npm install

      - name: Run E2E Tests
        run: npm run test:e2e

This workflow has helped maintain code quality while fostering a collaborative culture of continuous improvement.

Reducing Flakiness in Tests

Flaky tests can be a nightmare. I've spent a good part of my career dealing with them, and here are some strategies that worked for me:

Avoid Overlapping Tests: Isolate execution contexts using before and after hooks to set up and tear down test data.
Keep Tests Small and Focused: Testing a single functionality per test simplifies debugging and reduces complexity.
Regular Reviews: Periodically refactor flaky tests and align them with current application behavior.

Example:

cy.intercept('POST', '/api/checkout', { statusCode: 200, body: { order: '12345' } });
cy.get('[data-testid="checkout-button"]').click();
cy.get('[data-testid="order-confirmation"]').should('contain', 'Order 12345');

Stubbing network requests like this has been key in controlling external dependencies and reducing test failures.

By implementing these practices, I’ve significantly improved test reliability and maintainability in my projects. While advanced E2E testing requires balancing real-world interactions with stable test design, these lessons have been invaluable in my journey. I hope they help you too!

WET vs. DRY: Testing Principles You Should Know

artshllaku — Sun, 24 Nov 2024 10:48:34 +0000

In software development, writing clear and maintainable tests is as crucial as writing the code itself. Two commonly discussed principles in this context are WET (Write Everything Twice) and DRY (Don’t Repeat Yourself).

These principles help guide how we structure tests, balancing readability, maintainability, and efficiency. Let’s dive into what they mean, explore examples, and understand when to apply each approach.

📝 What is WET Testing?

WET testing is a style where repetition in test cases is allowed. While often seen as less ideal, this approach can prioritize simplicity and clarity—particularly for straightforward tests.

Pros of WET Tests:

Simplicity: Easy to read and understand, especially for newcomers.
Isolation: Each test stands on its own, avoiding dependencies.
Quick to Write: Ideal for smaller projects or simpler scenarios.

Example of WET Testing:

describe('Login Tests - WET', () => {
  test('should allow user to login with valid credentials', async () => {
    await page.goto('https://example.com/login');
    await page.fill('input[name="username"]', 'user1');
    await page.fill('input[name="password"]', 'password1');
    await page.click('button[type="submit"]');
    await expect(page).toHaveURL('https://example.com/dashboard');
  });

  test('should show an error with invalid credentials', async () => {
    await page.goto('https://example.com/login');
    await page.fill('input[name="username"]', 'user1');
    await page.fill('input[name="password"]', 'wrongpassword');
    await page.click('button[type="submit"]');
    await expect(page).toHaveText('Invalid username or password');
  });
});

In this example, the login steps are repeated across tests.

✨ What is DRY Testing?

DRY testing focuses on minimizing redundancy by abstracting shared logic into reusable functions or setups. This approach shines in complex or large projects.

Pros of DRY Tests:

Reduced Redundancy: Centralizes logic, avoiding repetition.
Ease of Maintenance: Changes only need to be made in one place.
Cleaner Code: Focuses tests on behavior rather than setup.

Example of DRY Testing:

describe('Login Tests - DRY', () => {
  const login = async (username, password) => {
    await page.goto('https://example.com/login');
    await page.fill('input[name="username"]', username);
    await page.fill('input[name="password"]', password);
    await page.click('button[type="submit"]');
  };

  test('should allow user to login with valid credentials', async () => {
    await login('user1', 'password1');
    await expect(page).toHaveURL('https://example.com/dashboard');
  });

  test('should show an error with invalid credentials', async () => {
    await login('user1', 'wrongpassword');
    await expect(page).toHaveText('Invalid username or password');
  });
});

Here, the login function centralizes the shared steps, making the tests cleaner and easier to maintain.

💡 When to Use WET vs. DRY?

From personal experience, choosing between WET and DRY depends on your project’s complexity and requirements.

Use WET when:

Your tests are simple and isolated.
The code is unlikely to change frequently.
You prioritize clarity over abstraction.

Use DRY when:

You have repeated logic across multiple tests.
The codebase is large and maintainability is a concern.
You need to refactor tests for efficiency.

🔑 Key Takeaways
While the DRY principle is generally preferred, WET tests have their place. Strive for a balance that enhances both clarity and maintainability. For smaller projects or straightforward scenarios, a WET approach might suffice. However, in larger, more complex test suites, adopting DRY can significantly improve your workflow.

Ultimately, the goal is to write tests that are clear, maintainable, and efficient—whatever approach gets you there!