I took part in a project that had a two years old codebase and used Symfony 3.4 as the web framework. It was not the newest and shiniest project but it had a one big advantage: there were tests covering most of the critical actions in the application.
For the first time in my career I have witnessed how tests give you confidence over a codebase, start saving you time and help you tackle the business requirements.
Project achieved this with many Symfony functional tests and some unit tests that cover the gaps in between. Total coverage was around 50-52% but critical features coverage was significantly higher and provided enough confidence to add new features without manually testing.
For those who are coming from an another ecosystem, let me explain what the functional tests in Symfony world mean.
In Symfony documentation, functionals tests are defined like this:
“Functional tests check the integration of the different layers of an application (from the routing to the views).“
They are basically end-to-end tests where you write code that sends an HTTP request to the application. You receive an HTTP response from the app and you do assertions based on that response. However, you do have power to go to the database and assert the changes in the persistance layer which sometimes gives an extra opportunity to check the state.
I would like to share some of the tips I have accumulated working with Symfony writing functional tests.
1) Testing with the persistence layer
Probably the first thing you want to achieve while running your functional tests, is to seperate your testing database from your development database. This is done to create a clean slate for your test suite to run. It will allow you to control and create a desired state of the app for your test cases. Also, having a test suite write random data to your development copy of the database is not ideal.
This generally is achieved by making your Symfony app connect to a different database while running the test suite. Using Symfony 4 or 5, you can define environment variables that are going to be used while testing in a .env.test file. You should also configure PHPUnit to change the environment variable APP_ENV to test. Luckily this configuration is done by default when you install Symfony PHPUnit Bridge Component.
For Symfony versions running below 4, you can rely on kernel booting in test mode while running the functional tests. That way, you can define your test specific configation within config_test.yml files.
2) LiipFunctionalTestBundle
This bundle packs some functional test helpers to write Symfony tests. Sometimes it tries to do too much and gets in your way, but generally it is pleasent to have some things streamlined for writing tests.
For example, during testing you can simulate logins with a user, load data fixtures, count database queries to test performance regresssions and etc. I would consider installing this bundle when starting testing a new Symfony application.
3) Flushing the database with every test
Another important topic to cover is when to reset the database during testing. Symfony testing tools does not give you an opinion about this issue but I generally like to refresh the database after each test method. This basically makes the whole test suite look like this:
<?php
namespace Tests;
use Tests\BaseTestCase;
class SomeControllerTest extends TestCase
{
public function test_a_registered_user_can_login()
{
// Clean slate. Database is empty.
// Create your world. Create users, roles and data.
// Execute logic.
// Assert the outcome.
// Database is reset.
}
}
I find that loading empty fixtures in PHPUnit’s special setUp method is a great way of flushing the database. If you have installed the previously mentioned LiipFunctionalTestBundle, you can install empty fixtures to flush the database.
<?php
namespace Tests;
class BaseTestCase extends PHPUnit_Test_Case
{
public function setUp()
{
$this->loadFixtures([]);
}
}
4) Creating Data
Starting every test with an empty database, means you have to have some utilities or helpers to create data for testing. These can be creating database model or entity objects.
Laravel has a very streamlined method with model factories. I try to follow a similar approach and create some traits that creates objects that I frequently use in my tests. For example this is a simple trait that creates User entities.
<?php
namespace Tests\Helpers;
use AppBundle\Entity\User;
trait CreatesUsers
{
public function makeUser(): User
{
$user = new User();
$user->setEmail($this->faker->email);
$user->setFirstName($this->faker->firstName);
$user->setLastName($this->faker->lastName);
$user->setRoles([User::ROLE_USER]);
$user->setBio($this->faker->paragraph);
return $user;
}
}
I can then include these traits in TestCases I want:
<?php
namespace Tests;
use Tests\BaseTestCase;
use Tests\Helpers\CreatesUsers;
class SomeControllerTest extends TestCase
{
use CreatesUsers;
public function test_a_registered_user_can_login()
{
$user = $this->createUser();
// Login as user. Do some tests.
}
}
5) Swap services in a container
While very easy to do so in a Laravel application, swapping services in the container is a bit tricky when it comes to the Symfony projects. Services in the container are marked private between Symfony 3.4 and 4.1 releases. This means, while writing your tests, you simply cannot access the service from the container and you cannot set another (mock) service.
While some developers state that during functional tests you should not mock anything, in practice there may be situations where you don’t have a sandbox environment for a 3rd party service and you don’t want to feed these 3rd party services with random test data.
Luckily as of Symfony 4.1, during testing Symfony will allow you to access the container and change services as you wish.
You may consider this approach:
<?php
namespace Tests\AppBundle;
use AppBundle\Payment\PaymentProcessorClient;
use Tests\BaseTestCase;
class PaymentControllerTest extends BaseTestCase
{
public function test_a_user_can_purchase_product()
{
$paymentProcessorClient = $this->createMock(PaymentProcessorClient::class);
$paymentProcessorClient->expects($this->once())
->method('purchase')
->willReturn($successResponse);
// this is a hack to make the container use the mocked instance after the redirects
$client->disableReboot();
$client->getContainer()->set(PaymentProcessorClient::class, $paymentProcessorClient)
}
}
But please do note, during functional testing Symfony kernel may boot up couple of times in a single test, basically rebuilding all the dependencies and leaving your mocked service left out. This may be tricky to spot first and I have yet to find an elegant solution to it.
The hacky way I have come up with is to disable kernel reboot and make sure the same app kernel instance is used during the litetime of that test method. This will ensure that the kernel will not be recompiled and mocked service will not be lost. Make sure this does not create more side effects.
6) Running SQLite in memory
It is very common to use sqlite as the persistence layer during testing since it is very portable and very easy to setup. These qualities also make it very suitable for using in CI/CD environment during testing.
SQLite is serverless, meaning that the SQLite program will write and read all the data from a file.
Reading and writing from a file will likely to be come a performance bottleneck during testing as it is an extra I/O operation that you code will have to wait. For that reason, you may use the in-memory option for the SQLite. This will write all the data to memory and hopefully make the operations faster.
While configuring your database in your Symfony app, instead of pointing to a ‘database.sqlite’ file, it is enough to feed it with ‘:memory:’ keyword.
7) Running SQLite in memory with tmpfs
Using the in memory option is great but I had a very hard time configuring the in memory option under the old version of the LiipFunctionalTestBundle.
If you face a similar sitation here is a trick.
On a Linux system, you can allocate a space in RAM that behaves as a normal file storage space. This feature is called tmpfs. Basically you can create a tmpfs folder, put your sqlite database file there and then use that to run your tests on.
You can also follow a similar approach with MySQL testing, however that may be a little complicated to setup.
8) Testing the Elasticsearch layer
Just like connecting to a testing database instance you can connect to a test Elasticsearch instance as well. Or better yet, you can use different index names during testing have a testing/development environment seperation that way.
While looking easy, testing Elasticsearch can be a struggle in practice. We have strong tools to generate database schemas, create fixtures, populate the database with test data easily. These tools may not exist when it comes down to Elasticsearch and you may have to create your own solutions. It may not be straightforward to start testing right away.
There are also a problem of indexing new data and that data being available. A commong gotcha is the Elasticsearch’s refresh interval. Generally indexed documents become avaiable for search after the Elasticsearch’s configurable refresh interval. The default value is 1 second and it may trip up your tests randomly if you are not careful.
9) Using Xdebug filters to speed up coverage report
Coverage is an important asset when it comes to testing. It should not be treated as just a number, but a way to find untested branches and flows in your code.
Generating coverage info is generally handled by Xdebug.
You’ll notice that running the coverage analysis, generally causes your test suite speed to crumble. This may be a problem in CI/CD environment where each minute is billed.
Luckily there are some optimizations to be made. When Xdebug generates coverage info for the running tests, it generates coverage info for the every PHP file run in that test. That includes the PHP files residing in the vendor folder, a code that we don’t own.
Using the Xdebug code coverage filter configuration we can prevent Xdebug to generate coverage for the files that we don’t want and same some significant time.
How do we generate that filter? Well. PHPUnit can do that for us. You just have to a single to command to create a filter configuration file:
phpunit --dump-xdebug-filter build/xdebug-filter.php
And feed that configuration file when you run your tests:
phpunit --prepend build/xdebug-filter.php --coverage-html build/coverage-report
10) Paralel testing tools
Running functional tests can take up significant time. As an example, a test suite with 77 tests and 524 assertions can take up to 3-4 minutes to fully run. This pretty normal considerin with each test you are making bunch of database calls, rendering twig templates, running the crawler on those templates and do some assertions.
If you open up the activity monitor, you can see that while running tests only single core of your computer is utilized.
Some paralel testing tools such as paratest and fastest have come out to use more than single core to execute your tests. While running unit tests with these tools are straightforward, running tests that connect to a database may need some tweaking.
I was able to utilize 6 cores and 12 threads on my local dev machine to bring down 4 minute test suite times to about 25-30 seconds.
Edit: Some very helpful comments appeared on r/php when I shared this article. Click here to read more tricks and tools.
Top comments (2)
Thanks for taking the time to document your experience! I've recently had to move from symfony 2.x to 4.x and really wish someone had written these tests 😭 . This post motivates me to write tests haha
Great article! Will make a great reference for me.