DEV Community: Ethan Bray

Streaming Buffered PHP Output in Zend Framework

Ethan Bray — Sun, 11 Aug 2019 14:47:30 +0000

I was recently working on a feature that required me to retrieve large amounts of data (7 figure document counts) from Elasticsearch, modify the data and then output a CSV file. This feature needed to be performant and not have an adverse effect on the memory usage of the application's EC2 instance.

An original implementation of the feature retrieved all documents, stored them in memory and then created a file on disk that would be served to the user. This resulted in memory exhaustion, as well as taking up large amounts of disk space on production instances.

This pseudocode represents the initial implementation we had:

class ExportController() {
    public function exportAction(): Stream {
        $exportQuery = $this->params()->fromPost('query');
        $this->dataExporter->export($exportQuery);

        $response        = $this->getResponse();
        $responseHeaders = $response
            ->getHeaders()
            ->addHeaders([
                'Content-Description'       => 'File Transfer',
                'Content-Type'              => 'text/csv',
                'Content-Transfer-Encoding' => 'binary',
                'Expires'                   => 0,
                'Cache-Control'             => 'must-revalidate',
                'Pragma'                    => 'public',
            ]);
        $response->setHeaders($responseHeaders);
        $response->setContent($fileContents);
        return $response;
    }
}

class DataExporter {
    public function export($exportQuery)
    {
        // Get all of the results
        $results = $this->elasticsearchService->getData($exportQuery);

        // Iterate through data and output to CSV file
        $filePointer = fopen('/tmp/data.csv', 'wb');
        foreach ($results as $result) {
            fputcsv($filePointer, $result, ',');
        }
    }
}

fputcsv takes a resource as its first parameter, including a resource that points to a PHP I/O stream such as php://output. By replacing /tmp/data.csv with php://output we removed our need to create CSV files on the server.

Now that we're writing the data straight to the output stream, we need to send the correct headers before we flush the buffer:

class ExportController() {
    public function exportAction(): Stream {
        $exportQuery = $this->params()->fromPost('query');
        $this->dataExporter->export($exportQuery);
    }
}

class DataExporter {
    public function export($exportQuery): void
    {
        // Get all of the results
        $results = $this->elasticsearchService->getData($exportQuery);

        // Iterate through data and output to CSV file
        ob_start();
        $outputPointer = fopen('php://output', 'wb');
        foreach ($results as $result) {
            fputcsv($outputPointer, $result, ',');
        }

        // Close our file pointer, send the file headers and flush our output buffer
        fclose($filePointer);
        header('Content-Description: File Transfer');
        header('Content-Type: text/csv');
        header('Content-Transfer-Encoding: binary');
        header('Expires: 0');
        header('Cache-Control: must-revalidate');
        header('Pragma: public');
        ob_end_flush();
        exit();
    }
}

We use exit() to avoid a headers already sent error as Zend Framework will try to send a response, and more headers, to the user.

If we were to write a test for the export method, we'd swiftly run into the issue where our usage of exit() ends our PHPUnit process. Short of monkey patching the exit() function, there's not much we can do about this. Other than removing exit() that is. Instead of flushing the buffer in our export method, we can instead create a Stream response and return that to our controller. The controller will then return the Stream to Zend Framework which will send it to our user.

class ExportController() {
    public function exportAction(): Stream {
        $exportQuery = $this->params()->fromPost('query');
        return $this->dataExporter->export($exportQuery);
    }
}

class DataExporter {
    public function export($exportQuery): Stream
    {
        // Get all of the results
        $results = $this->elasticsearchService->getData($exportQuery);

        // Create our Stream
        $outputPointer = fopen('php://output', 'wb');
        $stream = new Stream();
        $stream->setStream($outputPointer);
        $stream->setHeaders((new Headers())->addHeaders([
            'Content-Description'       => 'File Transfer',
            'Content-Type'              => 'text/csv',
            'Content-Transfer-Encoding' => 'binary',
            'Expires'                   => '0',
            'Cache-Control'             => 'must-revalidate',
            'Pragma'                    => 'public',
        ]);

        // Iterate through data and output to CSV file
        foreach ($results as $result) {
            fputcsv($outputPointer, $result, ',');
        }

        return $stream;
    }
}

When a response is returned by a controller, Zend MVC emits a SendResponseEvent event that is listened to by several Zend Framework listeners. A listener is available for each type of response we may return. In our case, the StreamResponseSender will be responsible for sending our response onwards.

Zend\Mvc\SendResponseListener\PhpEnvironmentResponseSender with a priority of -1000
Zend\Mvc\SendResponseListener\ConsoleResponseSender with a priority of -2000
Zend\Mvc\SendResponseListener\SimpleStreamResponseSender with a priority of -3000

class SimpleStreamResponseSender extends AbstractResponseSender
{
    public function sendStream(SendResponseEvent $event)
    {
        if ($event->contentSent()) {
            return $this;
        }
        $response = $event->getResponse();
        $stream   = $response->getStream();
        fpassthru($stream);
        $event->setContentSent();
    }

    public function __invoke(SendResponseEvent $event)
    {
        $response = $event->getResponse();
        if (! $response instanceof Stream) {
            return $this;
        }
        $this->sendHeaders($event);
        $this->sendStream($event);
        $event->stopPropagation(true);
        return $this;
    }
}

Now unfortunately, fpassthru does not support resources using php://output. Attempting to use the StreamResponseSender will result in an error about an invalid resource. Replacing fpassthru with ob_end_flush is all that is required to make the SimpleStreamResponseSender work.

Now we'll need to create our own response sender to handle these Stream responses using php://output. We also need to differentiate our responses using php://output from a standard Stream response.

class BufferedStream extends Stream {
    public function __construct($filePointer)
    {
        $this->setHeaders((new Headers())->addHeaders([
            'Content-Description'       => 'File Transfer',
            'Content-Type'              => 'text/csv',
            'Content-Transfer-Encoding' => 'binary',
            'Expires'                   => '0',
            'Cache-Control'             => 'must-revalidate',
            'Pragma'                    => 'public',
        ]));
    }
}

class BufferedStreamResponseSender extends AbstractResponseSender
{
    public function sendStream(SendResponseEvent $event): self
    {
        if ($event->contentSent()) {
            return $this;
        }

        // fpassthru does not work with php://output so we have to flush the buffer instead
        ob_end_flush();
        $event->setContentSent();
        return $this;
    }

    public function __invoke(SendResponseEvent $event)
    {
        $response = $event->getResponse();
        if (! $response instanceof BufferedStream) {
            return $this;
        }

        $this->sendHeaders($event);
        $this->sendStream($event);
        $event->stopPropagation(true);
        return $this;
    }
}

Now we have our listener and our new BufferedStream, we simply need to update our DataExporter to use the correct response and attach our listener to the SendResponseEvent.

class ExportController() {
    public function exportAction(): BufferedStream {
        $exportQuery = $this->params()->fromPost('query');
        return $this->dataExporter->export($exportQuery);
    }
}

class DataExporter {
    public function export($exportQuery): BufferedStream
    {
        // Get all of the results
        $results = $this->elasticsearchService->getData($exportQuery);

        // Create our Stream
        $outputPointer = fopen('php://output', 'wb');
        $stream = new BufferedStream();
        $stream->setStream($outputPointer);

        // Iterate through data and output to CSV file
        foreach ($results as $result) {
            fputcsv($outputPointer, $result, ',');
        }

        return $stream;
    }
}

class Module implements ConfigProviderInterface
{
    /**
     * @inheritdoc
     */
    public function getConfig()
    {
        return include __DIR__ . '/../config/module.config.php';
    }

    /**
     * @inheritdoc
     */
    public function onBootstrap(EventInterface $e)
    {
        /** @var EventManager $eventManager */
        $eventManager  = $e->getApplication()->getEventManager();
        $eventManager->attach(SendResponseEvent::EVENT_SEND_RESPONSE, new BufferedStreamResponseSender(), 1);
    }
}

We've now removed any need for processing data in to an intermediary
file. Data is processed and then streamed straight to PHP's buffered
output.

Elasticsearch offers the Scroll API for processing large amounts of data. It operates in a similar way to a cursor on a traditional database. By replacing our single getData call with a loop utilising the Scroll API, we can minimise the amount of data we have stored in memory at one time.

class DataExporter {
    public function export($exportQuery): BufferedStream
    {
        // Create our Stream
        $outputPointer = fopen('php://output', 'wb');
        $stream = new BufferedStream();
        $stream->setStream($outputPointer);

        // Start our scroll
        $resultSet = $this->elasticsearchService->startScroll($exportQuery);
        $this->writeDataToResource($outputPointer, $resultSet);

        // Continue the scroll till we have no further results
        while (count($resultSet->getResults()) !== 0) {
            $resultSet = $this->elasticsearchService->continueScroll($resultSet);
            $this->writeDataToResource($outputPointer, $resultSet);
        }

        return $stream;
    }

    private function writeDataToResource($outputPointer, ResultSet $resultSet): void
    {
        $results = $resultSet->getResults();
        foreach ($results as $result) {
            fputcsv($outputPointer, $result, ',');
        }
    }
}

By utilising streams and the Scroll API, we reduced our memory usage and removed our file usage completely.

PHPUnit: Returning values and throwing exceptions from the same mocked method

Ethan Bray — Sun, 11 Aug 2019 14:35:31 +0000

Recently I was writing a test case where an initial call to a method would return a value and a subsequent call would throw an exception. I assumed that it could be achieved by using a combination of will, willReturn and willThrowException.

$mockFoo     = $this->createMock(Foo::class);
$mockFactory = $this->createMock(FooFactory::class);

$mockFactory
    ->expects($this->exactly(2))
    ->method('make')
    ->with('Foo', 'Bar')
    ->will($this->willReturn($mockFoo), $this->throwException(new BarException()));

PHPUnit only supports one parameter to will and as such the exception was never thrown. I was surprised to find that PHPUnit has no documented solution for this issue and I instead resorted to scouring StackOverflow. I found this answer which put me on the right track.

Instead of creating the InvokedCount matcher and immediately passing it to expects, we can create it earlier and assign it to a variable. We can then replace our will method with willReturnCallback, using $matcher in the callback. InvokedCount provides the method getInvocationCount which allows us to track how many times the method make has been invoked. Using our callback, we can then handle each invocation separately.

$mockFoo     = $this->createMock(Foo::class);
$mockFactory = $this->createMock(FooFactory::class);
$matcher     = $this->exactly(2);

$mockFactory
    ->expects($matcher)
    ->method('make')
    ->with('Foo', 'Bar')
    ->willReturnCallback(function () use ($matcher, $mockFoo) {
        if ($matcher->getInvocationCount() === 1) {
            return $mockFoo;
        }

        throw new BarException();
    });

It's worth noting that this solution is not a replacement for willReturnOnConsecutiveCalls and is only for use when your mocked method need to do more than return values.

Intermittent RDS Errors When Triggering Lambda Functions

Ethan Bray — Sun, 21 Jul 2019 13:32:03 +0000

In recent years we've started moving our infrastructure from dedicated hosts such as OVH to cloud providers like AWS. Now this post is not about the pros and cons of cloud providers, though there are many, but rather a specific issue we faced when using AWS RDS in conjunction with AWS Lambda.

We store our user details in an RDS instance which is then consumed by an internal API. We also have an Elasticsearch cluster which stores certain modified user information for analysis and other purposes. MySQL triggers are used to invoke a Lambda function that upserts a modified version of the user information to our Elasticsearch cluster.

At first this system seemed to be working perfectly. Indeed we had many other RDS instances triggering Lambda functions hundreds of times a day successfully. However, it then became clear that the MySQL triggers were occasionally failing, resulting in outdated data in our Elasticsearch. The issue appeared to be random. We could not consistently replicate it despite trying various approaches. We removed the triggers to negate the impact it was having on our production infrastructure and created a clone of the RDS instance for testing. This is the error we were faced with when we managed to break the trigger:

Statement could not be executed (42000 - 1064 - Unknown trigger has an error in its body: 'Access denied; you need (at least one of) the Invoke Lambda privilege(s) for this operation')

Access denied, how could that only be occurring part of the time? I checked that our cluster had the correct IAM role as well as the correct VPC security groups. They were the same as our working instances. Googling the issue did not unearth anyone with similar symptoms to us, only the same error message when they had misconfigured their IAM roles. We were stumped.

Whilst reading the documentation on MySQL triggers on RDS, we noticed that we were using MySQL version 5.6 where RDS now supported 5.7 too. The documentation did not mention that we may be facing our issue due to the version difference, but we figured it was worth performing the upgrade regardless. Secretly, we hoped it would cause our problem to disappear too. The upgrade was performed and... our triggers were still breaking.

At this point we decided to call in backup and contact AWS support. Realistically we should have done this much earlier. We opened a ticket, explaining our issue and the error messages we were getting as well as the things we had tried.

Now some background on our infrastructure on AWS. Our infrastructure is inside a VPC so that we can define security groups and access control lists to our resources. We have several subnets in this VPC, three of which have access to a NAT Gateway. The NAT Gateway allows our private instances to access the internet without exposing their private IP. The reason we have three subnets is that the subnets are availability zone specific and there are three availability zones in the London region.

While waiting for a reply from AWS, I compared the configuration of a working RDS instance to our user information instance for what felt like the hundredth time. Suddenly I spotted it, our user information instance was only in subnet a and b. Was it possible that the subnet was causing these occasional errors? Unfortunately you cannot modify the subnets of an existing RDS instance so I created a new clone of the instance with all 3 subnets configured.

Once our new testing instance was ready, I created our triggers and started to test them. They worked every time. It then became obvious to me that the reason the issue appeared random was that our requests were being load balanced across the 3 subnets. Any requests to subnet c would fail, resulting in a 66% success rate for the trigger invocation.

We recreated our RDS instance with the correct subnets and pointed our applications at the new host, fixing the issue for good.

Looking back, the issue and the solution seem obvious. At the time it left us puzzled, especially since we could not find any similar cases online.

Logging in Zend Framework Using the Event Manager

Ethan Bray — Sun, 21 Jul 2019 13:16:20 +0000

Last year I was working on a refresh of our client control panel. The client control panel is a web application that allows our users to gain an insight into their various services and campaigns with us. The aim of the project was to create a brand new control panel using Zend Framework 3 components, as well as updating to the latest PHP version and using Material Design Components for the front end.

In our previous version of the control panel, there was very little infrastructure in place for logging. Usage of logging was inconsistent and various different logging libraries were used, depending on which developer had worked on that section of the site.

After some research, we narrowed it down to three potential options:

Use a Logger class with static methods for each of the potential log levels. We had used this approach in several smaller applications but felt that it would not suit a large project such as this one. While this approach is easy to use and create, we didn't want to pollute our codebase with hundreds of static calls that cannot be mocked.
Inject an instance of a logger implementing LoggerInterface into any
class in which we wanted to log information. At the time, this seemed like it added unnecessary overhead to the construction of each object.
Use the Zend event-manager. Emit log events (LogEvent) and attach a LogListener to the event manager. The LogListener then logs the information from the event using an instance of LoggerInterface.

The third approach seemed easy to implement and to understand, so we decided to use it in our application. This post will cover how we implemented this, as well as reviewing this approach now that we've spent over a year using it in production.

Creating the Log Event

This is our LogEvent below. The event will be emitted by the class wishing to log information and then caught by a listener attached to our event manager. We define public constants that map to log levels from PSR-3. We could have used the PSR-3 levels directly, however we wanted to provide a layer of abstraction between our LogEvent and whichever logging library we used. This step is optional as most people will use PSR-3 compliant loggers.

declare(strict_types=1);

namespace Application\Event;

use Psr\Log\LogLevel;
use Zend\EventManager\Event;

class LogEvent extends Event
{
    public const DEBUG     = LogLevel::DEBUG;
    public const INFO      = LogLevel::INFO;
    public const NOTICE    = LogLevel::NOTICE;
    public const WARNING   = LogLevel::WARNING;
    public const ERROR     = LogLevel::ERROR;
    public const CRITICAL  = LogLevel::CRITICAL;
    public const ALERT     = LogLevel::ALERT;
    public const EMERGENCY = LogLevel::EMERGENCY;
}

Creating and Registering the Listener

Now that we have our event, we need to create a listener and register it with the event manager. Listeners are attached to an event manager which is then responsible for reacting to events that are emitted in the application.

Our listener implements the ListenerAggregateInterface which requires two methods: attach for attaching one or more listeners and detach for removing those listeners. We use the ListenerAggregateTrait to implement the detach method.

declare(strict_types=1);

namespace Application\Listener;

use Application\Event\LogEvent;
use Application\Model\LoggerAwareInterface as LoggerAware;
use Psr\Log\LoggerInterface;
use Zend\EventManager\EventInterface;
use Zend\EventManager\EventManagerInterface;
use Zend\EventManager\ListenerAggregateInterface;
use Zend\EventManager\ListenerAggregateTrait;

class LogListener implements ListenerAggregateInterface
{
    use ListenerAggregateTrait;

    /**
     * @var LoggerInterface
     */
    private $logger;

    /**
     * @param LoggerInterface $logger
     */
    public function __construct(LoggerInterface $logger)
    {
        $this->logger = $logger;
    }

    /**
     * @inheritdoc
     */
    public function attach(EventManagerInterface $events, $priority = 1)
    {
        $sharedManager = $events->getSharedManager();
        $this->listeners[] = $sharedManager->attach(LoggerAware::class, LogEvent::DEBUG, [$this, 'addDebug'], $priority);
        $this->listeners[] = $sharedManager->attach(LoggerAware::class, LogEvent::INFO, [$this, 'addInfo'], $priority);
        $this->listeners[] = $sharedManager->attach(LoggerAware::class, LogEvent::NOTICE, [$this, 'addNotice'], $priority);
        $this->listeners[] = $sharedManager->attach(LoggerAware::class, LogEvent::WARNING, [$this, 'addWarning'], $priority);
        $this->listeners[] = $sharedManager->attach(LoggerAware::class, LogEvent::ERROR, [$this, 'addError'], $priority);
        $this->listeners[] = $sharedManager->attach(LoggerAware::class, LogEvent::CRITICAL, [$this, 'addCritical'], $priority);
        $this->listeners[] = $sharedManager->attach(LoggerAware::class, LogEvent::ALERT, [$this, 'addAlert'], $priority);
        $this->listeners[] = $sharedManager->attach(LoggerAware::class, LogEvent::EMERGENCY, [$this, 'addEmergency'], $priority);
    }

    private function addRecord(string $loggerLevel, EventInterface $event)
    {
        $logMessage   = $event->getTarget();
        $logContext   = $event->getParams();

        $this->logger->log($loggerLevel, $logMessage, $logContext);
    }

    public function addDebug(EventInterface $event)
    {
        return $this->addRecord(LogEvent::DEBUG, $event);
    }

    public function addInfo(EventInterface $event)
    {
        return $this->addRecord(LogEvent::INFO, $event);
    }


    // Methods must be added for each log level listed in the LogEvent
}

We specifically use the SharedEventManager as it can be used when we want to attach listeners without yet having an instance of the class that composes an EventManager. In our case, that would be any class that wishes to emit LogEvent. As you can imagine, attaching the LogListener to every single class emitting events would be unwieldy.

We attach a listener and define a method for every log level defined in our LogEvent. Some of the methods have been omitted from the example above for the sake of readability. This allows us to log the events at different levels depending on the constant specified by the emitting class.

For example:

// This event will be logged as INFO due to the constant specified
$this->getEventManager()->trigger(LogEvent::INFO, 'Example log message', ['context']);

// Whereas this event will be logged as CRITICAL
$this->getEventManager()->trigger(LogEvent::CRITICAL, 'All connections down', ['context']);

The LoggerAwareInterface

As we are using the SharedEventManager, we need to provide an identifier when attaching a listener. The identifier is used to identify classes that may emit the LogEvent and should therefore be listened to. We use LoggerAwareInterface aliased to LoggerAware as our identifier for the listener. It extends the EventManagerAwareInterface as in order to trigger the LogEvent, the emitting class must have access to an instance of the event manager. The EventManagerAwareInterface requires two methods to be implemented: setEventManager and getEventManager.

Any classes retrieved through the Zend service manager that implement the
EventManagerAwareInterface will automatically have the event manager set. This makes emitting events very easy, but does have a certain 'magic' quality to it that can be unclear for developers who have not worked with the event manager before.

declare(strict_types=1);

namespace Application\Model;

use Zend\EventManager\EventManagerAwareInterface;

interface LoggerAwareInterface extends EventManagerAwareInterface
{

}

For example, the class Foo implements the LoggerAwareInterface and has a method bar. When the bar method is called, a LogEvent is emitted in the event manager and handled by any listeners attached to the LogEvent. We've implemented the setEventManager and getEventManager methods ourselves. In the setEventManager method we use the class name as well as any interfaces it implements as identifiers for the events. You'll recall earlier that we used LoggerAwareInterface as our identifier.

class Foo implements LoggerAwareInterface {
    private $events;

    public function bar(): void
    {
        $this->getEventManager()->trigger(LogEvent::INFO, 'Example log message', ['context']);
    }

    /**
     * @inheritDoc
     */
    public function setEventManager(EventManagerInterface $events)
    {
       $className = get_class($this);

       $nsPos = strpos($className, '\\') ?: 0;
       $events->setIdentifiers(array_merge(
           [
               __CLASS__,
               $className,
               substr($className, 0, $nsPos),
           ],
           array_values(class_implements($className))
       ));

       $this->events = $events;

       return $this;
    }

    /**
     * @inheritDoc
     */
    public function getEventManager()
    {
       if (!$this->events) {
           $this->setEventManager(new EventManager());
       }

       return $this->events;
    }
}

EventManagerAwareTrait

Duplicating the setEventManager and getEventManager in every class we wish to log from would be a maintenance nightmare, as well as increasing 'visual debt'. We separated these methods into an EventManagerAwareTrait that can be used in any class that wishes to emit events.

declare(strict_types=1);

namespace Application\Model;

use Zend\EventManager\EventManager;
use Zend\EventManager\EventManagerInterface;

trait EventManagerAwareTrait
{
    private $events;

    /**
     * Set the event manager instance used by this context
     *
     * @param  EventManagerInterface $events
     * @return $this
     */
    public function setEventManager(EventManagerInterface $events)
    {
        ...
    }

    /**
     * Retrieve the event manager
     *
     * Lazy-loads an EventManager instance if none registered.
     *
     * @return EventManagerInterface
     */
    public function getEventManager()
    {
        ...
    }
}

This simplifies our example to this:

use Application\Model\EventManagerAwareTrait;

class Foo implements LoggerAwareInterface {
    use EventManagerAwareTrait;

    public function bar(): void
    {
        $this->getEventManager()->trigger(LogEvent::INFO, 'Example log message', ['context']);
    }
}

One thing to note is that you will not need to use the EventManagerAwareTrait when emitting events from a Zend controller, as it already implements setEventManager and getEventManager.

Attaching the Listener on Application Bootstrap

Almost there. There's still one more thing to do before we can start logging. We need to ensure our LogListener is attached to the event manager during the bootstrap phase of the application. We can do this in the Module.php for the Application module.

declare(strict_types=1);

namespace Application;

use Application\Listener\LogListener;
use Zend\EventManager\EventInterface;
use Zend\ModuleManager\Feature\BootstrapListenerInterface;
use Zend\ModuleManager\Feature\ConfigProviderInterface;

class Module implements ConfigProviderInterface, BootstrapListenerInterface
{
    /**
     * @inheritdoc
     */
    public function getConfig()
    {
        return include __DIR__ . '/../config/module.config.php';
    }

    /**
     * @inheritdoc
     */
    public function onBootstrap(EventInterface $e)
    {
        $serviceManager     = $e->getApplication()->getServiceManager();
        $eventManager       = $e->getApplication()->getEventManager();
        $sharedEventManager = $eventManager->getSharedManager();

        // Logging Listener
        $logListener = $serviceManager->get(LogListener::class);
        $logListener->attach($eventManager);
    }
}

Great! Now we can easily trigger LogEvent from any class by simply implementing the LoggerAwareInterface and using the EventManagerAwareTrait.

Event Based Logging: 1 Year On

The new control panel was a success and was a huge step up from the previous version, both for developers and our users. The logging works perfectly and has given us greater visibility into our application whilst providing a consistent experience to developers.

However, this approach has heavily coupled our application with Zend's event manager component. If we were to refactor our application to use a different framework, it would take much longer due to the coupling. This issue would not be as prevalent with the other implementations I listed at the beginning of this post.

In the past year we also welcomed some new developers to the team. These developers were in junior positions and had no experience with an application like our control panel before. The event management system features a large amount of 'magic' behaviour that isn't clear to newer developers, especially ones that are unfamiliar with event-driven architecture. I think it's important that something as vital as logging is immediately debuggable by everyone in the team, regardless of skill level.

If I were to start this project again, I would pass an instance of LoggerInterface into any class that requires it. This approach may be more 'manual' but is much more explicit and does not tie us to any framework in particular. If you are already working in an event-driven
system, you may decide that coupling with the event manager component isn't an issue. It's about using the right tool for the job.

If you have any questions or feedback for this article, you can email me at ethan@messagecloud.com.

Unit Testing Zend Framework Forms

Ethan Bray — Sun, 21 Jul 2019 13:07:38 +0000

The Zend Framework documentation on unit testing focuses on testing controllers rather than components such as forms, filters or validators.

Testing these components is vital for catching bugs and regressions in the behaviour of key forms in your web application.

It can be unclear how to test these components in the least brittle fashion. Elements with special behaviour such as file uploads can be particularly tricky to test.

Testable Behaviour

There are several parts of a form that can be tested. Tests can be written to ensure the form and its elements are configured properly, as well as ensuring the filtering and validation are behaving correctly.

The construction of a form and a filter can be tested by simply instantiating the class. You can then test the individual elements for specific configuration options such as their type, labels or required state. Any of the options or attributes can be retrieved by using the getOption or getAttribute method on a form element.

use Foo\Bar\ExampleForm;
use PHPUnit\Framework\TestCase;
use Zend\Form\Element\Text;
use Zend\Form\Element\Password;

class ExampleFormTest extends TestCase {
    public function test_username_configuration(): void
    {
        $exampleForm = new ExampleForm();   
        $usernameInput = $exampleForm->get('username');

        $this->assertInstanceOf(Text::class, $usernameInput);
        $this->assertEquals('Username: ', $usernameInput->getOption('label'));
        $this->assertTrue($usernameInput->isRequired());
    }

    public function test_password_configuration(): void
    {
        $exampleForm = new ExampleForm();   
        $passwordInput = $exampleForm->get('password');

        $this->assertInstanceOf(Password::class, $passwordInput);
        $this->assertEquals('Password: ', $passwordInput->getOption('label'));
        $this->assertTrue($passwordInput->isRequired());
    }
}

The filtering and validation can be tested by instantiating the input filter, setting the test data and then retrieving the specific validator you're testing. You can then assert whether the validation was successful, what the error message(s) are and what the value of the input is post-filtering.

use Foo\Bar\ExampleFormFilter;
use PHPUnit\Framework\TestCase;

class ExampleFormFilterTest extends TestCase {
    public function test_username_validator(): void
    {
        $formFilter = new ExampleFormFilter();
        $formFilter->setData(['username' => 'foobar']);
        $validator = $formFilter->get('username');

        $this->assertTrue($validator->isValid());
        $this->assertEquals([], $validator->getMessages());
    }

    public function test_username_filter(): void
    {
        // For this test, we're testing that our input is having whitespace trimmed
        // and any tags removed

        $formFilter = new ExampleFormFilter();
        $formFilter->setData(['username' => ' foobar <script>']);
        $validator = $formFilter->get('username');

        $this->assertTrue($validator->isValid());
        $this->assertEquals('foobar', $validator->getValue());
    }
}

I recommend using a data provider to try an array of different values. This will help cut down on the amount of duplicated tests you have to write.

use Foo\Bar\ExampleFormFilter;
use PHPUnit\Framework\TestCase;

class ExampleFormFilterTest extends TestCase {
    /**
     * @dataProvider getUsernameData
     */
    public function test_username_validator($username, bool $expected, array $messages): void
    {
        $formFilter = new ExampleFormFilter();
        $formFilter->setData(['username' => $username]);
        $validator = $formFilter->get('username');

        $this->assertEquals($expected, $validator->isValid());
        $this->assertEquals($messages, $validator->getMessages());
    }

    private function getUsernameData(): array
    {
        return [
            ['user', true, []],
            ['User', false, ['username' => ['notLowercase' => 'Username must be lower case']]],
            ['1234', true, []],
            ['', false, ['username' => ['isEmpty' => 'Username is required.']]],
            ['^9bhy&', false, ['username' => [
                'notAlnum' => 'Username not valid. Use only alphanumeric characters.',
            ]]],
            ['bl ah', false, ['username' => [
                'notAlnum' => 'Username not valid. Use only alphanumeric characters.',
            ]]],
            [000, false, ['username' => ['stringLengthTooShort' => 'Username not valid. Minimum length 4 characters.']]],
            ['a', false, ['username' => ['stringLengthTooShort' => 'Username not valid. Minimum length 4 characters.']]],
            ['123456789012345678901234567890123456789012345678900', false, ['username' => [
                'stringLengthTooLong' => 'Username not valid. Maximum length 15 characters.']]],
        ];
    }
}

Testing File Elements and Validators

Testing elements such as file uploads can be slightly trickier due to the
more complex validators such as Size or WordCount.

You could test these validators by using example files that you commit to your repository. However, this bloats your repository with large amounts of files.

We can use vfsStream as a stream wrapper for a virtual file system. This allows us to programatically create virtual files that can then be used during our tests.

use Foo\Bar\ExampleFormFilter;
use org\bovigo\vfs\content\LargeFileContent;
use org\bovigo\vfs\vfsStream;
use PHPUnit\Framework\TestCase;

class ExampleFormFilterTest extends TestCase {
    public function test_file_size_validator_fails(): void
    {
        $exampleFile = vfsStream::newFile('example_file.csv')
            ->withContent(LargeFileContent::withMegabytes(2));

        $formFilter = new ExampleFormFilter();
        $formFilter->setData(['file' => $exampleFile]);
        $validator = $formFilter->get('file');

        $this->assertFalse($validator->isValid());
        $this->assertEquals(
            ['size' => 'File size must be under 1MB'],
            $validator->getMessages()
        );
    }

    public function test_file_size_validator_passes(): void
    {
        $exampleFile = vfsStream::newFile('example_file.csv')
            ->withContent(LargeFileContent::withMegabytes(1));

        $formFilter = new ExampleFormFilter();
        $formFilter->setData(['file' => $exampleFile]);
        $validator = $formFilter->get('file');

        $this->assertTrue($validator->isValid());
        $this->assertEquals([], $validator->getMessages());
    }

    public function test_file_extension_must_be_csv(): void
    {
        $exampleFile = vfsStream::newFile('example_file.zip')
            ->withContent(LargeFileContent::withMegabytes(1));

        $formFilter = new ExampleFormFilter();
        $formFilter->setData(['file' => $exampleFile]);
        $validator = $formFilter->get('file');

        $this->assertFalse($validator->isValid());
        $this->assertEquals(
            ['fileExtensionFalse' => 'File has an incorrect extension'],
            $validator->getMessages()
        );
    }

    public function setUp(): void
    {
        vfsStream::setup();
    }
}

The virtual files can be treated exactly like normal files, allowing you to test validators such as WordCount easily. Documentation for vfsStream can be found here.

Working Remotely: 2 Months In

Ethan Bray — Sat, 20 Jul 2019 07:41:17 +0000

I've been working at MessageCloud for around 3 and a half years now. After taking a break to travel at the beginning of this year, I returned to work in a 75% remote position. I spend the majority of my time working from home and spend a week in the office each month.

We've had several other remote employees in the past, but they've always started as on-site. I've read plenty of articles about the pros and cons of working remotely, tackling issues such as loneliness and lack of communication, so I felt prepared.

This post will cover some of the teething issues we've encountered as well as the solutions we've put in place to help combat them. It's likely that this post will be the first in a series detailing my experience working remotely.

Using video standups instead of Slack messaging

Previously I would speak with each of the team members individually and discuss our plans for the day. It was rare that I needed to ask for more information as being on-site made it a lot easier to passively soak up information regarding their progress and workload.

Once I began remote working, I tried to continue the daily chats using Slack. However, Slack isn't particularly conducive to long conversations. Especially several at once in the same room. People's replies would often be delayed as they were sidetracked by other people or simply did not see that I had replied to them amidst other conversations.

Our solution for this was to have video standups every morning. This way each member of the team has their chance to speak and have everyone's attention. These standups help both myself and the team stay up to date with everyone's work and any blockers they might be facing.

Having all members of a meeting using a webcam

This issue became apparent when we introduced the standups as well as the regular catchup meetings I have with other colleagues. Without a webcam it can be difficult to know when one person is done speaking or who is speaking to who in the case of meetings with more than 2 people.

It also just feels better. Being able to see the person you're conversing with is an underrated requirement of working remotely, in my opinion.

At first I imagine some members of the team felt uncomfortable with being required to use a webcam, but since it has become a regular affair in our standups everyone has consistently used a webcam in other meetings.

Detailed issue tracking

At MessageCloud, we use GitHub issues to record our work. However, it's fair to say that at times we were less than verbose with the details we left in our issues. We unknowingly relied on the passive information we received throughout the day to inform ourselves on the issues we were tackling.

This occurred when team members would start conversing with me about an issue and I would struggle to grasp what they were talking about immediately. I would often have to ask my colleague to start from the beginning and explain the issue to me in greater depth. This can become slightly frustrating as you spend more time asking for clarification than actually solving the issue.

I spoke with the team and we worked together to record our issues with greater detail, as well as keeping them up to date with any progress, blockers and extra information that came to light. We're not yet doing this constantly but we're working on making it a consistent habit among the whole team.

Missing Break/Lunch

Although I work remotely, I'm required to work the same hours as if I was in the office as my role mainly consists of team leadership. A surprising symptom of working remotely was that without the break time hubbub of office to remind me, I frequently missed my scheduled breaks and lunch time. When focusing in on a particularly tricky piece of work, I tend to turn on do not disturb and full-screen my editor, hiding the time and exacerbating the issue.

At MessageCloud, we have a Go based Slack bot called Ava. Ava is used to kick off Jenkins deployments as well as having several fun little features.

I added a small piece of code to Ava that sends me a message on Slack when it's break time.

As a side note, I'll cover how we built Ava in a future blog post.

Using my on-site time wisely

As much as everyone likes to sing the praises of remote working, there are definite issues. Especially so in a company that does not have many remote workers and a 'remote-first' attitude. Certain tasks such as event storming and helping junior developers are infinitely easier in person.

In the week that I spend on-site, I focus on performing the tasks that may otherwise be difficult remotely. Meeting with domain experts as well as having in-person catchups with other members of the organisation.

It's also good to spend this time bonding with the team and ensuring that you're not just a face on a computer screen to them!