DEV Community: Barry O Sullivan

Estimating software: How to deal with requests for estimates

Barry O Sullivan — Mon, 02 Mar 2020 11:17:02 +0000

A question that often comes up in conversation with other developers is "How do I deal with requests for estimates?". Usually the person asking the question is frustrated, they know that any estimates they create are pure fiction, yet management keeps asking for them anyway. Then they ask the real question; "How do you make them stop?".

Well, I usually give the same answer, "You don't.". Instead you need to ask why are they asking for estimates in the first place. What problem are estimates solving for them? Once you understand that you can figure out the best way to help.

Why we're asked for estimates

When I listen to developers discuss the problem with estimates, they always come at it from the perspective of management not getting that estimates don't work. Their focus is on making management understand their perspective, taking a competitive rather than connective approach, hammering it home in any way they can. But communication is a two way street, if we want management to understand us, we need to understand them. We need to empathise.

First off, management and stakeholders don't ask for estimates for the sheer hell of it, they have reasons. Here are some common ones:

They want to plan for the future
They want to know if the work is on schedule or not
Someone higher up the chain has asked them how long something will take

No matter the reason, turning around to this person and saying "I dunno" is supremely unhelpful. In case one you're making it impossible to plan anything, in case two you're robbing them of any sense of control and in case three you're making them look bad in front of their boss. Any of these outcomes will ruin any chance you have at co-operation. You're effectively shutting down the conversation and presenting any communication as adversarial. Not good footing for a working relationship.

Can we give accurate estimates

No conversation about estimates is complete without a dive into accuracy. So what's the reality of the situation? Can we give them accurate estimates? Well . . . no. Hell no. Definitely not at the start of a piece of work anyway.

We all know that estimates are usually wrong, at best they're educated guesses, at worst they're delusional and unrealistic. Estimates are so difficult because we're asked for them when we have the least information about the problem. But that doesn't mean we can't lean on experience and manage how we produce them.

Dealing with requests for estimates

Given that accurate estimates are impossible, what do we do? Well here's the thing, management may say they want ACCURATE estimates, but what they actually want is a sense of control. They want to know that the work is in safe hands, and estimates are just one way they go about getting that sense of security.

So the first step is to empathise. They want to know how long it will take, you want to know how long it will take; what do we need to figure this out? Start the conversation with this perspective and you'll find they're much more open to exploring the problem and figuring out a useful middle ground.

Now it boils down to knowledge and confidence. What you do next depends on how well you understand the problem and the solution. I've broken it down into three steps based on your level of knowledge, least knowledgeable at the start, most knowledgeable at the end. These are not proscriptive, it's just how I've broken down difficult projects in the past.

Confidence level 1: Don't have a clue:

Strategy: Research spike

First off you need to understand the problem. If you don't understand the challenge then estimates are impossible. In agile these are called research spikes and they're a great way to scope out potentially difficult work. I usually time-box two days for these, but it can vary. I'll arrange meetings with stakeholders and dive into the codebase and get a feel for how difficult it is to work with. At the end of the time-box I'll report to a manager on my progress. If things are going badly I'll ask for more time. If things are going well I'll move onto stage 2 or stage 3, depending on my level of confidence.

Confidence level 2. I have an idea:

Strategy: Throwaway Prototype

At this stage you have an idea of what you need to do, but you have no idea as to how you'll actually do it. This usually happens when the codebase is legacy/messy and implementing a solution is non-trivial. You don't really know where you need to change yet and you don't know how long the final product will take, but you probably have an idea of how long a quick and dirty prototype should take. The goal here is to focus on the happy path and hammer in the functionality, see what works and what breaks, discover what parts of the system will fight you. The goal is to gain knowledge, not to write a well written solution. As long as you're clear on this to stakeholders then you'll be fine.

Confidence level 3: I know what needs to be done and where:

Strategy: Estimate and do the work

This is the part were you can give management actual estimates. You have the most information you can have about the project without actually completing it, so you're in the best position to plan out and estimate the work. You know what needs to be done, you know what needs to change in the codebase and you know the pain points you'll face. The estimate doesn't need to be accurate, but now you're able to give reasons for your estimates and more importantly you'll be able to present a plan.

What if they want estimates at the start?

I always try to avoid project length estimates, but sometimes there's no way around around them. For whatever reason estimates are considered the main priority and we need to provide answers. In these cases I find communicating risk is key. Give an estimate in month increments (3 months, 6 months, etc ...), but gauge it as rough and subject to change. If they're experienced they'll accept this and appreciate your candor. If they don't accept this, then that's fine, they're just inexperienced and they'll quickly learn that estimates are just that, estimates. Personally I've met very few of the latter variety, most managers and stakeholders are open to honest communication and collaboration. They know that plans and reality rarely align.

It's all about communication

The main issue I've found with developers if that we want to dictate, to get across our ideas and explain why our knowledge is superior. We think this will persuade management, despite the fact it doesn't even convince other developers. This isn't how things go, it's a two way relationship and that requires give and take. It requires communication.

So at each stage of a project you need to report to stakeholders on your progress. Don't be too detailed, just enough so they know if things are on track or not and if you're blocked and need help. Frame the messages for their context, tell them what's relevant to them. E.g. Product people don't care about the patterns and infrastructure you use, only tell them what's relevant to them. You can do this at standups, or you can send them a PM, it doesn't matter. What does matter is that you are open and honest about your progress. They want security, so give it to them.

Story points

You'll notice a suspicious absence of story points in the above. This is because they are not useful when reporting estimates for the purpose of building confidence. Story points were never intended to be a proxy metric for time, they exist so that teams can reach consensus on how "big" a piece of work is amongst themselves, ensuring everyone is on the same page with the difficulty and details of a story. They are literally made up numbers and your velocity one sprint will not match your velocity for the next. (If they do, someone is fudging the numbers . . .)

So as a metric to report upwards, it's not very useful. This is why I avoid story points when reporting estimates and instead use actual time. It's much more useful and it's what management actually want to hear. In short, I'd advise you use story points internally in a team and time estimates when reporting externally.

Closing thoughts

I've found that once you frame estimates as a collaborative problem and work with others to succeed, then management will relax. They know that everything is in safe hands and will gladly let you work as you need to. You've proven that you can communicate risk and that you understand their needs. Working like this presents the relationship as collaborative, people will want to work with you and will happily let you manage how you work. Remember, if you make their job easy for them, they'll let you make your job easy for yourself. They don't want to control you, they want to succeed. Enable their success.

Template Driven Development: Why it doesn't work

Barry O Sullivan — Mon, 06 Jan 2020 15:45:29 +0000

Software development is hard, and there have been many attempts to simplify the process over the years, some good, most bad. I'd like to discuss one of the bad attempts, a recurring problem I've seen in software development that myself and my friends have dubbed "Template Driven Development" (the bad TDD).

Template Driven Development is the practice of reducing software development down to a series of templates that developers should follow. Developers are given a user story and a rigid schema for how their system should be structured, they are to use a set of pre-defined patterns that they must follow when modelling concepts in the story. Following the template is paramount.

Now, you're probably thinking, "Wait a minute, isn't that just consistency?", but don't worry, I'm not attacking consistent code. Consistently formatted code is a must, as is consistent naming conventions and use of patterns. No, what I'm attacking is the idea that you can reduce software development down to a series of templates and steps that people can just follow without thought.

The dream is that it makes software development quicker and cheaper. The reality is messy code that's hard to navigate and fails to express business intent. The models end up being anemic and the system is littered with hacks to workaround failings in the prescribed patterns. The codebase rots exceptionally fast, the resulting product is broken, and the developers have no idea how to fix it. Congrats, you played yourself.

Why it doesn't work

Naively "Template Driven Development" seems like it should work; there are recurring patterns in software development after-all and we are encouraged to use them.

When you look at a well designed system it is consistent and easy to navigate. From there it appears trivial to reduce it down to a series of templates that junior developers can follow. Bam, now you don't need as many senior devs, you just need to teach your junior devs those patterns and they'll be able to add new features with ease.

So if this sounds too good to be true, that's because it is. It is an illusion. The system only looks like it could be turned into a series of templates because the senior developers spent so long iterating on the codebase to make it easy to understand and consistent in style. It didn't start out that way, and to strive for that level of consistency at the start of a project is akin to putting the cart before the horse. It's classic waterfall style thinking and it just doesn't work. If the system changes then the patterns must change to accommodate it, and developers that can only follow patterns cannot handle that change.

Why we want it to work

"Template Driven Development" comes from the desire to turn a complex/complicated process (working as a team to write software) into a simple process (putting together Ikea furniture). It tries to shortcut all the design, iteration, training and collaboration that leads to a well written system. The idea that this is even possible is re-enforced by the fact that well designed systems seem obvious in hindsight. When we see the final product we forget all the trial and error it took to actually get there.

Development managers and seniors like this idea because it potentially allows them to shortcut finding and training developers (an expensive task). Instead they just need to have some meetings, come up with the templates, and then fire a small army of inexpensive juniors at the problem.

Junior to mid-level developers also want to learn the secrets of writing good code and ideally they'd like to skip all the ambiguity and messing about. They want to write well designed systems and they don't want to go through the pain of getting it wrong. They believe it is possible to leapfrog all the iteration and mistakes and just jump straight to the finish line.

Combine these two mindsets together and you can see why the idea is so attractive, it feels like a shortcut that should work, even though it's obvious that it doesn't. If it did work then that's just how we'd write software and this article wouldn't exist. At this stage you'd think we'd realise that it isn't possible, yet we keep falling into the same mental trap, the idea is just too alluring it seems.

Remember, there are no silver bullets.

The problems it causes

When you teach developers to work like this then you encourage them to stop thinking. In fact, you've explicitly told them not to think and instead they should follow the process laid down from on high. They don't know why the patterns are good or what problem they're solving, they just view them as the way you do things and will not put anymore thought into it than that. The resulting systems are messy and are typically nonsensical because no design thought was put into them. The developers don't know how to work outside the patterns you've given them, they can't choose patterns based on context, so they will follow those patterns even when it's obvious that they're getting in the way.

Just because you can put together an Ikea flatpack does not mean you can design and manufacture a desk from scratch with no supervision. That's not how carpentry works and it's definitely not how coding works.

Teaching developers to work like this does them a massive disservice and potentially ruins their professional development. Developers that want to get better will quickly realise that this doesn't work and will get frustrated and leave. Developers that are isolated and trust in management will buy into it and will stop progressing, producing bad code without understanding why it's bad. Since it doesn't work, management believes that the devs need tighter reigns, and it devolves into dark agile with a stringent command and control structure . . . which just makes everything worse.

Once people buy into this idea it is incredibly hard to make them stop, it becomes faith driven, an ideology about how software development "should" work, and it takes a LOT of failure before they will admit it doesn't.

So how do we combat this?

Thankfully it's not all doom and gloom, there are ways to fight this mindset. Below is how we dealt with it at one company I consulted (obvious self plug).

1. Kill your darlings

Well first off we need to kill the idea that there is one way to write software, one perfect way to architect an application that doesn't require design and iteration. This idea is toxic and is the ultimate pre-optimisation.

Basically we as developers are looking for the "God pattern", the pattern that is perfect and always applicable. This pattern does not exist, and any attempt to find it is a fools errand. That isn't to say there aren't generic patterns, of course there are, but the patterns are only applicable in a particular context. You let the patterns emerge to help you express the problem you're solving, you don't force them to appear.

2. Don't present patterns as how-to

This is a trap that has caught myself and many others. You show off a well designed system and explain all the patterns you used, hoping that it will help other developers. You teach these patterns as "the way a system should be designed", under the belief that it'll shortcut training time. This isn't enough, they need to understand the design choices you made, not just the end result. You have to showcase the trade-offs in design, show how you got there. Which leads us to PRs . . .

3. Express the evolution of your designs through small PRs

Instead of showing a final product, you should instead show the evolution of your design over time. And the best way to do this is through small PRs.

Small PRs show how you changed a system over time, introducing patterns and designs when applicable. You didn't choose the design at the beginning, you let it emerge naturally, and it's important to show that to developers. If you don't, they'll try to jump straight to the finish line and the result will be design for the sake of design, rather than design to express a particular solution in a given context.

Quick aside, when writing these PRs you should be explicit about design choices in the description, this makes things clearer and opens up the design to discussion, which helps spread knowledge and understanding.

4. Allow time to sharpen the axe

Knowledge comes from experience. If you want you developers to understand when and where patterns are applicable, you have to let them try out those patterns and review the results. Simply teaching them the patterns isn't enough, information is not knowledge, they have to experience using the patterns firsthand. I.e. they need to sharpen the axe.

One way to do this is to carve out time for a developer to try out a new idea on a codebase. Maybe a day every two weeks or so. Afterwards review the code with the team and discuss what worked and what didn't. If it's good, keep it, if it's not then drop the PR, no harm done. The goal is to learn and explore, not to push all code to production. This time isn't wasted time, this time is an investment in your staff and the work they'll produce in future. It's pays off massively and surprisingly quickly. You're saving yourselves a lot of future bugs and headaches.

Don't fall victim

"Template Driven Development" is a real problem in our industry, it's a seductive idea that leads to crap software and frustrated teams. Thankfully it is possible to combat it, we just need to be realistic and humble, accept that there are no shortcuts in training. We need to create an environment were knowledge is shared and skills can be improved. The simplest way to start is to have seniors show junior and mid-level developers how they do things, they just need to take the time. (Small PRs are a God send here!)

Software development is about iteration and collaboration at it's core, there's no way to shortcut it, and we must accept that.

Introducing DDD to your Company

Barry O Sullivan — Mon, 15 Jul 2019 13:39:48 +0000

The following are my notes from the last DDD IE meetup, 6th June 2019, on how to introduce DDD to your company. This post is about the concepts we discussed and discussions we had around the topic. There's some great stuff in here, the kind that can only happens through discussion and collaboration.

First off, the meetup itself was structured as half talk, have group discussion, with discussion interspersed throughout the slides. We've done a few of these style of meetups in the past and they've worked out really well.

Here are the slides: https://slides.com/barryosull/event-storming-workshop-building-noteworthy-9

Have you Introduced DDD to your company in the past?

As people went in we asked them if they had tried to introduce DDD and how that worked out form, here's a sample of the responses. (apologies for potato quality picture)

At this stage there were six responses, three attendees had tried and three had not.

Of the three that tried, two of them succeeded (one on the second attempt) and one of them didn't. The one that failed stated that it was because management and development didn't really engage with the concept (more paying it lip service) and it stalled in its tracks.

Quick bit of honesty, the one that failed the first time and succeeded the second was me. A lot of the information I presented was based on my personal experience with introducing DDD, supplemented by discussions I've had with others since.

Why are you introducing DDD?

After a brief introduction to the meetup, we asked people why they wanted to introduce DDD to their company. If they want to introduce it, they should be able to articulate why. I presented some of the reasons I usually give, then opened the floor to a discussion on why the attendees wanted to introduce DDD. We broke into groups of 4 and spent 10 minutes discussing, then at the end we articulated our reasons:

The Reasons

Allow us to engage domain experts so we can decouple our software
Maintainable designs
Avoid DRY on superficial similarity, e.g. Eric Evans Toilet/Sink story
Add a public API and build a vocabulary for it
Figure out if we need to build software at all

These were really interesting, as there was a clear focus on software, but from different perspectives. Through out the meetup I talked to each group and dive a little further into their answers. I was delighted that I did, here's what we discussed.

1. Allow us to engage domain experts so we can decouple our software

This team had a giant monolith that was causing them big headaches. They wanted to break it apart into separate modules that would be easier to manage. The lack of domain knowledge really hindered this, as they had lots of God objects and didn't know how to split them up. They hoped that gaining domain knowledge would allow them to rename and the split concepts. A solid strategy and a great use for DDD.

2. Maintainable designs

This team wanted to build software that was easy to maintain. Their current systems were fine but were far from easy to work with and they knew that they'd just get worse over time. They wanted to introduce DDD so they could build better software that is actually maintainable, instead of a codebase that slowly rots.

3. Avoid DRY on superficial similarity, e.g. Eric Evans Toilet/Sink story

This was a great one. This attendee was a CTO and wanted to make sure that devs knew how to build modular code, specifically bounded contexts. A hindrance to this was our over-reliance on DRY, our tendency to aggressively remove duplication.

You see, sometimes different parts of a system have the similar code, but they have different rates of change. Coupling them together hinders change. The example story told was an Eric Evans' one on toilets and sinks. They both get rid of water in a pipe, so a "smart" developer would notice the similarities and merge the two concepts together, job done. Then later a feature request comes into for a shredder to be installed in the sink. But uhoh, the pipe concepts are merged, so adding a shredder to the sink also adds a shredder to the waste system. Next time the toilets flushed, well ... it doesn't end well. These were the situations they wanted to avoid, and DDD was a way to get that mindset across.

4. Create a public API and build a vocabulary for it

An interesting problem. Two of the attendees were tasked with taking an existing legacy system and exposing it's functionality via a HTTP API. This was a mammoth task given the age and size of the codebase, so they wanted to ensure they were exposing it's functionality in a way that made sense while also hiding some of the messier details. They'd heard that DDD could help and I can't help but agree (I recommended they looked into an Anti-corruption layer for a start).

5. Figure out if we need to build software at all

Another interesting reason and one that isn't brought up often enough, especially by developers. This group wanted to use DDD to understand the problems the business faced, and what options there were to solve them. Do you need to write software? Can they use existing tools? Can you solve the problem by gluing tools together (e.g. Typeform, Trello and Zapier)? These are the kinds of questions we need to ask and this team wanted to make sure they introduced the right level of complication. Great reasoning and DDD can definitely facilitate it.

How to Introduce DDD

At this point we moved backed to the slides, which began by looking at "how" you introduce DDD. We knew why, so now we had to figure out how to phrase our "whys" so that the rest of the business would understand the benefit. In other-words, we had to understand their domain and how we could help them. We had to sell the concept.

I presented three known techniques that have worked for me in the past. Click the links beside the techniques for more details.

Event Storming session:
- The concept: https://www.eventstorming.com/
- Tips: https://slides.com/barryosull/event-storming-workshop-building-noteworthy-9#/10
Talk to the domain experts:
- The concept: https://www.quora.com/What-does-it-mean-to-be-a-domain-expert
- Tips: https://slides.com/barryosull/event-storming-workshop-building-noteworthy-9#/12
Separate Domain Code from Infra Code:
- The concept: https://barryosull.com/blog/cleaning-up-your-codebase-with-a-clean-architecture/
- Tips: https://slides.com/barryosull/event-storming-workshop-building-noteworthy-9#/14

With those three techniques highlighted we then had a short break for food and beverages, then went back for the final discussion session.

Mapping your Why to a How

This is what it all leading up to. We wanted attendees to discuss their problems (their "why") and then figure out how they could introduce it (the "how"). We gave this session about 25 minutes with teams of 4 to 5, at the end each team presented their technique for introducing DDD, be it one they've tried already or one that they thought would work in their context. This is what we ended up with:

I started writing ValueObjects and it snowballed from there (love this one)
Micro-services are not an end goal, they're a technique along the way
Identify the contexts that will change per feature, allows you to understand/demonstrate cost
DDD is slower initially, like any new idea or techniques, make this cost known up front
Wait for pain, use DDD to solve it. Demonstrate the value

I was delighted with these, great suggestions from different perspectives yet again. As before, we discussed each further with the groups, here's what we gathered.

1. I started writing ValueObjects and it snowballed from there

This one was hilarious and completely honest. The attendee in question was originally a dev at a company and wanted to improve their ball of mud, so he started introducing value objects to bring at least a little bit of clarity. It kinda snowballed from there, with him introducing more and more DDD concepts over time, slowly massaging the messy system until it started to make more sense. This didn't go unnoticed and when his dev manager left, he was made head of dev and he is still applying those practices today. This just goes to show that you don't need the entire team to be behind it, you just need to demonstrate the value.

2. Micro-services are not an end goal, they're a technique along the way

Now this one was different. This group was in the process of migrating from a monolith to a micro-service architecture and they were noticing problems with it. The main issue was that having micro-services was considered the end goal, it didn't seem to matter if these services mirrored the actual domain and allowed greater agility in the business. They realised they could sell DDD by focusing on the agility and clarity it brings and how it naturally leads to micro-services as a byproduct, rather than micro-services being the objective from the start. Definitely agree with this one, you need a vision for your architecture, and "micro-services" are not a vision, they're a strategy to enable a vision. DDD can help define and clarify that vision.

3. Identify the contexts that will change per feature, allows you to understand/demonstrate cost

This team decided to focus on cost first. If they could demonstrate how DDD could help the business understand how difficult/expensive a feature will be to implement, then management will be able to make better decisions. This demonstrates the value of DDD straight away. Contexts were chosen as the starting point, as that was the DDD concept they thought would help the company the best. Contexts are a great way to classify a system, understand its boundaries and its upstream/downstream dependencies. By defining the contexts they would be able to demonstrate how complicated a change is. A great way to introduce DDD.

4. DDD is slower initially, like any new idea or techniques, make this cost known up front

Thought this was an odd one, at least initially. The team was thinking about how they'd introduce it and wondering how much it would affect things upon introduction. We all agreed that introducing a new way of doings things would slow down development, there's always a learning curve. The idea was to demonstrate that the cost of this is far out weighed by the benefits over time. This team felt that quantifying the cost in some way would help show management that you're not just hyping up the latest sexy silver bullet, instead they actually understand the costs and have factored them into the pitch. I think this a solid idea and can only help but improve your chances of the concept being embraced.

5. Wait for pain, use DDD to solve it. I.e. demonstrate the value

Another great one. The simplest way to demonstrate value is to solve a problem. But how do we find problems? Well, we don't, they'll find us, masquerading as the feeling of pain. When something goes wrong, or someone complains, that pain is a sign of a problem in the system, something went wrong or is getting in the way (bugs are the obvious example). This is where you use DDD and discuss the pain with those affected, figure out how you could solve it, using the language of the business and the techniques offered by DDD. It's a great way to get your foot in the door and for you to get direct access to the business owners. Once you do this and solve problems it just greases the wheels towards it happening more often. I've seen this in action and it works great.

Ending the meetup

At this point we reviewed the above and ended the meetup. Some of us went for a drink and discussed concepts further, but nothing strongly related to the above. We had a chat about event broadcasting and bounded context, should you share domain events (the answer we all agreed on was "no").

Overall it was a fantastic meetup and I hope to have another like it soon. I felt that the above was far too useful to keep to ourselves and I wanted to share it with others. Thanks to all that attended and to all those that read the above, I appreciate your time.

Folder Structure and Frameworks: What is exerting control?

Barry O Sullivan — Fri, 17 May 2019 11:31:11 +0000

Recently I've been thinking about folder structures, specifically how we structure our web apps to encourage the design we want and to enable other developers to explore and understand the codebase. This train of thought was spurred by a problem we faced with one of our apps, which I'll go into shortly.

Folders Give Context

When we open up an application the folder structure is the first thing we see, even before we glance down at the readme. It conveys the hierarchy of concepts and hopefully how they relate to each other. A haphazard folder hurts more than it helps, especially if you have to hop around from folder to folder. Choosing the right structure is important, it's why so many frameworks come with a structure already defined, it's a foundation you can easily build on.

Now some developers advocate putting all your code into a single folder that's "per feature", e.g. controllers, DB accessors, repos, configs, etc... Honestly I've never seen this work out, it's always a jumbled mess that falls apart once you have more than 7 classes, so I'm just gonna disregard that notion straight off the bat.

With that out of the way let's look at the folder structure that was causing us issues.

The Status Quo

/app
    /Funding
        /App
        /Domain
        /Infra
/bootstrap
/config
/database
/public
/resources
/routes
/storage
/tests
/vendor

This is the structure of our web app, as you can see it's a fairly standard Laravel app, the only thing that's different is the internals of the "app" folder structure.

Quick bit of background, we structure our codebases using a Clean Architecture/Onion Architecture. I won't go into too much detail but here's a quick overview:

Domain: The core code of the system, models the problem you're solving. Contains no technical details (e.g. no SQL or DB concepts), focuses on business language/concepts instead.

Application: Compose domain objects into a single business operation (e.g. CreateUser), allow domain code to interact with external systems via interfaces (e.g. a NotificationService or a UserRepository).

Infrastructure: The implementation of domain/app concepts. All technical details and framework bindings live here. This is where you glue your domain/app code to technical concepts such as databases and/or libraries. HTTP controllers live here, as they are technical concepts that plug into domain/app code.

The main reason for this is to decouple your system from implementation details, making it easier to design, understand and test.

The Problem

The issue that sparked my introspection into folder structure was the high level folder "app". "app" is the default folder created by Laravel for your applications code (thus the name). However you can see that within "app" is another folder called "App". We didn't like this as it meant there were two folders in a hierarchy with the same name, despite the fact that they serve different purposes. One is the framework's concept of an "app", the other is the defined interface for our "App", i.e. the input and outputs decoupled from the framework and technical details. This is potentially confusing.

We had a discussion about changing the folder name to be clearer, since "app" isn't great. We iterated on a couple of names, including "components", "src", even "code" (I'm not joking). None of these really fit. We realised that changing the name would break Laravel convention which would confuse new developers.

This got me thinking, why are we letting the framework control this? It's an implementation detail after all. On top of that our business code is now wrapped and contained by the framework code. This implies that our code is a subset of the system, even though the opposite is true. This affects our thinking and constantly re-enforces the idea that the framework is in control (even though it shouldn't be).

This is a problem that I've seen developers spend an awful lot of time on, they focus on how they're business concepts fits into the framework rather than the opposite way around. This leads to every concepts being seen through the lens of the framework, muddying the concept and making it harder for people to understand intent.

If you think about it, this structure breaks the Dependency Inversion Principle. We have abstractions (the business concepts) being controlled by the details (the framework). That's bound to cause issues.

Instead I'd like a structure that makes encourages the separation between the code we write to solve a business problem and the code we use to do that job.

So with all the above in mind, how would we structure our codebase? (You can probably guess where I'm going with this).

The New Structure

/contexts
    /Funding
        /App
        /Domain
        /Fund
/framework
    /bootstrap
    /config
    /database
    /resources
    /routes
    /storage
/public
/tests
/vendor

First of, you'll notice that our "app" code is now called "contexts". By naming it contexts we make it very clear that the code inside is solving a particular problem for a specific sub domain (here's some detail on the concept of bounded contexts). As the application grows we'll add more contexts. "app" was far too generic a name, whereas "contexts" gives context straight away (I'm so sorry for that pun).

Second you'll notice that the framework code is now contained is its own folder structure, independent of the contexts. This makes it very clear that the framework is a detail, rather than the controller of the system. It is a component that our contexts use (much like the "vendor" folder), rather than a system exerting design control on our contexts.

Tests is still outside, as the test code should be coupled to the contexts and not the framework. You're testing that your system works as expected, not that the framework works as expected. This structure encourages this thinking which I believe leads to better system design (don't couple test code directly to frameworks).

Public is at the root level as it usually contains lots of resources that are framework independent, the only thing that's framework/system specific is the code within the index page that boots the app, and that's not a good enough reason to bundle it and all the other resources (css, js, images, etc...) with the framework, it really is a separate thing.

Effectively I've inverted the dependencies to the folder structure, now the abstractions (the business code) is no longer contained within the details (the framework).

Structure matters

How you structure your codebase will influence how you think about it, and our practice of building our application code inside our framework code can lead to problems. It's forces a way of thinking that just muddies the water, for junior and senior developers alike.

My goal with the above structure is to make it very clear that the contexts are the heart of the application, not the framework. This guides developers to focus on writing solid context code, written from the perspective of the domain rather than the implementation details. I believe this structure encourages better design and aids developers in understanding the distinction and responsibility of each folder. In short, less noise, more signal.

What about you? Have any interesting structures you'd like to share? Feel free to reach out in the comments below or message me on twitter.

Managing projectors is harder than you think

Barry O Sullivan — Mon, 13 Aug 2018 12:32:23 +0000

We've discussed the bones of projectors in the past, this time let's go deeper and look at how to manage them.

At it's simplest a projector is something that takes in a stream of events and does some work on them, projecting them into whatever shape or operation is needed. Like anything though, there's more to it than that, lots more. That's what this article is, my attempt to discuss the complications and problems you will run into while working with projectors day to day.

Run modes

Let's start simple, let's talk about the different modes of projectors and how they behave. In my experience it boils down to three modes.

Run from beginning (standard projector)
Run from now (what Laravel are calling "Reactors")
Run once (special case)

1. Run from Beginning

This one is pretty simple, start at the first event and play forward from there. These projectors will play through all historical events, and then continue processing any new events that occur. Most projectors will run in this mode by default.

2. Run from Now

Some projectors don't need historical events, they only care about events that happen after they're released. For example, say you wanted to release a new email service. It's job is to send welcome emails to new users when there is a UserRegistered event, but you only want this to happen for new users, existing users shouldn't receive anything.

This projector mode only processes UserRegistered events created after the projector is released, so only new users will get emails. Nice and simple. This projector mode is rarer than "Run from beginning", but it's still a must have.

3. Run Once

The rarest projector type. I've only created four of these, but they were essential to adapting a living system. These projectors play forward from the first event, but they only run once. Once they're run, they'll never run again.

Why would you need this? Well we used them for difficult migration issues. Say you need to update a domain model so that it has new data. Now, it's easy to add this change to the code so newly created objects have the data, but what about historical objects? This is where run-once mode comes into play. You write a projector that back-fills the missing data from historical events, at deploy you run it, adding the missing data, then on release it stops running and never runs again. Now all your data is up to date. Think of them as migrations that upgrade existing data structures via events, usually in prep for a release.

How that's we've looked at the different modes, let's discuss the lifecycle of a projector, and see how modes come into play.

Projector lifecycles

There are three stages in the lifecycle of a projector, from start to finish they are.

Boot
Play
Retire

We're going to start with number 2, play, because it's how most people are used to thinking about projectors.

2. Play

The most basic task, a projector must be able to play events. This is the bread and butter of projectors; when an event is triggered, the projector will process that event. This is typically called "playing" a projector. Usually you'll have a process that manages this and records the position of each projector in the event stream.

This is all very quick for active projectors, but it can be a bit of a problem for new projectors, which have to play though the entire event steam to catch up, that's why we have a separate booting process.

1. Boot

The goal of the boot stage is to prepare a projector for release. When a "Run from Beginning" projector is introduced to an app, it will need to play all the events in the log. This process takes time, and if you've pushed the code live, anything reading the data will get old, possibly dodgy, data. E.g. Reports with incorrect numbers. To solve this you should have a distinct boot process that preps projectors before release.

Booting should only happen during the deploy process, and it's best if it's the last step before pushing code live. Booting only affect new projectors, i.e. ones without a position record. Once all projectors has been booted, you can safely deploy the new code, letting the standard play system for projectors take over. To users of the system it's seamless.

3. Retire

Of course after a while you'll no longer need some of your projectors and you'll want to remove them from your system.

Retiring projectors is pretty simple and involves three steps:

Delete the projector code
Delete all data stored in the projections controlled by the projector
Delete the record keeping track of the projector's position

BTW, for steps 2 and 3 I'd suggest using migrations, it's explicit and you have a full historical record of changes.

When to retire

You probably think you should retire projectors all in one go as soon as you don't need them, but that's not a great idea. Imagine you have to rollback your system to a deploy with a retired projector. Before the rollback can complete it would need to rebuild the projector, which takes time, time you don't have if the rollback is critical. This is why I usually leave leave step 2 and 3 for a few days, only deleting the code from the repo to start.

Another tip, you should only completely retire side-effect free projectors. Projectors that trigger side-effects (e.g. sending emails) should not have their position record deleted (step 3), otherwise you run the risk of accidentally replaying them, triggering the side-effects again. It's safer to delete the code and projection data (steps 1 and 2), but leave the actual position record alone.

Projector failure

I've got some sad news, at some point your projectors will fail. I know, it should never happen, but bugs are inevitable and it's wise to prepare for them. Some examples I've seen, invalid event schemas, missing DB columns, missing data in DB (bad assumptions about event order), broken external service, etc... you get the idea.

Projectors can fail during the boot or the play processes, and how you handle failure depends on which.

Boot failures

When a projector fails during boot, it should stop the boot process immediately and mark the projector as broken. If there are other projectors being booted at the same time, they should be marked as stalled (more on this soon). Then your team should be alerted of the issue, probably via a bug tracker. It goes without saying that "deploy" process should fail as well.

Now it's a case of fixing the broken projector and triggering a new deploy. When boot runs the second time around, it should attempt to play both the broken and stalled projectors forward from their last position. If the bug was fixed, then they'll all boot successfully and you can deploy the code. Done. If not, then repeat the process until it's all fixed.

Play failures

Boot will catch most bugs, but sometimes you'll encounter an error while a projector is running as part of the live system. If a projector fails while being played, it should record the failure, mark the projector as broken, alert the team, and then stop playing that projector. All other projectors should keep going as normal, as there's no point in one failing projector bringing down the entire application.

Once the projector is fixed, trigger a deploy, which will cause boot to attempt to play the broken projector forward. If it's fixed, then it will successfully play the events and you'll be able to do a release.

Conclusion

To summarise, projectors are more complex than people think. If you want to build a stable, maintainable projector system, keep the following in mind.

Run modes
Lifecycle
Failures

If you have a handle on all three of these, then you're set. You'll be able to deploy and manage projectors with ease, fixing issues when they arise, with minimal downtime for you or your customers.

I've built a projectionist system to solve the above problem, but it's not ready for the limelight, it's more of an experiment in distilling the above down into a single system. Feel free to have a look though! https://github.com/barryosull/the-projectionist

And with that, happy projecting!

Domain Driven Design for Everyone Else

Barry O Sullivan — Thu, 19 Jul 2018 09:50:55 +0000

I've been talking a lot about Domain Driven Design (DDD) lately, be it at meetups or with clients, so I thought I'd write down my thoughts and see if it helps.

Now, lots of people have written about DDD from a technical perspective (see the end for links), so I'm not going to do that, instead I'm going to discuss DDD from a non-technical perspective.

This is DDD for everyone else.

Solutions Always Overrun

Designing and building a solution is not a trivial problem. It never goes smoothly, and even if it's completed on time (which is never) the solution is usually ineffective and needs to be changed, often drastically. This leads to more delays, bigger budgets and even larger problems down the line.

Why does this keep happening?

Complex vs Complicated

Building a profitable business is a complex problem, and complex is different to complicated.

For example, "taxes" are complicated. There are many, many interweaving rules and processes, but once you know how to apply them (the process), you've solved the problem. It becomes procedural; no thought is required. Just follow the process and you'll be fine.

Building a business is not like that at all, there are just too many unknowns. In other words, it's complex. This is fairly obvious if you think about it. If such processes existed then everyone would just use them. There would be zero risks whatsoever (and this article wouldn't need to exist).

Complex problems cannot be controlled, they can only be managed. Yet we still try to treat business development as a process, because we really want it to be one. Just look at the popularity of "Agile"* and "Lean", especially the heavily marketed, process orientated versions, they re-enforce this illusion (and they also don't work). It's wishful thinking.

DDD acknowledges this fact, and instead of focussing on rigid processes and hard rules (i.e. one size fits all solutions), it presents techniques to manage and remove ambiguity. The secret isn't following a process, it's about iterating on the problem you're trying to solve.

Focus on the (right) problem

In DDD the domain, i.e. the problem and its resulting knowledge/activities, is the driver of everything else. All solutions flow from problems, so putting the core domain at the centre will naturally lead to better solutions.

For example, say your business is online news. Your "core" domain (the one that drives your business) is content generation, everything else is secondary. By producing better content faster, you'll grow your business. However, if you focus on solving the problem of resizing images (and hire a bunch of developers to write the software) then you're probably not going to grow your business.

DDD brings clarity, and through clarity, we're able to focus.

Talk to the experts

If you want to understand a problem, then you need to talk to the domain experts. A domain expert is someone that understands the problem better than anyone else, and they are able to tell you what's important and what isn't.

In most organisations, the domain experts are not the ones building a solution. DDD helps to bridge the knowledge gap between what the domain experts know and what those building the solution are trying to understand.

This is why a large chunk of DDD techniques have nothing to do with technology, instead they focus on people and ways to unearth complexity and ambiguity. If we're all on the same page, it's easier to move forward.

Building a shared understanding

One key way to gain clarity is to build a shared understanding of the problem, i.e. a domain model. If everyone has the same model in their head, then there's no ambiguity. It helps us avoid overly complicating our solutions, or worse, building the wrong one (e.g. the image resizer above).

Sprechen Sie Talk, huh?

Language is core to DDD. We use language to express our ideas, to explore problems and define solutions. If the domain is complex, then that language will be rich and complex, with its own subtleties and nuance.

The thing is, most people in your business may not share that language. Instead they use their own language to solve problems, which is fine. However, problems arise when two or more people try to communicate using different language without realising it, e.g. the same words but with different meanings. This causes ambiguity and its a leading cause of errors and misunderstandings.

The more people you add to the mix, the worse the problem gets. Take the typical chain of command for example.

That's a lot of room for mis-communication. If you've ever requested a feature and gotten back something that is way off the mark, this is why.

The solution is for everyone to work closely with domain experts to understand the language as the business defines it, and figure out where these language boundaries exist, i.e. when a word is being using in a different context. This enables everyone to communicate (i.e. have shared domain model), leading to less siloing, better collaboration and simpler solutions.

Your solution reflects your understanding of the problem

Any solution you build is a direct reflection of how well you understand the problem. If the solution is good, then you understand your domain, if it's bad, then you don't. This is why quick iteration is so important, it's about tightening feedback loops and iterating on the problem, not building the right solution the first time you try (which never happens).

By using your solution as feedback, you can have further discussions and figure out if you were close or not, which leads to a better domain model, which results in a new solution, which feeds back into your understanding of the problem. It's cyclic and it encourages us to better understand what we're doing, rather than focussing on the solution as the be all and end all.

Faster feedback is better feedback

The faster you can test a solution the better. This doesn't mean you need to build software, that's the most expensive way to test. Instead why not prototype a solution with mockups or even pen and paper? You'll get the same feedback at a fraction of the timescale. DDD encourages upfront discovery and iteration, not writing software for the sake of it.

That's not to say DDD doesn't talk about software a lot, it definitely does, and it has lots of patterns for writing it, but software isn't the core. A common adage in DDD is that the best software is no software. You see, software adds complexity, so if you can avoid that complexity, you should. A good DDD practitioner will try to find existing solutions to problems, only falling back on custom software if it brings the most value.

Self Improvement

DDD is about the problem of understanding and iterating on problems. This is cyclic, meaning DDD is frequently used to understand itself and improve on itself. You can imagine how fast this feedback loop is, and DDD practitioners are always iterating, discovering and naming patterns and techniques. It only gets better with time.

Conclusion

In short, DDD puts the business and its domain in the driving seat, where it should be. I've been applying DDD for 5 years now and I have to say that my ability to build useful solutions has increased massively. It's helped me hone the techniques and skills I need to understand and iterate on problems. If you want to get better at building solutions, I'd recommend DDD wholeheartedly.

Communication styles - Working effectively as a team

Barry O Sullivan — Mon, 11 Jun 2018 12:15:12 +0000

This idea for this article came from a twitter thread by @ErynnBrook.

// Detect dark theme var iframe = document.getElementById('tweet-1001881209733988352-499'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1001881209733988352&theme=dark" }

The thread explained the concept of man-splaining (which was eye-opening as I am a chronic explainer), but it also delved into communication styles, which is where things got really interesting.

Styles of communication

In short, there are two styles of communication:

Competition Style
Connection Style

Competition style is about coming out on top. Every statement is viewed as a move in a game, the goal is to say something that "beats" whatever the other players are saying, leading to a constant back of forth until there is a "winner".

Connection style is about connecting to the other person. Every statement is viewed as an attempt to empathise and an opportunity to explore a topic with them, creating a shared understanding of the issue and deepening the social connection between parties.

What's fascinating is that neither style is inherently better than the other, competition is good in certain contexts whereas connection is better in others. Most of the time we're not even aware that we're using a certain style, it's just how we communicate. This can lead to tension, as two people communicating using different styles are not going to reach consensus.

With the above understanding under my belt, I realised I'd seen both styles used everywhere, in software development and every other aspect of business. Great teams used each style effectively, creating team consensus and better solutions, whereas poor teams used them incorrectly, creating tension and poor results. Let's go through some styles of collaboration and explore when each style works best.

Exploring solutions

Say your team needs to come up with a solution to a problem (be it implementing a new feature or fixing a broken one), what's a good way to do this?

First off, you should start with a connective style, as you need everyone to be on the same page about the problem at hand. Your goal is to reach consensus, if you can't do that then there's no point in moving forward.
Once you've agreed on what you're there to do, it's time to brainstorm, which means a switch to competitive style. The goal here is to compete with each other to come up with the "best" solution, creating as many as you can. Ego will play a roll here and that's fine. It's also ok to point out flaws in someone else's solution, but don't focus on that, the goal is to come up with solutions, not filter them.
Now that you've brain stormed, it's time to reach consensus, that's where connective style comes into play again. Go through each of the ideas and make sure that each one of you has a shared understanding of the pros and cons of each. This isn't about being right, it's about a shared understanding.
Now that everyone is on the same page, we can switch back to competitive and try to figure out which is best. At this point, ego should not be in play. Winning here isn't about choosing an idea you created, it's about figuring out which idea is best overall. This might involve choosing one solution, or combining solutions (which I find is often the case). It's a competition between solutions, not people.
Once you've done this and you think you've reached the best solution, don't end the conversation, instead be sure to switch back to connective style and make sure everyone is on the same page and is in agreement. This ensures that there's no unresolved tension and everyone is agreed on next steps. At the end, everyone should be happy and even feel comfortable saying "Go team!".

Looking at the above, it's clear there's a pattern, which at it's simplest is the following.

Connective -> Competitive -> Connective -> Competitive -> Connective

It turns out this is a recurring pattern.

The feedback loop

At it's core it's all about the feedback loop. The above can be distilled down to the following, a constant loop where you switch between styles when appropriate, starting and ending at connective.

If you need to come up with lots of ideas, go competitive, when you need consensus, switch to connective. It really is that simple.

It's everywhere

What's funny is the same process works in so many other contexts.

Say you're trying to figure out the fix to a bug, it will follow the same process.

First you reach agreement on the bug (connective)
Then you compete on figuring out potential causes (competitive)
Then you reach agreement on which is most likely (connective)
Then compete on ways to prove it's actually the bug (competitive)
Then agree on who will do the work and how it will be verified (connective)

What about pair programming? Same thing, it's a constant switch between the two styles. If you need to explore solutions, compete to come up with ideas. When you need to reach consensus on the next step, connect.

Using the wrong style

A little word of warning, I mentioned above that I've seen styles misused. Let's quickly go into that.

Say you're trying to create ideas, if you're in a connective communication style, the team will settle on the first solution presented, even if it has glaring flaws that healthy competition would unearth.

The same is true with competing at the wrong time. If you're trying to reach consensus and instead of agreeing you're competing, then you'll get nowhere, you'll just end up frustrated and tired.

Finally you could have team members using different styles at the same time, so it always feels like you're never on the same page and that your discussions have no momentum, they're always stopping and starting jarringly.

A quick guide to spotting the above:

Are you always agreeing with each other but your solutions fall apart as soon as you implement them? Then you're not competing enough.
Is your team tense and you always feel like you're never getting your say, you're always doing what someone else says? Then you're not connecting enough.
Are you uncomfortable when communicating with certain members of the team? You might be connecting when they are competing, so you may need to switch styles (temporarily).

Conclusion

So that's that. When you're working with others, figure out what is the best style to apply for the given situation.
Are you exploring ideas? Then compete. Are you trying to reach consensus so you can move forward? Then connect.

In my mind, everyone is a leader, so if you notice that the team is stuck in an ineffective mode, attempt to switch them out of it.
(The twitter thread linked above has some great advice on how to switch from competitive to connective, have a read through so you're better prepared).

For myself, I'm going to pay more attention to how I communicate and I'm going to encourage those on my team to do the same. Ultimately, it's about working together to be effective.

Acceptance testing your PHP app with ease

Barry O Sullivan — Mon, 23 Apr 2018 11:50:27 +0000

Acceptance tests are core to any stable system, they're how you make sure it actually works, start to finish (My preference is to write them first, use them a guideline to make sure the feature I'm writing works as expected).

When writing acceptance tests, it's best to treat the system as a blackbox, inputs go in and outputs go out, that's it. This proves our app works and can be interacted with by other systems. Some frameworks come with this built in, like Laravel, but not every app is written in those frameworks, infact most are not (especially legacy apps), and they still need to be tested.

If your app is a HTTP driven API, you'll need to test it accepts HTTP requests. So you'll need to boot up a webserver, configure it, send it HTTP requests and then check the responses. You'll probably have some console commands as well, so you'll also need to write tests for them. And let's not forget checking the database, that's an output after-all.
How do you do all this in PHP?

PHPs Built-in WebServer

First off, it turns out the web server part is very easy, PHP comes with one built-in (as of PHP 5.4). Simply run php -S 127.0.0.1:8000 at the entry point for the application and you're good to go.

Of course, there's a little more to it than that. As we're using PHPUnit for our tests (because why would you use anything else?), we want to launch the web server from our acceptance tests and then send it requests. We also want the web server to shut down once the tests have completed.

In order to make things easier for ourselves, I've written a simple class that takes care of the above. Have a look, and I'll explain the details below.

<?php
declare(strict_types=1);

namespace Root\ProjectTests\Acceptance;

use GuzzleHttp\Client;

class WebApp
{
    private $host;
    private $entryPoint;

    private static $localWebServerId = null;

    public function __construct(string $host, string $entryPoint) 
    {
        $this->host = $host;
        $this->entryPoint = $entryPoint;
    }

    public function startWebServer()
    {
        if ($this->isRunning()) {
            return;
        }

        $this->launchWebServer();
        $this->waitUntilWebServerAcceptsRequests();
        $this->stopWebserverOnShutdown();
    }

    private function isRunning(): bool
    {
        return isset(self::$localWebServerId);
    }

    private function launchWebServer()
    {
        $command = sprintf(
            'php -S %s -t %s >/dev/null 2>&1 & echo $!',
            $this->host,
            __DIR__.'/../../'.$this->entryPoint
        );

        $output = array();
        exec($command, $output);
        self::$localWebServerId = (int) $output[0];
    }

    private function waitUntilWebServerAcceptsRequests()
    {
        exec('bash '.__DIR__.'/wait-for-it.sh '.$this->host);
    }

    private function stopWebServerOnShutdown()
    {
        register_shutdown_function(function () {
            exec('kill '.self::$localWebServerId);
        });
    }

    public function makeClient(): Client
    {
        return new Client([
            'base_uri' => 'http://'.$this->host,
            'http_errors' => false,
        ]);
    }
}

This bit of code launches a web server and waits for it to start accepting requests, then it registers a shutdown function to kill the web server once all the tests have completed. We're using a script called 'wait-for-it' (found here) that waits for the web server to go live before continuing. This was added because sometimes the tests would start before the server was actually active. We've also ensured that calling launchWebServer multiple times won't cause any issues. If there's a web server currently running it just stops.

Once the server is running you can call makeClient, which gives us a http client, specifically a Guzzle one (again, why would you use anything else), configured to send requests to that server. Now you can begin testing HTTP requests.

Configuration

We can launch a webserver and send it requests, but how do we configure it? What database does it use, where does it log errors? You're most likely using environment variables to configure these details (and .env files to store those values). A solution could be to create different .env files for each environment, then loading the right one at runtime. This is a bit of a pain in the ass, and thankfully, it is not required.

PHPUnit has a config section for environment vars, these vars are auto loaded for each PHPUnit process. Here's an example.

<php>
    <env name="APP_ENV" value="testing"/>
    <env name="DB_NAME" value="test_db"/>
    <env name="DB_HOST" value="127.0.0.1"/>
    <env name="DB_USER" value="root"/>
    <env name="DB_PASSWORD" value=""/>
    <!-- Ohh no, my precious DB credentials, please don't hack me! -->
</php>

This is where things get interesting; any sub processes created by the parent test process will also have access to these env variables. In other words, the web server we launched via exec from our tests will automatically have all these env variables pre-loaded, so we don't have to worry about setting up any vars. It's super handy and makes testing a breeze.

Continuous Integration

At this stage we have a web server that's configured for our local tests (through phpunit.xml). That's fine, but what happens when we want to run these acceptance tests on a CI server, such as CircleCI or TravisCI? They won't have the same config details as our local machine, so how do we configure them to work correctly?

Again, it turns out this pretty simple. CircleCI, for example, allows you to define environment variables in your config, which it pre-loads into the server. Now, you may think that these vars will be overwritten by PHPUnit, but don't worry, phpunit.xml env vars will not override existing env values. I.e. If you've already setup the env vars for the database through the CI boot script, then PHPUnit will leave them as is. That means the above web server will just work on the CI system, no code modification is required.

Console commands

Of course, not all requests are HTTP, sometimes you'll want to run commands via the console. How do we configure that? Again, this is pretty simple, we run the command via exec from within our test code. For instance, here's a symfony style command call.

exec("php bin/console app:create-user");

As I said previously, any process started by our tests will inherit the same env vars as the parent, so our app has everything it needs to run via exec (or system if you prefer), no changes needed.

Data storage

Checking the response usually isn't enough for an acceptance test, you'll also need to check that right data was sent to the database. How we access the DB from within the tests?

Yet again, this is very simple. The PHPUnit process has access to the env vars, so you can just create a PDO instance and run queries against it directly. Same with any other services, be they file storage (s3) or APIs (though this is trickier, I may need to write about this at some stage). To things even simpler, use your application container (like PHP-DI) to inject these services into your test code already pre-configured. Job done.

It's that easy

So it turns out that acceptance testing a PHP app locally is trivial. Not only that, but it's easy to run the exact same same process on a CI system, with little to no changes. At this stage I hope you're seeing the appeal, you can easily treat your system like it's a blackbox, which is a great way to get stability and confidence in your code.

Immediate vs eventual consistency

Barry O Sullivan — Mon, 09 Apr 2018 12:07:37 +0000

In the last article we looked at projectors, the backbone of any CQRS/Event Driven system. This article was originally meant to be about implementing projectors, but I realised there was an important question to answer first, one that would shape the solution, "When do we project the events, now, or later?". Turns out this question has far reaching effects, so it's important we dig into it before moving onward.

Immediate vs Eventual Consistency

When it comes to projectors there are two choices, immediate or eventual consistency. With immediate, events are processed by projectors as soon as they happen. With eventual, events get processed in a different process at a later time (usually a split second later).

Immediate is an all or nothing operation, if anything goes wrong then the entire process is halted. No events are stored and no events are processed. Eventual is a staggered operation, once the events are stored each of the projectors will process them at a later time (and potentially fail).

Reasons to not use Immediate Consistency

From the above you may think that immediate is the obvious choice, it seems simpler and has less moving parts. Well, that simplicity is an illusion. It turns out immediate is far more complex and the following questions will illustrate why.

1. What happens if one of the projectors has an error?

Say a projector fails and throws and exception, what do you do? The ideal solution is to roll back all your changes, ie. act like it never happened. This is easy enough if you're using a single DB to store everything (transactions FTW), but if you're using multiple technologies (e.g. Redis/MySQL/MongoDB/etc...) then this problem becomes a lot harder. Do you roll back all of them? How do you manage that? How do you test it? What happens if you make two API calls that you can't roll back? Hmmm, things just got very complicated.

2. What happens if one of the projectors has a temporary error?

Some errors are temporary. Say you have a projector that connects to an API that's rate limited.

A user makes a request causing an event
The projector tries to process that event
It makes an API request, you're over the rate limit, so it fails
Another user makes a request causing an event
The projector tries to process that event
It makes an API request, you're under the limit, so it goes through

How do you handle this? Do you just accept it and allow processes to fail? Do you force the user to retry the request and hope it works this time? That's not a great user experience and will definitely annoy people.

3. What happens if one of the projectors is slow?

Say one of the projectors performs an expensive process, like connecting to a slow external service (eg. sending an email). With immediate consistency we have to wait for it to complete before we can let the user continue. Worse, if we're using transactions to ensure data integrity across a domain (e.g. email address is unique), then you're potentially slowing down other processes, not just this one. These become bottlenecks that affect the entire system.

4. What happens when you launch a new projector in a running system?

Even if you opt for immediately consistent projectors, you'll still need some way to play historical events into projectors, otherwise you'll be unable to launch new ones. While new projectors are spinning up, they are not consistent with the live system, but they will be. You can architect the process so that you only release the code when all the projectors have finished (we did this, worked really well), but even so, to do this you had to build the core of an eventually consistent system.

5. What if you need to process events on a different service?

Ahh, the classic problem of distributed systems. Processing events within a single service can get complicated, but it's nothing compared to immediately processing events on a different service/server. The laziest solution is to force other services to process the events immediately via a synchronous call, but now you've coupled yourself to that system; if it goes down; you go down, and what do you do then? Immediate consistency becomes a lot harder (and next to impossible) once you're communicating with another service, even if it's one you control yourself.

Reasons to use Eventual Consistency

Now that's we've seen the problems caused by forcing immediate consistency, let's look at how things fare when we take an eventually consistent approach.

1. What happens if one of the projectors has an error?

That's fine, if there's an error, report it, maybe disable the projector (depends on the error). Once an event has happened, it's happened, so if one projector fails we don't need to roll back the events or the changes to other projectors. Instead we fix the projector, roll out a new release and let it catch up. Once you embrace eventual consistency, problems like this become a lot easier to handle.

2. What happens if one of the projectors has a temporary error?

Again, pretty simple. We simply swallow the error and try again. We know the request will eventually get through, we just need to keep sending it. If it's a rate limiting issue, we can throttle the projector, slowing down the speed at which it's processing events.

3. What happens if one of the projectors is slow?

This isn't an issue at all. Projectors run in their own background process, so they can take as long as they want. If we find out one projector in particular is slow and is affecting others, we can simply move it into it's own process and move on. Nice and easy.

4. What happens when you launch a new projector in a running system?

Not much to say here. Running a new projector is the same as running any other projector, it will play though the events until it eventually catches up. This simply isn't a problem anymore.

5. What if you need to process events on a different service?

Yep, no issues here either. Events are consumed by other services at their own pace. The producing service doesn't need to wait for them to handle the events, so it doesn't matter if they're running slow, or even that they're running at all.

Immediately consistent views

At this point I've hopefully convinced you that eventual is better, but there's still one problem to address, one you're probably asking right now, "What happens when you need views to be immediately consistent?".

Let's take an example, say you've processed a request to add an item to a cart, and the user is redirected to the cart page, what do you do if the cart is rendered without the latest item because the cart projector is running slow? Simple, you fake it. You render the page as if the item is actually in the cart, even if the view says it isn't.

This isn't as crazy as it sounds, in-fact most apps do this all the time and you barely notice. Have you ever posted to Facebook, seen your post appear, then refreshed the page and noticed your post isn't there? They were faking it. This fakery is mostly done on the client side, and it's made even easier by the likes of the reflux. This pattern is more commonly know as an optimistic UI, here's an article on the concept.

What I'm trying to say is that it really isn't a big deal, apps do this all time and it's very easy to implement, so there's really no reason not to do it.

Choosing Immediate or Eventual

At this stage it should be clear that there's a trade-off between the two. Immediate is easier to reason about, as it's an all or nothing operation, but it opens the door to lots of potential problems, especially once you move to a distributed architecture. Eventual on the other hand gives you more freedom and scalability, but it makes debugging harder. When deciding which to use, be sure to ask yourself how you'll handle failures. If you're using multiple storage technologies or APIs, then you should seriously consider moving to eventual consistency.

Protip: When running acceptance tests, run all your projectors as immediately consistent, this makes it easier to spot errors during tests and makes things a lot less complicated to debug.

Personally, I opt for immediate consistency when dealing with domain projections (i.e. projections required to validate domain wide business constraints) and eventual consistency for everything else.

What about you, do you opt for immediate or eventual consistency? What kind of issues have you had and how have you solved them? Let me know in the comments!

Writing a DSL parser using PegJS

Barry O Sullivan — Mon, 12 Feb 2018 15:19:38 +0000

In the previous article I wrote about Domain Specific Languages (DSLs) and how useful they are, but I didn't get into the details of parsing them, that's where this article comes in.

Previously we made this DSL:

User.ScheduleAppointment has { 
  a UserId userId 
  an AppointmentDatetime appointmentDatetime
  a Location location from {
    a LocationName locationName from location
    a Latitude latitude
    a Longitude longitude
  }
}

We want to take the above, and parse it. That means turning the text into an Abstract Syntax Tree (AST). An AST is a tree structure that's easy to navigate and interpret. Once we have an AST, we can interpret it.

To do this, we're going to use PegJS, a Parsing Expression Grammar (PEG) parser written in Javascript. PegJS (like most parsers) is based on regular expressions, they allow you to build named regexes (rules) that you combine together to form a tree. The results of your rules can be turned into data structures, letting you build up your AST.

The AST

The first thing we need to do is design our AST, as we need to know what the end result looks like. Once we know this, we can reverse engineer the rules.

So in our perfect world, we want to take the DSL above and turn it into the following.

{
   "entity": "User",
   "command": "ScheduleAppointment",
   "values": [
      {
         "class": "UserId",
         "param": "userId",
         "alias": null,
         "type": "value"
      },
      {
         "class": "AppointmentDatetime",
         "param": "appointmentDatetime",
         "alias": null,
         "type": "value"
      },
      {
         "class": "Location",
         "param": "location",
         "type": "composite",
         "values": [
            {
               "class": "LocationName",
               "param": "locationName",
               "alias": "location",
               "type": "value"
            },
            {
               "class": "Latitude",
               "param": "latitude",
               "alias": null,
               "type": "value"
            },
            {
               "class": "Longitude",
               "param": "longitude",
               "alias": null,
               "type": "value"
            }
         ]
      }
   ]
}

This structure is easy to navigate and interpret, so it's a great end goal for our parser. Now we know what we want, let's figure out how to get there.

Writing a PEG

PEGs work via regular expressions (regexes). If you're like me, then your regex-fu is probably a bit weak, so writing a parser can seem like a daunting task. Thankfully, there are easy ways to learn regexes. I'd recommend playing this regex crossword game. Once you've completed the "experienced" level crossword, you'll understand regexes well enough that you'll be able to write a parser without looking up regex documentation. I'd highly recommend this game to anyone that wants to learn regexes.

Assuming we understand Regular Expressions, here's an example of single simple rule.

Sample PegJS rule

Var = name:[A-Za-z0-9_]*
  {
    return name.join("");
  }

The above is a PegJS rule that matches variable names like the following "positionId", "canidateId", "variable_name", etc... .
It then returns the result as a string. Here this is defined as a "rule" called Var that can be reused throughout the parser, that way we don't have to repeat code, making the parser easier to read and use.

The rules

A PegJS parser is made up of rules. Our goal is to take the above DSL and figure out the rules for each type of AST structure. Rules are composable, so once we have a few basic rules, we can start building more complex ones.

Whitespace (_): match all the spaces, newlines and tabs, usually ignored
Var: match valid variable names
Alias: An alias (Var) for a request parameter, optional
Value: A composite of a class (Var), a param name (Var) and an optional alias (Var)
CompositeValue: A composite of a class (Var), a param name (Var) and a collection of values
Values: a collection of Values and CompositeValues
Command: a composite of an entity (Var), a command (Var) and a collection of values (Values)

Now we know what the object types are, we can write the PEGJs rules to parse the DSL and create the AST.

The full PEG

/**********
 Starting rule, our entry point to the parser.
 The first part of the PEG extracts the entity name as a string, 
 and makes the "entity" accessible in the JS below, allowing us to the AST and return it. 
 It matches certain expected keywords, eg. "has", but does nothing with them. 
 Finally it extracts the values, which have already been turned into a Values AST. 
 You'll see the "_" rule used here, this means "accept/skip whitespace", it's defined below.
**********/
Command = entity:Var "." command:Var _ "has" _ "nothing"* _ values:Values* _
  {
    // Handle case that there are no values
    values = values.length === 0 ? [] : values[0];

    // Return the matched results with this object structure
    return {
      entity: entity,
      command: command,
      values: values
    }
  }

/**********
 Matches a collection of inputs, 0 to many, that are wrapped in parentheses
**********/
Values = "{" _ values:(CompositeValue/Value)* _ "}"
  {
    return values;
  }

/**********
 Value and CompositeValues always have the same initial shape, so I extracted 
 this into a partial result that is extended by both Value and Composite
**********/
ValuePartial = _ "a" [n]? _ cls:Var _ name:Var _
  {
    return {
      class: cls,
      param: name
    }
  }

Value = value:ValuePartial alias:(Alias)? _
  {
    value.requestParam = (alias) ? alias: value.param;
    value.type = 'value';
    return value;
  }

/**********
 Extract the alias value, ignore the "from" string
**********/   
Alias = _ "from" _ alias:Var 
  {
    return alias;
  }

CompositeValue = value:ValuePartial "from" _ values:Values
  {
    value.type = 'composite';
    value.values = values;
    return value;
  }

Var = name:[A-Za-z0-9_]*
  {
    return name.join("");
  }

/**********
 Match any sequence of "whitespace" characters
**********/   
_ = [ \t\n\r]*

That's the full parser. The above will turn our DSL into the AST above.
You can check this out yourself. Simply go to the PegJS, open their online editor and paste the above DSL and parser in. You'll see the results straight away.

As a side note, we're using a PegJS extension that outputs a PHP version of the parser, so we can use the same parser on the server as well as the client.

Conclusion

As you can see writing a parser isn't that hard. Using that simple parser grammar, I'm able to automate part of my teams workload, ie, writing boilerplate (and error prone) adapters that turns HTTP requests into commands. These simple DSLs make that trivial.

After seeing the above in action, I hope you're thinking of all the things you could define and automate with a DSL. So why not write a simple DSL and parser, and try it out?

Projection Building Blocks: What you'll need to build projections

Barry O Sullivan — Tue, 30 Jan 2018 13:40:39 +0000

(Originally posted on my blog)

Let’s talk about projections. This topic is quite large, so this is the first part in a four part series on projections.

Projection Building Blocks: What you'll need to build projections
Broadcasting events in PHP: Techniques and technologies
Designing Projections: How to design and implement real world projections
Projection DevOps: Continuously deploying new/updated projections with zero downtime

If you've read my previous articles, you should have the basics of event sourced/event driven Command Query Responsibility Segregation (CQRS) systems. At its core there are two concepts, a command side that outputs events and a query side that reads them.

Up until now I've focussed primarily on the command side, i.e. how we model state changes in our apps. I mention projections, but always with a hand-wavy statement and without any real detail. Let's do something about that.

What are projections?

Projections are a necessary part of any event sourced or CQRS system. These systems don't rely on a single generic data source such as a normalised MySQL database. Instead you build up your data sets by playing through the events, i.e the “film”, "projecting" them into the shape you want. This allows lot of flexibility as you're no longer bound by a single data model on which you have to run increasingly monstrous SQL queries (12+ joins anyone?). With projections you can build a data model specifically for the problem/question at hand.

For instance, say your app has the following requirements:

Webapp needs to fetch a user, their active cart, and the cart's items, all as a single document.
Marketing needs a list of how much each user spends over a 6 month period, broken down month by month.

Building a generic data-model that can produce both answers is possible, but it's difficult, and leads to complex SQL statements and brittle data structures. Instead, it's much easier to build up a custom dataset for each use-case, keeping them independent and minimal.

In the above, the webapp would listen for the appropriate events and build a tree of the user, their cart and its items. This structure would get stored in a document DB, such as MongoDB, and fetched later when it’s needed. Nice and easy.

For the Marketing report, it’s a little different. We listen for the same events, but we don’t build the same structure. Instead we keep track of three pieces of data: the users_id, the month, and how much they spent that month. We store that data in a RDBMS, such as MySQL, so that it’s easy to query.

These two datasets are simple yet completely different. They are each designed to answer their specific question and are thus simpler to understand and build. That's the power of projections.

The Building Blocks

Building a robust* projection system is not a trivial task, as there are many concepts and moving parts. Before you can build one, you need to understand each piece in isolation, then see how they all work together.

*An apt word that has been ruined for me due to overuse in college

Event

An event is a named object that represents some discreet change that occurred in your system. It's usually modelled as a class with a collection of properties, giving just enough formation to be useful.

Eg.

<?php namespace Domain\Selling\Events;

use ...;

class CartCreated extends Event
{
    /** @var Uuid */
    public $cart_id;

    /** @var Uuid */
    public $customer_id;     
}

Events also contain some generic meta information, info that each event should contain to make easier to work with. I'd recommend the following.

the ID of the event (unique)
when the event happened
the actor that triggered the event (could be a person or a system process)
the version of the event (events can change shape over time, we'll get into this later)

<?php namespace Domain\Events;

use ...;

abstract class Event 
{
    /** @var Uuid */
    public $id;

    /** @var Carbon */
    public $occurred_at;

    /** @var Actor */
    public $actor;

    /** @var integer */
    public $version = 1;
}

EventStore and EventStream

The EventStore is your access point to all the events that have ever occurred in your system. You give it a position/point in time and it gives you an EventStream that you can iterate through. From there it’s a simple as iterating through the stream until there are no events left.

<?php
$last_position = "b70931a6-b330-4866-97b4-0710c8ad5d3e";
$event_store = new EventStore();
$event_stream = $event_store->getStream($last_position);
while ($event = $event_stream->next()) {
    // Do things
}
?>

Under the hood Event stores and streams can be implemented in any number of technologies, SQL, NoSQL, Kafka, even the FileSystem. As long as they meet the following constraints you're good.

You can start reading from any point in the stream
The events are read in the order that they occurred
The event stream is append only (history doesn't change)

Now that we have a way to read events from the log, let's look at what we do with them.

Projector

In order to build up a data set, you need to listen for a set of events. That's where projectors come in. Their job is to listen for events then pass them through to the projection that's building up the dataset.

There are many ways to do this, but this is my preferred one.


<?php namespace App\Projections\Carts;

use Domain\Shopping\Events;

class Projector
{
    private $projection;

    public function __construct(Projection $projection)
    {
        $this->projection = $projection;
    }

    public function whenCartCreated(Events\CardCreated $event)
    {
        $this->projection->createCart($event->cart_id, $event->customer_id);
    }

    public function whenItemAddedToCart(Events\ItemAddedToCart $event)
    {
        $this->projection->addItemToCart($event->cart_id, $event->item);
    }

    public function whenItemRemovedFromCart(Events\ItemRemovedFromCart $event)
    {
        $this->projection->removeItemFromCart($event->cart_id, $event->item_id;
    }

    public function whenCartCheckedOut(Events\CartCheckedOut $event)
    {
        $this->projection->checkoutCart($event->cart_id);
    }
}

As you can see above, the method signature defines the event we're listening for, and the method itself extracts the data and feeds it through to the projections.

Projection

A projection is the result of "projecting" a sequence of events. It has two categories of functions: commands and queries (standard CQS pattern). Commands change the shape of the underlying dataset. Queries fetch results from the dataset, usually to answer business questions or to present data.

Here's a simple example that looks after a customer's cart and it's items.

<?php namespace DoctrinePlayground\App\Projections\Carts;

use Doctrine\DBAL\Connection;
use Ramsey\Uuid\UuidInterface;
use Carbon\Carbon;
use DoctrinePlayground\Domain\Selling\Entities\Item;

class Projection
{
    private $db;

    const CARTS_TABLE = 'carts';
    const CART_ITEMS_TABLE = 'cart_items';

    public function __construct(Connection $db)
    {
        $this->db = $db;
    }

    /** Example Commands **/

    public function createCart(UuidInterface $cart_id, UuidInterface $customer_id, Carbon $created_at)
    {
        $this->db->insert(self::CARTS_TABLE, [
            'cart_id'     => $cart_id,
            'customer_id' => $customer_id,
            'active'      => true,
            'created_at'  => $created_at->format("Y-m-d H:i:s"),
        ]);
    }

    public function addItemToCart(UuidInterface $cart_id, Item $item)
    {
        $this->db->insert(self::CART_ITEMS_TABLE, [
            'cart_id'     => $cart_id,
            'item_id'     => $item->item_id,
            'item_ref'    => $item->reference
        ]);
    }


    /** Example Queries **/

    public function getActiveCart(UuidInterface $customer_id)
    {
        $cart_arr = $this->db->fetchAssoc('SELECT * FROM '.self::CARTS_TABLE.' WHERE customer_id = ? AND active = 1', [$customer_id]);

        if (!$cart_arr) {
            return null;
        }

        $cart = (object)$cart_arr;

        $items = $this->db->fetchAll('SELECT * FROM '.self::CART_ITEMS_TABLE.' WHERE cart_id = ?', [$cart->cart_id]);

        $cart->items = array_map(function($item) {
            return (object)$item;
        }, $items);

        return $cart;
    }
}

An aside on implementation

If you plan to have many different implementations of the same projection, I'd recommend extracting the methods into an interface, otherwise don't bother. If you do write an interface, some devs advocate separating the Command and Query sides into their own interfaces, but I think this is overkill; We did this for each of our projections and it just made the code harder to navigate, understand and change, ie. it didn't bring any real benefit.

Projectionist

In keeping with the projection metaphor we also have a projectionist. The projectionist is responsible for playing a collection of projectors. Internally, the projectionist does this by keeping track of where each projector is in the stream (by event position or event id), then plays each projector forward from that point, recording the last event that each projector has seen.

Projector Position Ledger

As mentioned above, the projectionist needs some way to keep track of where each projector is in the event log. That's where the Projector Position Ledger comes in. This is a simple data store that keeps track of each projector and it's position. Its fairly simple and can be implemented in any storage technology.

A handy idea, it should also keep track of whether a projector is broken or not. If a projector tries to run and an unexpected exception is thrown, the projector should be marked as "broken" and the projectionist should stop attempting to play it. This way you won't try to keep playing a broken projector, filling up your bug tracker with duplicate exceptions.

Event Upgrader

This component is a little more advanced, but it's still worth mentioning. Sometimes you'll need to change the shape of events, usually by adding or changing properties. When this happens, you'll need to "upgrade" the event shape. This is why each event has a "version" attribute, so we can check the version of the event and apply the appropriate upgrader if it's required.

This component lives in the event stream and manipulates the event data before it is deserialized into the actual event classes. It is used by both the command and query side.

I won't get into too much detail here, as this is quite complex, just be aware that it exists. Think of event upgraders as migrations for events that are run on the fly and you're most of the way there.

Putting it all together

Those are all the components, so let's look at how they all work together.

The projectionist is given a collection of projectors. They go through each projector, and fetch an event stream that starts after the last event each has seen. The new events are played through the projector, and on completion the projectionists records the last even seen by the projector.

And that's that, those are the pieces you need to build an effective projection system, at least at the start.

Next Steps

There's one thing missing from above, how do you trigger your projectionist to play a projector? This is a complex question, especially if you're using PHP. So that's why I'm dedicating the next article to just that. We'll go through some implementation details, exploring the pros and cons of each, then settle on what I think is the best solution.

The third article will dive into building complex projections using various technologies designed to solve specific problems. In it we’ll highlight some of the techniques our team has found useful when building, designing and implementing projections.

The fourth article will dig into projection versioning, seamless releases and some more advanced concepts related to the projector/projectionist side of things.

Until then, best of luck!

Write DSLs and Code Faster

Barry O Sullivan — Tue, 23 Jan 2018 15:09:08 +0000

(Originally published on my blog.)

What is a DSL

DSL stands for Domain Specific language, this means that it is a language that is designed to solve a specific problem in a domain. DSLs are great if you want to write a generic solution to a specific problem without all the boilerplate code.

A good DSL is easy to write and understand. Once someone understands the domain language, they can read the DSL, and understand the problem, even if they're not a coder. DSLs allow us to automate many things, including aspects of development, greatly increasing our speed and lowering codes and error rates.

Here's one I made earlier

At work, I was getting annoyed with how it always took us so long to write request handlers that decode HTTP requests into domain objects. So in my spare time I designed a DSL for automatting this process, and then added it to our codebase. Here is a full example of one of these DSL files.

Exampe Request DSL

User.ScheduleAppointment has { 
  a UserId userId 
  an Appointmentatetime appointmentDatetime
  a Location location from {
    a LocationName locationName from location
    a Latitude latitude
    a Longitude longitude
  }
}

This simple DSL defines a request coming into a server, that is then turned into a command, which is then handled by the system.
The DSL defines the command, the command's inputs, and request parameters used to create those inputs. We're using a CQRS based system with Valueobjects and Commands, so our language is very simple, strongly defined and well structured. (If you don't know that a valueoject is, you can find out here)

So, in the above DSL, we have a Command "User.ScheduleAppointment" (guess what this does), which has many inputs. The first input is called "userId", it is a valueobject of type "UserId". The request parameter that's passed into the ValueObject is implied from from the input's name, which becomes "userId". You can use a different request parameter by using the "from" keyword, which you can see in line 6. This allows us to handle the case that a client is sending a request that has mispelled or misnamed fields (ie, legacy clients).

In practice, sometimes a ValueObject's input is made from other ValueObjects, rather than just a request parameter (string). That's why we have the tabbed tiers of inputs surrounded by parentheses. This allows us to build very complex, tiered objects in a simple, automated manner. From this, we can easily build the tree of inputs required to create a command.

And that's it, we've defined our language.

What are the advantages of this DSL?

You can already see the main advantage of this DSL, it allow us to autoamte a complex, error prone process. It turns out though, that we can use this DSL to solve various other issues, further highlighting how useful they are. Here are a few examples.

1. Generating requests

Making a request object client side is no longer a pain. We can use the same DSL to define the shape of the request that should be sent, and you can validate the request before you send it. Not only that, but you can automate the generation of the request itself. We created a request factory that takes in a form and tries to extract the request parameters out by key. Then we can set any remaining parameters manually. Very handy.

2. Sending and receiving requests

Sending requests to a server is easy, but tiring to implement, especially if the URLs are not uniformly named and structed. You end up writing a lot of boilerplate code that is painful to change. Well, you can automate this. We created a single Command endpoint that takes in the above request, and then decodes the command and runs it. Now we have only one endpoint to hit, so our code is drastially reduced, both client and server side.

3. Documenting requests

We can use the above DSL to create a request schema. That schema can be used in documentation, or you can make the endpoint self documenting. If someone sends a request with the method "OPTIONS", you return the Schema as JSON, making it easier for people to use and understand your API.

4. Speed of development

Once you understand the language, changing or writing a new request is quick and painless. With this DSL, I was able to write and validate an entire collection of requests (20 in total) in under 30 minutes. This would have taken me at least a day if I didn't have the DSL handy.

5. Modular

This DSL is not just specific to our current app, it can used in any app, as long as it uses commands and valueobjects (which all our apps do/will, we're DDD/CQRS junkies). So now have another tool that can rapidly speed up development across all our applications.

DSLs can be used anywhere

What you saw above is one simple DSL that has drastically improved our development speeds. It's automated a painful, error-prone process and it's also offered a host of other benefits. Think about that, that's just one DSL. Imagine what you could do with multiples ones, all working together! You can write DSLs to automated practically any process that can be defined in language. Which, if you think about it, is every process, as we define and comminicate processes to others (and ourselves) via language. The sky's the limit.

So to close, I say this. Start writing DSLs. You'll do what developers do best, automate processes. Learn how to do it, and you won't look back, infact, you'll wonder how you worked without it.

If you want to look at parsing the above DSL in Javascript, stay tuned for my next article.

DEV Community: Barry O Sullivan

Estimating software: How to deal with requests for estimates

Why we're asked for estimates

Can we give accurate estimates

Dealing with requests for estimates

Confidence level 1: Don't have a clue:

Confidence level 2. I have an idea:

Confidence level 3: I know what needs to be done and where:

What if they want estimates at the start?

It's all about communication

Story points

Closing thoughts

Template Driven Development: Why it doesn't work

Why it doesn't work

Why we want it to work

The problems it causes

So how do we combat this?

1. Kill your darlings

2. Don't present patterns as how-to

3. Express the evolution of your designs through small PRs

4. Allow time to sharpen the axe

Don't fall victim

Introducing DDD to your Company

Have you Introduced DDD to your company in the past?

Why are you introducing DDD?

The Reasons

1. Allow us to engage domain experts so we can decouple our software

2. Maintainable designs

3. Avoid DRY on superficial similarity, e.g. Eric Evans Toilet/Sink story

4. Create a public API and build a vocabulary for it

5. Figure out if we need to build software at all

How to Introduce DDD

Mapping your Why to a How

1. I started writing ValueObjects and it snowballed from there

2. Micro-services are not an end goal, they're a technique along the way

3. Identify the contexts that will change per feature, allows you to understand/demonstrate cost

4. DDD is slower initially, like any new idea or techniques, make this cost known up front

5. Wait for pain, use DDD to solve it. I.e. demonstrate the value

Ending the meetup

Folder Structure and Frameworks: What is exerting control?

Folders Give Context

The Status Quo

The Problem

The New Structure

Structure matters

Managing projectors is harder than you think

Run modes

1. Run from Beginning

2. Run from Now

3. Run Once

Projector lifecycles

2. Play

1. Boot

3. Retire

When to retire

Projector failure

Boot failures

Play failures

Conclusion

Domain Driven Design for Everyone Else

Solutions Always Overrun

Complex vs Complicated

Focus on the (right) problem

Talk to the experts

Building a shared understanding

Sprechen Sie Talk, huh?

Your solution reflects your understanding of the problem

Faster feedback is better feedback

Self Improvement

Conclusion

Further reading

Communication styles - Working effectively as a team

Styles of communication

Exploring solutions

The feedback loop

It's everywhere

Using the wrong style

Conclusion

Acceptance testing your PHP app with ease

PHPs Built-in WebServer