DEV Community: Committed Software

Storybook Docs First Page

Committed Software — Mon, 14 Oct 2019 00:00:00 +0000

Storybook has been a great tool for helping the development and testing of our components. Now with the addition of the DocsPage it will likely become our default platform for documentation and communication of our visual design libraries.

However, one problem we have found is that there is no way to declare which page the documentation should open on. You can sort the stories and the first will be displayed. You can also add a parameter to the story metadata and use that for sorting.

For .mdx documentation:

<Meta title="Components|Button" component={Button} parameters={{ order: 5 }} />

or for Component Story Format

export default {
  title: "Components|Button",
  component: Button,
  parameters: {
    order: 5
  }
};

Then sort the stories in your config.js:

import { addParameters } from "@storybook/react";

addParameters({
  options: {
    storySort: (a, b) =>
      a[1].kind === b[1].kind
        ? 0
        : a[1].parameters.order - b[1].parameters.order
  }
});

But after sorting your desired first page may not be the first in the ordering. To set a specific first page we create our own simple addon. As it’s small, you can put it straight into the addons.js file along with any other registered addons:

import { STORIES_CONFIGURED } from "@storybook/core-events";
import addonAPI from "@storybook/addons";

addonAPI.register("my-organisation/firstpage", storybookAPI => {
  storybookAPI.on(STORIES_CONFIGURED, (kind, story) => {
    if (storybookAPI.getUrlState().path === "/story/*") {
      storybookAPI.selectStory("Kind", "story");
    }
  });
});

This is only triggered on load when the stories are configured. Also by testing the url for /story/* we do not interrupt requests for a specific story or docs page.

Limitations

This description applies to v5.2 upwards, though I think similar things can be done using sortStoriesByKind for earlier versions. Also, if an incorrect story URL is entered then the first page by the ordering will be used. If this is a concern you can add into the addon:

import { STORIES_CONFIGURED, STORY_MISSING } from "@storybook/core-events";
import addonAPI from "@storybook/addons";

addonAPI.register("my-organisation/firstpage", storybookAPI => {
  storybookAPI.on(STORIES_CONFIGURED, (kind, story) => {
    if (storybookAPI.getUrlState().path === "/story/*") {
      storybookAPI.selectStory("Kind", "story");
    }
  });
  storybookAPI.on(STORY_MISSING, (kind, story) => {
    storybookAPI.selectStory("Kind", "story");
  });
});

Speedy Spotless

Committed Software — Thu, 03 Oct 2019 10:17:27 +0000

Speedy Spotless is a Maven plugin for easy formatting of staged changes. It is inspired by pretty-quick but for Java.

Spotless is a tool to automatically format your code. It can format in various languages in a multitude of different ways. Regularly running an automatic code formatter like Spotless is a great way to maintain a consistent code style and to keep changes readable. It is good practice to use a Git hook to format new code as it is being committed.

Existing approaches for formatting staged changes don't address what to do for the formatting of partially staged files. If the formatter makes changes to the file it is no longer obvious what changes to include in the commit! Speedy Spotless will format such files, but will cancel the commit, allowing the developer to decide what to do instead of doing something that the developer may not have intended.

Speedy Spotless is a drop in replacement for Spotless Maven Plugin and supports the same configuration options, but also includes the new goal speedy-spotless:staged to trigger the formatting staged changes and speedy-spotless:install-hooks that creates & installs the Git hook to format staged changes automatically. It will even preserve any existing hooks!

Speedy Spotless on Github.

Vaadin Flow and Reactive Spring

Committed Software — Sun, 23 Sep 2018 00:00:00 +0000

Vaadin 11 and Spring Boot 2

A couple of week back saw the release of Vaadin 11 which builds on the recent major changes to Vaadin 10. Vaadin allows Java developers to build web user interfaces in Java with Java components.

As we are heavy users of Java, and specifically Spring, Vaadin is an interesting technology for us. In this blog post, we will look at the use of some of the newer functionality of Spring Boot 2 with Vaadin.

This isn’t an introduction or tutorial for Vaadin, they already have some great resources and documentation for that. We’ll cover some tips (and gotchas) when using Vaadin and Spring.

Everything here works with Vaadin 10, which is the LTS version, but with Vaadin now following a 3-month release cycle the majority of developers will want to keep their apps up to date.

Spring and Vaadin

Vaadin 10+ plays well with Spring Boot 2 and has some specific annotations which can be used within components. This is covered in a whole chapter in the use Flow with Spring documentation.

We would highly recommend using Vaadin alongside Spring. If you are used to Vaadin, the Spring integration is very light but Spring will provide you with a considerable amount of additional engineering support:

Database access through with support for
Access to caching locally and externally
Authentication, including social sign on
Messaging support
External configuration mechanism
Executable JAR and service wrappers

In short, combining Spring and Vaadin you have a complete stack before you’ve written any code yourself.

If you are starting out with Vaadin they have a project starter for Spring Project Base with Spring.

If you have a paid Vaadin subscription, they also have a full stack Vaadin Spring starter too.

Using Reactive Spring

Spring Framework 5 and Spring Boot 2 offer the option to use the reactive types Flux and Mono.

Reactive types are a complex topic but consider them an improved, standardised and more flexible version of Java’s Stream (now reactive Flux) and Optional (now reactive Mono). Behind the scenes, they are designed for high throughput and non-blocking operations. Reactive types are an efficient and composable way to process data in applications.

Reactive types provide flexibility on which thread events are published, processed and consumed. Like many UI frameworks, Vaadin enforces strict control over which thread can update a UI component.

Here we have a counter of items, we can update by clicking the refresh button:

Text text = new Text("...");
Button button = new Button("Refresh", event -> {
  int count = 20;
  text.setValue(count + " items")
})

When the button is clicked, the event callback is run in a manner that allows UI updates and the text value updated.

So far, so un-reactive.

Suppose we have a Reactive Spring Data Repository:

// The repository
public interface PersonRepository extends ReactiveCrudRepository<Person, String> {

}

and use it in the following way

PersonRepository repo = ... // Typically injected by Spring

Text text = new Text("...");
Button button = new Button("Refresh", event -> {

  // Ask for the count
  Mono<Long> mono = repo.count();

  // WARNING: DOESN'T WORK!
  mono.subscribe(count -> text.setValue(count + " items"));
})

Here subscribe will call the lambda (count -> text.setValue) is called when then count operation is finished. This will happen asynchronously - the code above is basically just wiring an ‘on complete count’ callback. It will likely happen on the non-UI thread and hence without the proper UI locking that Vaadin requires. Clicking the button will cause an exception from Vaadin.

You could mono.block() in order to force waiting for the count, but that loses the benefits for reactive types. You don’t really want to force your UI to wait for long-running queries.

To run code on a UI thread you can use the UI.access() method. You can access the UI using the getUI() method on a Vaadin Component which returns an Optional. Thus an implementation of the above would be:

PersonRepository repo = ... // Auto wired / injected by Spring

Text text = new Text("...");
Button button = new Button("Refresh", event -> {
  Mono<Long> mono = repo.count();

  // In effect this says:
  // - when you get the count
  // - if you can get the UI
  // - schedule a ui update
  // - in that ui update, set the text value.
  mono.subscribe(count -> {
    getUI().ifPresent(ui -> {
      ui.access(() -> text.setValue(count + " items"));
    });
  }
})

That is quite a lot of code, and quite complex to read.

Taking a step back, really what we want is:

When the mono returns
Convert to it a set of commands (text.setValue)
Pass the commands to UI to execute

This can be done more neatly as:

repository.count()
    .map(count -> () -> text.setValue(count + "items"))
    .subscribe(doInUi(getUi()));

// where
public static Consumer<Command> doInUi(Optional<Ui> ui) {
  return command -> ui.ifPresent(u -> u.access(command));
}

You can make this nicer with Spring, by creating a UiSupport component which you can inject into your components:

@Component
@UIScope
public class UiSupport {

  private final UI ui;

  public UiSupport(@Inject UI ui) {
    this.ui = ui;
  }

  public Consumer<Command> doOnUi() {
    return command -> updateUi(command);
  }

  public void updateUi(Command command) {
    ui.access(command);
  }
}

Notice the component is scoped to the each individual Vaadin UI (@UiScope) because we will need one UiSupport per Vaadin UI (ie one for each user in each tab).

So our callback becomes:

@Autowired
    public MyComponent(UiSupport uiSupport) {

      Button button = new Button("Refresh", event -> {
          repository.count()
            .map(count -> () -> text.setValue(count + "items"))
            .subscribe(uiSupport.doOnUi());
      });
      //...
    }

Or perhaps more cleanly out of the lambdas:

@Autowired
    public MyComponent(UiSupport uiSupport) {
      this.uiSupport = uiSupport;

      Button button = new Button("Refresh", event -> refreshData());
      //...
    }

    private void refreshData() {
        repository.count()
          .map(this::setTextValue)
          .subscribe(uiSupport.doOnUi());
    }

    private Command setTextValue(long count) {
      return () -> text.setValue(count + "items")
    }

Reactive repositories as a Data Provider

Vaadin use DataProviders to provide the data items for tables, etc. If you are using Spring, you’ll often want this to be generated from a Spring Data Repository.

You can neatly wrap a reactive repository to a DataProvider using the fromCallbacks method:

public class ReactiveDataProvider {

  public static <T> DataProvider<T,?> fromReactiveRepository(ReactiveCrudRepository<T, ?> repository) {
    return DataProvider.fromCallbacks(
        query -> repository.findAll()
            // Basic sorting will work if you implement Comparable<T> on your data item (T)
            // but it's better to so this in the database using Sort
            // which is more performant and flexible
            // .sort(query.getInMemorySorting())
            .skip(query.getOffset())
            .take(query.getLimit())
            .toStream(),
        query -> repository.count().map(Math::toIntExact).block()
    );
  }
}

The above doesn’t use the DataProvider query, so it’s a bit more effort if you want to access a filtered result set. For a simple, but flexible, approach to filtering look at the QueryByExampleExecutor<T> which provides a neat way of passing a filter based on the data item class.

The ReactiveMongoRespository has query by example, so we can use:

public class ReactiveDataProvider {

  /// ...

  public static <T> DataProvider<T,Example<T>> fromReactiveMongoRepository(ReactiveMongoRepository<T, ?> repository) {
    return DataProvider.fromFilteringCallbacks(
        query -> 
            findByExample(repository, query.getFilter())
            .skip(query.getOffset())
            .take(query.getLimit())
            .toStream(),
        query -> countByExample(repository, query.getFilter())
            .map(Math::toIntExact).block()
    );
  }

  private static <T> Flux<T> findByExample(ReactiveMongoRepository<T,?> repository, Optional<Example<T>> filter) {
    if(filter.isPresent()) {
      return repository.findAll(filter.get());
    } else {
      return repository.findAll();
    }
  }

  private static <T> Mono<Long> countByExample(ReactiveMongoRepository<T,?> repository, Optional<Example<T>> filter) {
    if(filter.isPresent()) {
      return repository.count(filter.get());
    } else {
      return repository.count();
    }
  }
}

You can use this in your components with:

dataProvider = ReactiveDataProvider.fromReactiveMongoRepository(repository);
filteredDataProvider = dataProvider.withConfigurableFilter();

// You'd likely set this in response to user input but...
filteredDataProvider.setFilter(Example.of(new Person("hello")));

There’s a lot more you can do with Example using ExampleMatchers.

Broadcaster and Spring application

As Vaadin code exists on the server it has access to the full state of the application. For example, all the logged in users.

So functionality such as broadcast messaging or shared state is easily implemented in Vaadin. The Vaadin documentation uses a broadcast pattern to achieve this.

In Spring, we already have nice functionality for passing events around the application using @EventListener. However, it seems that you can not use this within Vaadin components, see Github issue 272.

In order to bridge Spring events to Vaadin we use the same broadcaster pattern - combining broadcast via the registered listeners and Spring components:

// This is a unified broadcaster outputting to Spring and (registered) Vaadin components
public abstract class AbstractBroadcaster<T> {
  private Executor executor = Executors.newSingleThreadExecutor();

  private LinkedList<Consumer<T>> listeners = new LinkedList<>();

  @Autowired
  private ApplicationEventPublisher publisher;

  public synchronized BroadcastRegistration register(Consumer<T> listener) {
    listeners.add(listener);

    return () -> {
      synchronized (Broadcaster.class) {
        listeners.remove(listener);
      }
    };
  }

  public synchronized void broadcast(T message) {
    // Send message to Spring components
    publisher.publishEvent(message);

    // Send message to our listeners in Vaadin
    for (Consumer<T> listener : listeners) {
      executor.execute(() -> listener.accept(message));
    }
  }
}

You can then create your own broadcaster for your specific event with:

@Service
public class PersonChangedBroadcaster extends AbstractBroadcaster<PersonsChangedEvent> {

}

You can now register a Vaadin route to respond to the event. Notice we register with the broadcaster onAttach and deregister onDetach.

@Route
@Push
public class MainView extends VerticalLayout {

  private final VaadinFlux vf;
  private final PersonRepository repository;
  private final PersonChangedBroadcaster notifier;
  private final DataProvider<Person,Example<Person>> dataProvider;
  private BroadcastRegistration notifierBroadcastRegistration;

  public MainView(VaadinFlux vf,
      @Autowired PersonRepository repository, PersonChangedBroadcaster notifier) {
    this.template = template;
    this.vf = vf;
    this.repository = repository;
    this.notifier = notifier;

    dataProvider = ReactiveDataProvider.fromReactiveMongoRepository(repository);

    // Create components here...
  }

  private void refreshData() {
    // When we are prompted refresh the data
    vf.updateUi(() -> dataProvider.refreshAll());
  }

  @Override
  protected void onAttach(AttachEvent attachEvent) {
    super.onAttach(attachEvent);

    // Register for events as soon as we attach
    this.notifierBroadcastRegistration = notifier.register(p -> this.refreshData());
  }

  @Override
  protected void onDetach(DetachEvent detachEvent) {
    super.onDetach(detachEvent);

    // When we 'detach' we no longer want events any more
    if (notifierBroadcastRegistration != null) {
      notifierBroadcastRegistration.deregister();
      notifierBroadcastRegistration = null;
    }
  }
}

Our MainView view will automatically update (thanks to @Push) whenever a PersonChangedEvent is sent.

A typical use case, inside Spring would be to automatically refresh all users’ UI when a data item is added or deleted (by any user). In Spring Mongo we can take advantage of the AbstractMongoEventListener to listen for save and delete functions. We can then push new events into our broadcaster:

@Service
public class PersonChangedListener extends AbstractMongoEventListener<Person> {

  private final PersonChangedBroadcaster notifier;

  public PersonChangedListener(ApplicationEventPublisher publisher,
      PersonChangedBroadcaster notifier) {
    this.notifier = notifier;
  }

  @Override
  public void onAfterSave(AfterSaveEvent<Person> event) {
    super.onAfterSave(event);
    notifyListeners(new PersonsChangedEvent(event.getSource()));
  }

  @Override
  public void onAfterDelete(AfterDeleteEvent<Person> event) {
    super.onAfterDelete(event);
    notifyListeners(new PersonsChangedEvent(event.getSource()));
  }

  private void notifyListeners(PersonsChangedEvent event) {
    notifier.broadcast(event);
  }
}

Using Vaadin

If, like us, you develop your Web UI with a framework such as React, the idea of moving back to Java might seem an odd step.

Where we feel Vaadin really shines is to support our backend services. These don’t have consumer user interfaces, but they do benefit from management UIs. An example might be data microservice, where we offer an API to programmatically create, list or delete data, but we’d also like to provide a management UI to allow that to be easily entered by a system manager. We might also expose other functionality via the Vaadin UI such a data export and batch import, which aren’t part of the API.

For these types of application, Vaadin allows us to quickly develop a UI which:

Has a simple unified build process for a service, since all code is compiled through Maven
Is deployed as part of the service itself, integrated into a single JAR
Takes advantage of the common Java code base (no duplication of code or opportunity for data transfer objects to diverge)

Outside that use case, Vaadin really shines where the server state is complex and needs to be shared. The above shows how in a few lines we can update all the users on important changes in state, triggered by the database. Thanks to Spring and Reactor we have non-blocking, performant code.

Photo by Philip Swinburn on Unsplash.

Hack: Accessing Time Machine backups

Committed Software — Thu, 20 Sep 2018 00:00:00 +0000

A natural choice for macOS backup is to use Time Machine to output to an external device. To protect the data, the backup disk is encrypted and each backup is encrypted independently. That’s all standard functionality for Time Machine.

Time Machine works great when you wish to recover a deleted file or overwritten file. We don’t often see the need as we working in the cloud (versioning automatically on GDrive) or using Git (which is a time machine in its own right) but the safety net is reassuring.

In the last week, we had reason to access a backup but the computer in question had been reformatted for a different purpose, we couldn’t use it to access its own backup.

What doesn’t work

The theory is you can access another computer’s Time Machine by clicking on the Time Machine icon in the menu bar and then holding down holding Option (alt). (You can add the Time Machine icon to the menu bar using System Preferences). When you press Option ‘Enter Time Machine’ changes to ‘Browse other backup disks…’ which prompts for a backup disk.

Sadly, selecting the other machine’s Time machine backup doesn’t seem to work for us. It always displayed our current machine’s backup. A quick search shows others have seen the same problem.

Disk Utility to the rescue

Access your backup server in Finder - it’ll be listed in the Shared section in the Sidebar. You can log in (with your disk encryption password) and navigate to the folder where your backups are stored.

They are .sparsebundle files, in recent versions of Time Machine. If you haven’t been using encryption, you can probably review them just by right-clicking and selecting ‘Show Package Contents’.

If you are using encryption, then ‘Show Package Content’ issues an error box, stating ‘Device not found’.

Load up Disk Utility. Then use the menu to File > Open Disk Image and navigate to the backup you want. You should enter your backup encryption password when prompted. It’ll take a moment to think, but then your backup will be mounted and accessible in Finder.

The backups are organised into folders which are timestamped ‘date and time of backup’ and their content is ‘what has changed since the last backup’. It is not that easy to find the latest content of a folder, but that’s the way it is. Copy the files you want out of the Backup.

Warnings

Don’t change or add files in the backup. It’s probably fine but you want your backup to work. This is an emergency!
Always unmount your backup volume when you are done. Otherwise, next time to try to access it you may get an error saying it’s locked.

Use at your own risk!

Credit: Photo by James Newcombe on Unsplash

Using SonarQube and GitHub without losing your comments

Committed Software — Fri, 03 Aug 2018 17:00:00 +0000

If you happen to use the official sonar-github plugin to integrate Sonar into your GitHub pull requests you may have found that the plugin has a tendency to delete any comments it considers stale. This is especially annoying when the Github account used for the integration is also a member of your team, causing many valuable review comments to disappear!

While there exists an existing PR to fix this precise issue, the support for this Github Sonar plugin has dried up (with Sonar moving towards a new solution for its paid-for offerings). For this reason, we decided to fork the sonar-github project, applying the fix for this issue. Releases of our fork are available here.

Cypress.io - A JavaScript E2E Testing Framework

Committed Software — Wed, 09 May 2018 23:00:00 +0000

End to end (E2E) testing is a crucial part of any development cycle and until now, E2E UI testing has been primarily done using Selenium WebDriver. Now though, a new testing framework, Cypress, is available. It differs from Selenium and offers several extremely useful game-changing features, such as:

Asynchronous DOM element finding
Server stubbing
‘Time Travel’ debugging

Setup

Setup could not be simpler - Cypress is just an npm package, and so can be installed as a package and run interactively with cypress open, or run in headless mode with cypress run

Asynchronous by design

Anyone who has any experience writing Selenium tests will likely know the pain of synchronisation: dealing with the problem of tests running before certain elements have loaded into the DOM. Cypress solves this by using JavaScript promises. Manually writing an implicit or explicit wait, as was the case in Selenium, is no longer needed as implicit waits are effectively built in.

describe('a sample test', () => {
  it('should do stuff', () => {

    cy.visit(appUrl) // Go to appUrl

    cy.get('.some-button').click() // Wait for .some-button to be present, then click on it

    cy.get('.some-other-button', ($btn) => {
      // do stuff with the element $btn
    })

    cy.get('.yet-another-button', ($btn) => {
      // do stuff with the element $btn
    }, {timeout: 60000}) // If Cypress doesn't find the element within 60000 ms, then error
  })
})

Server stubbing

Cypress has built-in functionality to stub/mock any backend servers linked to the UI. This has 2 major advantages:

We can remove any dependencies on a backend by stubbing responses for every request made to the API.
We can mock some responses - for example, login attempts, or 3rd party APIs that we don’t want our tests to depend on.

describe('another sample test', () => {

  beforeEach(() => {
    cy.server() // Enable response stubbing
    cy.route({
      method: 'GET', // when the app sends a GET request
      url: 'http://www.some-api.com/something', // to this URL
      response: {message: 'Some mocked message'} // then return this response
    })

    cy.visit(appUrl)
  })

  // Tests...
})

Time travel debugging

Even with a trivial setup and built-in support to tackle synchronisation issues, Cypress has another gem for developers. Interactive, ‘time travel’ debugging. In a simple UI, the user can see all the steps in the tests being run and see what happened before, during, and after each step.

Continuous Integration

Continuous Integration can often be tricky when it comes to E2E testing. In terms of running Cypress tests, this is simple - it’s just an npm command. The potential difficulty comes in the form of running a UI (and maybe an API) as part of the pipeline. We achieved this by spinning up instances of the UI and backend as Docker containers using Docker Compose, although this is obviously dependant on a project’s infrastructure and deployment methodology.

Conclusion

Cypress has amazing potential as a UI testing solution. Not only is it easy to get up and running, it also solves one of Selenium’s biggest headaches - synchronisation. By doing this using promises, getting used to this should be easy for most JavaScript developers. Time travel debugging allows for easy insight into where and why tests are failing, and the ability to stub part or all of the backend allows for testing just the UI, an entire system, or a mix of both.

Conversational Bots With Dialogflow

Committed Software — Fri, 13 Apr 2018 17:00:00 +0000

Building Conversational Bots With Dialogflow

Conversational bots are becoming increasingly popular, with AI assistants available on most mobile phones, devices like the Amazon Alexa and Google Home and in various chat applications like Slack and Facebook Messenger. However, developing conversational bots can produce a number of problems:

Speech recognition to understand a spoken user request.
Extracting query parameters contained within a query.
Understanding the intent of a query to determine the correct action.
Generating an appropriate response for the query.
Integration with a growing number of assistant devices or chat applications.

Each platform usually has their own suggested approach for developers creating chatbots, for example Slack offers various APIs and suggests the use of JS libraries like “Botkit”, whilst Amazon offer the “Alexa Skills Kit” and “AWS Lambda functions”. Ideally, we don’t want to support multiple solutions for each device, so how can we write a conversational bot once whilst integrating with a variety of platforms?

Dialogflow

Google’s Dialogflow platform provides us with a suitable solution; handling query parsing, parameter extraction, intent detection and offering easy integrations with a variety of common applications.

Intents

Dialogflow allows us to define different Intents which it will match incoming queries to. We define examples of phrases that would trigger our intent and Dialogflow will use these to train a model used in query recognition. The more example queries provided, the more accurate the query recognition will be.

We can also state the parameters that should be extracted from each request. For each parameter, we specify its name, type and default value. Dialogflow offers a number of existing types for common types like numbers, dates or locations whilst also allowing us to define our own entity types.

We state if each parameter is required, and we can also provide prompts to be used when a user does not provide all the required parameters. In the example above, the question regarding lunch requires a FoodType parameter.

User: “Where should I go for lunch?”

Bot: “Any preferences?”

User: “A burger”

Dialogflow automatically handles these parameter filling exchanges, in a conversational manner utilising our provided prompts, before proceeding to build a response.

Response Creation

Once a user’s Intent has been determined we need to provide a response for their request. Dialogflow allows us to provide basic text responses in the browser, with specific responses for different platforms like Google Assistant or Slack.

However, we probably want to provide a customised response, and for this Dialogflow offers integration with Firebase Cloud Functions. Once again it allows us to edit and deploy this function completely within the browser. Dialogflow provides us with a templated function to edit, in this function we write our request fulfilment code and provide Dialogflow with a map stating the functions to be called to handle each Intent.

For each Intent handler, we can use the provided request to extract specific parameters and build a custom response. We can also detect the request source, so that we can build a response specific to that platform, for example adding Cards to a Google Assistant response, or making use of Slack’s styled responses.

exports.dialogflowFirebaseFulfillment = functions.https.onRequest((request, response) => {
  const agent = new WebhookClient({ request, response });

  function lunchHandler(agent) {
    var source = request.body.originalDetectIntentRequest ? request.body.originalDetectIntentRequest.source : undefined

    // Your code here - using request.body parameters to build a response

    switch(source){
        case 'slack': buildSlackResponse(agent); break;
        case 'google': buildGoogleAssistantResponse(agent); break;
        default: agent.add("Default response"); break;
    }

    // Add context for chained intents to use
    agent.setContext({ name: 'preference', lifespan: 2, parameters: { foodType: 'burger' }});
  }

  function buildSlackResponse(agent){
      // Building slack specific response
  }

  function buildGoogleAssistantResponse(agent){
      // Building Google Assistant cards
  }

  // Run the proper function handler based on the matched Dialogflow intent name
  let intentMap = new Map();
  intentMap.set('Lunch Intent', lunchHandler)
  agent.handleRequest(intentMap);
});

We may want to make an asynchronous request from within our cloud function, calling an API or requesting data from a data store to populate a response. In this case, Dialogflow allows us to return a Promise from our intent handler function. Dialogflow will wait for the Promise to resolve before processing the agent response.

function lunchHandler(agent) {
    return Promise.resolve(someAsyncRequest()
      .then(function (res) {
        agent.add("Here is a service specific response")
      })
      .catch(function (err) {
        console.log(err.stack)
        agent.add('Sorry, the service is not available right now')
      })
    );
}

Integrations

One of the main advantages of Dialogflow is its application integrations. Dialogflow offers simple integration with Google Assistant, Facebook Messenger, Slack, Twitter, Alexa, Skype and more. In most cases, these integrations are very simple to set up, with many being a case of activating the integration and providing a set of credentials required for the two services to communicate. In each case, Dialogflow provides a simple set of steps to follow to set up the integration.

Once an integration is setup, any query requests from that platform are forwarded to Dialogflow for processing and results are returned, all requests from each platform are all handled by the same underlying service.

Wrapping Up

Dialogflow is incredibly easy to use, conversational bots providing basic responses can be created entirely within the browser, without having to download or setup any development kits or having to write a single line of code. A user only requires any programming knowledge once they need to produce complex responses.

Its many integrations truly seem to be its main advantage, allowing the developer to have one single service support a wide range of clients provides a maintainable solution. Every request is handled by the same parsing service and parameter extraction, leaving the developer to only be concerned with producing a suitable user response.

As with most fits all solutions, Dialogflow does have its drawbacks. Many applications offer differing features within their chatbots, and supporting a large number of integrations may mean that some of these platform specific features cannot be taken advantage of. For example, at the moment of writing this, there are issues with the Slack integration and responding to direct @bot mentions.

However, Dialogflow is constantly being updated and improved, new features are gradually being implemented and so these issues may be addressed over time.

Credit: Blog photo by “Ben Kolde” on “Unsplash”

Jenkins Slack Notifications

Committed Software — Sun, 14 Jan 2018 00:00:00 +0000

To add a slack notification from our continuous integration builds on Jenkins Blue Ocean we use the Slack Notification plugin. Adding a set of functions to the Jenkinsfile which is then called in the post section of the pipeline. We use the git command to extract the committers name and the hash of the commit so we can link to the commit in bitbucket and add the author’s name if the build has failed or is unstable so it will be brought to their attention.

pipeline {
  agent { 
    ...
  }

  stages {
    stage('env') {
      steps {
        notifyStarted()
      }
    }
        ...
  }
  post {
    success {
      notifySuccess()
    }
    unstable {
      notifyUnstable()
    }
    failure {
      notifyFailed()
    }
  }
}

def notifyBuild(String buildStatus = 'STARTED', String colorCode = '#5492f7', String notify = '') {

  def project = 'projectName'
  def channel = "${project}"
  def base = "https://bitbucket.org/committed/${project}/commits/" 

  def commit = sh(returnStdout: true, script: 'git log -n 1 --format="%H"').trim()
  def link = "${base}${commit}" 
  def shortCommit = commit.take(6)
  def title = sh(returnStdout: true, script: 'git log -n 1 --format="%s"').trim()
  def subject = "<${link}|${shortCommit}> ${title}" 

  def summary = "${buildStatus}: Job <${env.RUN_DISPLAY_URL}|${env.JOB_NAME} [${env.BUILD_NUMBER}]>\n${subject} ${notify}"

  slackSend (channel: "#${channel}", color: colorCode, message: summary)

}

def author() {
  return sh(returnStdout: true, script: 'git log -n 1 --format="%an" | awk \'{print tolower($1);}\'').trim()
}

def notifyStarted() {
  notifyBuild()
}

def notifySuccess() {
  notifyBuild('SUCCESS', 'good')
}

def notifyUnstable() {
  notifyBuild('UNSTABLE', 'warning', "\nAuthor: @${author()} <${RUN_CHANGES_DISPLAY_URL}|Changelog>")
}

def notifyFailed() {
  notifyBuild('FAILED', 'danger', "\nAuthor: @${author()} <${RUN_CHANGES_DISPLAY_URL}|Changelog>")
}

Debugging Elasticsearch Queries in Java: how to convert queries to JSON

Committed Software — Fri, 03 Nov 2017 17:00:00 +0000

Elasticsearch has excellent documentation with detailed descriptions and plenty of examples. The Elasticsearch Java API has handy builder classes accessible from org.elasticsearch.index.query.QueryBuilders to craft queries and aggregations in an intuitive and fluent way that is idiomatic to Java.

Unintuitively, however, QueryBuilder and AggregationBuilder subclasses do not implement toString, making it harder to debug and analyze these Java calls than you would expect, as it is not always clear how they compare to the Elasticsearch-native JSON format used extensively in Elasticsearch’s documentation. This issue is compounded further when the construction of queries is out of your control!

One way we could view our queries in JSON format is to enable elasticseach’s Slow Log, but there is a simpler way, using the Java Elasticsearch API to achieve what we want:

import java.io.IOException;
import org.elasticsearch.common.xcontent.ToXContent;
import org.elasticsearch.common.xcontent.XContentBuilder;
import org.elasticsearch.common.xcontent.json.JsonXContent;

...

public static void toJSON(ToXContent queryBuilder) {
  try {
    XContentBuilder builder =
        queryBuilder.toXContent(JsonXContent.contentBuilder(), ToXContent.EMPTY_PARAMS);
    log.info(builder.string());
  } catch (IOException e) {
    log.error("Failed to log elasticsearch query", e);
  }
}

...

This can be used for classes implementing either QueryBuilder or AggregationBuilder, as both of these extend the ToXContent interface.

We can use the same approach when using Spring Data Elasticsearch by calling NativeSearchQueryBuilder.getQuery(), NativeSearchQueryBuilder.getFilter() or NativeSearchQueryBuilder.getAggregations() and passing the resulting objects using the same approach.

Now we can see the exact query that will be sent to Elasticsearch:

{ "term" : { "user" : "Kimchy" } }

If you are using Kibana to view your indices, try copying & pasting your query across on the Discover page for further analysis.

Docker in JUnit tests

Committed Software — Mon, 23 Oct 2017 00:00:00 +0000

Unit testing in Java is made simple by libraries such as JUnit and e.g. Mockito, but there’s not the same panacea for integration testing. Alongside your mocked out unit tests, integration tests are important to ensure the correct function of your software when accessing external services. Previously, we have taken different approaches to the problem, such as:

Embedding the service
Running the service
Running a Docker container for the service

Embedding the service can be hard (especially if it is not Java). Running them directly, or remotely, adds significant work to configure, maintain and make available for all developers and your continuous integration service. Running a Docker container for the service was our preferred method but setting them up locally and as part of the CI build seems out of sorts with the fundamental approach to testing. This has been a common and frustrating pain point for us.

Test Containers

We have recently switched to using TestContainers to solve this problem. TestContainers is an MIT licensed open source project for running a Docker container for a JUnit test.

Benefits

This has a number of benefits over our previous approaches. First, we are running the tests against a real instance of the service, not some mocked or embedded version, which may behave differently. As the containers run just for the test they are isolated from other test and become much more reproducible. The setup and configuration are done in the test class, where they belong. It also allows you to easily repeat the test against different versions of the service allowing you to correctly determine a compatibility and check new versions of services do not require changes to your own code.

Use cases

This technique for integration testing can easily be applied to multiple use cases, some of which have more specialised support:

Databases
Logging services
Web services
Complex multi-service interactions using Docker Compose
UI testing with Selenium

Example

The following example shows how to set up an integration test for Redis.

import org.junit.Rule;
import org.junit.Test;
import org.testcontainers.containers.GenericContainer;

import redis.clients.jedis.Jedis;

public class RedisIntegrationTest {

  private static final String DOCKER_IMAGE = "redis:3.2.9";

  private final Jedis jedis;

  @Rule // 1
  public static GenericContainer redis = new GenericContainer(DOCKER_IMAGE).withExposedPorts(6379); // 2, 3

    @Before
    public void setUp() throws Exception {
        Jedis jedis = new Jedis(redis.getContainerIpAddress(), redis.getMappedPort(6379)); // 4
    }

  @Test
  public void yourTest() {
    //Your test code here
  }

}

@Rule Runs a new container for every test, you can change to @ClassRule for a single container for the test class.
It also supports the use of Dockerfiles or Docker Compose files for non-standard, of not published containers by using DockerComposeContainer.
Other standard docker config like .withVolume(), withEnv(), withCommand() are available.
Obtain the configuration you need from the GenericContainer object.

Disadvantages

None! O.K. maybe there are some disadvantages, but we think they are well worth it:

You need docker available to run the tests. (i.e. on your local device and on the CI server) but who doesn’t?
You need to make docker available (which may require exposing the docker socket)
You have to download the image but this is only needed once.
Windows support is only in Alpha.

Credit: Image by Richard North, TestContainers Project.

First impressions with GraphQL in Java

Committed Software — Mon, 16 Oct 2017 12:40:00 +0000

GraphQL is an open query language for exposing data in a flexible way. It was developed by Facebook and now in production use in many projects. The main concepts are that GraphQL describes the data available and then the consumer can ask for what they need. All using the same endpoint and language.

We are investigating its use to expose natural language processing results to replace, or augment, REST.

Java ecosystem

GraphQL is programming language agnostic and different implementations exist in many languages for this project we are using Java and Spring Boot so the natural choice is graphql-java. It may not be the only java implementation but it is certainly the most active at this time. This core project stays true to the specification and there are a number of supporting projects to ease integration and use.

Approach

In GraphQL you need to declare your data schema so clients can introspect the system and query correctly. However, the schema needs to be an accurate representation of your underlying data classes. While you could create both of these things manually, generating one from the other will reduce effort and errors. One approach is to define your schema and generate the data classes. This approach is provided by graphql-apigen and graphql-java-tools. While this approach could be very useful if you have a new project with a strict schema specification, we already have data classes. Therefore, we take the classes-first approach and generate the schema. There is a graphql-java-annotations project based on this approach, however, development of it seems to have stopped and the community seems to be moving towards graphgql-spqr (pronounced “speaker”). It looks likely that this will become the official graphql-java class-first approach.

Getting started

It was very easy to get started using the graphql-spring-boot. This gives:

A graphql-java servlet, to serve the schema, and accept GET and POST GraphQL requests
A GraphiQL UI, for writing and executing GraphQL queries against the published schema.
~~grapghql-java-tools~~ schema first
~~graphql spring common~~ class first

We chose not to use the latter and added graphql-spqr simple by providing a graphql.schema.GraphQLSchema Bean generated using graphql-spqr and annotations on DemoService and the POJOs used.

package io.committed;

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.EnableAutoConfiguration;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.ComponentScan;
import org.springframework.stereotype.Controller;
import org.springframework.web.servlet.config.annotation.ViewControllerRegistry;
import org.springframework.web.servlet.config.annotation.WebMvcConfigurerAdapter;

import graphql.schema.GraphQLSchema;
import io.committed.query.DemoService;
import io.leangen.graphql.GraphQLSchemaGenerator;

@Controller
@EnableAutoConfiguration
@ComponentScan
public class GrahpQLDemo {

  @Autowired
  DemoService demoService;

  @Bean
  GraphQLSchema schema() {
    return new GraphQLSchemaGenerator()
        .withOperationsFromSingleton(demoService)
        .generate();
  }

  public static void main(String[] args) throws Exception {
    SpringApplication.run(GrahpQLDemo.class, args);
  }

}

Getting going

It was simple to add the graphql-spqr annotations to our existing JPA/MongoDB data access objects (DAO). In fact, if you are using the same names it is not even necessary as they will be picked up automatically. Alternatively, if you want to separate the DAO from the GraphQL definition you can define a set of data transfer objects (DTO) and use those. This may give you more flexibility if you can’t change your DAO layer. We exposed the types to the GraphQL by adding a root query method to each service i.e.:

package io.committed.query;

import java.util.List;

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Component;

import io.committed.dao.repository.DemoRepository;
import io.committed.dto.Document;
import io.leangen.graphql.annotations.GraphQLQuery;

@Component
public class DocumentService {

  @Autowired
  DemoRepository repository;

  @GraphQLQuery(name = "allDocuments", description="Get all documents")
  public List<Document> getDocuments() {
    return repository.findAll();
  }
}

Arguments can be added to these methods for filtering, limiting, etc and if using Streams these can be returned directly:

@GraphQLQuery(name = "allDocuments", description="Get all documents")
  public Stream<Document> getDocuments(
      @GraphQLArgument(name = "limit", defaultValue = "0") int limit) {
    Stream<Document> stream = repository.streamAll();
    if (limit > 0) {
      stream = stream.limit(limit);
    }

    return stream;
  }

Similarly, we can get a specific document by id, and directly return an Optional:

@GraphQLQuery(name = "document")
  public Optional<Document> getDocument(@GraphQLArgument(name = "id") String id) {
    return repository.findById(id);
  }

In GraphQL the client asks explicitly for what data the result should contain. This does not have to be restricted to the fields of your data objects, methods can also be used to provide calculated results, here is a trivial example:

package io.committed.dto;

import io.leangen.graphql.annotations.GraphQLId;
import io.leangen.graphql.annotations.GraphQLQuery;
import lombok.Data;

@Data
public class Document {

  @GraphQLId
  private String id;
  private String content;

  @GraphQLQuery(name = "length")
  public int length() {
    return content.length();
  }

}

This will expose a length field on the Document class. This method is only called if requested by the query so the client does not have to pay the penalty for any calculations that they do not need. Such methods also allow the client to save on data transfer by getting the server to do the calculations and only transferring the result.

Getting graphing

The real power of GraphQL comes from the ability to traverse data type by their joining properties. For example, if I have Entities extracted from a Document then I want to be able to query for the entities contained in a document. If these are already stored in your Document object this is trivial, however, they may be stored in a different table or collection in your database. To embed entities in the document we add the following:

@GraphQLQuery(name = "entities")
  public Stream<Entity> getByDocument(@GraphQLContext Document document) {
    return repository.getByDocumentId(document.getId());
  }

where the @GraphQLContext annotation provides the linking logic to build the schema.

Getting querying

You can query using the GraphiQL UI hosted on /graphiql or send HTTP requests to /graphql and get the schema from /schema.json, I’m sure all these are configurable too. You can also use the GraphQL internally, for example by creating a QueryService:

package io.committed.query;

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;

import graphql.ExecutionResult;
import graphql.GraphQL;
import graphql.schema.GraphQLSchema;

@Service
public class QueryService {

  private final GraphQL graphQL;

  @Autowired
  public QueryService(GraphQLSchema graphQLSchema) {
    graphQL = GraphQL.newGraphQL(graphQLSchema).build();
  }

  public ExecutionResult query(String query) {
    return graphQL.execute(query);
  }

}

Or by exposing the GraphQL object as a bean.

Getting better

We were able to get a quick GraphQL endpoint up and running in less that a day on our existing DAO objects providing a powerful querying mechanism for clients. During this investigation, we uncovered and reported a bug in graphql-spqr and got a rapid response from the author. It looks likely that this ecosystem will develop fast, out suggestions for improvements to graphql-spqr are:

Ability to ignore certain fields
Out of the box support for limiting and filtering
Support for graphql-java v5

Overall, this is a very promising route for rapid integration of GraphQL into an existing spring-boot server.

React Router Redux Authenticated Container

Committed Software — Mon, 22 Aug 2016 15:23:00 +0000

Separate authentication from your Components with a dedicated authentication component.

When writing React applications with Redux and react-router-redux some routes in your application may require an authenticated user and others not. To separate the authentication concern we use a specific container that checks for the authentication and before rendering the child. If the authentication fails then you can respond by display an error or more likely redirect to login.

Route

To simplify the main render method we use a function to wrap the child in an AuthenticatedComponent, so it is clear in your main render method which parts of your application are protected.

ReactDOM.render(
  <Provider store={store}>
    <Router history={history}>
      <Route path="/" component={AppParent}>
        <Route path="login" component={LoginContainer}/>
        <Route path="logout" component={LogoutContainer}/>
        <Route path="about" component={AboutPage}/>
        <Route path="help" component={HelpPage}/>
        <Route path="user" component={requireAuth(UserContainer)}/>
        <Route path="admin" component={requireAuth(AdminContainer)}/>
        <Route path="main" component={requireAuth(MainContainer)}>
          <Route path="one" component={FirstPage}/>
          <Route path="two" component={SecondPage}/>
        </Route>
      </Route>
    </Router>
  </Provider>,
  document.getElementById('app')
)

Authenticated Component

We then implement the requireAuth function as follows. You may need to change the detail of the auth check depending on your authentication actions and you can make the authFailed function do whatever you like, in this example we redirect to login with the path so we can redirect back after a successful login.

import React, { Component, PropTypes } from 'react'
import { connect } from 'react-redux'
import { push } from 'react-router-redux'

export function requireAuth(ChildComponent) {

  class AuthenticatedComponent extends Component {

    componentWillMount() {
      this.checkAuth(this.props.isAuthenticated);
    }

    componentWillReceiveProps(nextProps) {
      this.checkAuth(nextProps.isAuthenticated);
    }

    checkAuth(isAuthenticated) {
      if (!isAuthenticated) {
        this.props.authFailed();
      }
    }

    render() {
      return (
        <div>
         { this.props.isAuthenticated === true ?  
           <ChildComponent { ...this.props } /> : null }
        </div>
      )

    }
  }

  const mapStateToProps = (state) => ({
    auth: state.auth,
    isAuthenticated: state.auth.login.isAuthenticated,
  });

  const mapDispatchToProps = (dispatch, ownProps) => {
    return {
      authFailed: () => {
        let location = ownProps.location
        let redirect = encodeURIComponent(location.pathname + location.search)
        dispatch(push(`/login?next=${redirect}`))
      },
    }
  }

  return connect(mapStateToProps, mapDispatchToProps)(AuthenticatedComponent);

}

The auth details can be useful for the child components, say to show the username but not all children are likely to need it. A good way to do this is to make it a context object for the children. This can be done by adding the following to the class:

class AuthenticatedComponent extends Component {

  \\...

  static childContextTypes = {
    auth: PropTypes.object.isRequired
  }

  getChildContext() {
    return {
      auth: this.props.auth
    }
  }

  \\...
}