DEV Community: Sonalake

An Open Source Rust SNMP Simulator

Frank Horvath — Tue, 10 Jan 2023 09:35:01 +0000

In much of the work we carry out for communication service providers (CSPs), either as part of VisiMetrix integrations or when developing custom software products, we need to integrate with IP networking devices that power the packet core, circuit-switched core, backhaul, transmission, radio access, and IMS networks. While many of these integrations use proprietary management protocols, others conveniently use SNMP.

SNMP defines a standard protocol to collect status data, configuration information, and asynchronous notifications from such devices. Slightly less convenient is the fact that during development, we often don’t have access to the actual devices we need to integrate.

Developing and testing against production devices is not an option. While sometimes clients may have test devices, they’re often in high demand for exclusive access by other departments within the CSP. They can also suffer from sporadic downtime or are hidden behind remote desktop solutions rendering them unreachable from development and continuous integration environments.

For these reasons, a very long time ago, we developed an SNMP agent simulator. We used Java because it was our go-to language at the time. Cut to today, and while we still do most of our server-side work in Java and Kotlin, we also have a growing number of teams developing products for our clients in Rust.

Recently when looking for a way to provide more of our engineers an opportunity to develop and hone their Rust skills and hit on the idea of rewriting the agent simulator in Rust. Not only that, we decided to develop it in the open.

We’ll be sharing more about our progress in future posts, but in this one I’m going to talk about a particular challenge we hit early on.

The Problem

Our goal is to implement a service that can simulate one or more SNMP agents concurrently and allow such agents to be created, deleted, started, or stopped at runtime.

Our functional requirements can be summarised as:

Expose REST endpoints that allow clients to control the simulator
Support multiple UDP listeners, each on their own port
Support dynamically creating and destroying UDP listeners

We also want to avoid excessive thread creation.

One of the first problems we hit was how to approach handling a dynamic set of UDP listeners in addition to a HTTP listener for REST API requests. Here’s what the (very) high level architecture looks like.

While there are plenty of Rust HTTP server frameworks [1], none support our UDP requirement. What to do?

Standing on the Shoulders of Giants

Before we did anything, we needed to make some key technical decisions. Based on previous experience and in some cases research, we settled on using the following crates [2].

Actix

Actix is an actor framework for developing concurrent applications built on top of the Tokio asynchronous runtime. It allows multiple actors to run on a single thread, but also allows actors to run on multiple threads via Arbiters. Actors can communicate with each other by sequentially exchanging typed messages.

The Actix framework is stable and mature and will ultimately help us create a scalable solution.

Actix Web

To handle REST API requests, we chose the Actix Web framework. By default, its HttpServer automatically starts a number of HTTP workers equal to the number of logical CPUs in the system. Once created, each receives a separate application instance to handle requests. Application state is not shared between threads, and handlers are free to manipulate their copy of the state with no concurrency concerns.

rasn ASN.1

We also need to read and write SNMP messages. The rasn ASN.1 codec framework supports handling ASN.1 data types according to different encoding rules.

Conveniently, it also provides implementations for a bunch of relevant SNMP IETF RFCs:

SNMP v1: RFC 1157
SNMP v2: RFC 3416
SNMP v2c: RFC 1901
SNMP v3: RFC 3412, RFC 3414

Application Architecture

On startup, the SNMP Agent Simulator creates a number of UdpServerProvider actors to handle incoming requests. The number of actors is determined by the host environment CPU resources. The UdpStreamHandler actor is responsible for serializing and deserializing UDP messages and for forwarding the decoded SNMP messages to the SNMPAgentEngine for further processing.

UDP Socket Creation

So how do we go about creating a UDP socket for each agent? There are a couple of potential options. We can either use UdpSocket from the Rust standard library, or UdpSocket from the Rust async standard library, or UdpSocket from Tokio. Which to choose?

Rust Standard Library

The standard net UdpSocket represents a raw UDP socket bound to a socket address.

let std_socket = std::net::UdpSocket::bind(binding_address)
    .map_err(|error| UdpServerError::StartFailed(error.to_string()))?;

Standard net UdpSockets are created in blocking mode by default, and adding a stream backed by a socket created in blocking mode to an Actix stream handler actor leads the actor to end up in deadlock state.

To work around this, we need to ensure we create the socket in non-blocking mode before adding it to the Actix stream handler actor.

std_socket
    .set_nonblocking(true)
    .map_err(|error| UdpServerError::StartFailed(error.to_string()))?;

From here we can take that socket and wrap it in a Tokio UdpFramed object.

let socket = TokioUdpSocket::from_std(std_socket)
    .map_err(|error| UdpServerError::StartFailed(error.to_string()))?;

This works, but we have to explicitly set the socket to no-blocking mode. Perhaps the Rust async standard library might offer a solution.

Rust Async Standard Library

Unfortunately this isn’t an option. Tokio does not support async_std sockets so we can’t wrap the socket in a UdpFramed object.

The last option to explore is the Tokio UdpSocket.

Tokio UdpSocket

The Tokio UdpSocket is created in non-blocking mode directly, so we don’t need to change that property manually.

let socket = TokioUdpSocket::bind(binding_address)
    .await
    .map_err(|error| UdpServerError::StartFailed(error.to_string()))?;

This does the job and is succinct. Now that we’ve settled on how to create sockets, the next challenge is to figure out how to convert data to and from SNMP messages.

UDP data to SNMP Messages

Raw UDP sockets work with datagrams (byte buffers), but it is more convenient for us to work with domain specific types (messages). Tokio UdpFramed is an interface to an underlying UDP socket that returns a single object that is both a Stream for reading messages and a Sink for writing messages. However, the StreamHandler actor expects the Stream part of the UdpFramed object only. Conveniently, the split method allows us to split the UdpFramed object into separate Stream and Sink objects as in the following code snippet.

// create a UdpFramed object using the SnmpCodec to work with the encoded/decoded frames directly, instead of raw UDP data
let (sink, stream) = UdpFramed::new(socket, SnmpCodec::default()).split();

The UdpFramed constructor must be supplied by an instance of a codec object, which implements the Encoder and Decoder Tokio crate traits. It’s used on top of the socket to handle frames. SnmpCodec is used to decode the received UDP socket data into a stream of SNMP messages, and to encode the SNMP messages into ASN.1 encoded UDP socket data. The SnmpCodec is a wrapper of rasn ASN.1 codec framework, which implements the Encoder and Decoder traits to encode and decode UDP frames. The decoder and encoder implementations leverage the rasn ASN.1 codec framework.

In the Actix StreamHandler actor, the stream handler helper trait is used to implement a UDP receiver data handler in a similar way to standard actor messages. The UdpStreamHandler represents an UDP socket receiver. The actor UdpMessage handler is called, when an SNMP message is received from the socket and successfully decoded.

// Add the stream to the actor's context.
// Stream item will be treated as a concurrent message and the actor's handle will be called.
ctx.add_stream(stream.map(|a| UdpMessage(a.map_err(|e| e.to_string()))));

The StreamHandler actor has no method to deregister the stream from the actor’s context. The solution here is to simply close the socket representing the stream.

What next?

We love Rust! It’s a great language. However, it’s still possible for developers to write code that results in deadlocks. This often happens when the synchronous and asynchronous worlds accidentally collide. Perhaps we’ll write more about that in a future post.

In the meantime there’s lots more work to do on the SNMP simulator. If you’re interested in contributing, please let us know on Github!

[1]: For example Actix Web, Rocket, Axum, Warp, Tide.

[2]: 'Library' or 'package' for readers new to Rust

SSO without Standards: Simple Yet Secure Authentication in Legacy Systems

zfattuhi-sonalake — Tue, 01 Mar 2022 09:26:21 +0000

Single Sign-On is a solved problem, right? There are even standards! SAML, OAuth 2.0, SCIM, OpenID Connect… So many standards. But what if you’re working with a legacy application that doesn’t support any of these? Well, that’s a situation we found ourselves in. I want to share how we overcame the challenge and what we learned along the way.

First, a little context.

We needed to be able to include UI elements from a newly developed web application (Spring Boot, ReactJS, RESTful APIs) into a legacy content management system (CMS) via embeds. We also needed to provide a seamless experience for users so that they don’t feel that they’re moving between the two systems.

No problem, we said. Let’s use one of the standards-based SSO mechanisms.

Hold On

Not so fast. The legacy CMS doesn’t support modern SSO mechanisms. How do we solve the challenge of fulfilling the authentication requirements when we can’t use a standard approach to SSO?

We immediately identified two options.

Option 1: Shared Session

Pass the sessionId in the browser from a CMS page to the new system via URL
Create a session validation service in the CMS to provide the new system a mechanism to validate any sessionIds it receives
Update the new system so that it calls the session validation service on request to validate the received sessionId

We discounted this option as it introduces tighter coupling between the two systems. It could also lead to performance issues, as the session would potentially need to be validated on each request to the new system.

Option 2: Shared Credentials

Pass the user credentials to the new system via URL
New system validates the credentials either by the authentication service in the CMS, or by directly accessing the database of the old system to verify the credentials

Again, we discounted this option because sharing credentials, even encrypted credentials, is poor practice, and accessing the CMS database from the new system introduces tighter coupling.

At that point, we decided to see if JWT might be a solution.

JWT to the Rescue

After a little exploration, we settled on the following approach:

CMS system generates a JWT token (using a secret that it and the new system know) and passes that token to the new system in each URL
New system receives the JWT token, stores it securely in the browser (using, for example, HttpOnly cookie), which then uses the token to subsequent REST API calls.
New system can validate JWT tokens because it knows the token secret. This ensures tokens are valid, authentic, and have not been tampered with.

Success! This approach worked out great – authenticated users were able to seamlessly and securely access both CMS content as well as content from the new system.

JWT gave us:

Portability: JWT token carries its own unit of identity, and can be validated without having to call a 3rd party system (no performance impact!)
Decentralisation: Both the JWT token issuer and consumer can be completely separate systems, as long as they share the same secrets. In our case the CMS is the JWT issuer, and the new system is the JWT consumer.
Extensibility: Since the JWT token is not tightly coupled to any system, if a new system is to be integrated with the CMS in the future then the same JWT token can be used for this integration.
Self contained: JWT tokens have sufficient information to identify the caller.

In addition to the above, JWT tokens are currently the de-facto standard mechanism for authentication, and following the standards is a great way of reducing the possibility of introducing yet another legacy system to the world.

Some Code

The following code snippets show how we achieved our goal.

CMS

In the CMS, the following code generates the JWT token. It’s in a JSP (did I mention that the CMS was legacy?).

<%@ page import="com.auth0.jwt.JWT,com.auth0.jwt.algorithms.Algorithm" %>

<% 
       Algorithm algorithm = Algorithm.HMAC256("secret value");
        String jwtToken = JWT.create()
                .withIssuer("auth0")
                .withExpiresAt(expiryDate)
                .sign(algorithm);

%>
<iframe src="....?token=<%=jwtToken %>">

When the page is rendered, the JWT token is generated and passed as a URL parameter in any iframe links.

New System

We introduced two classes to support JWT validation.

JWTTokenAuthRequestFilter is a Spring HTTP filter class. It ensures the JWT token is valid and has not expired.


import com.auth0.jwt.JWT;
import com.auth0.jwt.JWTVerifier;
import com.auth0.jwt.exceptions.JWTVerificationException;
import org.springframework.web.filter.OncePerRequestFilter;


public class JWTTokenAuthRequestFilter extends OncePerRequestFilter {
private final JWTVerifier verifier;

public JWTTokenAuthRequestFilter(ApiProperties apiProperties) {
   Algorithm algorithm = Algorithm.HMAC256(apiProperties.getJwtSecret());
   verifier = JWT.require(algorithm)
           .build();
}

@Override
protected void doFilterInternal(HttpServletRequest request, HttpServletResponse response, FilterChain filterChain) throws ServletException, IOException {

// Obtain jwt token from the http request
String authToken = ....

try {
   verifier.verify(authToken);
   filterChain.doFilter(request, response);
} catch (JWTVerificationException jwtVerificationException) {
   logger.info("Error while verifying the JWT token:", jwtVerificationException);
   response.sendError(HttpStatus.FORBIDDEN.value(), String.format("Invalid JWT Token, error: %s", jwtVerificationException.getMessage()));
   return;
}

...

}

JWTWebSecurityConfigurer registers the request filter (JWTTokenAuthRequestFilter above) in the Spring Context and handles requests as soon as they are received.


@Configuration
@EnableWebSecurity
public class JWTWebSecurityConfigurer extends WebSecurityConfigurerAdapter {
   private final JWTTokenAuthRequestFilter jwtTokenAuthRequestFilter;

   public JWTWebSecurityConfigurer(JWTTokenAuthRequestFilter jwtTokenAuthRequestFilter) {
this.jwtTokenAuthRequestFilter = jwtTokenAuthRequestFilter;
   }


@Override
protected void configure(HttpSecurity http) throws Exception {
   http.addFilterBefore(jwtTokenAuthRequestFilter, UsernamePasswordAuthenticationFilter.class)
           .addFilter(jwtTokenAuthRequestFilter)
           .authorizeRequests().anyRequest().permitAll();
}
...
}

Conclusion

We were really happy with this approach as it met our requirements without making compromises on security, and required minimal code change to implement. On top of that, should we need to integrate other applications with the CMS in the future, we’ll be able to use the same JWT token.

If you hit a challenge like this in the future, be sure to take advantage of JWT if you can!

Integration Testing Done Right

winciu — Wed, 02 Feb 2022 13:54:34 +0000

Writing integration tests is not always straightforward. By definition, integration tests require interaction between several components, and we need to deal with them in many different ways. Let’s look at some tools that will make writing and reading tests easier. I believe that Testcontainers and the Spark Framework will allow you to write shorter and more descriptive tests.

Is this how you test?

What is your approach to writing an integration test (IT)? Maybe some of the following sound familiar:

Write mocks or stubs of external services
Create a dedicated remote environment for ITs (playground, sandbox) and run them there
Setup all the components (where the ITs are supposed to run) locally

No, I’m not saying that you’ve been doing it all wrong if you do that! But the truth is each of those approaches has drawbacks. By way of example, let’s look at the first option.

When you mock or stub some external services that are not crucial for the component you are testing, there is a chance you might miss some aspects of that mocked service that can only occur when running live.

Of course, you could invest more effort into replicating the logic of how the actual component works, but it might be difficult to replicate accurately and will also be time-consuming to develop. Even then, there are no guarantees it will be correct, so your test might still be unreliable.

What if there was a more effective way? Let’s see what we can do to make real integration tests and not imitation ones!

Meet Testcontainers

Testcontainers is a Java library that supports JUnit tests providing lightweight instances of anything that we can run in a Docker container.

I will go through a use case where Testcontainers can provide substantial benefits.

In the project I’m currently working on, we have a component called the Integration Component (IC). IC is a Spring Boot service that acts as a consumer and a producer of RabbitMQ messages. As a consumer, it listens to a queue where another service sends job requests. IC reads those messages (job requests), processes them and finally sends an HTTP request to Databricks to run a job. Before we submit the request (step #4 on the diagram below), we need to do a few other things, and we have divided this logic into several steps in the IC.

For testing purposes, those requests are handled by the Spark framework, but I’ll get back to that later.

As mentioned before, the service logic is divided into several steps, where each step has a process() method. Let’s look at the SendRequestToJobQueueStep method (#3 on the diagram above).

@Slf4j
@RequiredArgsConstructor
@Component
public class SendRequestToJobQueueStep implements JobRequestConsumerStep {

   private final AmqpAdmin amqpAdmin;
   private final Exchange exchange;
   private final RequestQueueProvider requestQueueProvider;
   private final RabbitTemplate rabbitTemplate;
   @Value("${config.request-queue-ttl}")
   private Duration requestQueueTtl;

   @Override
   public boolean process(JobRequestProcessingContext context) {

       String queueName = createAndBindRequestQueue(runSettings.getTraceId(), context.getJobType());
       try {
           Supplier<? extends SpecificRecordBase> requestProvider = context.getRequestProvider();
           sendJobRequestToRequestQueue(requestProvider.get(), queueName);
       } catch (AmqpException e) {
           String customMsg = String.format("Sending '%s' request using routing key '%s' for jobId=%d failed.", context.getJobType(), queueName, job.getJobId());
           log.error(prepareExceptionLogMessage(deliveryTag, e, customMsg), e);
           requeue(context, deliveryTag);
           return false;
       }
       return true;
   }

   private <R extends SpecificRecordBase> void sendJobRequestToRequestQueue(R requestObject, String routingKey) {
       rabbitTemplate.convertAndSend(Amqp.EVENTS, routingKey, requestObject);
       log.info("Job request is sent to job queue. Routing key: '{}'", routingKey);
   }


   private String createAndBindRequestQueue(String traceId, JobType jobType) {
       Queue requestQueue = requestQueueProvider.getRequestQueue(jobType, traceId);
       amqpAdmin.declareQueue(requestQueue);
       String routingKey = requestQueue.getName();
       Binding binding = BindingBuilder.bind(requestQueue)
               .to(exchange)
               .with(routingKey)
               .and(Map.of("x-expires", requestQueueTtl.toMillis()));
       amqpAdmin.declareBinding(binding);
       return routingKey;
   }
}

When the process() method is invoked, the IC is sending a job request to the dynamically created and bound queue. The creation and binding happen in the createAndBindRequestQueue() method.

There’s quite a lot going on in that class. Imagine writing an integration test that would cover all that logic!

There’s another challenge. Consider the createAndBindRequestQueue() method. If you mock all the methods used in it, namely declareQueue() and declareBinding(), will it really help you? Sure, you can verify if those methods were invoked, or try to return a value (if it’s possible), but it’s not actually the same as running the code live.

An approach using mocks might look like this:

@Test
void queueShouldBeDeclaredAndBoundDuringCreation() {
   when(queue.getName()).thenReturn(QUEUE\_NAME);
   when(exchange.getName()).thenReturn(EXCHANGE\_NAME);

   step().process(context);

   verify(amqpAdmin).declareQueue(queue);
   ArgumentCaptor<Binding> bindingArgumentCaptor = ArgumentCaptor.forClass(Binding.class);
   verify(amqpAdmin).declareBinding(bindingArgumentCaptor.capture());
   Binding binding = bindingArgumentCaptor.getValue();

   assertEquals(EXCHANGE\_NAME, binding.getExchange());
   assertEquals(QUEUE\_NAME, binding.getRoutingKey());
   assertEquals(QUEUE\_NAME, binding.getDestination());
}

This might be considered a unit test, but it’s definitely not an integration test. What we need here is to verify whether the queue has been created for real and a message has been sent to it.

Here’s how to achieve that using Testcontainers.

@Slf4j
@SpringBootTest
@Testcontainers
class CommonJobRequestConsumerIT {

   private static final int SPARK\_SERVER\_PORT = 4578;

   @Container
   private static final RabbitMQContainer container = new RabbitMQContainer(DockerImageName.parse("rabbitmq:3.8.14-management")) {
       @Override
       public void stop() {
           log.info("Allow Spring Boot to finalize things (Failed to check/redeclare auto-delete queue(s).)");
       }
   };

   @Autowired
   private RabbitTemplate template;
   @Value("${databricks.check-status-call-delay}")
   private Duration statusCallDelay;
   private static SparkService sparkServer;

   @BeforeAll
   static void before() {
       sparkServer = SparkService.instance(SPARK\_SERVER\_PORT);
       sparkServer.startWithDefaultRoutes();
   }

   @AfterAll
   static void after() {
       sparkServer.stop();
   }

   private <R extends SpecificRecordBase> void assertJobRequest(String expectedQueueName, Supplier<R> requestSupplier, JobSettingsProvider<R> jobSettingsProvider) throws IOException {

       Message receivedRequest = template.receive(expectedQueueName);
       assertNotNull(receivedRequest);
       R serializedRequest = AvroSerialization.fromJson(requestSupplier.get(), receivedRequest.getBody());
       log.info("Request received in '{}' '{}'", expectedQueueName, serializedRequest.toString());
       RunSettings jobSettings = jobSettingsProvider.getJobSettings(serializedRequest);
       assertEquals(JOB\_ID, jobSettings.getJobId());
       assertEquals(TRACE\_ID, jobSettings.getTraceId());
   }

   @ParameterizedTest
   @MethodSource
   <R extends SpecificRecordBase> void jobRequestShouldBeSentToDedicatedQueue(String requestRoutingKey, R request, String expectedQueueName, Supplier<R> requestSupplier, JobSettingsProvider<R> jobSettingsProvider) throws InterruptedException, IOException {

       template.convertAndSend(Amqp.EVENTS, requestRoutingKey, request, m -> {
           m.getMessageProperties().setDeliveryTag(1);
           return m;
       });

       // finish all steps
       // + give rabbit some time to finish with dynamic queue creation
       long timeout = statusCallDelay.getSeconds() + 1;
       TimeUnit.SECONDS.sleep(timeout);

       assertJobRequest(expectedQueueName, requestSupplier, jobSettingsProvider);
   }
}

As you can see, the test is readable and really easy to follow, which is not always the case when you mock. We start (in the before() method) with some Spark related logic (more on that later), and then we send a message to the queue, starting the entire process. This is exactly how the system under test (IC) works. It’s listening to a particular queue and once the message is there, it picks it up and starts processing it. In some cases, we need to wait a bit, since otherwise a test will finish too early and assertions will fail.

I think that proves that there is a good reason why using Testcontainers in similar cases could be an excellent choice. In my opinion, there is no better way to be certain this code works as expected.

I’m sure there are many other examples where mocking is not a viable solution, and the only reasonable option is to be able to run those components live. This is where the Testcontainers library shows its power and simplicity. Give it a try next time you write an integration test!

Hero #2: The Spark framework

In the same component (IC) I’m also using the Spark framework to handle HTTP calls to an external service, in this case, Databricks API. Spark is lightweight and perfect for such use cases. Instead of mocking RestTemplate calls, we are using a real HTTP server!

Why is that so important? Well, if you look at how our test is organised, I think it will become apparent. As mentioned earlier, I’m using Testcontainers to make the test as real as possible. I do not want to mock anything. I want my REST calls to be real as well.

Let’s look at how Spark is being used in this test. While working with Spark in integration tests, I’m using a wrapper class called SparkService.

@Slf4j
public class SparkService {
   private final Service sparkService;

   private SparkService(Service service) {
       this.sparkService = service;
   }

   static File loadJsonPayload(String payloadFileName) {
       ClassLoader classLoader = SparkService.class.getClassLoader();
       URL resource = classLoader.getResource(payloadFileName);
       return new File(resource.getFile());
   }

   public static SparkService instance(int dbServerPort) {
       return new SparkService(Service.ignite().port(dbServerPort));
   }

   public void startWithDefaultRoutes() {
       DatabricksRoutes.JOB\_LIST.register(sparkService);
       DatabricksRoutes.JOB\_RUN\_NOW.register(sparkService);
       DatabricksRoutes.JOB\_RUNS\_GET.register(sparkService);
       DatabricksRoutes.JOB\_RUNS\_DELETE.register(sparkService);
   }

   public void stop() {
       service().stop();
   }

   public void awaitInitialization() {
       service().awaitInitialization();
   }

   public Service service() {
       return sparkService;
   }
}

Notice the startWithDefaultRoutes() method. It contains several lines where particular endpoints (which I would like to stub) are defined. I’m using enum classes for those endpoints, and each of the enum keys implements the SparkRoute interface.

public interface SparkRoute {
   HttpMethod httpMethod();

   String path();

   Route route();

   default void register(Service sparkService) {
       register(sparkService, route());
   }

   default void register(Service sparkService, Route route) {
       switch (httpMethod()) {
           case GET:
               sparkService.get(path(), route);
               break;
           case POST:
               sparkService.post(path(), route);
               break;
       }
   }
}

Here is an example of the JOB_LIST enum from the DatabricksRoutes class.

public enum DatabricksRoutes implements SparkRoute {
   JOB\_LIST {
       @Override
       public String path() {
           return "/api/2.0/jobs/list";
       }

       @Override
       public Route route() {
           return JobController.handleJobList("json/spark/jobs\_list\_response.json");
       }

       @Override
       public HttpMethod httpMethod() {
           return HttpMethod.GET;
       }
   }
}

Ok, so how are all of these used in actual integration tests? In a simple scenario where no special logic for the stubbed endpoints is required, it could look like this.

private static SparkService sparkServer;

@BeforeAll
static void before() {
   sparkServer = SparkService.instance(SPARK\_SERVER\_PORT);
   sparkServer.startWithDefaultRoutes();
}

@AfterAll
static void after() {
   sparkServer.stop();
}

@DynamicPropertySource
static void properties(DynamicPropertyRegistry registry) {
   registry.add("databricks.url=", () -> "http://localhost:" + SPARK\_SERVER\_PORT);
   registry.add("databricks.token.token-host=", () -> "http://localhost:" + SPARK\_SERVER\_PORT);
   registry.add("spring.rabbitmq.host", container::getContainerIpAddress);
   registry.add("spring.rabbitmq.port", () -> container.getMappedPort(5672));

   log.info("RabbitMQ console available at: {}", container.getHttpUrl());
}

I have not mentioned this earlier but in the properties() method above, you can see how RabbitMQ can be configured with Testcontainers. This method is where all the properties in which we need to specify a URL are overridden, so we could use Spark as a handler for the original REST calls to those default services.

That was a simple Spark usage scenario within an integration test. For more sophisticated logic, we need a bit of a different approach.

What if we have a parameterised test, and we need each given endpoint to return a different response for every run? In the example above, all the endpoints were defined before the test started. However, we can configure particular endpoint handlers inside each test. This is where Spark can show its power of configuration and customisation in handling incoming requests.

For instance, consider this example:

sparkServer.registerGetRoute(JOB\_RUNS\_GET\_PATH, handleRunsGet(jobRunDataPath));
sparkServer.awaitInitialization();

jobRunDataPath is one of the parameters in a parameterised test, so we can register a different request handler and return a custom response (a JSON file) for every test run.

Try it out!

To sum up, I believe that Testcontainers and the Spark framework will change your habits when writing integration tests.

By leveraging the power of containerisation, you can move your tests to the next level by making them more reliable and even easier to write. You will be able to verify your system under test in almost the same conditions as if it was running on production. Furthermore, your test can eventually become even more readable.

Give it a try and see how easy it is to write integration tests now!

OAuth2 Authentication with API Gateway in a Distributed Environment

znaczek — Fri, 03 Dec 2021 12:36:37 +0000

Recently I was involved in a project in which the application was structured as a group of microservices. Users were interacting with the system through a web portal. The challenge we were facing was to prepare a tailored solution that would allow users to easily authenticate across the multiple components of the system.

A lot has been written about distributed systems, as well as OAuth2, API gateways and Kubernetes separately. But when I was looking for ways to combine these technologies to build an end-to-end solution, I found that they are mostly described in isolation from other parts of the system. That’s why I decided to do a small experiment and find an elegant way to authenticate within this specific infrastructure.

If you ever get a similar challenge, I hope that this will show you an example of integrating systems that don’t necessarily show you an easy way to do it.

Requirements

Before jumping straight to the project, let’s take a look at the specifics of the environment I was working with. You’ll need these if you try to reproduce the solution:

Kubernetes as an orchestration platform for the distributed environment.
Users managed by an identity provider (IDP) following OAuth2/OIDC specification (e.g. Keycloak, Azure Active Directory etc.).
Backend services authenticate/authorize users using stateless access token added to the request as Authorization header.
AGW is the client application in OAuth2 terminology. In case the frontend needs data about authenticated users, it gets it from AGW.
Session timeout – access to the system is denied after a given period of time since the last request.

Solution overview

This diagram shows the relationships between the individual components:

Let’s follow the request path (red arrows). Once a client browser reaches the K8s cluster, it enters the system through Kubernetes’ specialized load balancer, the Ingress controller, which in our case is implemented with Nginx. As we can see, all traffic goes through AGW. A natural consequence is that all routing happens inside AGW.

We can see two streams of calls to IDP (the black arrows on the diagram). The first, coming out of AGW, is responsible for driving the authentication process which consists of acquiring the token and refreshing it within the requests so it doesn’t timeout. Additionally, AGW adds a login page URL (as a header) to unauthenticated requests to allow the client to start authentication. That’s the only small bit of FE participation in the authentication process.

Why couldn’t AGW just send a redirect itself and mention the redirect location in a header? Because we are talking here about API calls only (which in a browser are XMLHttpRequest, not browser top-level navigation), we return 401 for unauthenticated requests, hence the browser won’t do top-level redirection based on these calls.

The second stream coming out from the microservices is fetching public keys for token validation. We will take a closer look at that later on a sequence diagram.

It is worth mentioning that AGW doesn’t authenticate requests. Someone could ask “why”, as authentication is very often associated with API gateways. Well, based on the above we can say that AGW drives the authentication, knows how to authenticate users, can acquire a token for a user (after user consent is given), but doesn’t authenticate the requests going through it. The reason behind this approach is that some of the services are protected, and some are public. Even in a single service, sometimes only a couple of endpoints could be protected instead of each one of them. That’s why I left the authentication/authorization of requests to the specific services. The implementation of course doesn’t prevent us from doing authentication also in AGW. It’s just a matter of choice.

AGW (as we will see in the next section) is backed with spring-cloud-gateway and spring-security-oauth2-client. In this setup, the user session is kept inside the AGW. On the other hand, AGW is a regular Kubernetes deployment, which means that it can be evicted, restarted, scaled up and down at any time. That’s why we need a distributed session that persists throughout the service lifecycle. I decided to use a Redis backed WebSession.

Implementation details

In order to keep this clear and easy to read, I will focus only on the most important building blocks. If you’d like to see the full source code, you can find it on Github.

The project consists of three components; two Java modules and a client portal. The first Java module named agw is the AGW component, and the second called customers is a BE microservice. Under the portal directory we will find a React app that serves as a FE client.

AGW

Let’s start with the dependencies. If we look inside build.gradle, we will find a few crucial dependencies:

spring-boot-starter-oauth2-client provides all the security mechanisms around OAuth2
spring-cloud-starter-gateway is the spring implementation of gateway functionality, e.g. routing, enhancing the requests via filters etc.
spring-boot-starter-data-redis together with spring-session-data-redis provide the distributed session functionality backed with Redis

If we look at the application properties, we will find two especially interesting sections:

spring.security.oauth2.client – the configuration of OAuth2 client that the spring security will consume, for development purposes I used Keycloak as IDP (check README file in the source code for more reference)
spring.cloud.gateway configures static routing rules and default filters. The first route exposes jwks-uri as an internal Kubernetes endpoint and the next two are rules for routing traffic to the portal and the customers service.

Now, let’s go through the code. We shall start from spring security configuration, as most of the functionality of this component relies on it.

@Bean
  public SecurityWebFilterChain springSecurityFilterChain(ServerHttpSecurity http) {
    return http
        .formLogin(ServerHttpSecurity.FormLoginSpec::disable)
        .authorizeExchange(ae -> ae     //1
        .anyExchange()
        .permitAll()
      )
      .oauth2Login(l -> l
        .authorizedClientRepository(authorizedClientRepository())   //2
        .authenticationSuccessHandler(authenticationSuccessHandler) //3
        .authenticationFailureHandler(authenticationFailureHandler) //4
      )    
      .logout(l -> l
        .logoutSuccessHandler(logoutSuccessHandler)         // 5
      )
      .csrf()
      .disable()
      .build();
  }

Let’s look at what this does, other than some basic configuration:

1) As mentioned earlier, AGW doesn’t authenticate requests, so we permit all of them.

2) We need to make spring security work with a distributed session. Almost all classes that store some data are leveraging WebSession. Only the ServerOAuth2AuthorizedClientRepository needs to be defined to use the WebSession backed solution.

3), 4), 5) Minor behaviours customisation in case of login success/failure and logout, mainly redirecting to the right place in the app after these actions complete.

We also add 2 filters in the filters package:

Global401Filter – when downstream service returns 401 it adds login page URL in X-auth-entrypoint header to allow the caller client to navigate the user to it.
SessionFilter – spring maintains the session itself, but it doesn’t expose session expiration to the requests, that’s why we calculate the expiration and add it as a cookie to every request so that the client app can react to it to improve user experience.

The controller package exposes the /whoami endpoint that a client app can call to figure out whether the user is authenticated or not. If the user is authenticated, the endpoint returns status 200 with the user details payload. Otherwise, 401 status is returned with the login page URL in a header.

At this point, let’s discuss the idea behind the above setup. spring-security will handle everything related to authentication, such as:

redirecting to the IDP login page after user reaches /oauth2/authorization/<client-registration-name> where – in our case – the registration name is iam,
performing the OAuth2 authorization code exchange and automatically refreshing the access token,
storing users authentication details data in the session,
clearing the session on logout.

The session is persisted between requests using a SESSION cookie containing the session id. No other sensitive data (like OAuth token) is exposed to the frontend.

One of the spring-cloud-gateway functionalities is routing traffic to the underlying services. A TokenRelay filter will also extract the access token stored in the user session and add it to outgoing requests as an Authorization header. That allows downstream services to authenticate the request.

If the client app wants to authenticate the user, it can call the /whoami endpoint. Based on the result it can perform any operation it wants (e.g. redirecting to the login page to start the authentication process). AGW doesn’t force any client behaviour, it only exposes the necessary data to perform authentication.

The ‘customers’ microservice

As stated in the requirements, BE microservices should do the authentication/authorization. We achieve that by using org.springframework.boot:spring-boot-starter-oauth2-resource-server. To make it work, we need two things:

In application properties under the spring.security.oauth2.resourceserver.jwt.jwk-set-uri key, we define a URL under which the public keys are available,
when configuring HttpSecurity, we enable the JWT resource server like this:

http
  …
  .oauth2ResourceServer()
  .jwt()
  .jwtAuthenticationConverter(jwtAuthenticationConverter)

where jwtAuthenticationConverter\ is responsible for converting token claims to Collection<GrantedAuthority>\. We store the roles as a comma-separated list in one of the user claims.

Portal

As mentioned above, the portal isn’t too involved in the authentication process. However, there are a few points that are worth highlighting.

In src/index.tsx, before the application starts, we call /whoami (endpoint already discussed in the AGW section) and store the result for later usage. Once the user reaches the protected section of the app, which technically means entering an src/AppInternal.tsx component, the component will consume the whoami call result and it will either let the user in or redirect the user to the login page.

To improve the user experience, we store the page URL when the whoami call returns 401. Then, after the user comes back authenticated, we can redirect the user back to the stored URL.

Additionally, the src/components/session/Session.tsx component will track the session expiration. It consumes the session expiration stored in the cookie set by the AGW. If the session is about to finish, the user gets notified with a pop-up. When the session expires, the user is logged out.

Sequence diagram

Let’s see what an example user scenario would look like. We assume that the user hasn’t been authenticated yet. The user wants to see the /customers page. If we have looked at the portal behaviour for /customers we would also notice that for an authenticated user, it will try to fetch data from the /api/customers endpoint.

Summary

As we can see, this is not a complicated setup. It’s just a bit of configuration. The spring-cloud-gateway and spring-security-oauth2-client libraries put together cover the claimed requirements almost out of the box. However, to correctly understand the capabilities of these libraries, I had to spend some time going through their source code.

Additionally, due to the fact that spring-cloud-gateway is implemented using Spring WebFlux, it should by definition provide better resource utilisation. I strongly recommend looking at Spring Cloud solutions for any cloud-related topic.

Using Shotgun to Find and Limit Indirect Dependencies

Daniel Bray — Tue, 26 Oct 2021 08:25:16 +0000

On a well-run project, over time, novelty tends to zero, and all good software, if it’s being used at all, will eventually go into maintenance.

At the beginning of a project, when designs are still coming together, commits are likely to be in many different components of the application. As the application moves into maintenance, then – if the SOLID principles have been applied – one would expect commits to impact on smaller and smaller sections of the codebase. This would indicate that the changes for issue fixes and the like don’t require lots of small changes all over the codebase.

Shotgun (and its related gradle plugin) is a new tool that can identify overly complex and interdependent elements of your code base that other tools can’t.

What does this give me that a cyclomatic complexity rating doesn’t?

We already have tools for calculating the complexity of code (e.g Cyclomatic Complexity) but they rely on finding the elements of code that talk directly to each other. Shotgun, however, reports on what elements of code are updated at the same time, measuring how coherent your commits are within a code hierarchy.

For example, if you had an event-based architecture, where services interact through a queue, then you may find that changing one service might require changes in the events being sent, and so on down to the receiving services. A cyclomatic complexity rating wouldn’t take account of this, since the services don’t directly interact with each other. Shotgun, however, will report that when these elements are updated at the same time over and over again.

The idea is that – if we’ve been doing our job right – then over time the complexity of commits is getting smaller. If it’s not, then it’s likely that “small” commits are touching too many files because the code is overly interdependent.

The report looks like this:

With:

A heatmap showing the complexity of each day’s commits.
A list of active commit sets: these are sets of files that are regularly committed in one go. These file sets have a high inter-dependence.
A list of active files: these are single files that are being updated a lot.

You can also click on any given day and view the details of the commits for that day.

What do I do with this information?

Shotgun tells you if every change you’re making is a big change in lots of different components.

At the beginning of a project this is normal – you’re just figuring things out. If, after a while, most commits are touching elements all over the codebase, then you need to start thinking about refactoring.

The active commit sets will point to groups of elements that are committed together a lot. These are highly coherent. If the sets are too big then there’s probably a need to abstract some common behaviours to reduce the interdependence.
It’s also worth checking the larger commits for the same issues.

How is shotgun coherency calculated?

The aim is to derive a score for each commit, that is lower if:

The commits are limited to small numbers of files.
The commits are limited to files in the same package.
The commits are limited to files in the same package hierarchy.

The actual score is calculated as follows.

Imagine a commit like this:

The process is a simple:

Ignore any merge commits – we don’t want to risk double-counting, or be at the mercy of whether or not the merge was fast-forwarded.
Ignore any files that were deleted – removing code doesn’t add to the complexity of the application.
Split the files up into different source sets, e.g.
- src/main/java
- srs/main/resources
- src/test/java
- srs/test/resources
Build a set of simple directed graphs, where the vertices are the directories and files in the commit.
Prune out the roots of these graphs so we’re left with only the common root of the commit. In this example we’d be removing “com.sonalake”
Finally, add up all the edges of the graphs.

This is the shotgun coherency score.

Where there are multiple commits on a given day, then it is the median score that is used for that day.

Some examples:

A single file commit | Score: 1

Two files in the same directory | Score: 2

Two files in the same hierarchy | Score: 3

Two files in parallel directories | Score: 4

How can I use it?

There is a basic library that comes with a command line: shotgun; where you can find full details of the configuration parameters; but in short you can define things like:

The different source sets.
How small should commits get before they are not included in the home page.
The size of the heatmap buckets.

The easiest way to use the tool in a project is to use the shotgun-gradle-plugin

Just drop this in as a plugin in your gradle configuration (full details here):

plugins {
id 'com.sonalake.shotgun\-gradle-plugin' version "1.0.0"
}

And configure it appropriately for your project:

shotgun {
  inputDirectory            = "$projectDir"
  outputFile                = ".shotgun/report.htm"
  sourceSets                = \["src/main/java",
                                "src/main/resources",
                                "src/main/webapp",
                                "src/test/java",
                                "src/test/resources"\]
  minimumCommitInterest     =   3
  topCommitValueForFileSets =   10
  topCommitValueForFiles    =   40
  legendLevels              =   \[10, 20, 30, 50, 80, 120\]
}

Why is it called shotgun?

The purpose of this app is to spot when changes are consistently being applied in a scattergun approach over the entire codebase.

Also because of this.

Can I contribute to this project?

We hope this tool is useful to everyone, so we made it public, along with some other tools, libraries and examples, in our Sonalake github project.

If you think there are improvements to make, please fork the project and submit them, and we’d be delighted to review and merge them.

Identity vs Resource-based AWS IAM Policies

Cezary Czekalski — Tue, 07 Sep 2021 09:47:26 +0000

Identity and Access Management (IAM) is a global AWS service dedicated to managing access to AWS services and resources. It’s a cornerstone of AWS, but it’s also very powerful and there are many ways to achieve the same goal. In this post, I’m going to look at two approaches to securing access to AWS resources. Let’s first start by looking at some fundamental IAM concepts.

Key IAM Concepts

An IAM user is an entity that you create in AWS to represent the person or application that uses it to interact with AWS. Users can be given access to the AWS console or to AWS APIs. Given the appropriate permissions, users can invoke actions on AWS infrastructure resources. Users can also be assigned to groups – a collection of IAM users. Groups let you specify permissions for multiple users, which can make it easier to manage the permissions for those users. IAM users can be assigned to many different groups (one user can be assigned to many groups, but a group itself cannot be assigned to another group).

IAM permissions are assigned via policies. There is a principle which states that what is not explicitly allowed is denied by default and it’s a good idea to prevent accidentally being over-privileged.

An IAM role is an IAM identity that has specific permissions. It is similar to an IAM user, in that it is an AWS identity with permission policies that determine what the identity can and cannot do in AWS. However, instead of being uniquely associated with one person, a role is intended to be assumable by anyone who needs it.

A policy is a set of permissions written in JSON format. There are six types of policies, but this post will focus on two of them; identity-based policies, and resource-based policies.

Identity-based Policies

Identity-based policies grant permissions to an identity. An identity-based policy dictates whether an identity to which this policy is attached is allowed to make API calls to particular AWS resources or not. For example, the following policy would allow a user to invoke any Get or List request on any S3 resource.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:Get*",
                "s3:List*"
            ],
            "Resource": "*"
        }
    ]
}

The key elements of the policy definition are:

Effect – whether to allow or deny the action(s)
Action – the API(s) the policy applied to requests on resources (‘*’ means wildcard)
Resource – identifier of the resource(s) to which the policy applies. Resources are identified by an Amazon Resource Name (ARN), or by wildcard.

Resource-based Policies

Resource-based policies grant permissions to the principal that is specified in the policy. They specify who or what can invoke an API from a resource to which the policy is attached.

For example, the policy below specifies that S3 events on the bucket arn:aws:s3:::test-bucket-cezary can be handled by the Lambda (lambda-s3) in account id 1234567890 in eu-west-1 region.

{
  "Version": "2012-10-17",
  "Id": "default",
  "Statement": [
    {
      "Sid": "s3-event-cezary_for_lambda-s3",
      "Effect": "Allow",
      "Principal": {
        "Service": "s3.amazonaws.com"
      },
      "Action": "lambda:InvokeFunction",
      "Resource": "arn:aws:lambda:eu-west-1:1234567890:function:lambda-s3",
      "Condition": {
        "StringEquals": {
          "AWS:SourceAccount": "1234567890:"
        },
        "ArnLike": {
          "AWS:SourceArn": "arn:aws:s3:::test-bucket-cezary"
        }
      }
    }
  ]
}

In this case, the principal is “a caller” who can invoke a particular action on the specific resource arn:aws:s3:::test-bucket-cezary.

Now that we’ve covered these two policy types, let’s set up some demo policies and explore the types of issues that can result when mixing the two policy types and setting too restrictive or too permissive access.

Demo Environment

Assuming you have AWS account setup, create two IAM users:

An “admin user” with the following policies attached
- AdministratorAccess – This policy allows for almost unrestricted access to all services. This identity-based policy is available by default. It is created & managed by AWS. It’s also one of the so-called AWS managed policies.
A “restricted user” with the following policies attached
- IAMReadOnlyAccess – Read-only access to the IAM console
- ListAllMyBuckets (Permission) – list all S3 buckets (but not their content). You need to create an identity policy to set permission inside of it.
- AWSCloudShellFullAccess – Provides fully featured access to CloudShell in AWS Console

The policy for the admin user is:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "*",
            "Resource": "*"
        }
    ]
}

The policy for the restricted user is:

{
   "Version":"2012-10-17",
   "Statement":[
      {
         "Effect":"Allow",
         "Action":[
            "iam:GenerateCredentialReport",
            "iam:GenerateServiceLastAccessedDetails",
            "iam:Get*",
            "iam:List*",
            "iam:SimulateCustomPolicy",
            "iam:SimulatePrincipalPolicy"
         ],
         "Resource":"*"
      }
   ]
}{
   "Version":"2012-10-17",
   "Statement":[
      {
         "Sid":"VisualEditor0",
         "Effect":"Allow",
         "Action":"s3:ListAllMyBuckets",
         "Resource":"*"
      }
   ]
}{
   "Version":"2012-10-17",
   "Statement":[
      {
         "Action":[
            "cloudshell:*"
         ],
         "Effect":"Allow",
         "Resource":"*"
      }
   ]
}

Now let’s create a bucket via our admin user and test these two policies. You can do this via the command line, or if you don’t want to install the AWS Command Line Interface (CLI), via CloudShell.

# Create a bucket
aws s3api create-bucket --bucket YOUR_GLOBALLY_UNIQUE_NAME
--region YOUR_CHOSEN_REGION
--create-bucket-configuration LocationConstraint=YOUR_CHOSEN_REGION

# Response
{"Location": "http://YOUR_GLOBALLY_UNIQUE_NAME.s3.amazonaws.com/"}    
# Prohibit public access
aws s3api put-public-access-block --bucket YOUR_GLOBALLY_UNIQUE_NAME
--public-access-block-configuration "BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true"
# Put one object into a bucket (optional)
aws s3api put-object --bucket YOUR_GLOBALLY_UNIQUE_NAME
--key YOUR_FILE --body YOUR_FILE
# List your buckets (optional)
aws s3api list-buckets
#Response
{  "Buckets": [
        {
            "Name": "YOUR_GLOBALLY_UNIQUE_NAME",
            "CreationDate": "2021-06-12T11:12:55+00:00"
        },...

Policies Problems and How to Avoid Them

Having created and configured the bucket check from either console or via an API call that this bucket policy (the resource-based policy) is empty.

aws s3api get-bucket-policy --bucket YOUR_GLOBALLY_UNIQUE_NAME --query Policy --output text

Ok, now we are ready to list the content of our bucket.

# As the admin user
aws s3 ls s3://YOUR_GLOBALLY_UNIQUE_NAME
# Response: 2021-06-15 15:13:33     612870 mountains.jpg
# As the restricted user
aws s3 ls s3://YOUR_GLOBALLY_UNIQUE_NAME
# Response: An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied

As you can see, the restricted user does not have permission to list the objects in the bucket. To fix this you could use either Identity-based policies or resource-based policies (or both, but that would be redundant). Let’s go ahead and create a resource-based policy. To do this, we could use the console, but let’s use the AWS Policy Generator instead.

Go to the AWS console and select the bucket you created earlier. Next go to permissions, choose bucket policy, and then edit and policy generator. After filling the template click Add Statement and then Generate policy.

In this case I’ve chosen a bucket policy, because I know that I will add more users in the future.

Copy and paste policy to bucket settings

{
  "Id": "Policy1624125393305",
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "Stmt1624125391986",
      "Action": [
        "s3:ListBucket"
      ],
      "Effect": "Allow",
      "Resource": "arn:aws:s3:::YOUR_GLOBALLY_UNIQUE_NAME",
      "Principal": {
        "AWS": [
          "arn:aws:iam::ACCOUNT_ID:user/read-only-console-user"
        ]
      }
    }
  ]
}

Sometimes you have to wait a bit for the policy to kick in. Now you should be able to list content of your bucket.

aws s3 ls s3://YOUR_GLOBALLY_UNIQUE_NAME
#Response: 2021-06-15 15:13:33     612870 mountains.jpg

What will happen when we explicitly deny something? Remember that IAM is very strict. Explicit denial is a real, true, strict rejection and it doesn’t matter whether you allowed something earlier. When you say you want to block it, it’s blocked.

The easiest way to check it is to add resource-based policy to our bucket.

{
    "Version": "2012-10-17",
    "Id": "Policy1623531542448",
    "Statement": [
        {
            "Sid": "Stmt1623531541369",
            "Effect": "Deny",
            "Principal": {
                "AWS": "*"
            },
            "Action": "s3:ListBucket",
            "Resource": "arn:aws:s3:::YOUR_GLOBALLY_UNIQUE_NAME"
        }
    ]
}

Let’s test it

#IAM Admin bucket perspective
aws s3 ls s3://YOUR_GLOBALLY_UNIQUE_NAME
#Response: An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied

Remember, You can always simulate your calls in the IAM console like in this screenshot. It’s safer especially in production environments. This simulator takes into account already set identity-based, resource-based policies. That’s why I used an already created bucket and not a fake one.

Yet I think that testing it manually is much more fun even though it’s much more dangerous. Don’t you think?

If you wonder, You can brick your access to the bucket even for IAM Admin by attaching this policy.

WARNING: Don’t try this at home!

{
    "Version": "2012-10-17",
    "Id": "Policy1623526933543",
    "Statement": [
        {
            "Sid": "Stmt1623526931104",
            "Effect": "Deny",
            "Principal": {
                "AWS": "*"
            },
            "Action": "*",
            "Resource": "arn:aws:s3:::YOUR_GLOBALLY_UNIQUE_NAME"
        }
    ]
}

Then you would need to switch to the root account and remove this policy.

Final words

Policies can be quite tricky but they are also very powerful. I hope this post served as a useful introduction to some of the concepts. For more information be sure to check out the AWS Identity and Access Management documentation. It’s really good and It has helped me a lot many times. Have fun!

React State Management with Recoil

Jakub Abramowski — Thu, 01 Jul 2021 09:51:43 +0000

Deciding which state management library to use in a new React project can be quite a challenge – there are so many options. Obviously Redux is high on the list, however it requires a lot of boilerplate even in small apps that don’t require a lot of global state management. On the other hand, React Context is very simple, but any change to the value prop causes all the consumer’s children to rerender, even when only a small bit of the data is being used by them. You could use Mobx, but there’s a steep learning curve there, especially if you’re not quite familiar with observables.

That said, there is nothing wrong with any of these solutions, but they all have one downside – they are ‘external’ to React. Wouldn’t it be better to have something created especially for React?

The new kid on the block comes from the React team themselves – it’s called Recoil. The library is still in the experimental phase and has not been released in a stable version, but it’s a good time to start playing around with it, since it might prove the best choice for React developers in the near future.

So what is Recoil and what makes it so special? I will try to explain while creating a small shop that allows users to purchase some products.

Let’s start with the basic stuff – first we need to wrap our app (or just the part that needs access to the global state) in RecoilRoot (remember to import it from recoil, I won’t include all the imports here).

ReactDOM.render(
 <React.StrictMode>
   <RecoilRoot>
     <App />
   </RecoilRoot>
 </React.StrictMode>,
 document.getElementById("root")
);

That’s it, we’re all set to use Recoil in our code!

The smallest unit of state in Recoil is called an atom. It is a function that requires a unique key and a default value, and returns a writable piece of state.

const cartState = atom({
  key: "cartState",
  default: [],
});

To write/read from that atom, we need to use the useRecoilState hook.

const [cart, setCart] = useRecoilState(cartState);

Looks nice and simple, doesn’t it? And the best thing is, it has the exact same syntax as standard React hooks, which dramatically flattens the learning curve once you get familiar with some basic Recoil concepts.

Let’s stock our little shop with some products.

const products = [
 { name: "Toothbrush", price: 10 },
 { name: "Smart TV", price: 800 },
 { name: "Laptop", price: 600 },
 { name: "Chocolate", price: 12 },
 { name: "Apple juice", price: 5 },
];

Next, we’ll display them to the user so they can add them to their cart.

const ShopProducts = () => {
 const setCart = useSetRecoilState(cartState);

 const addToCart = (product) =>
   setCart((cart) =>
     cart.find((item) => item.name === product.name)
       ? cart
       : [...cart, product]
   );

 return (
   <>
     PRODUCTS:
     <div>
       {products.map((product) => (
         <p key={product.name}>
           {product.name} (${product.price})
           <button onClick={() => addToCart(product)}>Add</button>
         </p>
       ))}
     </div>
   </>
 );
};

In this component we do not need access to the cart, we only require a setter function to be able to add products to it. We can use the useSetRecoilState hook in the setCart function to achieve this.

Now all we need to do is render this component within our app.

const App = () => (
   <div>
     <ShopProducts />
   </div>
 );

Now we can see a list of products, each having a button next to it that says ‘Add to cart’. But what about the cart? We need a way of showing it’s content to the user. Let’s take care of that.

const Cart = () => {
 const [cart, setCart] = useRecoilState(cartState);

 const removeFromCart = (product) =>
   setCart((cart) => cart.filter((item) => item.name !== product.name));

 return (
   <>
     CART:
     <div>
       {!cart.length
         ? "Empty"
         : cart.map((product) => (
             <p key={product.name}>
               {product.name} (${product.price})
               <button onClick={() => removeFromCart(product)}>Remove</button>
             </p>
           ))}
     </div>
   </>
 );
};

And remember to include Cart in the app.


const App = () => (
   <div>
     <ShopProducts />
     <Cart />
   </div>
 );

Now we are able to see the cart’s content or the string ‘Empty’ it’s empty. Of course – I know, I know – we could use some fancy icon to represent an empty cart, and we could also allow users to buy multiple products of the same type, but hey it’s just a simple app to demonstrate how easy it is to manage the state with Recoil, so bear with me please.

Next thing we need is to display the total price. We can do that using a selector.


const cartDetailsState = selector({
 key: "cartDetailsState",
 get: ({ get }) => {
   const cart = get(cartState);

   const total = cart.reduce((prev, cur) => prev + cur.price, 0);

   return {
     total,
   };
 },
});

Simple as that! Selectors let you create derived state from atoms, but you can also pass props to them, or even access setter functions, but we’ll get there later.

The next step is to add a component that will read the cart details state.

const CartDetails = () => {
 const { total } = useRecoilValue(cartDetailsState);

 return <p>TOTAL: ${total}</p>;
};

const App = () => (
   <div>
     <ShopProducts />
     <Cart />
     <CartDetails />
   </div>
 );

Item Discounts

Let’s say we want to allow the shop manager to set discounts for all the products. We can create a simple select with a few options, prepare an atom for holding the shop’s configuration, which will include the discount, and then expand the CartDetails selector logic to include the new state:

const shopConfigState = atom({
 key: "shopState",
 default: {
   discount: 0,
 },
});

const cartDetailsState = selector({
 key: "cartDetailsState",
 get: ({ get }) => {
   const { discount } = get(shopConfigState);
   const cart = get(cartState);

   const total = cart.reduce((prev, cur) => prev + cur.price, 0);
   const discountAmount = total * discount;

   return {
     total,
     discountAmount,
   };
 },
});

const CartDetails = ({ total, discount, discountAmount, setCart }) => (
 <div>
   <p>
     DISCOUNT: {discount * 100}% (${discountAmount})
   </p>
   <p>TOTAL: {total}</p>
   <p>TOTAL AFTER DISCOUNT: {total - discountAmount}</p>
   <button onClick={() => setCart([])}>Clear cart</button>
 </div>
);

const ShopConfig = () => {
 const [shopState, setShopState] = useRecoilState(shopConfigState);

 const updateDiscount = ({ target: { value } }) => {
   setShopState({ ...shopState, discount: value });
 };

 return (
   <div>
     Discount:
     <select value={shopState.discount} onChange={updateDiscount}>
       <option value={0}>None</option>
       <option value={0.05}>5%</option>
       <option value={0.1}>10%</option>
       <option value={0.15}>15%</option>
     </select>
   </div>
 );
};

const App = () => (
   <div>
     <ShopConfig />
     <ShopProducts />
     <Cart />
     <CartDetails />
   </div>
 );

Now our little shopping app is complete! The code looks very simple compared to Redux, but it has even more out-of-the-box features which make it a great competitor for all the other state management tools, and the coolest one is that it supports asynchronicity! Just like that, no additional libraries are needed, no extra code needs to be added.

Asynchronous

Let’s change the way we provide the product list to the shop.

const getProductsFromDB = async () =>
 new Promise((resolve) => {
   setTimeout(() => resolve(products), 2500);
 });

const productsQuery = selector({
 key: "Products",
 get: async () => getProductsFromDB(),
});

getProductsFromDB is a function that waits 2.5 second (to simulate a network latency, or database response time) to simply resolve a promise with the product list that we’ve created at the beginning. The important part here is the selector with an async getter function – it simply calls the aforementioned function.

Recoil is built to work with React Suspense, and this is how they can be connected.

const ShopProducts = () => {
 const shopProducts = useRecoilValue(productsQuery);
 const setCart = useSetRecoilState(cartState);

 const addToCart = (product) =>
   setCart((cart) =>
     cart.find((item) => item.name === product.name)
       ? cart
       : [...cart, product]
   );

 return (
   <>
     PRODUCTS:
     <div>
       {shopProducts.map((product) => (
         <p key={product.name}>
           {product.name} (${product.price})
           <button onClick={() => addToCart(product)}>Add</button>
         </p>
       ))}
     </div>
   </>
 );
};

const App = () => (
   <div>
     <ShopConfig />
     <React.Suspense fallback={<div>Loading products...</div>}>
       <ShopProducts />
     </React.Suspense>
     <Cart />
     <CartDetails />
   </div>
 );

From now on, before the product list gets displayed on the screen, the user will see a nice fallback loader component for 2.5 seconds.

What about handling errors? Simply wrap the component in an error boundary.

But what if I don’t really want to use React Suspense? It’s still experimental and using it in production is still discouraged (well, so is Recoil).

There is a way of using async Recoil without Suspense.

const ShopProducts = () => {
 const setCart = useSetRecoilState(cartState);
 const shopProducts = useRecoilValueLoadable(productsQuery);

 const addToCart = (product) =>
   setCart((cart) =>
     cart.find((item) => item.name === product.name)
       ? cart
       : [...cart, product]
   );

 switch (shopProducts.state) {
   case "loading":
     return <div>Loading products...</div>;
   case "hasError":
     throw shopProducts.contents;
   case "hasValue":
     return (
       <>
         PRODUCTS:
         <div>
           {shopProducts.contents.map((product) => (
             <p key={product.name}>
               {product.name} (${product.price})
               <button onClick={() => addToCart(product)}>Add</button>
             </p>
           ))}
         </div>
       </>
     );
 }
};

When using useRecoilValueLoadable, we get access to an object containing the current state of an async selector’s promise and, when it’s resolved, either to contents or error. This works exactly the same as in case of using Suspense.

Summary

To sum up, there are numerous ways of dealing with global state in React apps. Some have a steep learning curve, others are inefficient and in most cases and dedicated to just some pieces of state that don’t change very often. Now that we have Recoil within our reach, we can write state logic in a very similar way to how we write UI logic. The result is that creating React apps is even simpler and even more fun!

If you’d like to experiment, the recoil-demo source is available on Github.

Happy coding!

Introduction to Progressive Web Applications – Part 2

Krzysztof Dębczyński — Tue, 08 Jun 2021 13:52:32 +0000

In the first part of the series, we covered the main techniques that allow us to create a basic Progressive Web Application. In this part we will introduce the Service Worker API to allow our web application to work offline.

Service Workers

To achieve our goal we need to learn a little more about Service Workers. In the first part, we created Service Worker Javascript File (sw.js) which was imported in index.html. To make our application installable, we added a single event listener.

self.addEventListener('fetch', (event) => {
  console.log('fetch');
});

The ‘fetch’ event is one of the key events that allow us to make our application work offline. In this post we will use this event to handle requests and responses in combination with the Cache API. Before we do that, we’ll look at some lifecycle events to get a fuller picture of Service Workers.

“Service workers essentially act as proxy servers that sit between web applications, the browser, and the network (when available).”

— Developer.mozilla.org

Service Workers are a type of web worker – they execute in a separate thread from the main browser thread. They:

Are Promise based
Are only available on secure origins served through TLS, using the HTTPS protocol (working locally from localhost origin is not subject to this requirement)
Depend on the Fetch and Cache APIs
Have access to the IndexedDB API

Service Workers sit inside the web browser and are set up to handle browser requests and server responses from the Internet when we are online, or from the cache when offline.

Service Workers are designed to:

Handle network requests and store content for offline use)
Handle push events

Life Cycle

At first glance, the Service Worker lifecycle appears quite complicated, but once you know how it works you can make use of it to its full potential.

source: Introduction to Service Worker

We can see the installation and activation phase but before that you need to register as a Service Worker. Note that only one version of your service worker is running at a time on registered scope.

<!DOCTYPE html>
<script>
  navigator.serviceWorker.register('/sw.js')
    .then(reg => console.log('SW registered!', reg))
    .catch(err => console.log('Registration failed', err));
</script></pre>

The diagram above shows the life cycle for newly registered Service Worker. There are some differences when updating a previously registered worker to a newer version.

Events

Service Workers have six basic events.

Install Event

After the Service Worker is registered (i.e. downloaded to the client), the “install” event is the first one that the Service Worker receives. This event is fired once per Service Worker upon registration.

In the install event listener usually you can cache your static assets on the client web browser to make your web application working offline. All the JavaScript, CSS, images and other assets can be stored by the browser for use when offline.

To register an “install” event listener:

self.addEventListener('install', (event) => {
  console.log('install');
  // forces the waiting service worker to become the active service worker.
  self.skipWaiting();
  // delay install by caching assets and open database
  event.waitUntil(cacheStaticAssets());
});

You can also see that the install event offers the waitUntil method. This method accepts a Promise and the Promise resolution (success or failure) will tell the Service Worker if the installation process is completed as expected. cacheStaticAssets is a function which returns a Promise. We will cover the body of this function in the Cache API section in this post.

Once successfully installed, the updated worker will wait until any existing worker is service no clients. skipWaiting functionality prevents the waiting, meaning the service worker activates as soon as it’s finished installing.

Activate Event

If there are no clients controlled by another Service Worker and if a current Service Worker is installed then the “activate” event triggers. Here you can do additional setup such as cleaning up old cache data.

“Activate” also exposes two additional functions:

event.waitUntil() – you can pass a Promise that will delay activation. Usually, when new cache policy was implemented then in this event you can do cleanup and remove the old cache
self.clients.claim() – allows an active service worker to set itself as the controller for all clients within its scope (without reloading).

self.addEventListener('activate', (event) => {
  self.clients.claim();
  event.waitUntil(caches.delete(CACHE_NAME).then(cacheStaticAssets));
});

Message Event

This event allows us to react to communication between our web application and Service Worker.

There are few ways of communication between application and the Service Worker:

In the post, we’re going to focus on the Clients API.

Application.js

// Listen to the response
navigator.serviceWorker.onmessage = (event) => {
  // event.data - contains data received from Service Worker
};

// Send message
navigator.serviceWorker.controller.postMessage({
  type: 'MY_EVENT',
  data: 'some-data'
});

ServiceWorker.js

// Listen for messages
self.addEventListener('message', (event) => {
  if (event.data && event.data.type === 'MY_EVENT') {
    // Select the clients we want to respond to
    self.clients.matchAll({
      includeUncontrolled: true
    }).then((clients) => {
      if (clients && clients.length) {
        // Send a message
        clients.forEach(client => client.postMessage({
          type: 'MY_EVENT',
          data: 'some-data'
        }))
      }
    });
  }
});

As you can see we have two-way communication here. We can use the postMessage function to pass an object with a type property which can be a message type identifier.

Working Offline

Now that we know how to register Service Workers, what their life cycle and events look like, let’s see how we can tie it all together to make our application work offline.

First, we need to learn more about the Cache API and the Service Worker fetch event.

Cache API

“The Cache interface provides a persistent storage mechanism for Request / Response object pairs that are cached in long lived memory.”

— developer.mozilla.org

The Cache API is exposed to window scope and workers. As it’s quite a simple API, you need to take care of housekeeping activities such as purging stale cache data. Let’s look look at an example:

ServiceWorker.js

const CACHE_NAME = 'cache-and-update';
const STATIC_ASSETS = [
  './',
  './index.html',
  './index.bundle.js',
  './assets/',
  './assets/my-logo-128.png',
  './manifest.json'
];

self.addEventListener('install', (event) => {
  // forces the waiting service worker to become the active service worker.
  self.skipWaiting();

  // delay install by caching assets and open database
  event.waitUntil(cacheStaticAssets());
});

self.addEventListener('activate', (event) => {
  // allows an active service worker to set itself as the controller for all clients within its scope.
  self.clients.claim();

  // remove old cache and then cache new static assets
  event.waitUntil(caches.delete(CACHE_NAME).then(cacheStaticAssets));
});

function cacheStaticAssets() {
  return caches.open(CACHE_NAME).then((cache) => cache.addAll(STATIC_ASSETS))
}

Fetch Event

Fetch event contains information about the fetch, including the request and how the receiver will treat the response. Let’s update our code and connect the Cache API with the Service Worker fetch event.

ServiceWorker.js


const CACHE_NAME = 'cache-and-update';

self.addEventListener('fetch', (event) => {
  // respond from cache first
  event.respondWith((async function() {
    // fallback for navigate requests
    if (event.request.mode === 'navigate') {
      return getCachedIndex();
    }

    const response = await fetchFromNetworkFirst(event.request);

    return response;
  })());
});

async function fetchFromNetworkFirst(request) {
  try {
    const response =  await fromNetwork(request);

    await updateCache(request, response.clone());

    return response;
  } catch(e) {
    const responseFromCache = await fromCache(request);

    if (responseFromCache) {
      return responseFromCache;
    } else {
      throw e;
    }
  }
}

function getCachedIndex() {
  return caches.open(CACHE_NAME).then((cache) => cache.match('index.html'));
}

function fromCache(request) {
  return caches.open(CACHE_NAME).then((cache) => cache.match(request));
}

function fromNetwork(request) {
  return fetch(request);
}

function updateCache(request, response) {
  return caches.open(CACHE_NAME).then((cache) => cache.put(request, response));
}

As you can see we are using the event.respondWith method that prevents the browser’s default fetch handling, and allows you to provide a Promise for a Response yourself. In our implementation we are trying to fetch data from the network first and when the network is not available then trying to get the response from cache. Notice that when the fetch request is successful, we update our cache with the data from the response.

This implementation is one of the approaches to make the application work even if the user doesn’t have an Internet connection. But of course, this is not the perfect solution. In the case where a user needs data that’s not in cache they will not see on a screen what they need on screen. Fortunately, Service workers offer the Web Background Synchronization and Web Periodic Background Synchronization APIs that can help us solve this problem. Note that these APIs are still in draft and may not work in all browsers and devices.

Web Background Synchronization

As the name says, this API enables web applications to synchronize data in the background.

Key facts:

Enables client and server data synchronisation of, for example, photo uploads, document changes, or draft emails
Allows the service worker to defer work until the user has connectivity
Requires the service worker to be alive for the duration of the fetch
Suited to short tasks like sending a message

On the web application side, first we need to wait for the Service Worker registration and then we can use sync.register function in the following way.

Application.js

const registration = await navigator.serviceWorker.ready; 

registration.sync.register('my-tag');

On a Service Worker side we react to a sync event as follows:

ServiceWorker.js

self.addEventListener('sync', event => {
  if (event.tag == 'my-tag') {
    event.waitUntil(doSomeWork())
  }
})

As you can see we are only allowed to pass a string parameter which is called “tag” and is a kind of identifier of the sync registration.

Unlike the fetch event, here on the Service Worker side, we don’t have access to the request, so we cannot use event.request and use it to handle background synchronisation as we did with the Cache API.

We can experiment a little with this API and try to use “tag” to pass information about the request that we want to send. Because we can pass only a “tag” which is a string, let’s stringify the configuration object and pass it as a tag.

First on the web application, we will handle cases when users don’t have access to the Internet. Let’s create two files in our web application side called Application.js and requestWithBackgroundSync.js.

Application.js

import requestWithBackgroundSync from "utils/requestWithBackgroundSync";

const someApi = {
    getItems: () => requestWithBackgroundSync("https://domain.name/api")
        .then(response => response.json())
};

// make request
someApi.getItems();

Here we are just calling the server using requestWithBackgroundSync function. This function returns a Promise, and then we can parse the response (similar as when using fetch API). Let’s implement the requestWithBackgroundSync function.

requestWithBackgroundSync.js

function requestWithBackgroundSync(url, config) {
  return fetch(url, config)
    .catch(() => backgroundSyncRequest(url, config));
}

export default requestWithBackgroundSync;</pre>

We attempt to get data using the fetch function and if the fetch fails (for example because of the network connection issues) then we will catch it and return a Promise implemented inside the backgroundSyncRequest function. Now we will implement this function using sync functionality if the Service Worker registration.

requestWithBackgroundSync.js

import uuidv4 from "uuid/v4";

async function backgroundSyncRequest(url, config) {
  // data that are passed to sync event
  const jsonTag = createFetchSyncDataObj(url, config);

  await registerBackgroundSync(JSON.stringify(jsonTag));

  // background sync data receive experiment
  const { data, headers } = await getDataFromBackgroundSyncByJsonTag(jsonTag);

  return prepareResponse(data, headers);
}

function createFetchSyncDataObj(url, config) {
  // method name used to extract data from body by service worker
  // TODO: detect method name by "Content-Type" header
  const bodyExtractMethodName = 'json';

  return {
    type: 'fetch-sync',
    requestId: uuidv4(),
    url,
    config,
    bodyExtractMethodName,
    link: document.location.href
  };
}

async function registerBackgroundSync(tag) {
  const registration = await navigator.serviceWorker.ready;

  registration.sync.register(tag);
}

function getDataFromBackgroundSyncByJsonTag(jsonTag) {
  // TODO: add timeout and remove event listener after timeout
  return new Promise(resolve => {
    const handler = createFetchSyncMessageListener(jsonTag, onDone);

    function onDone(data) {
      navigator.serviceWorker.removeEventListener('message', handler);
      resolve(data);
    }

    navigator.serviceWorker.addEventListener('message', handler);
  });
}

function createFetchSyncMessageListener(jsonTag, done) {
  function handler(event) {
    const receivedJsonTag = parseJson(event.data.jsonTag);

    if (receivedJsonTag) {
      const isFetchSyncMessage = receivedJsonTag.type === 'fetch-sync';
      const isTheSameRequestId = jsonTag.requestId = receivedJsonTag.requestId;

      if (isFetchSyncMessage && isTheSameRequestId) {
        done(event.data);
      }
    }
  }

  return handler;
}

function prepareResponse(data, headers) {
  // TODO: build blob based on "Content-Type" header (for now JSON is created)
  const blob = new Blob([JSON.stringify(data)]);
  const response = new Response(blob, { headers });

  return response;
}

function parseJson(str) {
  try {
    return JSON.parse(str);
  } catch(e) {
    return undefined;
  }
}

We are using “tag” (which is a string) to pass to Service Worker information about the request that we want to make.

Note that the object that we want to send to the Service Worker contains among others a requestId which will serve us for identifying if the response that we will get from the Service Worker will match the request that we want to make. We do this because we cannot use event.respondWith function and return a response. Of course, this approach has some limitations. We are losing all other information that the Request object has out of the box like for example the credentials, cache, mode or the methods implemented inside the Request object.

To get the data back from the Service Worker we are using a message event. As you probably noticed, we are always expecting to have a JSON response so that, after getting the data back, we are also preparing a JSON response. In the future, if this experiment goes well, we could extend this solution to support more response formats.

Let’s check now the code inside Service Worker.

ServiceWorker.js


self.addEventListener('sync', (event) => {
  const receivedJsonTag = parseJson(event.tag);

  if (receivedJsonTag && receivedJsonTag.type === 'fetch-sync') {
    const { url, bodyExtractMethodName, config } = receivedJsonTag;

    event.waitUntil(
      (async function () {
        try {
          const response = await fetch(url, config);

          const headers = {};
          response.headers.forEach((val, key) => {
            headers[key] = val;
          })

          await updateCache(url, response.clone());

          // extract data from body by received method name
          const data = await extractDataFromResponse(response, bodyExtractMethodName);

          self.registration.showNotification(`Background sync finished with success`, { data: { link: receivedJsonTag.link } });

          return sendMessageToAllClients({ jsonTag: event.tag, data, headers });
        } catch(e) {
          if (event.lastChance) {
            self.registration.showNotification(`Can't get ${url}`);
          }
          throw e;
        }
      })()
    );
  }
});

function parseJson(str) {
  try {
    return JSON.parse(str);
  } catch(e) {
    return undefined;
  }
}

function updateCache(request, response) {
  return caches.open(CACHE_NAME).then((cache) => cache.put(request, response));
}

async function extractDataFromResponse(response, methodName) {
  if (BODY_EXTRACT_METHOD_NAMES.includes(methodName)) {
    return response[methodName]();
  }

  throw new Error(`Can't extract data from response body by method ${methodName}`);
}

function sendMessageToAllClients(msg) {
  return clients.matchAll()
    .then(clients => {
      clients.forEach(client => client.postMessage(msg))
    });
}

We registered a sync event handler, parsed stringified JSON and then used event.waitUntil function. This function accepts a Promise and will try to execute this Promise until it is resolved with success or reached the limit. In this Promise we are making the request and if it is made successfully then we are putting the response to cache and then sending response to clients by posting a message.

Take a note that sync event has the lastChance property. This property tells us if this was the last attempt of making our request. After that sync event will fail.

Summary

This is the end of our journey with Service Workers and related APIs. We learned how to make our application offline using them but you need to know that this is not the only way to do it. Everything depends on your needs and requirements. The Service Worker API is quite new and some of its features are still in draft, so can behave differently on different platforms.

At the end I will leave you some links that can be helpful in further exploring progressive web apps:

Constraint Programming: Solving Sudoku with Choco Solver library

Daniel Bray — Tue, 27 Apr 2021 09:51:20 +0000

Why solve sudoku?

Enterprise application development is, for the most part, solving one of these types of problems:

“Let me create, read, update and delete these things”
“Do this same task to as many things as possible, as quickly as possible”
“What’s the best way to allocate the resources I have to do these tasks, if whatever does task X, can’t be used to do task Y?”

This last problem type is a graph colouring problem, and the nature of these is that solving one of them is much the same as solving another.

Sudoku is one of these types of problems, but it has very simple rules, so it’s a nice playground to try out different ways to solve graph colouring problems. This post outlines a solution using constraint programming with choco solver.

What is constraint programming?

Constraint programming is a paradigm for programming that can be a little unusual the first time you come to it, since it’s completely different to imperative programming.

In short: you tell the program what problem needs to be solved, but not how to solve that problem.

What this means in practical terms is:

First you define your variables.
- These are my tasks
- These are my workers
Then you define the domains in which these variables exist. For example:
- This variable has to have the value of 3
- This variable can have any value between 1 and 42
- This variable is a set of between 4 and 10 numbers, that are all taken from a domain running from 1 to 99.
Then you define the constraints for these variables, for example:
- If one variable has a value, the other variable must have a different value.
- One variable must be the sum/max/min of a few other variables.
- One set must be a compliment of another set.
If you’re looking for any solution, then you’re done. If you’re looking for the best solution, then you need to define a set of cost variables that you aim to minimise or maximise. For example:
- Find a solution that minimizes the “cost of doing business” variable.
- Find a solution that maximises the “how many messages are being transmitted” variable.
Finally, you give this to the framework to solve, and it will use different AI approaches to find solutions to the problem you’ve defined.

Solving Sudoku in Choco Solver

So, to make these ideas more concrete, we’ll use them to solve a simple problem.

For this example, we will choose the world’s hardest sudoku problem, you can find a full example of this in Sudoku.java

If you don’t know Sudoku, the rules are:

The grid is a 9 X 9 area of squares.
Each square must contain a single number, from 1 to 9.
The same number can’t appear in the same row twice.
The same number can’t appear in the same column twice.
The grid is broken down into 9 distinct sub-grids of 9 squares each. The same number can’t appear in the same sub-grid twice.

That’s it. If you’re wondering what this has to do with resource usage, you could imagine the numbers represent available channels in a cell tower, and the squares represent the messages that need to be sent. The solution to this problem will tell you how you could send out these messages in an evenly distributed manner, without getting any resource contention on the channels.

Getting started with Choco Solver

The complete code for this is available here: Sudoku

Before we do anything else, we must first create an empty model for the game, this is as simple as:

Model model = new Model("sudoku");

This model is where new variables, constraints and optimizations are created.

Defining variables and domains

For the sudoku problem, there are 81 variables, one for each square in our grid.

IntVar[][] grid = new IntVar[9][9];

These come in one of two flavours:

If the value is not known at the start, then we need to define it with a domain of possible values, in our case, they are values from 1 to 9

grid[row][col] = model.intVar(row, col), 1, 9);

If the value is known at the start, then we define it as a simple constant that can’t change. This is, in effect, saying that the variable has a domain of a single value.

grid[row][col] = model.intVar(value)

Define constraints

Once we have the variables, we need to set up their constraints, we have 9 rows, columns, and sub-squares where the values must all be different. We use the allDifferent constraint for this.

for (int i = 0; i != 9; i++) {
  model.allDifferent(getCellsInRow(grid, i)).post();
  model.allDifferent(getCellsInColumn(grid, i)).post();
  model.allDifferent(getCellsInSquare(grid, i)).post();
}

Solve it

Finally, we solve it as follows.

Solver solver = model.getSolver();
solver.solve();

Choco solver will keep looking for solutions until it gives up, because it’s used them all, or because it thinks it won’t find anything better, or because you’ve told it to give up after doing enough work.

Once this is done you can get at the value that choco solver found for each cell as follows

grid[row][column].getValue()

And that’s it, the world’s hardest sudoku is solved in under a second.

So what else can you do with choco solver?

Whatever you want it to, so long as you can turn it into a combinatorics constraint problem.

For example, here’s a more complex graph colouring program: GraphColouring.java

What it’s doing is:

Given a graph: imagine these are tasks that can’t be done at the same time.
Given you have N possible colours to choose from: imagine these are workers that are available to do these tasks.
Given that no colour can be used more the M times: imagine a worker can only do so many things in a day.
Given you want to use the least number of colours: use the fewest workers required to do this work.
Colour the graph in: suggest a roster for the tasks and workers that uses the least number of people without overworking them.

Where do we go now?

This post refers to two simple examples of what is possible, but really most combinatorial problems can be solved using this approach.

For a more formal and complete description of constraint programming, check out Explaining Constraint Programming
For a nice explanation of how to formalise actual problems, check out lectures 7, 8 and 9 of this AI lecture series from MIT (actually, all of this series is worth watching)

Code Sharing with React Native & Webpack

Łukasz Nawrocki — Mon, 29 Mar 2021 13:10:17 +0000

Time is a crucial factor in modern software development – clients need features delivered in a flash. Time – we could always do with more of it. As software engineers, though, we are often faced with such situations and have developed strategies to better use time. We work to reduce the amount we spend on unnecessary tasks. On repetitive work. On duplication of effort.

One target of such efforts is code duplication – something of particular relevance for multi-platform applications.

Code sharing is a technique that enables applications to reuse the same code across desktop, web, and mobile platforms. When done successfully, an application does not sacrifice its native look and feel on each platform. As it’s a large topic, I will limit the discussion in this post to sharing a single component across multiple platforms. Also, in reality, it’s a more likely scenario – one where we want to add a shareable component to a long-running project with an established codebase.

A conservative approach to code sharing would be to share business logic, for example, a set of JavaScript functions in a common Node module, and develop the UI of the application separately. However, this approach would not fulfil our primary objective. We would still need to implement the UI from scratch, using two different technologies, making one developer create the same feature twice, or even worse – engaging two developers for the same job.

Instead, I’ll go with a strategy based on the React Native framework that has more code sharing potential. It enables developers to create mobile applications in JavaScript and deliver them as native applications with a native look and feel.

But the best feature of React Native is yet to be fully utilised in the programming world. It is the package called React Native Web that allows web developers to use React Native components and run on the web using React DOM. React Native Web is widely used in the React Native community and makes it easy to reuse components used to create mobile React Native applications in ReactJS web apps. Let’s create an example component and see what it takes to “learn once, write everywhere”.

Project Structure

In this post we’ll only be talking about a single component. In the real world, it would likely be part of a larger component library. We’ll be using TypeScript, but there’s nothing preventing you from using JavaScript should you wish – obviously you’d need to change the configuration and webpack loaders though. Apart from including typings and creating a need for tsconfig.json file, this assumption also matters when it comes to choosing appropriate webpack loaders.

Let’s create the main structure of our project:

-src/
-web/
-package.json
-tsconfig.json
-webpack.config.js

The UI and login of our shareable component will sit in the src directory. We create a simple round button which will accept onPress callback and imageSource as properties:

src/components/Button.tsx:

import React from "react";
import { TouchableOpacity, StyleSheet, ImageSourcePropType } from "react-native";

Interface ButtonProps {
  onPress: () => void
  imageSource: ImageSourcePropType
}

export default function Button({ onPress, imageSource }: ButtonProps) {
  return (
    <TouchableOpacity style={styles.button} onPress={onPress}>
      <Image
        style={styles.image}
        source={imageSource}
      />
    </TouchableOpacity>
  );
}

const styles = StyleSheet.create({
  container: {
    width: 56,
    height: 56,
    borderRadius: 28,
    justifyContent: "center",
    alignItems: "center",
    backgroundColor: "#fff",
  },
  image: {
    width: 28,
    height: 28,
  },
});

We will keep all the components in the src directory. This way, every time someone installs the package, the code will appear in node_modules of the mobile React Native app. The other thing we could do would be to build it into a separate lib directory, for example with the tsc command, but we’re not going to do that here today.

The other important directory in the project tree is web. This is a place where we will keep the JavaScript component bundle. This is important because we can’t rely on React Native and React Native Web being installed in the React app, so in order to use our component in a web application we should have a pure React version of it prepared. How to make this bundle is discussed in the next section – for now it is important to note we will have a separate directory for the web.

We should also have a way to differentiate between the web, bundled version and React Native version of the component when importing from our library. We will do it by differentiating import paths. For web applications, we will be using /web directory to import, as presented below. For the React Native we will use the root directory represented by the name of the library. We should not have one entry point for it, because that will bring a risk of importing the bundle in React Native or importing React Native version in the React app. This is why inside the web directory we will add a new package.json file:

web/package.json:

{
  "main": "build/bundle.web.js",
  "types": "build/src/index.d.ts"
}

bundle.web.js file is the file with React bundle and the entry point to our components’ library from within the web application. Because we have put package.json inside the web directory, if we want to use the Button component in the React web application we will import it like:

/**
* importing from build/bundle.web.js under the hood
*/

import { Button } from 'our-components-library/web'

While in React Native app it will simply be:

import { Button } from 'our-components-library'

Except from src and web directories, in the project root we also defined three files:

package.json as the main configuration file of the whole library, containing dependencies, scripts, entry point for React Native etc.
tsconfig.json with definitions of typescript rules we use in the app
webpack.config.js with settings for bundling the code for web usage.

We will now take a closer look at the last file as bundling for the web is one of the focal points of the whole process of creating sharable components.

Bundling React Native for web

To bundle the code I will use the most common solution by far – Webpack. In order to use it we first need to install a couple of modules. Depending on preference you could use Yarn or NPM, I will use Yarn in the examples. We install to dev dependencies using command:

yarn add --dev webpack webpack-cli

Because we have installed the webpack-cli package, we can now use the webpack command to bundle our code. Let’s do it and also let’s add a script to package.json and name it accordingly so other developers can easily identify the purpose of it.

package.json:

"scripts": {
   "build-react": "webpack"
 }

Now, the most important part of the bundling process – we need to define configuration file, so the Webpack knows how to translate our React Native code into a React bundle. Let’s first take a look at the whole config and go through it section-by-section later.

webpack.config.js:

const path = require("path");

module.exports = {
 entry: path.join(__dirname, "src/index.ts"),
 output: {
   filename: "bundle.web.js",
   path: path.join(__dirname, "/web/build"),
   library: "our-components-library",
   libraryTarget: "umd",
   umdNamedDefine: true,
 },
 module: {
   rules: [
     {
       test: /\.(ts|tsx)?$/,
       include: path.resolve(__dirname, "src"),
       exclude: /node_modules/,
       use: [
         {
           loader: "babel-loader",
           options: {
             cacheDirectory: true,
             presets: ["module:metro-react-native-babel-preset"],
             plugins: ["react-native-web"],
           },
         },
         {
           loader: "awesome-typescript-loader",
           options: {
             configFileName: "tsconfig.json",
           },
         },
       ],
     },
     {
       test: /\.(gif|jpe?g|png|svg)$/,
       use: {
         loader: "url-loader",
         options: {
           name: "[name].[ext]",
           esModule: false,
         },
       },
     },
   ],
 },
 resolve: {
   extensions: [".ts", ".tsx", ".js", ".jsx"],
   alias: {
     "react-native$": "react-native-web",
   },
 },
 externals: {
   react: {
     commonjs: "react",
     commonjs2: "react",
     amd: "React",
     root: "React",
   },
   "react-dom": {
     commonjs: "react-dom",
     commonjs2: "react-dom",
     amd: "ReactDOM",
     root: "ReactDOM",
   },
 },
};

First thing we need to define for Webpack is the input file. This is where we export all the components in our library, for example:

src/index.ts:

import Button from "./components/Button";

export { Button };

In the next section, we define what should be the result of the bundling process:

output: {
   filename: "bundle.web.js",
   path: path.join(__dirname, "/web/build"),
   library: "our-components-library",
   libraryTarget: "umd",
   umdNamedDefine: true,
 },

Filename and path are self–explanatory, but it’s worth remembering that the path to the final bundle and its name must be the same as what’s defined in web/package.json, otherwise we won’t be able to import components of the library correctly as they won’t be found. Setting library name and libraryTarget to “umd” and umdNamedDefine allows us to import components from the library individually.

In the module section one needs to define all the loaders used to build particular file types. While awesome-typescript-loader and url-loader probably sound familiar as they are often used with React applications, I’d like to focus on configuration for babel-loader. Two settings are worth mentioning here – we need to use metro-react-native-babel-presets to obtain Babel presets for our React Native components, and we also add react-native-web Babel plugin. The plugin basically rewrites paths to import only the modules needed by the app. That allows us to vastly reduce the size of the result bundle.

Size concern is also the purpose for externals part of the config:

react: {
     commonjs: "react",
     commonjs2: "react",
     amd: "React",
     root: "React",
   },
   "react-dom": {
     commonjs: "react-dom",
     commonjs2: "react-dom",
     amd: "ReactDOM",
     root: "ReactDOM",
   },
 },

Thanks to this configuration, libraries mentioned are not bundled into the final file. Without it, we could have the whole React framework bundled and passed between packages. Because React and ReactDOM have to be installed in the parent project of our components library anyway, it wouldn’t make any sense to have it in the bundle.

Finally, we instruct webpack to use React Native Web library whenever we import something from react-native. This is obtained with alias configuration:

resolve: {
   extensions: [".ts", ".tsx", ".js", ".jsx"],
   alias: {
     "react-native$": "react-native-web",
   },
 },

Full list of React Native modules supported by React Native Web is available in it’s documentation.

After running the build-react script, we obtain a nicely compressed bundle of about 80kB. Thanks to web/package.json configuration, it is enough to import from the bundle like described before:

import { Button } from 'our-components-library/web'

Summary

Module bundlers allow us to develop React Native Web and React Native component libraries that can be shared across web and native mobile applications. In the example presented, we used the webpack bundler, but could have used an alternative such as rollup.

It’s also worth noting the way presented above is still a bit complex because it requires a lot of manual setup. A vast effort has been recently put into development of React Native and React Native Web libraries, together with an awesome React Native dev environment called Expo, which makes building React Native apps even simpler. Taking that into account, we can expect even more examples of cross-platform development in the upcoming years.

People Skills & Collaboration in the World of Design

Daragh Martin — Tue, 02 Mar 2021 10:41:56 +0000

Back in November, I attended DataVizLive, Europe’s top conference focusing on the intersection between people, data and technology, as well as the mechanics, the presentation and leadership in the field. It brings together both leaders and passionate contributors in Data Visualisation, all of whom looking to share their passion for creating, delivering and exploring the future of data!

One standout for me was Steve Hulmes’ talk on the importance of people-skills in data-visualisation. One of Steve’s main focus areas is the art of collaboration, to help improve productivity and to enable designers to make a greater impact when working with data!

My attempts to learn more about the topic revealed a scarcity of material to tap into. As designers, we learn and develop working processes, ones that are discussed in the broader community, whether it’s the latest tools or ways to improve one’s efficiency. However, I have seldom heard about how honing people skills and building relationships can improve everything from productivity to job satisfaction!

Steve’s talk motivated me to learn more about the importance of these skills in the world of design, and to reflect on how I am engaging with the stakeholders I work with.

So, without wasting any more time, let’s jump into it.

Why Soft-skills People skills?

Hard skills are the abilities or technical skills required to do a particular job. For example, a designer might need to be proficient in Figma; a developer might need to know Kotlin. People skills are traits such as communication, flexibility and the ability to collaborate with others. These are difficult things to measure.

People skills are often referred to as soft skills, implying that they are less important than hard skills. In many ways soft skills are more difficult to learn than hard skills as they are innate and less measurable.

People-skills refer to the personalities, attributes, qualities and personal behaviour of individuals. Soft skills include abilities such as; communication, problem-solving, self-motivation, decision-making, and time management skills (Majid et al., 2012, p. 1036).

Designers must have hard-skills if they want to be able to complete projects from start to finish. However, designers must also develop their people skills if they want to be able to become well-rounded and deliver end-products that are truly user-orientated.

“To be a great designer, you have to master the soft skills. A huge part of the job will involve stepping away from the design software of choice and dealing with people.”

– Co-founder of Charming Robot & Hard Candy Shell. Dan Maccarone

Research suggests that building on people-skills can lead to better job success. Moreover, research around the importance of these skills for education and career success suggests that appropriate people skills can play an important role in a successful career and that these particular skills are highly sought after by employers, specifically when recruiting graduates.

Other studies have investigated the perceptions of business employers on the importance of having different skills. It has been reported that when asked to pick their most favourable from a wide array of skills, up to 80% of those selected were people-skills. These included the ability and willingness to learn, collaboration, interpersonal communication, energy and passion, and problem-solving skills.

Furthermore, research suggests that people skills are identified by a wide array of employers as the major competency in nearly every industry, even in the technical environments that place a premium on hard skills.

So let’s look into some of these skills and why they’re crucial within the design world.

Communication

“Design, unlike art, is not just a representation of the designer’s own self or personal ideas. One cannot just get to work as soon as a brief is provided and then submit the design files once the work is done without any communication in between these phases.”

Co-Founder of Semly Pro

Surya Ravindran Pillai

From the get-go, designers need to be vocal and ask many questions, even if some aren’t met with patience or understanding. I have seen this happen when stakeholders are asked to elaborate frequently.

Working with others in different roles (using other technologies and ways of communication) is an excellent opportunity to develop your communication skills.

Personally, working with developers, business stakeholders and data scientists, I have learned that each individual has their way of communicating. Their mode of communication fits into their role or domain. These are ways of speaking, abbreviations and even humour.

Each person you work with is a reservoir of knowledge, asking questions and allowing them to provide insight is beneficial to your overall productivity and final product. By only reading from the design brief you will miss out.

Collaboration

Put very simply, collaboration is the action of working with someone! Better collaboration can help designs reach their potential and address the requirements of the users.

Steve’s talk stresses that we want to be working with stakeholders and not just for them. Designers and analysts alike are often left frustrated when they find themselves simply working for stakeholders. Symptoms of this may be; requests made last minute and often finding yourself kept out of the loop! It’s important to recognise when this becomes a standard thing.

To both draw attention to and counter this, Steve described two analytical service models; the Passive and the Collaborative.

The ‘Passive model’ of working is when a stakeholder takes over the first three stages and then hands over to the analyst or designer. This is often the ‘can you do this by Friday’ model, as the stakeholder is very busy. Rework is very common with this model due to the lack of engagement and often leads to missed opportunities in areas such as; relationship growth, user research and overall product development.

On the other hand, we have the Collaborative Model. This process has the designer in the loop from the start, which often leads to designs and features being built right the first time, more often than not (saving time and resources). It also embraces better engagement between stakeholders and designers. When there is a strong relationship and better communication there is more opportunity for learning and growth on both ends. This leads to a better understanding of the problem everyone is attempting to solve, and this leads to better idea creation and implementation of the designs.

If you find yourself working like the passive model there are some things that you can do to help break free.

Get into the Right Mindset Remind yourself that you are here to help the stakeholders. Act as the navigator and not the driver. The knowledge and experience a stakeholder has are instrumental to the development of the designs, make sure to get feedback regimentally and ask open-questions.
Build Credibility and Trust If stakeholders find it difficult to trust you as a designer, then they may feel that they must be prescriptive with requirements (e.g. Use a drop-down menu here). If this is the case, it’s important to demonstrate your credibility and while continuing to build trust. By working competently, showing your capabilities, while delivering work that is timely and accurate can help build credibility and trust. It’s important to consider that stakeholders may need guidance through this process, especially if it’s their first time working with a designer.

Looking for ways to improve your people skills?

Speak, Talk, Communicate

Use every opportunity you can to think about how you communicate. When talking to others, think about how you are addressing them and how clear your message is, think about your tone and body language (this may be equally important in zoom meetings, i.e. just because you’re sitting in your living room doesn’t mean we can’t see you roll your eyes!)

Don’t shy away from Teamwork

Often in design, you may think it’s easier to solve problems on your own? This may be true in some cases, but working in teams opens up new avenues of learning and helps improve your ability to work with others!

Embrace Feedback

“The most basic act of reaching out and letting others critique your design is a sign of a very mature designer”

Vasudha Mamtani

Experience Designer at Amazon

Some designers may spend lots of time working on solo projects, and this can result in a lack of collaboration skill development. By inviting peers to critique your designs allows you to get a fresh perspective on things and avoid getting fixated on just one idea!

Step Out of Your Comfort Zone

Don’t like public speaking? Try doing an internal talk on a topic for colleagues. This will help build confidence in the areas you feel unfamiliar with and help you improve, or if that’s a step too far, try rehearsing with a few close colleagues and friends first!

Learn From Others

Pay attention to more experienced people around you, whether they be teammates or colleagues in a different role! I often watch how those from the ‘People Team’ communicate and impart information to the wider community. Try watching how others navigate through difficult conversations, this can be a great way to hone your own skills, for when they are put to the test.

Be Flexible

Whether you love them or not, some ideas have to be let go, and you need to be able to focus on new ones! A new feature request or a change in scope may challenge your current design, so it’s important to be open to going in a different direction.

That’s all from me, I’d like to hear about your experience around honing your people skills, and how it’s benefited your stakeholder relationships, while hopefully building your confidence and allowing you to grow as a designer. As always, feel free to give feedback on the post and if you have any questions, please don’t hesitate to ask!

Part 4: Hypothesis Testing of frequency-based samples

Daniel Bray — Tue, 16 Feb 2021 15:59:19 +0000

In part one of this series, we introduced the idea of hypothesis testing, along with a full description of the different elements that go into using these tools. It ended with a cheat-sheet to help you choose which test to use based on the kind of data you’re testing.

Part two outlined some code samples for how to perform z-tests on proportion-based samples.

Part three outlined some code samples for how to perform t-tests on mean-based samples.

This post will now go into more detail for frequency-based samples.

If any of these terms – Null Hypothesis, Alternative Hypothesis, p-value – are new to you, then I’d suggest reviewing the first part of this series before carrying on with this one..

What is a frequency-based sample?

In these cases we’re interested in checking frequencies, e.g. I’m expecting my result set to have a given distribution: does it?

Are differences between the distributions of two samples big enough that we should notice it? Are the distributions between variables in a single sample enough to indicate that the variables might depend on each other?

Requirements for the quality of the sample

For these tests the following sampling rules are required:

Random	The sample must be a random sample from the entire population
Normal	The sample must be normal, for these tests either: The underlying population must be normal – this can be tricky – as a population might normally be normal, only to be non-normal the day you sample it 😉 If you can’t assume the underlying population is normal then you should use a sample size of at least 30 (as per the central limit theorem)
Independent	The sample must be independent – for these tests a good rule of thumb is that the sample size be less than 10% of the total population.

Tests for mean-based samples

All of these code samples are available in this git repository

Chi-squared quality-of-fit

Compare the counts for some variables in a sample to an expected distribution

In this test we have an expected distribution of data across a category, and we want to check if the sample matches that.

For example, suppose a network was sized to have the expected distribution, and a sample observed the following counts

Class of Service	Expected Distribution	Observed Count in sample (size 650)
A	5%	27
B	10%	73
C	15%	82
D	70%	468

Given a null hypothesis that the distribution is as expected, then the following python code would derive the probability that the sample fits into this expected distribution.

from scipy.stats import chisquare

# can we assume anything from our sample
significance = 0.05

# what do we expect to see in proportions?
expected_proportions = [.05, .1, .15, .7]

# what counts did we see in our sample?
observed_counts = [27, 73, 82, 468]

########################
# how big was our sample
sample_size = sum(observed_counts)

# we derive our comparison counts here for  our expected proportions, based on the sample size
expected_counts = [float(sample_size) * x for x in expected_proportions]

# Get the stat data
(chi_stat, p_value) = chisquare(observed_counts, expected_counts)

# report
print('chi_stat: %0.5f, p_value: %0.5f' % (chi_stat, p_value))

if p_value > significance:
   print("Fail to reject the null hypothesis - we have nothing else to say")
else:
   print("Reject the null hypothesis - suggest the alternative hypothesis is true")

Chi-squared (homogeneity)

Compare the counts for some variables between two samples

In this case, the test is similar to the best fit (above) but rather than estimate the expected counts from the expected distribution, the test is comparing two sets of sampled counts to see if their frequencies are different enough to suggest that the underlying populations have different distributions.

This is, in effect, the same code as above – only in this case we have actual expected values to match, rather than having to estimate them from the sample.


from scipy.stats import chisquare

# can we assume anything from our sample
significance = 0.05

# what counts did we see in our samples?
observed_counts_A = [32, 65, 97, 450]
observed_counts_B = [27, 73, 82, 468]

########################

# Get the stat data
(chi_stat, p_value) = chisquare(observed_counts_A, observed_counts_B)

# report
print('chi_stat: %0.5f, p_value: %0.5f' % (chi_stat, p_value))

if p_value > significance:
   print("Fail to reject the null hypothesis - we have nothing else to say")
else:
   print("Reject the null hypothesis - suggest the alternative hypothesis is true")

Chi-squared (independence)

Check single sample to see if the discrete variables are independent

In this case you have a sample from a population, over two discrete variables, and you want to tell if these two discrete variables have some kind of relationship – or if they are independent.

NOTE: this is for discrete variables (i.e. categories). If you wanted to check if numeric variables are independent you’d want to consider using something like a linear regression.

Suppose we had a pivot to see how people from different area types (town/country) voted for three different political parties.

The question we are asking is whether or not we can say whether or not there is likely to be a connection between these two variables (i.e. do town/country people have a strong preference to vote for a given party).

	Party
	Cocktail Party	Garden Party	Mouse Party
Voter Type
Town	200	150	50
Country	250	300	50

The python code to check this is:

from scipy.stats import chi2_contingency
import numpy as np

# can we assume anything from our sample
significance = 0.05

pivot = np.array([
  # town votes
  [200,150,50],
  # country votes
  [250,300,50]
])
########################
# Get the stat data
(chi_stat, p_value, degrees_of_freedom, expected) = chi2_contingency(pivot)

# report
print('chi_stat: %0.5f, p_value: %0.5f' % (chi_stat, p_value))

if p_value > significance:
  print("Fail to reject the null hypothesis - we have nothing else to say")
else:
  print("Reject the null hypothesis - suggest the alternative hypothesis is true")

Where do we go next?

Thank you for reading the final part of our introduction into hypothesis testing. I hope you found it a useful introduction into the world of statistical analysis. If you would like to look deeper into this field, I’d suggest the following.

I’ve not touched on issues of power or effect size in this series. For that I would direct you to Robert Coe’s always worth reading: It’s the effect size, stupid: what effect size is and why it is important
If you have more complex types of data to examine, then I’d suggest reading more into
- Analysis Of Variance – for when you have means in more than two sets of groups to compare, and using multiple t-sets would waste your power.
- Linear Regression – for when you want to predict the value of one continuous variable, based on the values of some other continuous value, or just want to see if different continuous variables are, in fact, related.
If our previous post – Quantitative analysis is as subjective as qualitative analysis – is making you doubt whether you can trust stats at all, then check out how meta analysis can be used to collect the results of multiple different analyses, and produce a single overall measure as to whether the underlying tests show a significant interaction.

If you would like to know more or have any suggestions, please don’t hesitate to reach out to us!

PART I: An Introduction to Hypothesis Testing
PART II: Hypothesis Testing of proportion-based samples
PART III: Hypothesis Testing of mean-based samples