DEV Community: Shiveen Pandita

Software Is Not a Race

Shiveen Pandita — Thu, 26 Aug 2021 05:32:43 +0000

Too often I see people building software and doing changes in an existing piece of code as if they're texting one...excruciating...line...at...a...time. As if it's important to get one disjointed piece of functionality out and making up what's next on the go.

This is bad. Software like any other piece of well thought out functionality should be deliberate. Don't rush into adding a change because the current circumstances confirm it. Check if the change needs to be done based on the requirements of the system. Confirm it with your teammates. More often than not, you will realise that you have missed a key detail of the system. And that the change can be done in another part of the system or not needed at all.

I have long held the belief that a key and critical milestone in the path of young precocious programmer progressing from making computers go bleep blop to building programming megastructures, is the ability to pause and question, deeply, pragmatically, WHY. It's easy to forget in the pace of building a new systems and delivering rapid value, that the best software is the one that's not needed at all. Each extra line of code in the system brings with itself an extra line to maintain and code reading overhead.

Since most of the code ever written is write once and maintain/read many times - it is imperative to question the need. Each line of code needs to earn its place there and should have a rock solid reason to be there.

There are some good strategies that can be applied to enable well thought out software in the org:

No matter what the change, always communicate the need. Writing a detailed why on pull requests is good enough for trivial changes. For non-trivial changes, write a small RFC or some form of Approach Validation Document ™ and get a consensus sign off from your team or a subset of the team. This adds a small overhead to delivering your work but in the end your team will thank you for it.
Treat your codebase not as a junkyard to throw shit in instead a garden to be cultivated. Encourage this mentality within your team as well.
Hone and cultivate a strong bullshit meter for changes that are done to cover and compensate the deficiencies of the system or the process. Quick hacks to get stuff shipped are okay, living with those hacks without even discussing fixing the problems elsewhere seldom is.

I'd like to end this post with a caveat that there are cases where it's necessary to add quick & dirty hacks to fix production or hustle changed to prod to meet a deadline. Such things are a part and parcel of any engineering org to a certain degree and no one should expect you to write a detailed write up for every single commit you make in these cases. But, recognising the time and place for such changes and balancing that with careful followup is the key to not let the system fall under ever increasing tech debt.

(x-post from my blog)

Dynamodb Gotchas

Shiveen Pandita — Wed, 19 May 2021 10:16:26 +0000

This is just a collection of things that required some google searches to resolve while working with DynamoDB at work recently. All the code examples here use Kotlin, though, I'm confident that they would still appear in Java.

Persisting a DynamoDb Object that extends a class

DynamoDB doesn't automatically work with Abstract classes. For example, let's say you have an abstract class which contains some common fields (such as the hashKey and the range fields),

abstract class AbstractBook @JvmOverloads constructor(
    @DynamoDBHashKey(attributeName = "isbn")
    var isbn: String? = null,

    @DynamoDBTyped(DynamoDBMapperFieldModel.DynamoDBAttributeType.S)
    @DynamoDBRangeKey(attributeName = "author")
    var bookType: String? = null
)

And then you define the actual class, that extends this base class,

class Novella @JvmOverloads constructor(
        isbn: String? = null,
        bookType: String? = null, 
        var content: String? = null
 ) : AbstractBook (isbn, bookType)

When you try to use a DynamoDBMapper to save the object it will throw an exception.

To fix the exception, you'll need to add the @DynamoDBDocument to the AbstractClass to let the mapper know that it's the abstract version of the actual persisted entity. This is what the JavaDoc for the annotation says:

An annotation that marks a class which can be serialised to a DynamoDB document or sub-document. Behaves exactly the same as DynamoDBTable, but without requiring you to specify a tableName.

Which means that this annotation is necessary to serialise objects types that are not directly part of the actual stored object type.

Persisting a List of Objects in DynamoDB

Another little snowflake behaviour I encountered was when persisting a list of objects. Let's say we go ahead and add some more data to the Novella object, like a List of Publishers.

data class Publishers(
    val name: String,
    val address: String?
)

class Novella @JvmOverloads constructor(
        isbn: String? = null,
        bookType: String? = null,
        var content: String? = null, 
        @DynamoDBTypeConvertedJson 
        val publishers: List<Publishers>? = null
 ) : AbstractBook (isbn, bookType)

@DynamoDBTypeConvertedJson is the annotation DynamoDB recommends for storing objects when using DynamoDBMapper. It has a strange behaviour where it can auto serialise a List<T> but it loses type information on deserialisation and deserialises the object as an amorphous map. Which means you get exceptions like:
java.lang.ClassCastException: java.util.LinkedHashMap cannot be cast to Publisher.

Based on this thread this has to do with type erasure, wherein T : Object which results in bad behaviour at deserialisation time (the default Jackson marshaller is smart enough during serialisation).

The best way to solve this is to define a custom serialiser and deserialiser for your object.

So, in this case it's a matter of defining a class like,

class PublishersMapListConverter : DynamoDBTypeConverter<String, List<Publishers>> {
    private val objectMapper = jacksonObjectMapper()

    override fun convert(publishers: List<Publishers>): String {
        return objectMapper.writeValueAsString(publishers)
    }

    override fun unconvert(publishers: String): List<Publishers> {
        val type = object : TypeReference<List<Publishers>>() {}
        return objectMapper.readValue(publishers, type)
    }
}

And then adding the @DynamoDBTypeConverted(converter = PublishersMapListConverter::class),

class Novella @JvmOverloads constructor(
        isbn: String? = null,
        bookType: String? = null,
        var content: String? = null,
        @DynamoDBTypeConverted(converter = PublishersMapListConverter::class)
        val publishers: List<Publishers>? = null
) : AbstractBook(isbn, bookType)

Hopefully this post helps someone else save some time as well.

x-post from my blog

Building a Reactive Oauth Client App with SpringBoot and Kotlin Coroutines

Shiveen Pandita — Sun, 14 Jun 2020 11:18:15 +0000

Background

In this post, I’ll walk through two new exciting things that have happened in the spring ecosystem recently. First is spring webflux support for kotlin couroutines and second is the overhaul of spring security, and the addition of the out of the box oauth2 client support for social logins.

For me one of the most impactful new features is the integration with spring coroutines throw the Flow primitive. The integration makes writing reactive code a lot more straightforward - no more subscribeOn and observeOn operations. Instead of thinking in terms of Mono and Flux based primitives borrowed from reactor, it allows a straightforward way to generate cold streams instead. Kotlin flow allows us to write purely asynchronous code in an sequential/imperative manner - which in turn means an existing codebase using blocking code can be converted to use the non-reactive paradigm a lot easier. Another advantage of using coroutines based non-blocking code is that coroutines have been implemented as lightweight threads whereas schedulers with reactor can incur a lot of performance overhead as they have to context switch between threads. As we progress forward in this post, people who are already familiar with composing non-reactive code in webflux will see what I mean.

What we will be building

To demonstrate the way this works, we will try to develop an asynchronous api that connects to a GitHub account and gets all the starred repos for a user. The api will use spring webflux with Kotlin Flow and we will integrate that with the spring oauth2 client to ensure the user is logged in.

To start off, let’s create a new gradle project in whatever IDE you prefer (I use Intellij IDEA) and add the following set of dependencies:



implementation("org.springframework.boot:spring-boot-starter-webflux")
implementation("org.springframework.boot:spring-boot-starter-security")
implementation("org.springframework.security:spring-security-oauth2-client")
implementation("org.springframework.security:spring-security-oauth2-jose")
implementation("com.fasterxml.jackson.module:jackson-module-kotlin")
implementation("io.projectreactor.kotlin:reactor-kotlin-extensions")
implementation("org.jetbrains.kotlin:kotlin-reflect")
implementation("org.jetbrains.kotlin:kotlin-stdlib-jdk8")
implementation("org.jetbrains.kotlinx:kotlinx-coroutines-core")
implementation("org.jetbrains.kotlinx:kotlinx-coroutines-reactor")

Once setup, lets start building our controller first, creating a new file Router.kt



@RestController
class Router {

    @Bean
    fun routes(handler: Handler) = coRouter {
        "/".nest {
            GET("", handler::getStarredRepos)
        }
    }
}

As you can see we have simply used the normal way of defining a new spring based REST api controller but instead of using the standard router annotations we took the approach of using the coRouter. It’s more of a personal preference since it allows me to see my whole api surface area in a more concise format. However, you could easily replace it with the more common Spring MVC style annotation based approach.

Next we will build a handler that let’s us handle our service response.



@Component
class Handler(val service: Service) {

    suspend fun getStarredRepos(req: ServerRequest): ServerResponse {

        return ok().bodyAndAwait(service.getUserStarredRepos())
    }
}

Nothing special about the handler except two main distinctions when compared to equivalent reactor based approach. First is that we have made the function getStarredRepos() a suspending function - which means it tells the compiler that this code will be run inside a couroutines context. The second interesting thing to note is theat we used a bodyAndAwait() instead of body() method for a server response. This extension allows us to correctly get the body from suspending couroutine context while the service is generating a response without blocking the calling thread.

Now let’s jump into the service code - we will use the new spring Oauth2 client with comes with first class webflux support to make our app support the GitHub Oauth login.

To begin with, register a new app in Github. Once done, add a new application.yml file and add your oauth details like:



spring:
  security:
    oauth2:
      client:
        registration:
          github:
            clientId: <add-github-client-id-here>
            clientSecret: <add-github-client-secret-here>

This tells the Oauth client what credentials to use when redirecting a non authenticated user to Github authentication page. The default authorization callback url setup by the spring is: http://localhost:8080/login/oauth2/code/github .In and of itself all oauth client will do is grab the github authorization grant and store that data in a JSESSION cookie. If that's all you want to do then that's fine - however, I would also like to show the user their starred github repositories. To achieve that we need to be able to somehow get the access token and make an authenticated call on behalf of the user. Spring provides a nice and secure interface to achieve that as well - enter the authenticated webclient.

Webclient is the reactive counterpart of the old and trusty RestTemplate from the Spring MVC days introduced in Spring webflux. It allows us to make calls to APIs in a non-blocking api and comes with nice composition and testing support. In our case, we will build a spring config that will populate the current context with an authenticated webclient for the current logged in user.



@Configuration
class GithubWebClientConfig {

    @Bean
    fun webClient(clientRegistrations: ReactiveClientRegistrationRepository?,
                  authorizedClients: ServerOAuth2AuthorizedClientRepository?): WebClient? {
        val oauth = ServerOAuth2AuthorizedClientExchangeFilterFunction(clientRegistrations, authorizedClients)
        oauth.setDefaultOAuth2AuthorizedClient(true)
        return WebClient.builder()
            .filter(oauth)
            .build()
    }
}

The key to setting the right authentication credentials is the line oauth.setDefaultOAuth2AuthorizedClient(true) which allows the github client bean to be automatically authenticated with the current users github tokens, simple and delightful.

Now we'll go ahead and wire up the last piece of this puzzle which is the service layer that allows us to tie the authenticated webclient with an API call. So let's go ahead and do that:



@Service
class Service(val client: WebClient) {

    suspend fun getUserStarredRepos(): Flow<String> {
        return client
            .get()
            .uri("https://api.github.com/user/starred?page=1")
            .retrieve()
            .bodyToFlow()
    }
}

To explain a bit of what's going in this code snippet, we have created another suspendable function getUserStaarredRepos() that retrieves the starred repos from github and uses our pre authenticated oauth webclient to do so. One callout here would be the return type which is of the type Flow<String> . This possible due to Kotlins extension based programming approach where the coroutines team have created an extension for the webclient which is functionally equal to reactor's bodyToFlux() method.

That concludes our build for this app. To test, let's fire it up using the ever helpful gradle bootRun command (you should run this in your terminal with the current directory set to this sample project root):



./gradlew bootRun

Once done, this will start up our Springboot app and you should be able to navigate to localhost:8080 to see it in action.

Conclusion

As someone who moved from springboot with spring mvc building blocking code to using spring webflux and spending hours to understand the non-blocking paradigm - the Flow support in webflux is a welcome evolution. The potential of migrating existing codebases to non-blocking stream based paradigm has been greatly amplified, and I can already see the impact it is having in making my own side projects a lot easier to build and reason about!

If you'd like to experiment yourself the source code for this post can be found on github.

This post can also be found on my blog

References:

Right Way To Shadow Jar When Using Jetty With Http4k

Shiveen Pandita — Sat, 18 Apr 2020 03:21:38 +0000

(x-post from my blog here)

If you’ve been spring boot for a while, you’re probably familiar with Spring’s bootJar functionality that lets you create a new executable “fat jar” with all its dependencies pre defined. However, while I was building a kotlin app using http4k, which doesn’t come with any such built in tooling I had to resort to using the shadowJar plugin to build a fat jar.

However post deploy, the app stopped starting up and I noticed the following error in the logs:

Exception in thread “main” java.lang.ExceptionInInitializerError
    at org.eclipse.jetty.http.MimeTypes$Type.<init>(MimeTypes.java:98)
    at org.eclipse.jetty.http.MimeTypes$Type.<clinit>(MimeTypes.java:56)
    at org.eclipse.jetty.http.MimeTypes.<clinit>(MimeTypes.java:175)
    at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:806)
    at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:275)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110)
    at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:100)
    at org.eclipse.jetty.websocket.server.WebSocketHandler.doStart(WebSocketHandler.java:84)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
    at org.eclipse.jetty.server.Server.start(Server.java:407)
    at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110)
    at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:100)
    at org.eclipse.jetty.server.Server.doStart(Server.java:371)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
    at org.http4k.server.Jetty$toServer$3.start(jetty.kt:33)
    at io.taggit.AppKt.main(App.kt:175)
    at io.taggit.AppKt.main(App.kt)
Caused by: java.lang.ArrayIndexOutOfBoundsException: 1
    at org.eclipse.jetty.http.PreEncodedHttpField.<clinit>(PreEncodedHttpField.java:68)
    … 20 more

Perplexed, since when I used the same plugin with another one of my projects I had never noticed this error.

After much googling, turns out Jetty bundles HttpField encoders by referencing META-INF/services/org.eclipse.jetty.http.HttpFieldPreEncoder so when shadow jar created the fat jar it didn’t have the data that Jetty was looking for which in turn meant Jetty threw an initialisation exception.

The fix?

Adding the mergeServiceFiles() attribute to the shadowJar task, as per the shadow documentation here.

shadowJar {
    baseName = ‘taggit-api’
    zip64 = true
    mergeServiceFiles()
}

And voila, the service was back up and running again. This is becauase multiple libraries potentially use the same service descriptor files (usually META-INF) and in case of creating fat jars its generally desired to merge the service descriptors to make sure all libraries have their service descriptors loaded at runtime.

P.S. - It might be worth mentioning here as well that the reason I didn’t encounter the same error for one of my other services is due to that project using Netty as the underlying webserver.

Cover Photo by Markus Spiske on Unsplash

Software Engineering Levels

Shiveen Pandita — Sun, 05 Jan 2020 02:17:02 +0000

For many software engineers, SE job titles hold a special place. No other topic elicits a more passionate response than when an engineer thinks that their ranking in the pecking order is threatened by an unworthy usurper in their company. The pattern is the same, whether it be a 1000+ tech drone corporate or small tight-knit startup.

Although initially used to define various steps in the career ladder ¹ which meant more prestige from the colleagues and more importantly, better compensation from the employer - the ranks have warped into a pissing contest between new developers in the industry. The way I like to look at it is that the ranks are purely indicative of the level of impact, and the breadth of responsibility that any engineering individual in the company has. Unlike popular opinion, I don't think it's necessarily tied to experience either - I have seen engineers with < 2 years of experience handling complex architectural problems and conversely seen someone with > 10 years of experience in the industry struggling with basic programming concepts. There's also the matter of company size, a mid level at a large company might be a senior at a small company and vice versa. The career progression paths are different in every company and where you might lie based on that should also be taken into account.

For someone who has been coding since the age of 13 and has had various jobs at different levels in a professional setting for the past five years, I felt like now was the time to put in writing my thoughts on what constitutes developers at various levels. It is worth noting however that whatever you read here isn't definitive, and as always factors outside of technical and soft skills should be taken into account.

Junior Software Engineer

This is someone who's just tasted the wonderful life of what it means to make machines do mans bidding. More often than not, people that are put in this category are either fresh out of university with a degree that had some programming background or have just graduated from an academy, or an institute specializing in teaching people how to code.

In my opinion, such candidates are the lifeblood of any organisation, and they should be given space, time and most importantly mentorship, to grow. Usually the expectations for people in this role are:

Write code. As simple as that. As a junior, the expectations are not multi dimensional, and the company would thank you for just writing code and getting s*&$ done! You might be good at leetcode and blazingly fast at writing linkedlist implementations, but if you can't write proper javascript and css as a frontend engineer, you're already on a sinking ship.
Network and be open to feedback. Perhaps the most important skill of them all when you're just starting is to always be open to feedback from peers, whether it be via pull requests submitted at work or via a mentorship sessions with someone with a bit more experience. Another important professional goal should be to network with as many people as possible within the industry. This can involve going to meetups, tech fairs, conferences, hackathons etc. Programming by nature attracts social recluses, so going to events like this can provide a safe environment to meet new people in the industry without the undue pressure of an alternative social setting.
Learn various auxiliary system. Professional programming isn't simply writing code, it involves using version control systems, dealing with frameworks, reading APIs and reasoning about the system from them, working with databases etc. No one would expect you to be an expert in these systems, but knowing your way around it is almost essential part of the role. Fortunately, any good company will be willing to spend time and mentorship hours to coach you on any of those topics if you'ev never done them before.
Fundamentals and algorithms are good, but pick a modern framework (and if possible, stick with it). As the saying goes, frameworks come and go, but the logic never changes (okay yes, I just made that up). It's hardly a secret that there always seems to be a new framework to achieve a result, this is especially true of frontend - but it's worth investing and picking up one of the popular framework to gain deep knowledge in, especially for juniors. I picked up VueJs and Spring boot as my front end and backend frameworks of choice and have never looked back. It was one of the best decisions I made as a junior.

Mid level Software Engineer

These are the underdogs of any dev shop. Almost always underappreciated but are the ones making sure the product guys are happy with the sprint velocity. Humor aside, mid level engineers ensure the rhythmic hum of new features being churned and priority bugs being squashed withing the organisation. The people that fit in this role usually have following expectations:

Working autonomously or with minimal direction. A solid mid level engineer, after an initial period of acclimatisation is in tune with the product that the company is selling is able to pick up technically challenging or unassigned work without little to no effort. They may still require some guidance here and there, but for the most part the company, and their peers can be confident that they will always deliver on their assigned work.
Mentoring or showing the ropes to juniors. Often companies will expect mid level engineers to be showing the ropes to interns and junior software engineers - this can mean many things, however, usually involves pairing with them through onboarding and making sure onboarding docs are up to date, guiding them on fixing issues with individual components and generally being a validation checkpoint for their development queries.
Comfortable in making component and micro level architectural decisions. This part of mid level engineer roles, overlaps with a senior engineer/solution architects role. As a mid level engineer you're also expected autonomously make micro level code and drive technical decisions within a small 2-3 team pod. Quite frankly, most companies won't expect it but in my opinion if someone is truly looking to breach into the senior engineer and then perhaps technical leadership ladder this is an essential skill.

Senior Software Engineer

People in these positions are usually regarded as the influencers and drivers of technical direction in a given stack/project domain for the company. Routine expectations for people in such roles are:

Deep knowledge of their stack and company's product domain. Senior engineers are usually expected to be the go to authority on complex technical questions and architectural design decisions. They're expected to know their technical domain very well, including any quirks, and are expected to use that knowledge to solve complex product and build challenges.
Technical stewardship. Most companies expect senior engineers to be the stewards of their chosen tech stack. This can manifest in various ways, for example, some are champions of their industry and are regularly given talks about a particular programming concept or tool. Others, tend to do things more privately and are consistently experimenting with better and smarter way to build systems and improve business outcomes within the company. This requires people in senior engineering position to be constantly updating their skills.
Long term focus. This probably the most underappreciated part of a senior engineers roles, but which I think is what really differentiates a true senior engineer from the lot. A senior engineer is expected to always prioritise maintainability/extensibility over everything. This is part of their programming DNA, and this is what allows them to dismiss the new fancy framework of the season, or the next -killer.

Communication

Communication is such an essential skill for engineers at all levels, so I decided to dedicate a whole section to it. Good communication skills are hard to quantify but very easy to identify. More often than not, technical and project management, inside a company would rather have engineers that can communicate but produce marginally less output over engineers that are rockstars but lack the desired communication skills. This is not universal, but it is definitely the trend.

For entry level/junior engineers it's important to be to consistently ask for feedback and articulate why the code they added/changed was done in the way it did.

For mid level engineers, it's important to be able communicate with juniors on the team as well as be able to document design decisions made over the course of delivering product features.

For senior engineers and above it is even more critical, as a big part of their job is to communicate their ideas and have them percolate through various teams inside the development org.

Why leave the rest

I understand that an engineering org can, and mostly is, made up of other varieties of engineers such as intern or graduate/entry level engineers, technical analysts, software architects, tech leads, devops ninja etc etc. This is post is not aimed at them and sometimes a lot of their skills will overlap with developers. I might do a follow up post in the future that targets people working in cross-cutting development roles.

References

This article was also posted on my blog here

Building a small S3 Browser in pure kotlin

Shiveen Pandita — Wed, 14 Aug 2019 11:13:10 +0000

(x-posted from my blog here)

Background

Recently I found myself with a little bit of extra development time on hand. Now usually, most of my projects start off as big grand ideas and as soon as I start working on them, I lose steam or life comes in the way and things just sit their eating dust as a private github repo.

However, I stumbled upon Kweb which is a server side rendered web app building library, but written entirely in Kotlin. Kweb provides a nice dsl like interface to build web apps by programmatically defining the html elements for the app instead of writing the html and CSS by hand. For those who don't know, Kotlin provides a really nice way to build type safe declarative builders.

Since it's all created declarative in the kotlin code, we get access coroutines, extensions and all the nice things that make working with Kotlin such a joy. For those of you who have worked with pure code based server side rendered frameworks before, this might remind you of Vaadin which is an industry leader in its space, but there are subtle and not so subtle differences which you can be found on the Kweb FAQS.

Going through their codebase, I though it would be a really good opportunity to try and a build a quick and easy app by just using pure kotlin. Why you ask? Cause it was quick and I could smash it out in a few hours.

Simple Kotlin S3 Client

To start off, I wrote a basic S3 browsing class, starting off small - I copied the code from AWS examples on creating a new S# client and then added the ability to search for all public keys in a given bucket and get some metadata and download links:

class S3Client(private val endpoint: String, private val bucketName: String) {


    private val client = AmazonS3ClientBuilder.standard()
        .withPathStyleAccessEnabled(true)
        .withEndpointConfiguration(AwsClientBuilder.EndpointConfiguration(endpoint, "ap-southeast-2"))
        .build()

    fun listAllKeys(): List<S3Data> {
        val req = ListObjectsV2Request().withBucketName(bucketName).withMaxKeys(10)
        val keyList = mutableListOf<S3Data>()
        client.listObjectsV2(req).objectSummaries.forEach {
            keyList.add(S3Data(it.key, "$endpoint/$bucketName/${it.key}", it.size.toString().toDouble() / 1000.0, it.lastModified.toString()))
        }
        return keyList
    }
}

the listAllKeys() functions returns a list of all keys in that given bucket, which I can then map to a custom S3Data class:

data class S3Data(
    val key: String,
    val downloadUrl: String,
    val size: Double,
    val lastModifiedAt: String
)

Building the UI with Kweb

Once done, I got cracking on the UI interface. I wanted something quick and simple, crude even, just to demonstrate that it all works as proposed. So I setup a container with some form fields, a search button and a table to input the S3 region link and the name of the bucket.

I also needed a table to display all the keys (sans pagination, who builds pagination in PoCs anyway? 🤠). To enable holding the data, I used something called KVAR which is simply a state store used by Kweb to support propagating state changes to the web app via Observer Pattern.

The following code fragment gets the initialises the S3 data Kvar (setup as an empty list initially) - which eventually propagates it to the table:

div(fomantic.ui.main.container).new {
    div(fomantic.ui.vertical.segment).new {
        div(fomantic.ui.header).text("Welcome to S3 Browser 💻")
    }

    val keyData = KVar(emptyList<S3Data>())

    val loader = div(mapOf("class" to "ui active text loader")).addText("Retrieving keys...")
    loader.setAttribute("class", "ui disabled text loader")
    createInputSegment(loader, keyData)
    createKeysTable(keyData)
}

Now, here is where Kwebs deep integration with kotlin really comes in handy, since it allows us to use kotlin coroutines to handle tasks with considerable i/o (such as retrieving data from an AWS bucket). I have recently started using coroutines frequently in production code and I can without doubt say they're the best way to write asynchronous tasks without worrying about threads. The low touch syntax setup and the results are so easy it almost feels like cheating.

The following code fragment uses the S3Client introduced earlier to launch a coroutine and when the user hits the search button, and displays a loading icon until all the data is retrieved or an error is thrown:

private fun ElementCreator<DivElement>.createInputSegment(
    loader: Element,
    keyData: KVar<List<S3Data>>
) {
    div(fomantic.ui.vertical.segment).new {
        div(fomantic.ui.input).new {
            val endpointInput = input(type = InputType.text, placeholder = "Enter S3 Endpoint Url")
            val bucketInput = input(type = InputType.text, placeholder = "Enter S3 Bucket Name")
            button(mapOf("class" to "ui primary button")).text("Search").on.click {
                GlobalScope.launch {
                    loader.setAttribute("class", "ui active text loader")
                    val s3Client =
                        S3Client(endpointInput.getValue().await(), bucketInput.getValue().await())
                    try {
                        keyData.value = s3Client.listAllKeys()
                    } catch (ex: Exception) {
                        p().execute(ERROR_TOAST)
                        loader.setAttribute("class", "ui disabled text loader")
                    }
                    if (keyData.value.isNotEmpty()) {
                        p().execute(SUCCESS_TOAST)
                        loader.setAttribute("class", "ui disabled text loader")
                    }
                }
            }
        }
    }
}

So far so good, now that we have successfully pulled in the data in out Kvar container, we can start rendering a table. Now I also wanted to show a nice little icon to show that the retrieved object was a file - and also allow the ability to click name of the key as a link so the user can download.

Now Kweb as far as I could tell didn't have the ability to specify that via a DSL object, however, it does provide the ability to specify nested HTML inside a table element to add my own custom behaviour.

private fun ElementCreator<DivElement>.createKeysTable(
    keyData: KVar<List<S3Data>>
) {
    table(mapOf("class" to "ui celled striped table")).new {
        thead().new {
            tr().new {
                th().text("Key")
                th().text("File Size (in KB)")
                th().text("Last Modified At")
            }
        }
        tbody().new {
            keyData.map {
                it.forEach {
                    tr().new {
                        td(mapOf("data-lable" to "Key")).innerHTML("<i class=\"file outline icon\"></i> <a target=\"_blank\" href=${it.downloadUrl} download=${it.key}>${it.key}</a>")
                        td(mapOf("data-lable" to "File Size")).text("${it.size} KB")
                        td(mapOf("data-lable" to "Last Modified At")).text(it.lastModifiedAt)

                    }
                }
            }
        }
    }
}

The code above creates a new table and generates a new row in the table for each public key present in the provided bucket. If no data is present, nothing gets rendered.

This probably also the time to give a shoutout to the Kweb creators for an integration with Fomantic UI which comes pre-configured with nice UI elements. Although, the integration doesn't end there and there is a nice APi for anyone to write a new plugin with their favourite UI elements library.

Final Notes

And that is all that's needed to write a simple S3 browsing web app using Kweb. Here is the app deployed on heroku and the full working code is on github.

Gif Demo:

Sending Multipart Form Data Using Spring WebTestClient

Shiveen Pandita — Fri, 03 May 2019 11:15:22 +0000

(x-posted from personal blog here)

Background

For the past year or so, I have been working extensively with spring, especially spring webflux; building scalable reactive micro services for our customers.

Coming from spring MVC, learning webflux and getting used to reactive programming in general has been a great and worthy learning experience and I highly suggest going through the references section if you haven’t heard of reactive programming and/or have been thinking about giving it a go and don’t know where to start. But essentially reactive programming involves a model of creating, requesting and manipulating data in a controllable (from a consumers perspective) and non-blocking manner.

WebTestClient is a reactive testing high level http client with fluent assertions, packaged in spring web flux. Recently, while integration testing an application that accepted data as multipart/form-data I had to figure out how to test the data effectively using the webtestclient and personally found the lack of comprehensive resources on the internet lacking, so I wrote this blogpost to share my own learnings.

Web Form Testing with Webflux

Let’s suppose that we’re trying to send the request to fill a form api that accepts a document (image, text, plain binary etc.) and some textual data.

To aid with our example, lets imagine the form is a profile setup for an document share service and takes the following input:

Profile Image (api label: profileImage)
Username (api label: username)
Email (api label: email)
PDF document to share (api label: userDocument)

For us to begin sending the data, we’ll use the spring library called MultipartBodyBuilder which provides a nice api for setting up the body for multipart requests.

To send the first part, the profile image we can set it up as:



val bodyBuilder = MultipartBodyBuilder()

bodyBuilder.part("profileImage", ClassPathResource("test-image.jpg").file.readBytes()).header("Content-Disposition", "form-data; name=profileImage; filename=profile-image.jpg")

To explain a bit about what’s going on there, we’re simply telling the body builder to upload an image found in src/test/resources folder with the name test-image.jpg as the profile image part of this body. The real kicker here is setting up the Header part as that is what’s used by the webtestclient internals (specifically the Synchronoss-nio library which webflux uses internally) to determine the type of form data being sent and how to process it.

Also, note that the real file name that will get uploaded in the web server receiving the request is the profile-image.jpg filename that gets sent as part of the headers,.

Similar to the profile image, we can also send the document part of the whole request payload:



bodyBuilder.part("userDocument", ClassPathResource("user-document.pdf").file.readBytes()).header("Content-Disposition", "form-data; name=userDocument; filename=my-thesis.pdf")

Similar to the previous payload we test the body builder 💪 to read a file in the test resources folder called user-document.pdf as bytes and send the document with the name my-thesis.pdf to the form web api.

As you can already see, compared to some other ways of doing it, such as in this excellent blog , using the MultipartBodyBuilder is rather conveneient.

Now for the last two remaining pieces of the form api, which are usually only plain text, we can set them up as:



bodyBuilder.part("username", "shiveenpandita", MediaType.TEXT_PLAIN).header("Content-Disposition", "form-data; name=username").header("Content-type", "text/plain")

bodyBuilder.part("email", "shiveenpandita@gmail.com", MediaType.TEXT_PLAIN).header("Content-Disposition", "form-data; name=email").header("Content-type", "text/plain")

Woohoo! 🎉 We’ve got all our form fields wired now.

Now to see it all in action and bring it all together, we can simply setup a spring integration test and use our freshly setup body builder as:



@RunWith(SpringRunner::class.java)
@SpringBootTest
@AutoConfigureWebTestClient
class WebClientTest {

    private lateinit var webclient: WebTestClient

    @Test
    fun `test webform api`() {
        val bodyBuilder = MultipartBodyBuilder()

        bodyBuilder.part("profileImage", ClassPathResource("test-image.jpg").file.readBytes()).header("Content-Disposition", "form-data; name=profileImage; filename=profile-image.jpg")

        bodyBuilder.part("userDocument", ClassPathResource("test-document.pdf").file.readBytes()).header("Content-Disposition", "form-data; name=userDocument; filename=my-thesis.pdf")

        bodyBuilder.part("username", "shiveenpandita", MediaType.TEXT_PLAIN).header("Content-Disposition", "form-data; name=username").header("Content-type", "text/plain")

        bodyBuilder.part("email", "shiveenpandita@gmail.com", MediaType.TEXT_PLAIN).header("Content-Disposition", "form-data; name=email").header("Content-type", "text/plain")

        webClient.post()
            .uri("/v1/test-api")
            .contentType(MediaType.MULTIPART_FORM_DATA)
            .body(BodyInserters.fromMultipartData(bodyBuilder.build()))
            .exchange()
            .expectStatus().isOk
    }
}

The above code snippet will successfully send the required data to our test api and the webtestclient asserts that the response is 200 OK.