Moving from elm-validate to elm-verify

At Zaptic, we use Elm for our business administration website to allow customers to configure the processes that they want to carry out. The site contains a number of different forms for adding & editing various entities in our system. All of these forms benefit from client side validation before the data is submitted to our servers.

When we started writing our Elm code, the recommended approach to form validation was the rtfeldman/elm-validate library. For a form backed by a data structure like this:

type alias ProductGroup =
    { name : String
    , mode : Maybe Mode
    , selectedProducts : Dict Int Product

    -- Used for storing validation errors
    , errors : List ( Field, String )
    }

We might have a validation function like this:

validateProductGroup : Validator ( Field, String ) ProductGroup
validateProductGroup =
    Validate.all
        [ .name >> ifBlank ( NameField, "Please provide a name" )
        , .mode >> ifNothing ( ModeField, "Please select a mode" )
        , .selectedProducts >> ifEmptyDict ( ProductsField, "Please select at least one product" )
        ]

This checks that the name field isn't blank, that a mode has been selected and that some products have been selected. The result is a list of errors corresponding to the criteria that aren't met:

validateProductGroup { name = "", mode = Just MultiMode, selectedProducts = Dict.empty }
-- [(NameField, "Please provide a name"), (ProductsField, "Please select at least one product")] 

The list can be saved in the model and then the view searches the list for relevant entries when displaying the various input fields in order to display the error state to the user if there is one.

With this in mind, we can create our form to send a SaveProductGroup message when the form is submitted and we can handle this message in our update function with:

        SaveProductGroup ->
            case validateProductGroup model.productGroupData of
                [] ->
                    { model | productGroupData = { productGroupData | errors = Nothing } }
                        ! [ postProductGroup model.productGroupData
                          ]

                results ->
                    { model | productGroupData = { productGroupData | errors = Just results } } ! []

Here, we run the validateProductGroup function and if we get an empty list then we know that there are no errors and we can post our data to the server using postProductGroup that will create a command for us to carry out the necessary HTTP request.

The problem we encounter is that the postProductGroup needs to encode the ProductGroup structure to JSON and when it does that it has to navigate the data that we've given it. We know that the mode value is a Just because we've validated it but the convert to JSON function cannot take any shortcuts because Elm doesn't let us (which is a good thing!) We can try to write the encoding function like:

encodeProductGroup : ProductGroup -> Json.Encode.Value
encodeProductGroup { name, mode, selectedProducts } =
    Json.Encode.object
        [ ( "name", Json.Encode.string name )
        , ( "mode"
          , case mode of
                Just value ->
                    encodeMode value

                Nothing ->
                    -- Help! What do we put here?
          )
        , ( "products", Json.Encode.list <| List.map encodeProduct <| Dict.values selectedProducts )
        ]

As you can see from the comment, it is hard to figure out what to write there. Elm gives us some escape hatches like Debug.crash but that feels wrong. We could not include the mode key if we don't have a value but then we're knowingly allowing a code path that sends incorrect data to the server.

So what do we do? Well, we need to recognise that this function shouldn't be making this decision. It shouldn't have to deal with the Maybe, especially when we have already validated that it has a Just value.

So we create a new type:

type alias ProductGroupUploadData =
    { name : String
    , mode : Mode
    , selectedProducts : List Product
    }

Which is the same data that we're interested in but with the Maybe resolved to an actual value and the Dict.values transformation already applied to selectedProducts. If we change our encodeProductGroup function to expect this type then the implementation becomes trivial:

encodeProductGroup : ProductGroupUploadData -> Json.Encode.Value
encodeProductGroup { name, mode, selectedProducts } =
    Json.Encode.object
        [ ( "name", Json.Encode.string name )
        , ( "mode", encodeMode mode )
        , ( "products", Json.Encode.list <| List.map encodeProduct selectedProducts )
        ]

But how do we convert our ProductGroup to ProductGroupUploadData? This is where the stoeffel/elm-verify library comes in. It allows us to both validate our data and transform it to another structure in the same operation. It does this by using the Result type to allow it to report validation errors, if any are encountered, or the new data structure for us to use. And it does this with a Json.Decode.Pipeline-like interface:

validateProductGroup : ProductGroup -> Result (List ( Field, String )) ProductGroupUploadData
validateProductGroup =
    let
        validateProducts productGroup =
            if Dict.isEmpty productGroup.selectedProducts then
                Err [ ( ProductsField, "Please select at least one product" ) ]
            else
                Ok (Dict.values productGroup.selectedProducts)
    in
    V.validate ProductGroupUploadData
        |> V.verify .name (String.Verify.notBlank ( NameField, "Please provide a name" ))
        |> V.verify .mode (Maybe.Verify.isJust ( ConfigField, "Please select a mode" ))
        |> V.custom validateProducts

Where V is the result of import Verify as V. You can see the "pipeline" approach that might be familiar from Json.Decode.Pipeline. That means we're using ProductGroupUploadData as a constructor and each step of the pipeline is providing an argument to complete the data. We use String.Verify to check that the name isn't blank and Maybe.Verify to check that the mode is specified. Then we use Verify.custom to provide a slight more complex check for the selectedProducts. Verify.custom allows us to write a function that takes the incoming data and returns a Result with either an Err with an array of errors or an Ok with the valid value. We use it to not only check that the dictionary is empty but also extract just the values from the dictionary. We don't have to run Dict.values here, we could also do that in the encodeProductGroup function when generating the JSON but I have a personal preference for the UploadData to match the JSON closely if possible.

With that in place, we can change our SaveProductGroup case in our update function to look like this:

        SaveProductGroup ->
            case validateProductGroup model.productGroupData of
                Ok uploadData ->
                    { model | productGroupData = { productGroupData | errors = Nothing } }
                        ! [ postProductGroup uploadData
                          ]

                Err errors ->
                    { model | productGroupData = { productGroupData | errors = Just errors } } ! []

Which means that the postProductGroup is given a nice ProductGroupUploadData record and no longer has to worry about the Maybe type.

Prior to using the elm-verify library, we used a separate function to convert between the types and we only called postProductGroup if both the validation & the conversion functions were successful. The conversion function always felt a little strange though and switching to elm-verify cleans that up nicely.

Further note: Whilst the elm-verify interface is similar to Json.Decode.Pipeline it isn't quite the same. It has a version of andThen but it doesn't provide some ways to combine operations like Json.Decode.oneOf or helper functions like Json.Decode.map. This is partly to keep the interface simple and partly because it is always manipulating Result values so with a bit of thought you can use helper functions like Result.map quite easily.

I only include this as originally sought to use elm-verify exactly as I would Json.Decode.Pipeline and ended up writing a bunch of unnecessary functions in order to allow me to do just that. I should have spent more time understanding the types in the interface and working with those.

Comments

[Kasey Speakman]

I've had a similar experience to you as far as the input form "allowing" (but displaying errors for) more permutations than what I can send off to the server. Our form-based data model evolved to look like this:

type alias SomeFormError = ( FormError, String )

type alias SomeForm =
    { form : UserEnteredData
    , result : Result (List SomeFormError) SanitizedData
    }

Whenever data is changed by the user, we run a validation route, which takes the form and converts it to a result.

The submit button renders something like this:

case x.result of
    Err _ ->
        button [ class "ui primary button disabled", disabled True ] [ text "Save" ]

    Ok data ->
        button [ class "ui primary button", onClick (SaveData data) ] [ text "Save" ]

The data literally cannot be saved until it is valid and in a form that the server accepts. Then eventually it evolved to this:

type alias ValidationResult err a =
    { errors : List err
    , data : Maybe a
    }

type alias SomeForm =
    { form : UserEnteredData
    , original : UserEnteredData
    , result : ValidationResult SomeFormError SanitizedData
    , saving : RemoteCommand Http.Error ()
    }

I have found that in the view, I almost always use a let statement to convert the Result into a List SomeFormError and a Maybe SanitizedData so I can check them independently. So I just made the validation do that conversion too. Also the RemoteCommand is to represent that the form is currently saving (or failed or finished), so you can affect the submit button or inputs in those cases too. And original is for reset.

I like some things that you showed about elm-verify that I may look into.

[Kasey Speakman]

Well, I compared to what we were doing for validation functions, and it is almost the same. We use simple functions that we wrote ourselves.

validationResult =
    Ok SanitizedData
        |> verify (Ensure.nonEmptyOr (NameRequired, "Name required") x.name)
        |> verify (Ensure.intOr (NumberRequired, "Number required") x.num)
        |> verify customFieldValidation

-- OR with custom operator
-- +* is supposed to mean combine on the left (errors), apply on right

validationResult =
    Ok SanitizedData
        <+*> Ensure.nonEmptyOr (NameRequired, "Name required") x.name
        <+*> Ensure.intOr (NumberRequired, "Number required") x.num
        <+*> customFieldValidation

For my ensure functions, I put the value last so it is chainable:

percentResult x =
    Ensure.nonEmptyOr (ScoreField, "Score required") x
        |> Result.andThen (Ensure.intOr (ScoreField, "Score is not a number"))
        |> Result.andThen (Ensure.rangeOr (ScoreField, "Score needs to be 0 to 100") 0 100)

-- OR with standard operator

percentResult x =
    Ensure.nonEmptyOr (ScoreField, "Score required") x
        >>= Ensure.intOr (ScoreField, "Score is not a number")
        >>= Ensure.rangeOr (ScoreField, "Score needs to be 0 to 100") 0 100

I really need to just take the ending "Or" off the names.

[Michael Jones]

Thanks for the response. That looks great! The elm-verify API allows you to chain checks with andThen but your API looks very nice too. I suspect you might have more experience with Haskell than I do, given your <+*> operator :)

I'm curious about your first comment. It sounds like you validate everything on any change from the user. How do you avoid showing errors on empty fields that the user hasn't reached yet when they fill in the first field? Is validation only triggered after the first save event?

I've got some forms where I check for changes before enabling the save button but I haven't tied in the validation yet. Interesting stuff!

[Kasey Speakman]

I have a passing familiarity with Haskell, but haven't written in it yet. I use Haskell as a reference when I'm looking for a specific kind of function or operator. I couldn't actually find one for combine errors and apply ok at same time, so I made up the <+*>. In general the inline operators just help you eliminate parenthesis and make things shorter.

...
    |> Result.andThen ( ... ... )
-- turns into
...
    >>= ... ...

Turns out I don't use the <+*>, though. Even the few very common operators we do use in the code base (e.g. >>=) can confuse people (including people such as Future Me), much less a custom one. So I just use the verify function and suffer with parenthesis.

As far as validating forms, here is the strategy I use:

• Instead of folding errors onto a general Field type (e.g. NameError), I name each error specifically. E.g. NameBlank, NameTooLong, NameAlreadyTaken, etc.

• In the edit view, I light up fields red on any error.

-- Name field
    ... 
    div [ class "ui horizontal relaxed divided list" ]
       (showErrors [ NameBlank, NameTooLong, NameAlreadyTaken ] m.state.errors)

• In the add view, I show errors on everything but blank errors. In other words, blank is not an error as far as the view is concerned. (But because it is internally an error, it will prevent the form from being submitted.)

-- Name field
    ...
    div [ class "ui horizontal relaxed divided list" ]
       (showErrors [ NameTooLong, NameAlreadyTaken ] m.state.errors)

I can indicate required fields with asterisks or some other affordance. I have a form where if any field is blank, I display a single message "All fields required" beside the Save button. It doubles for an instructive message as well as an answer to the question later: "Why can't I Save this?"

This allows me to avoid any complicated form state management whether it was before or after the user first typed. If you think about it, they will be just as scared when they blank out a field they typed in and it turns red as they will if it were red to start with. Because I just told them by turning it red that what they did was wrong. When it probably wasn't. They can blank it out as much as they want as long as something is in there before they hit Save.

For edit forms, I will validate and display errors even right after the form is loaded. For most data, this will not be a problem as it has already been validated when saved. But it could be that older validations allowed data through that newer ones consider errors. A human probably must correct the data or else it would have been back-filled already. So I'm ok with lighting up a field red immediately on load. That way the user knows what they are getting into if they want to edit that data. And it provides a built-in process to correct data.

[Michael Jones]

Thanks for the break down. Interesting to read. I think I'm still comfortable with the 'validate on save' approach but I appreciate learning about other strategies.