Continuous updates as you process data in TypeScript

Last week I wrote a post about how we can avoid glitches when processing data in TypeScript and animating. The example showed 1 million records being processed. One comment asked whether we could provide early updates to the user during processing so that charts and lists were dynamically changing as the user waits. This post addresses that interest.

Now clearly you'll only be processing so much data if you are writing an offline app that handles significant data volumes, but of course, the point is that even reasonably small data processing can take more than 17ms and cause a glitch.

js-coroutines allows you to run collaborative processes that share out the main thread between animations and data processing meaning you won't block the user from changing their mind, scrolling around or other ways of interacting.

Demo

Try typing in the search box or click on the animals/countries and color charts to apply filters

Realtime updates

In order to adapt the previous version of this routine to handle updating results as they are discovered I refactored the earlier code to process individual records matching the user query using Inversion of Control via an event emitter.

       if (
            parts.every(p => {
                const parts = p.split(':')
                if(parts.length === 1) {
                return record.description
                    .split(" ")
                    .some(v => v.toLowerCase().startsWith(p))
                }
                return record[parts[0]].toLowerCase().startsWith(parts[1])
            }
            )
        ) {
            output.push(record)
            events.emit('row', record)
        }

I've also updated the search so that we can add a predicate prefix in the search so color:red will only search the color column of the data, while red will continue to search for any word in any column starting with that value.

Updating the charts row by row is achieved with a simple custom hook:

function useChartData(
    column: (row: any) => string,
    forceLabelSort?: boolean
): ChartData[] {
    const [chartData, setData] = React.useState<ChartData[]>([])
    const localData = React.useRef<ChartData[]>([])
    useEvent("row", processRow)
    useEvent("progress", complete)
    return chartData
    function complete() {
        const next = localData.current.slice(0)
        if (forceLabelSort === false || (next.length > 20 && !forceLabelSort)) {
            next.sort(sortByValue)
        } else {
            next.sort(sortByName)
        }
        setData(next)
    }

    function processRow(row: any) {
        let value = column(row)
        let entry = localData.current.find(row => row.name === value)
        if (entry) {
            entry.value++
        } else {
            localData.current.push({ name: value, value: 1 })
        }
    }

    function sortByValue(a: ChartData, b: ChartData) {
        return +b.value - +a.value
    }

    function sortByName(a: ChartData, b: ChartData) {
        return a.name === b.name ? 0 : a.name < b.name ? -1 : 1
    }
}

We then emit an event called progress every 500ms to update the charts. js-coroutines allows this to happen alongside the continuing calculations.

Conclusion

This example is showing very complex calculations that are unlikely to exist in many real-world applications - however, many processes do happen on the front end, and maintaining a great experience can be helped by ensuring that data processing is shared out.