DEV Community: Colin Fay

Rlinguo — Why Did We Build It?

Colin Fay — Mon, 03 Feb 2025 18:00:00 +0000

We recently released something unprecedented: an app that brings R to mobile. Yes, you read that right — R, the statistical programming language, is now in the palm of your hand. Whether you're an environmental researcher, a student, or a logistics professional, R on mobile is here to revolutionize how you work with data on the go.

You can download Rlinguo today:

📱 iOS on the App Store
📱 Android on the Google Play Store

Bringing R to mobile wasn't an easy feat. Like any innovative project, it involved countless hours of trial and error, learning, and refining. But the question remains: Why did we do it? Why do this in the first place? What’s the real-world impact of having R in your pocket?

The answer is simple: mobility matters. In a world where data drives decisions, having the ability to analyze, visualize, and act on data anytime, anywhere, is a game-changer. Let’s dive into five real-world scenarios where R on mobile can make a difference.

1. Field Data Collection and Analysis for Environmental Researchers

Who?

Environmental scientists working in remote locations—forests, mountains, oceans—collecting data on-site.

The Problem

These locations typically lack reliable internet access, making it impossible to analyze data in real time with cloud-based tools.

How R on Mobile Helps

With R running on their phone, researchers can input data, perform statistical analyses, and generate visualizations directly on-site. This real-time feedback helps them adjust data collection methods on the fly, improving efficiency and accuracy in their research.

2. A Portable Learning Tool for Statistics Students

Who?

Students and educators who want to learn or teach statistics using R.

The Problem

Not everyone has access to a computer or a stable internet connection, especially when commuting or traveling.

How R on mobile Helps

R on Mobile can turn a phone into an interactive learning tool. Students can input datasets, apply statistical methods, and visualize results in real time. It’s an interactive, hands-on way to learn R and statistics — no laptop required.

3. Real-Time Quality Control in Manufacturing

Who?

Quality assurance teams monitoring production lines in manufacturing plants.

The Problem

These teams often don’t have access to computers while inspecting production lines, making it difficult to analyze data on the spot.

How R on Mobile Helps

With R on mobile, QA teams can analyze sensor data, apply statistical quality control methods, and detect anomalies on the spot. Immediate alerts help them take action before defects escalate, minimizing waste and improving efficiency.

4. Supply Chain Optimization for Logistics

Who?

Logistics professionals managing delivery routes, warehouse operations, or fleet management.

The Problem

Supply chains are dynamic, and decisions often need to be made on the fly — but without the right tools, optimization is challenging.

How R on Mobile Helps

R on Mobile enables logistics managers to apply predictive models and optimization algorithms directly from their phones. Whether it’s adjusting delivery routes, managing inventory, or responding to disruptions, Rlinguo ensures you’re always one step ahead.

5. Healthcare Data Analysis for Field Medics

Who?

Healthcare professionals working in remote or underserved areas.

The Problem

Field medics often lack access to cloud-based systems for analyzing patient data, which can delay critical decisions.

How R on mobile Helps

R on mobile allows medics to analyze patient health metrics, generate risk scores (e.g., for cardiac events or infectious diseases), and visualize trends directly on their devices. This enables rapid, data-driven decision-making, even in the most challenging environments.

Why R on Mobile?

R on mobile isn’t just for fun — it’s a tool that empowers professionals and learners alike to harness the power of R wherever they are. Whether you’re in a forest, a factory, or a classroom, R on mobile ensures that data analysis is always within reach.

Have questions or feedback? We’d love to hear from you! Reach out to us!.

🚀 download Rlinguo:

📱 iOS on the App Store
📱 Android on the Google Play Store

Setting values in R6 classes, and testing with shiny::MockShinySession

Colin Fay — Wed, 08 Jan 2025 17:00:00 +0000

Context

Recently, we worked on testing a {shiny} app that relies on values stored within the session$request object. This object is an environment that captures the details of the HTTP exchange between R and the browser. Without diving too deeply into the technicalities (as much as I’d love to 😅), it’s important to understand that session$request contains information provided by both the browser and any proxy redirecting the requests. Our app is deployed behind a proxy in a Microsoft Azure environment. Here, the authentication service attaches several headers to validate user identity (see documentation for details). Headers like X-MS-CLIENT-PRINCIPAL and X-MS-CLIENT-PRINCIPAL-ID are critical for identifying users, and the {shiny} app depends on these to manage authentication.

Testing headers

When a user connects to the app, their identifiers are retrieved from a header and stored for use throughout the app. Here's a simplified example of how this might work:

library(shiny)

ui <- fluidPage(
  textOutput("user_id")
)

server <- function(input, output, session) {
  r <- reactiveValues(
    email = NULL
  )

  observe({
    r$email <- session$request$HTTP_X_MS_CLIENT_PRINCIPAL_NAME
  })

  output$user_id <- renderText({
    req(r$email)
    sprintf("Hello %s", r$email)
  })
}

shinyApp(ui, server)

Testing this functionality, particularly in Continuous Integration (CI) environments, can be challenging. In our use case, we’d love to have something like this:

test_that("app server", {

  # Tweaking the session here

  testServer(app_server, {
    # Waiting for the session to be fired up
    session$elapse(1)

    expect_equal(
      r$email,
      "myemail@company.com"
    )
  })
})

But Authentication headers like HTTP_X_MS_CLIENT_PRINCIPAL_NAME are absent during automated tests, so we need a way to simulate their presence. {shiny} provides the MockShinySession class for testing, but it doesn’t natively simulate a realistic session$request object. Let’s explore how to work around this limitation.

Overriding `session$request`

We first attempt to directly modify session$request, but it doesn't work:

> session <- MockShinySession$new()
> session$request
<environment: 0x13a032600>
Warning message:
In (function (value)  :
  session$request doesn't currently simulate a realistic request on MockShinySession

Ok, maybe we can assign a new entry here?

> session$request$HTTP_X_MS_CLIENT_PRINCIPAL_NAME <- "test"
Error in (function (value)  : session$request can't be assigned to
In addition: Warning message:
In (function (value)  :
  session$request doesn't currently simulate a realistic request on MockShinySession

Ouch, it doesn't work, it can't be assigned to. But let's continue our exploration. What is session?

> class(session)
[1] "MockShinySession" "R6"
> class(session$request)
[1] "environment"

As we can see, it's an R6 object, an instance of the MockShinySession class, and session$request an env. What we want is being able to access, in our app, to session$request$HTTP_X_MS_CLIENT_PRINCIPAL_NAME. Maybe we could override request? request is contained in the active field of the R6 class:

> MockShinySession$active
# [...]

$request
function (value)
{
    if (!missing(value)) {
        stop("session$request can't be assigned to")
    }
    warning("session$request doesn't currently simulate a realistic request on MockShinySession")
    new.env(parent = emptyenv())
}
<bytecode: 0x11f25d8a8>
<environment: namespace:shiny

To override the request object, we can use the set() method of the R6 class. Here’s how we redefine the behavior:

MockShinySession$set(
    "active",
    "request",
    function(value) {
      return(
        list(
          "HTTP_X_MS_CLIENT_PRINCIPAL_NAME" = "myemail@company.com"
        )
      )
    },
    overwrite = TRUE
  )

Now, the session behaves as expected:

> session <- MockShinySession$new()
> session$request
$HTTP_X_MS_CLIENT_PRINCIPAL_NAME
[1] "myemail@company.com

Writing the Test

With the overridden request, we can now write a functional test:

test_that("app server", {
  MockShinySession$set(
    "active",
    "request",
    function(value) {
      return(
        list(
          "HTTP_X_MS_CLIENT_PRINCIPAL_NAME" = "myemail@company.com"
        )
      )
    },
    overwrite = TRUE
  )

  testServer(app_server, {
    # Waiting for the session to be fired up
    session$elapse(1)

    expect_equal(
      r$email,
      "myemail@company.com"
    )
  })
})

Cleaning Up After Tests

But, just one more thing: we need to clean our test so that the session object stays the same after our test. For this, we'll use on.exit to restore the old behavior:

test_that("app server", {
  old_request <- MockShinySession$active$request
  on.exit({
    MockShinySession$set(
      "active",
      "request",
      old_request,
      overwrite = TRUE
    )
  })
  MockShinySession$set(
    "active",
    "request",
    function(value) {
      return(
        list(
          "HTTP_X_MS_CLIENT_PRINCIPAL_NAME" = "myemail@company.com"
        )
      )
    },
    overwrite = TRUE
  )

  testServer(app_server, {
    # Waiting for the session to be fired up
    session$elapse(1)

    expect_equal(
      r$email,
      "myemail@company.com"
    )
  })
})

This setup ensures that our tests remain isolated and reliable, even in CI environments. By leveraging R6’s flexibility, we can fully control and mock session$request to test authentication-dependent logic. If you want to dig more into the details, you can visit this repo, where you'll find a reproducible example!

Do you need help with testing your apps?

Still unsure how to implement a good testing strategy for your app? Let’s chat!

Introducing Rlinguo, a native mobile app that runs R

Colin Fay — Mon, 23 Dec 2024 15:00:00 +0000

TL;DR

Rlinguo is a groundbreaking mobile application that uses R in the backend to handle its business logic.

Available for download now:

Yes, you read that right: the future is now, and we can run R on your mobile phone.

Once upon a time…

I've been building web apps with R for quite some time now—probably longer than I’d like to admit. At ThinkR, we’ve been developing {shiny} apps for a while, ranging from small widgets for researchers to large-scale dashboards used by thousands of bankers. But web development hasn’t been limited to {shiny} for us. We’ve also created REST APIs, flex/quarto dashboards, and dynamic documents generated with R. Over the past few years, we’ve explored various ways to deliver data products to our clients, primarily using {shiny} apps but also through other formats. Amid this diversity of formats, one platform has consistently evaded us: mobile. Yes, {shiny} apps can be optimized for mobile use, and JavaScript dashboards can display nicely on a phone. However, they’re not native mobile apps. They can’t be installed from an app store, don’t have access to your phone’s APIs, and making them work offline is a significant challenge.

`{shiny}` on my mobile

"{shiny} on my mobile" has been a recurring request from our clients for some time now. And it’s easy to see why: you might have a fantastic model or efficient data wrangling process, and you need people to access it on their smartphones—even in the middle of the forest with no cell connection (yes, that was an actual request from one of our clients). Yet, our response has always been the same: "Sorry, that’s not possible." {shiny} relies on a server, and since R can’t run directly on a phone, a remote server is required. That means an internet connection is necessary. And no, you can’t publish it to an app store. In recent months, however, something new has emerged—a development that R enthusiasts have been dreaming of for years: R, compiled for WebAssembly. If you’re unfamiliar with WebAssembly, think of it as a way to run R inside JavaScript. If you’re already acquainted with it, you’ll know it’s a bit more nuanced than that, but I’ll spare the technical details for now. This new tool is called webR.

The “What If” Moment

I’ve been experimenting with webR for several months now, primarily from a Node.js perspective. During this time, I developed two tools, webrcli and spidyr, specifically designed for projects that integrate R and Node.js. While these tools have the potential to revolutionize app development, they are still tied to a remote web-based approach: R runs on a server elsewhere, inside a Node.js process. This means there’s still no offline functionality, no app store deployment, and no native mobile features. This summer, we received yet another request for "{shiny} as a native mobile app." As we were about to deliver our usual response, I had a "What if?" moment. What if we could use webR in a JavaScript framework that creates mobile apps? With this idea, we developed a proof of concept (PoC): a mobile app capable of running one line of R code. A fully functional app of this nature had never been done before, but we believed it was achievable. Unfortunately, the client chose not to pursue the project. Building a mobile app requires skills beyond those typical of {shiny} developers, and they needed software they could maintain independently. Nevertheless, having created a PoC, we decided to go all in and develop a complete application—something user-friendly, slightly playful, and distributable via app stores, to prove that this could indeed be done. Over the next few weeks, the team at ThinkR brought this vision to life. Margot shaped the project and wrote the concept paper (more on that in a future blog post), Arthur crafted the app mock-up using Excalidraw and Figma, showcasing his exceptional UI skills, and I (Colin) focused on backend development (largely because my design skills are... lacking, and frontend work is not my forte). The result of this exciting journey is Rlinguo, a native mobile app that runs R. Yes, you read that correctly—R can now run on your phone, and this app is a proof of concept turned reality.

The Shape of R to Come

Rlinguo is a playful and quirky game that challenges users to identify which package a given R function belongs to. Admittedly, this may not be the most practical skill—knowing that mean() is in {base} while median() resides in {stats} won’t necessarily make you a better developer (although, I must admit, it still irks me that they’re in separate packages). The primary goal of this app was to serve as proof that native mobile apps leveraging the power of R are not only possible but can also pave the way for broader applications. The app demonstrates several groundbreaking capabilities:

R at your fingertips, straight from your pocket: The app showcases R’s functionality in multiple ways, such as performing a sample() from a list of installed packages and another from exported functions. On the About page, users can even execute a runif() to confirm that Rlinguo truly runs R behind the scenes.
Offline Functionality: Designed to work without an internet connection, Rlinguo makes it possible to deploy R models on smartphones—even for users in remote, disconnected locations, such as deep in the woods
Enabling a Real Data-Science Workflow: Rlinguo mirrors a typical data-science pipeline. Data is stored locally in an SQL database, queried, processed in R (via counting and pivoting), and then returned to JavaScript for visualization. The result? A user-friendly display of your game performance.
Integrating R Packages: The app uses R packages creatively. When you guess a function correctly, you’re rewarded with a praise::praise() message; if you’re wrong, a custom function provides feedback (though I’ve yet to find a package dedicated to offering words of encouragement).
Delivering a Great User Experience: Rlinguo prioritizes usability, addressing a common shortfall in tools incorporating R. The goal was to craft an app that feels polished and enjoyable to use.
Deploying R-Based Apps to Stores: Rlinguo demonstrates that it’s entirely feasible to create a mobile app powered by R and make it available for download on major app stores..

Rlinguo is available now on the Apple Store and Google Play. We see this as the first step in a new wave of innovation, and to borrow a phrase, "baby, you never felt this good."

You've got a use case?

The idea of running R on a phone has been a popular request from our clients — there are countless possibilities. Is this something you need? Do you have a fantastic use case for R on your phone? Perfect, let's chat!

Notes:

(*) We identified two applications on the App Store that incorporate webR. However, these appear to function primarily as wrappers around JavaScript integrated development environments (IDEs) and do not address our specific objective: creating a "real-life" mobile application that utilizes R for its core business logic.

webrcli & spidyr: What’s new

Colin Fay — Fri, 14 Jun 2024 00:00:00 +0000

Find all my posts about webR here.

Note: the first post of this series explaining roughly what webR is, I won’t introduce it again here.

In this blogpost, you’ll find some updates about some recent changes made in webrcli & spidyr.

What webrcli & spidy wants to achieve

In a nutshell, the goal of webrcli & spidyr is to provide an interface for bringing your R functions inside a NodeJS runtime.

Think of it as something that wants to achieve the following interation:

An R dev writes, builds and shares a package called {rfuns}, which contains a function called hello_world()
A NodeJS dev takes this package, can call it with rfuns.hello_world(), access and manipulate the results with JavaScript.

An important note

With webR & the tools described below, there is no R session running while the NodeJS app is running. spidyr imports an R function converted to a native JS.rfuns.hello_world does not call an R session running somewhere else.

That also means that if you build a container containing the app, you don’t need to install R in it. It’s a NodeJS app.

Below are some updates recently made to both webrcli & spidyr.

Making the template code smaller

When drafting the first code of webrcli & spidyr, I focused on providing a minimal template for an index.js loading your R functions, contained in a subdirectory inside your node project.

This worked, but might have been a bit complex to grasp. Provided you don’t change the default names of the folders (in that case you’ll need to modify the default parameters), the new default index.js look like this:

const {
  initSpidyr,
  mountLocalPackage
} = require('spidyr');

(async () => {

  await initSpidyr()

  const rfuns = await mountLocalPackage("./rfuns");

  const hw = await rfuns.hello_world()

  console.log(hw.values);

  console.log("✅ Everything is ready!");
})();

This minimal template makes it easier to read and understand what the code does. Note that you’ll get this template if you run webrcli init myproject.

Let’s decompose what this code does :

await initSpidyr() will create an object called spidyr_webR stored as globalThis.spidyr_webR, init() it (as in webr.init()), and load the packages contained in the ./webr_packages folder inside your project dir, the one containing the packages installed with webrcli install.
const rfuns = await mountLocalPackage("./rfuns"); will take the R package contained in ./rfuns, load it into the webR instance (spidyr_webR), recursively get all the exported functions from your package, and store them inside the rfuns object.
const hw = await rfuns.hello_world() will call the hello_world() function from {rfuns}. The values are then console.loged.

`new Library()` is now `library()`

Calling a package is now const spongebob = library("spongebob") instead of the previous approach which was const spongebob = new Library("spongebob");await spongebob.load(globalThis.webR);

Better installation from package.json & `DESCRIPTION`

The previous version of webrcli did not stored nor re-installed the R package installed in the project, meaning that if you git cloned a project, you had to guess what R packages to re-install.

Now, a project initiated with webrcli will store all the packages installed with webrcli install inside the package.json file. Then, when running the npm install command, the R packages will also be redownloaded.

You can also installed a series of packages based on a DESCRIPTION file with webrcli installFromDesc.

Other tools

shareEnv allows to copy one, several or all env var from node to the webR instance.

Workflow summary

webrcli init myproject will create a project skeleton at ./myproject
webrcli install spongebob will download the version of {spongebob} compiled for webR, and add it to ./webr_packages.
You can then modify you R code with your own functions.
Call your R function from rfuns.xyz() in Node
Run with npm start

Please do give it a shot

I think I’ve been the only user for now 😅, so please do try these and let me know what you think.

Future works

Here are what I plan on working in the upcoming weeks:

webrcli init should allow for specifying a webr version from npm issue 17
initSpidyr should allow to pass params to webR.init() issue 12
Allow to install from r-universe, something like webrcli installFromRUniverse issue 16. Probably in the future it will be merged to webrcli install and we’ll have a way to guess what the user wants based on the input.
A small converter from R named list to JS objects Issue 3
More documentation, and probably documenting the conversion of an existing {shiny} app to a spidyr based one.

webrcli & spidyr: A starter pack for building NodeJS projects with webR inside

Colin Fay — Wed, 06 Mar 2024 00:00:00 +0000

This post is the sixth one of a series of post about webR:

Using webR in an Express JS REST API
The Old Faithful Geyser Data shiny app with webR, Bootstrap & ExpressJS
Preloading your R packages in webR in an Express JS API
Using my own R functions in webR in an Express JS API, and thoughts on building web apps with Node & webR
Rethinking webR package & functions preloading with webR 0.2.2
A starter pack for building NodeJS projects with webR inside

Note: the first post of this series explaining roughly what webR is, I won’t introduce it again here.

Again, a new project?

Most of the previous posts have been about poking around with webR inside NodeJS, but nothing stable on how to build an app.

That’s something I’m trying to change. Based on the experience gathered building apps in R (in {shiny} with {golem}) and in JavaScript, I’ve tried to come up with a solution for a starter kit for building what I think will be the perfect skeleton for webR/NodeJS project.

The idea is to have one project with inside:

One R package
One JS app
A tool that brings the R package into the NodeJS app and allow to use the R functions inside JavaScript

This skeleton needed a toolkit to perform the following task:

1️⃣ From a cli point of view:

init the project with a specific skeleton
allow to install packages from the webrCRAN to a local webr library, in the spirit of previous post.
allow to install the deps from a DESCRIPTION file

2️⃣ From a NodeJS point of view:

Load the packages downloaded in a local webr library
Create an interface to load and manipulate the functions from a package downloaded from CRAN, and use them in JS.
Same but with a local package folder

For the function manipulation mechanism, I wanted something that would allow to think in terms of pkg::fun() but in JavaScript, which would translate as pkg.fun() in your Node app. Something I had implemented in another old NodeJS module called hordes, which I can now safely deprecate in favor of the following tools.

The reasoning being: the R dev writes a standalone package that exports an xyz function, then the web team can load this package, and launch xyz() without writing any R code.

webrcli & spidyr

So, here comes webrcli and spidyr.

❗️ Please note that these tools are work in progress and has been used very few times and only for example apps, will need a lot of bug fixes and documentation, so if ever you plan on using it be indulgent, and report any bug or feature request ❗️

Project init

These packages are made to be used together, and here is an example of how to use them:

# Global installing webrcli
npm install -g webrcli
# Init a webrcli project
cd /tmp
webrcli init webrspongebob


👉 Initializing project ----
(This may take some time, please be patient)
👉 Copying template ----
👉 Installing {pkgload} ----
✅ {pkgload} downloaded and extracted ----

✅ {cli} downloaded and extracted ----

✅ {crayon} downloaded and extracted ----

✅ {desc} downloaded and extracted ----

✅ {fs} downloaded and extracted ----

✅ {glue} downloaded and extracted ----

✅ {pkgbuild} downloaded and extracted ----

✅ {rlang} downloaded and extracted ----

✅ {rprojroot} downloaded and extracted ----

✅ {withr} downloaded and extracted ----

✅ {R6} downloaded and extracted ----

✅ {callr} downloaded and extracted ----

✅ {processx} downloaded and extracted ----

✅ {ps} downloaded and extracted ----

The project is now created, let’s move into it.

cd ./webrspongebob
tree -L 1


.
├── index.js
├── node_modules
├── package-lock.json
├── package.json
├── rfuns
└── webr_packages

4 directories, 3 files

Here is how it’s organized:

index.js is the main file for the app, node_modules the standard folder for the node deps
package-lock.json / package.json are standard Node metadata files
rfuns contains the R package that will be added to the NodeJS app
webr_packages contains the R dependencies

We can launch the app with node index.js and it will output:

node index.js
👉 Loading WebR ----
👉 Loading R packages ----
ℹ Loading rfuns
['Hello, world!']
✅ Everything is ready

Let’s dive a bit inside the index.js:

const path = require('path');
const { WebR } = require('webr');
const { loadPackages, LibraryFromLocalFolder } = require('spidyr');

const rfuns = new LibraryFromLocalFolder("rfuns");

(async () => {

  console.log("👉 Loading WebR ----");
  globalThis.webR = new WebR();
  await globalThis.webR.init();

  console.log("👉 Loading R packages ----");

  await loadPackages(
    globalThis.webR,
    path.join(__dirname, 'webr_packages')
  )

  await rfuns.mountAndLoad(
    globalThis.webR,
    path.join(__dirname, 'rfuns')
  );

  const hw = await rfuns.hello_world()

  console.log(hw.values);

  console.log("✅ Everything is ready!");

})();

Here are the bits that are specific to a spidyr project:

const rfuns = new LibraryFromLocalFolder("rfuns");

This function will take a local folder containing an R package, and load the functions from this package into the rfuns object. Here, for example, our R package contains one R function, hello_world(), it will then be available in NodeJS as rfuns.hello_world() once the mountAndLoad function is called.

await loadPackages(
  globalThis.webR,
  path.join(__dirname, 'webr_packages')
)

await rfuns.mountAndLoad(
  globalThis.webR,
  path.join(__dirname, 'rfuns')
);

The first bit loads the webr_packages folder, containing all the R dependencies, then the second bit mount the local folder into the webR file system and load the functions.

Finally, const hw = await rfuns.hello_world() calls the function from the R package, and its value is console.loged just after that.

With a CRAN package

And now, how do I load a CRAN package? For example, let’s say I want to use {spongebob} in my app?

First, let’s install {spongebob} via webrcli :

webrcli install spongebob


✅ {spongebob} downloaded and extracted ----

Then, let’s update our index.js:

const path = require('path');
const { WebR } = require('webr');
const { loadPackages, LibraryFromLocalFolder, Library } = require('spidyr');

const rfuns = new LibraryFromLocalFolder("rfuns");
const spongebob = new Library("spongebob");

(async () => {

  console.log("👉 Loading WebR ----");
  globalThis.webR = new WebR();
  await globalThis.webR.init();

  console.log("👉 Loading R packages ----");

  await loadPackages(
    globalThis.webR,
    path.join(__dirname, 'webr_packages')
  )

  await rfuns.mountAndLoad(
    globalThis.webR,
    path.join(__dirname, 'rfuns')
  );

  await spongebob.load(
    globalThis.webR
  );

  const hw = await rfuns.hello_world()

  console.log(hw.values);

  const said = await spongebob.tospongebob("hello from spongebob")

  console.log(said.values)

  console.log("✅ Everything is ready!");;

})();

Here:

const spongebob = new Library("spongebob"); will create a lib, ready to mount a package
await spongebob.load(globalThis.webR) will load all the functions from the {spongebob} package
const said = await spongebob.tospongebob("hello from spongebob") will run the spongebob::tospongebob() R function

Then :

node index.js


👉 Loading WebR ----
👉 Loading R packages ----
ℹ Loading rfuns
['Hello, world!']
['helLo fROm spONgEbOb']
✅ Everything is ready!

Some random notes

The webrspongebob example is available at https://github.com/ColinFay/webrspongebob

About the current implementation

Right now I’m not really sure how this handles the infix function like %>% for example. That being said, this might not be a function you’d want to use in the current way of building things.
The exported functions are read via getNamespaceExports("pkg") and getExportedValue("pkg", "fun"), if ever this is not the perfect way to do this in base R but I’ll be happy to find some other ways to do this.
webrcli and spidyr both encapsulate code run in webR, especially webrcli which has an R script which is sourced and run in a webR instance.

Future

Please do try these tools. I would be very happy to have your feedback on the philosophy, and on the general workflow.

If ever you find a bug or have an idea, feel free to open issues at :

Rethinking packages & functions preloading in webR 0.2.2

Colin Fay — Tue, 21 Nov 2023 00:00:00 +0000

This post is the fifth one of a series of post about webR:

Using webR in an Express JS REST API
The Old Faithful Geyser Data shiny app with webR, Bootstrap & ExpressJS
Preloading your R packages in webR in an Express JS API
Using my own R functions in webR in an Express JS API, and thoughts on building web apps with Node & webR
Rethinking webR package & functions preloading with webR 0.2.2

Note: the first post of this series explaining roughly what webR is, I won’t introduce it again here.

When I wrote my blogpost about Preloading your R packages in webR in an Express JS API, I mentioned that there was no native way to preload things in the webR filesystem — meaning that you had to reinstall all the R packages whenever the app was launched (and reported it in a Github issue). This also meant that I couldn’t easily take my own functions and run them in the webR environment, as described in Using my own R functions in webR in an Express JS API, and thoughts on building web apps with Node & webR. These needs made me write the webrtools NodeJS package, to do just that:

Download the R packages compiled for webR, to a local folder
Bundle them in the webR lib (by reading the folder tree and reproducing it in webR filesystem)
Load your local package to access its functions in webR

The last webR version now exposes Emscripten’s FS.mount, so this makes the process easier, as there is now no need to read the file tree and recreate it: the folder from the server can be mounted into webR filesystem.

That implied some rework (now integrated to webrtools) :

Use webR.FS.mkdir to create a local lib: the module can’t mount the package library into the libPath as is, because it would overwrite the already bundled library (in other words, if you mount into a folder that is not empty, the content from the server overwrites the content from webR).
Use webR.FS.mount to mount the local directory into the newly created library
Ensure that this new library is in the libPath()

Here is the new code for loadPackages, bundled into webrtools:

async function loadPackages(webR, dirPath, libName = "webr_packages") {
  // Create a custom lib so that we don't have to worry about
  // overwriting any packages that are already installed.
  await webR.FS.mkdir(`/usr/lib/R/${libName}`)
  // Mount the custom lib
  await webR.FS.mount("NODEFS", { root: dirPath }, `/usr/lib/R/${libName}`);
  // Add the custom lib to the R search path
  await webR.evalR(`.libPaths(c('/usr/lib/R/${libName}', .libPaths()))`);
}

I’ve also decided to deprecate the loadFolder function, as it is now native with webR.FS.mkdir + webR.FS.mount.

So here is a rewrite of the app from Using my own R functions in webR in an Express JS API, and thoughts on building web apps with Node & webR.

const app = require('express')()
const path = require('path');
const { loadPackages } = require('webrtools');
const { WebR } = require('webr');

(async () => {
  globalThis.webR = new WebR();
  await globalThis.webR.init();

  console.log("🚀 webR is ready 🚀");

  await loadPackages(
    globalThis.webR,
    path.join(__dirname, 'webr_packages')
  )

  await globalThis.webR.FS.mkdir("/home/rfuns")

  await globalThis.webR.FS.mount(
    "NODEFS",
    {
      root: path.join(__dirname, 'rfuns')
    },
    "/home/rfuns"
  )

  console.log("📦 Packages written to webR 📦");

  // see https://github.com/r-wasm/webr/issues/292
  await globalThis.webR.evalR("options(expressions=1000)")
  await globalThis.webR.evalR("pkgload::load_all('/home/rfuns')");

  app.listen(3000, '0.0.0.0', () => {
    console.log('http://localhost:3000')
  })

})();

app.get('/', async (req, res) => {
  let result = await globalThis.webR.evalR(
    'unique_species()'
  );
  try {
    let js_res = await result.toJs()
    res.send(js_res.values)
  } finally {
    webR.destroy(result);
  }

})

app.get('/:n', async (req, res) => {
  let result = await globalThis.webR.evalR(
    'star_wars_by_species(n)',
    { env: { n: req.params.n } }
  );
  try {
    const result_js = await result.toJs();
    res.send(result_js)
  } finally {
    webR.destroy(result);
  }
});

Full code is at ColinFay/webr-examples/.

Further exploration to be done: webR now bundles a way to package a file system in a file, which can then be downloaded and mounted into the runtime, as described here. This might come handy for our current structure, but I’ll have to explore it a bit more.

Using my own R functions in webR in an Express JS API, and thoughts on building web apps with Node & webR

Colin Fay — Tue, 17 Oct 2023 00:00:00 +0000

This post is the fourth one of a series of post about webR:

Using webR in an Express JS REST API
The Old Faithful Geyser Data shiny app with webR, Bootstrap & ExpressJS
Preloading your R packages in webR in an Express JS API
Using my own R functions in webR in an Express JS API, and thoughts on building web apps with Node & webR

Note: the first post of this series explaining roughly what webR is, I won’t introduce it again here.

The problem

Ok, so now that we have our webR / NodeJS machinery up and running, let’s try something more interesting: use our own R functions inside webR.

How can I do that?

There are at least three ways I could think of:

1️⃣ Writing a function inside the JS code to define a function, something like :

await globalThis.webR.evalR("my_fun <- function(x){...}");

But that doesn’t check what I would expect from something I’ll use in prod and I’m pretty sure you don’t need me to detail why 😅

❌ Well organized
❌ Documented
❌ Tested
❌ Safely installable

2️⃣ Simply create an R script and source it. Something like:

const fs = require('fs');
const path = require('path');
const script = path.join(__dirname, 'script.R')
const data = fs.readFileSync(script);
await globalThis.webR.FS.writeFile(
  "/home/web_user/script.R",
  data
);
await globalThis.webR.evalR("source('/home/web_user/script.R')");

That’s a bit better, we can at least organize our code in a script and it will be:

✅ Well organized (more or less)
✅ Documented (more or less)
❌ Tested
❌ Safely installable

3️⃣ I bet you saw me coming, the best way let’s put stuff into an R package, so that we can check all the boxes.

✅ Well organized
✅ Documented
✅ Tested
✅ Safely installable

Jeroen has written a Docker image to compile an R package to WASM, but I was looking for something that wouldn’t involve compiling via a docker container every time I make a change on my R package (even if that does sound appealing, I’m pretty sure this wouldn’t make for a seamless workflow).

So here is what I’m thinking should be a well structured NodeJS / WebR app:

Putting all the web stuff inside the NodeJS app, because, well, NodeJS is really good at doing that.
Putting all the “business logic”, data-crunching, modeling stuff (and everything R is really good at) into an R package.
load webR, write my R package to the webR file system, and pkgload::load_all() it into webR.

That way, I can enjoy the best of both worlds:

NodeJS is really good at doing web related things, and there are plenty of ways to test and deploy the code.
And same goes for the R package: if you’re reading my blog I’m pretty sure I don’t need to convince you of why packages are the perfect tool for sharing production code.

The how

Let’s start by creating our project:

mkdir webr-preload-funs
cd webr-preload-funs
npm init -y
touch index.js
npm install express webr
R -e "usethis::create_package('rfuns', rstudio = FALSE)"

Let’s now create a simple function :

> usethis::use_r("sw")


#' @title Star Wars by Species
#' @description Return a tibble of Star Wars characters by species
#' @import dplyr
#' @export
#' @param species character
#' @return tibble
#' @examples
#' star_wars_by_species("Human")
#' star_wars_by_species("Droid")
#' star_wars_by_species("Wookiee")
#' star_wars_by_species("Rodian")
star_wars_by_species <- function(species){
  dplyr::starwars |>
    filter(species == )
}

We can now add {dplyr} and {pkgload} to our DESCRIPTION (we’ll need {pkgload} to load_all() the package).

usethis::use_package("dplyr")
usethis::use_package("pkgload")
devtools::document()

Now that we have a package skeleton, we’ll have to upload it to webR.

As described in the previous post, I’ve started a webrtools NodeJS module, which will contains function to play with webR. Before this post, it had one function, loadPackages, that was used to build a webR dependency library (see Preloading your R packages in webR in an Express JS API for more info).

We’ll need to add two features :

Install deps from DESCRIPTION (not just a package name), so a wrapper around the Rscript ./node_modules/webrtools/r/install.R dplyr from before
Copy the package folder in NodeJS, so a more generic version of loadPackages, that can load any folder to the webR filesystem.

First, in R, we’ll need to read the DESCRIPTION and build the lib:

download_packs_and_deps_from_desc <- function (
  description,
  path_to_installation = "./webr_packages"
)
{
    if (!file.exists(description)) {
        stop("DESCRIPTION file not found")
    }
    deps <- desc::desc_get_deps(description)
    for (pak in deps$package) {
        webrtools::download_packs_and_deps(pak, path_to_installation = path_to_installation)
    }
}

Note: the code of webrtools::download_packs_and_deps() is a wrapper around the R code described in Preloading your R packages in webR in an Express JS API

And in Node, we’ll rework our loadPackages and split it into two functions — one to load into any folder, and one to load into the package library:

async function loadFolder(webR, dirPath, outputdir = "/usr/lib/R/library") {
  const files = getDirectoryTree(
    dirPath
  )
  for await (const file of files) {
    if (file.type === 'directory') {
      await globalThis.webR.FS.mkdir(
        `${outputdir}/${file.path}`,
      );
    } else {
      const data = fs.readFileSync(`${dirPath}/${file.path}`);
      await globalThis.webR.FS.writeFile(
        `${outputdir}/${file.path}`,
        data
      );
    }
  }
}

async function loadPackages(webR, dirPath) {
  await loadFolder(webR, dirPath, outputdir = "/usr/lib/R/library");
}

The end app

We now have everything we need!

npm i webrtools@0.0.2
Rscript ./node_modules/webrtools/r/install_from_desc.R $(pwd)/rfuns/DESCRIPTION

And now, to our index.js

const app = require('express')()
const path = require('path');
const { loadPackages, loadFolder } = require('webrtools');
const { WebR } = require('webr');

(async () => {
  globalThis.webR = new WebR();
  await globalThis.webR.init();

  console.log("🚀 webR is ready 🚀");

  await loadPackages(
    globalThis.webR,
    path.join(__dirname, 'webr_packages')
  )

  await loadFolder(
    globalThis.webR,
    path.join(__dirname, 'rfuns'),
    "/home/web_user"
  )

  console.log("📦 Packages written to webR 📦");

  // see https://github.com/r-wasm/webr/issues/292
  await globalThis.webR.evalR("options(expressions=1000)")
  await globalThis.webR.evalR("pkgload::load_all('/home/web_user')");

  app.listen(3000, '0.0.0.0', () => {
    console.log('http://localhost:3000')
  })

})();

app.get('/', async (req, res) => {
  let result = await globalThis.webR.evalR(
    'unique(dplyr::starwars$species)'
  );
  let js_res = await result.toJs()
  res.send(js_res.values)
})

app.get('/:n', async (req, res) => {
  let result = await globalThis.webR.evalR(
    'star_wars_by_species(n)',
    { env: { n: req.params.n } }
    );
  try {
    const result_js = await result.toJs();
    res.send(result_js)
  } finally {
    webR.destroy(result);
  }
});

Let’s now try from another terminal:

curl http://localhost:3000


["Human","Droid","Wookiee","Rodian","Hutt","Yoda's species","Trandoshan","Mon Calamari","Ewok","Sullustan","Neimodian","Gungan",null,"Toydarian","Dug","Zabrak","Twi'lek","Vulptereen","Xexto","Toong","Cerean","Nautolan","Tholothian","Iktotchi","Quermian","Kel Dor","Chagrian","Geonosian","Mirialan","Clawdite","Besalisk","Kaminoan","Aleena","Skakoan","Muun","Togruta","Kaleesh","Pau'an"]


curl http://localhost:3000/Rodian


{"type":"list","names":["name","height","mass","hair_color","skin_color","eye_color","birth_year","sex","gender","homeworld","species","films","vehicles","starships"],"values":[{"type":"character","names":null,"values":["Greedo"]},{"type":"integer","names":null,"values":[173]},{"type":"double","names":null,"values":[74]},{"type":"character","names":null,"values":[null]},{"type":"character","names":null,"values":["green"]},{"type":"character","names":null,"values":["black"]},{"type":"double","names":null,"values":[44]},{"type":"character","names":null,"values":["male"]},{"type":"character","names":null,"values":["masculine"]},{"type":"character","names":null,"values":["Rodia"]},{"type":"character","names":null,"values":["Rodian"]},{"type":"list","names":null,"values":[{"type":"character","names":null,"values":["A New Hope"]}]},{"type":"list","names":null,"values":[{"type":"character","names":null,"values":[]}]},{"type":"list","names":null,"values":[{"type":"character","names":null,"values":[]}]}]}

Yeay 🎉 .

You can find the code here, and see it live at srv.colinfay.me/webr-preload-funs/.

You can also try it with

docker run -it -p 3000:3000 colinfay/webr-preload-funs

Preloading your R packages in webR in an Express JS API

Colin Fay — Mon, 04 Sep 2023 00:00:00 +0000

This post is the third one of a series of post about webR:

Using webR in an Express JS REST API
The Old Faithful Geyser Data shiny app with webR, Bootstrap & ExpressJS
Preloading your R packages in webR in an Express JS API

Note: the first post of this series explaining roughly what webR is, I won’t introduce it again here.

Note 2 (or Disclaimer): this is kind of a long post that get into the weeds of package installation and library structure

The problem

In the two last posts, I’ve been deploying two REST APIs that use functions from the base distribution of R:

tools::toTitleCase() in express-webr-hello-world
cut() & faithful for express-webr-old-faithful

One of the strengths of the R ecosystem is its extensive list of contributed packages: a wide collection of code written by R nerds from all over the world. It would be unfortunate to use webR to create a web applications or API without taking advantage of this extensive library of packages: R wouldn’t be R without its packages.

Yet, bringing packages to webR is not that simple: it’s a distribution of R compiled for WASM, and you can’t just install a package from any CRAN repo.

Thankfully we don’t need to scratch our head too much anymore: the webR team has precompiled more than 18000 (at the time of writing this post) packages that we can install by simply calling webr::install() inside the webR instance.

That’s awesome, but there’s an issue with that: you can only install it from the webR JavaScript object As far as I know, there is no way to download them beforehand and upload them to the webR filesystem, as I reported in this issue. (Note: at the time of writing these lines, there has been some discussion on the GH issue, but there is at the moment no native, efficient way to do what I want to do 😅 — there might be one though if you’re reading this post in the future).

That means that every time the API is launched, webR has to download and reinstall the packages, as shown by this simple example:

const app = require('express')()
const { WebR } = require('webr');

(async () => {
  globalThis.webR = new WebR();
  await globalThis.webR.init();
  await globalThis.webR.installPackages(['attempt'])
  app.listen(3000, '0.0.0.0', () => {
    console.log('http://localhost:3000')
  })
})();

app.get('/', async (req, res) => {
  res.send("hello")
});

If I launch the app twice, the packages are downloaded twice 👇

$ node index.js
webR is ready
Downloading webR package: rlang

Downloading webR package: attempt

http://localhost:3000

$ node index.js
webR is ready
Downloading webR package: rlang

Downloading webR package: attempt

http://localhost:3000

This approach is definitely suboptimal: imagine having to reinstall packages every time you source() an R script, instead of benefiting from pre-installed packages in your lib. That’s kind of what is happening right now in this webR example.

Sure, it works, but it means that:

I’m downloading the (same) packages every time the app is launched, which feels weird
If I have a lot of packages, it would take a lot of time every time
I can’t deploy this on a service that has no access to internet
I can’t build my app just as any other language does: by relying on pre-downloaded /pre-installed modules
I can’t benefit from caching the package installation in a Docker layer

So, faced with something I needed but didn’t find, I behave like any other software engineer: I built my own solution to pre-load R packages in webR, without having to re-download them every time. The rest of this blogpost explains this process.

Preloading R packages in my ExpressJS/webR API

Building the R lib

If you look at the internal of the folder on your computer that contains your packages (which is at .libPaths()), you’ll see that it’s a series of folders with a bunch of files. Whenever you load R and call library(), it looks there to load the code.

Inside these folders (you can have several lib), you’ll find unzipped archives, downloaded from a repository (or installed from git/developed locally). These packages, at installation time, possibly performed some build and compilation, depending on the package and the OS you’re on. I won’t go into the details of this here, as the packages for webR have been pre-compiled, meaning that you simply have to download the tar file, and unarchive it at the correct place. In other words, the content of this archive is exactly what you’ll find in the library folder, and there is nothing to compile/install again.

Keeping this in mind, the first thing to do then is to create a folder that will contain all the folders and files necessary to load a package. In other words, if I want to use {dplyr}, I need to find a way to download the pre-compiled version of {dplyr} and all its dependencies from https://repo.r-wasm.org/, and untar them into the folder I wanna use inside my project.

R has a weird way of managing packages, and if ever you want to hear me monologue, catch me at a conference and ask me to talk about this for 30 minutes (just kidding, it will probably last one hour). I won’t go into details about how it works and why it’s weird, but here are some issues I’ve been facing for my current task:

By default R will not reinstall the dependencies if they are already installed on the machine. Which is great, but not in my case (I want the destination folder to contain the entire dependency tree). In other words, if I do install.packages("dplyr", lib="webr_packages"), R will only download the packages that are not already on my computer.
There is no (sane) way to avoid the default library. So basically unless you’re ready to write hacky code that will override the library environments and stuff, R will always look into the default lib. There is an old issue on the {withr} repo if you want to look into this, but basically the idea is that avoiding default lib is quite a challenge.
That means that if I try to do something as simple as install.packages("dplyr", lib="webr_packages", repos = "https://repo.r-wasm.org/"), it will only install {dplyr}, given that all the other dependencies already exist on my machine :

> setwd(tempdir())
> dir.create("webr_packages")
> install.packages("dplyr", lib="webr_packages", repos = "https://repo.r-wasm.org/")
# [...TRUNCATED]
* DONE (dplyr)
> list.files("webr_packages/")
[1] "dplyr"

I’ve tried the following:

Hacking .lib.loc as suggest by Andrie here, but it comes with a series of challenges like not having access to other packages and having to reinstall them and load them to the webR FileSystem which is not what I wanted.
Using {renv} but it feels a bit overkill for what I’m trying to do: download a bunch of zip files from the internet and unzip them in the correct folder. Also, {renv} failed to parse the dep tree in a NodeJS project for some reasons.

So I’ve ended up with my own R script, inspired by the internal of install.packages and webr::install (Read the source, Luke). I’ve first explored a version with {pak} and {pkgdepends}, but ended up using a base solution so that I don’t have to install anything in the Docker container.

#!/usr/bin/env Rscript

# This script only works if you pass it at least one arg
if (is.na(commandArgs(trailingOnly = TRUE)[1])) {
  stop("This script requires at least one argument")
} else {
  pk_to_install <- commandArgs(trailingOnly = TRUE)[1]
}

# If no second value, defaulting to webr_packages
if (is.na(commandArgs(trailingOnly = TRUE)[2])) {
  path_to_installation <- "./webr_packages"
} else {
  path_to_installation <- commandArgs(trailingOnly = TRUE)[2]
}

# Build the repos url
repos <- sprintf(
  "https://repo.r-wasm.org/bin/emscripten/contrib/%s.%s",
  # Get major R version
  R.version$major,
  substr(R.version$minor, 1, 1)
)

deps <- unique(
  unlist(
    use.names = FALSE,
    tools::package_dependencies(
      recursive = TRUE,
      pk_to_install
    )
  )
)

# Now the package list
pkg_deps <- c(
  pk_to_install,
  deps
)

# I don't want to re-download things so I'm listing all the already
# downloaded folders
already_installed <- list.files(
  path_to_installation
)

# Getting the list of available package
info <- utils::available.packages(contriburl = repos)

# Now we can install
for (pkg in pkg_deps){
  # Don't redownload things
  if (pkg %in% already_installed) {
    message("Package ", pkg, " already installed")
    next
  }
  # Not available: either it's not on the repo or it's included
  # in the base distribution of R
  if (!(pkg %in% info[, "Package"])) {
    message("Package {", pkg, "} not found in repo (unavailable or is base package)")
    next
  }
  message("Installing ", pkg)
  # Name of the targz, should be on the form of pkg_version.this.that.tgz
  targz_file <- paste0(pkg, "_", info[info[, "Package"] == pkg, "Version"], ".tgz")
  # Location of file on the repo
  url <- paste0(repos, "/", targz_file)
  # Download the file and untar the archive on the local lib
  tmp <- tempfile(fileext = ".tgz")
  download.file(url, destfile = tmp)
  untar(tmp, exdir = path_to_installation)
  unlink(tmp)
}

message("Done")

And now, let’s try this:

mkdir webr_packages
script tools/installR.r attempt
✔ Loading metadata database ... done
Installing attempt
trying URL 'https://repo.r-wasm.org/bin/emscripten/contrib/4.3/attempt_0.3.1.tgz'
Content type 'application/x-tar' length 93561 bytes (91 KB)
==================================================
downloaded 91 KB

Installing rlang
trying URL 'https://repo.r-wasm.org/bin/emscripten/contrib/4.3/rlang_1.1.1.tgz'
Content type 'application/x-tar' length 1062482 bytes (1.0 MB)
==================================================
downloaded 1.0 MB

Done

Rscript -e "fs::dir_tree('webr_packages')"
webr_packages
├── attempt
│ ├── DESCRIPTION
│ ├── INDEX
│ ├── LICENSE
│ ├── Meta
│ │ ├── Rd.rds
│ │ ├── features.rds
│ │ ├── hsearch.rds
│ │ ├── links.rds
│ │ ├── nsInfo.rds
│ │ ├── package.rds
│ │ └── vignette.rds
│ ├── NAMESPACE
│ ├── NEWS.md
│ ├── R
│ │ ├── attempt
│ │ ├── attempt.rdb
│ │ └── attempt.rdx
│ ├── doc
│ │ ├── a_intro_attempt.R
│ │ ├── a_intro_attempt.Rmd
│ │ ├── a_intro_attempt.html
│ │ ├── b_try_catch.R
│ │ ├── b_try_catch.Rmd
│ │ ├── b_try_catch.html
│ │ ├── c_adverbs.R
│ │ ├── c_adverbs.Rmd
│ │ ├── c_adverbs.html
│ │ ├── d_if.R
│ │ ├── d_if.Rmd
│ │ ├── d_if.html
│ │ ├── e_conditions.R
│ │ ├── e_conditions.Rmd
│ │ ├── e_conditions.html
│ │ └── index.html
│ ├── help
│ │ ├── AnIndex
│ │ ├── aliases.rds
│ │ ├── attempt.rdb
│ │ ├── attempt.rdx
│ │ └── paths.rds
│ └── html
│ ├── 00Index.html
│ └── R.css
└── rlang
    ├── DESCRIPTION
    ├── INDEX
    ├── LICENSE
    ├── Meta
    │ ├── Rd.rds
    │ ├── features.rds
    │ ├── hsearch.rds
    │ ├── links.rds
    │ ├── nsInfo.rds
    │ └── package.rds
    ├── NAMESPACE
    ├── NEWS.md
    ├── R
    │ ├── rlang
    │ ├── rlang.rdb
    │ └── rlang.rdx
    ├── backtrace-ver
    ├── help
    │ ├── AnIndex
    │ ├── aliases.rds
    │ ├── figures
    │ │ ├── lifecycle-archived.svg
    │ │ ├── lifecycle-defunct.svg
    │ │ ├── lifecycle-deprecated.svg
    │ │ ├── lifecycle-experimental.svg
    │ │ ├── lifecycle-maturing.svg
    │ │ ├── lifecycle-questioning.svg
    │ │ ├── lifecycle-retired.svg
    │ │ ├── lifecycle-soft-deprecated.svg
    │ │ ├── lifecycle-stable.svg
    │ │ ├── lifecycle-superseded.svg
    │ │ └── logo.png
    │ ├── paths.rds
    │ ├── rlang.rdb
    │ └── rlang.rdx
    ├── html
    │ ├── 00Index.html
    │ └── R.css
    └── libs
        └── rlang.so

Woot woot, seems to work 🎉.

Moving everything to the webR lib when the API is launched

And now, challenge number two: rebuilding this library in the webR filesystem.

webR has an interface to WASM filesystem (aka Emscripten Virtual File System), exposed via FS (doc is here). This interface allows to create a directory and write a file, which are the two things we want to do here:

Create the package folders/subfolders
Copy the files from the local machine (contained in a ./webr_packages folder) into the webR filesystem when launching the Node API.

Creating the directory tree

Creating a directory tree is not an easy task — it implies recursive function, keeping track of parents, and other weird things. R has list.files(recursive = TRUE) and we should be VERY happy about it: I looked on the internet and apparently there is no native way to do that in Node, you have to do it by hand.

As any sane modern JavaScript developer would do, I asked chatGPT to do it for me. It took me several iterations before getting the exact answer I was looking for but here it is:

const fs = require('fs');
const path = require('path');

// comments are mine

// Recursive function to get files, thank you chatGPT
function getDirectoryTree(dirPath, currentPath = '') {
  // Get a list of all the files/folders in the currently inspected folder
  const items = fs.readdirSync(path.join(dirPath, currentPath));
  // output
  const tree = [];

  // Looping on all the files/folders
  for (const item of items) {
    // Inspecting the item
    const itemPath = path.join(dirPath, currentPath, item);
    const isDirectory = fs.statSync(itemPath).isDirectory();
    const relativePath = path.join(currentPath, item);

    if (isDirectory) {
      // If it's a directory, we'll go inside and repeat the
      // current mechanism
      const subtree = getDirectoryTree(dirPath, relativePath);
      // pushing the current item, and its subtree
      tree.push({ path: relativePath, type: 'directory' });
      tree.push(...subtree);
    } else {
      tree.push({ path: relativePath, type: 'file' });
    }
  }
  // Getting the tree
  return tree;
}

Here is what it looks like on our ./webr_packages folder from before:

> getDirectoryTree("webr_packages")
[
  { path: 'attempt', type: 'directory' },
  { path: 'attempt/DESCRIPTION', type: 'file' },
  { path: 'attempt/INDEX', type: 'file' },
  { path: 'attempt/LICENSE', type: 'file' },
  { path: 'attempt/Meta', type: 'directory' },
  { path: 'attempt/Meta/Rd.rds', type: 'file' },
  { path: 'attempt/Meta/features.rds', type: 'file' },
  { path: 'attempt/Meta/hsearch.rds', type: 'file' },
  { path: 'attempt/Meta/links.rds', type: 'file' },
  { path: 'attempt/Meta/nsInfo.rds', type: 'file' },
  { path: 'attempt/Meta/package.rds', type: 'file' },
  { path: 'attempt/Meta/vignette.rds', type: 'file' },
  { path: 'attempt/NAMESPACE', type: 'file' },
  { path: 'attempt/NEWS.md', type: 'file' },
  { path: 'attempt/R', type: 'directory' },
  { path: 'attempt/R/attempt', type: 'file' },
  { path: 'attempt/R/attempt.rdb', type: 'file' },
  { path: 'attempt/R/attempt.rdx', type: 'file' },
  { path: 'attempt/doc', type: 'directory' },
  { path: 'attempt/doc/a_intro_attempt.R', type: 'file' },
  { path: 'attempt/doc/a_intro_attempt.Rmd', type: 'file' },
  { path: 'attempt/doc/a_intro_attempt.html', type: 'file' },
  { path: 'attempt/doc/b_try_catch.R', type: 'file' },
  { path: 'attempt/doc/b_try_catch.Rmd', type: 'file' },
  { path: 'attempt/doc/b_try_catch.html', type: 'file' },
  { path: 'attempt/doc/c_adverbs.R', type: 'file' },
  { path: 'attempt/doc/c_adverbs.Rmd', type: 'file' },
  { path: 'attempt/doc/c_adverbs.html', type: 'file' },
  { path: 'attempt/doc/d_if.R', type: 'file' },
  { path: 'attempt/doc/d_if.Rmd', type: 'file' },
  { path: 'attempt/doc/d_if.html', type: 'file' },
  { path: 'attempt/doc/e_conditions.R', type: 'file' },
  { path: 'attempt/doc/e_conditions.Rmd', type: 'file' },
  { path: 'attempt/doc/e_conditions.html', type: 'file' },
  { path: 'attempt/doc/index.html', type: 'file' },
  { path: 'attempt/help', type: 'directory' },
  { path: 'attempt/help/AnIndex', type: 'file' },
  { path: 'attempt/help/aliases.rds', type: 'file' },
  { path: 'attempt/help/attempt.rdb', type: 'file' },
  { path: 'attempt/help/attempt.rdx', type: 'file' },
  { path: 'attempt/help/paths.rds', type: 'file' },
  { path: 'attempt/html', type: 'directory' },
  { path: 'attempt/html/00Index.html', type: 'file' },
  { path: 'attempt/html/R.css', type: 'file' },
  { path: 'rlang', type: 'directory' },
  { path: 'rlang/DESCRIPTION', type: 'file' },
  { path: 'rlang/INDEX', type: 'file' },
  { path: 'rlang/LICENSE', type: 'file' },
  { path: 'rlang/Meta', type: 'directory' },
  { path: 'rlang/Meta/Rd.rds', type: 'file' },
  { path: 'rlang/Meta/features.rds', type: 'file' },
  { path: 'rlang/Meta/hsearch.rds', type: 'file' },
  { path: 'rlang/Meta/links.rds', type: 'file' },
  { path: 'rlang/Meta/nsInfo.rds', type: 'file' },
  { path: 'rlang/Meta/package.rds', type: 'file' },
  { path: 'rlang/NAMESPACE', type: 'file' },
  { path: 'rlang/NEWS.md', type: 'file' },
  { path: 'rlang/R', type: 'directory' },
  { path: 'rlang/R/rlang', type: 'file' },
  { path: 'rlang/R/rlang.rdb', type: 'file' },
  { path: 'rlang/R/rlang.rdx', type: 'file' },
  { path: 'rlang/backtrace-ver', type: 'file' },
  { path: 'rlang/help', type: 'directory' },
  { path: 'rlang/help/AnIndex', type: 'file' },
  { path: 'rlang/help/aliases.rds', type: 'file' },
  { path: 'rlang/help/figures', type: 'directory' },
  { path: 'rlang/help/figures/lifecycle-archived.svg', type: 'file' },
  { path: 'rlang/help/figures/lifecycle-defunct.svg', type: 'file' },
  { path: 'rlang/help/figures/lifecycle-deprecated.svg', type: 'file' },
  { path: 'rlang/help/figures/lifecycle-experimental.svg',type: 'file'},
  { path: 'rlang/help/figures/lifecycle-maturing.svg', type: 'file' },
  { path: 'rlang/help/figures/lifecycle-questioning.svg',type: 'file'},
  { path: 'rlang/help/figures/lifecycle-retired.svg', type: 'file' },
  { path: 'rlang/help/figures/lifecycle-soft-deprecated.svg',type: 'file'},
  { path: 'rlang/help/figures/lifecycle-stable.svg', type: 'file' },
  { path: 'rlang/help/figures/lifecycle-superseded.svg', type: 'file' },
  { path: 'rlang/help/figures/logo.png', type: 'file' },
  { path: 'rlang/help/paths.rds', type: 'file' },
  { path: 'rlang/help/rlang.rdb', type: 'file' },
  { path: 'rlang/help/rlang.rdx', type: 'file' },
  { path: 'rlang/html', type: 'directory' },
  { path: 'rlang/html/00Index.html', type: 'file' },
  { path: 'rlang/html/R.css', type: 'file' },
  { path: 'rlang/libs', type: 'directory' },
  { path: 'rlang/libs/rlang.so', type: 'file' }
]

Note: I need all the folders and subfolders, for example rlang/libs and rlang/libs/rlang.so, because I haven’t found a way to do a recursive mkdir in webR, and had to create all children level by level.

🚀 Yeay 🚀

Now I have a JavaScript object I can use to move my ./webr_packages library to a webR instance library.

Let’s write a function for that (no need for chatGPT here 😅):

async function loadPackages(webR, dirPath){
  // Our function from before
  const files = getDirectoryTree(
    dirPath
  )
  for await (const file of files) {
    // If the entry is a directory, we'll
    // call mkdir
    if (file.type === 'directory') {
      await globalThis.webR.FS.mkdir(
        `/usr/lib/R/library/${file.path}`,
      );
    } else {
    // case 2, the entry is a file, then we can use fs.readFileSync
    // That will return a ArrayBuffer that can be directly passed
    // to webR.FS.writeFile
      const data = fs.readFileSync(`webr_packages/${file.path}`);
      await globalThis.webR.FS.writeFile(
        `/usr/lib/R/library/${file.path}`,
        data
      );
    }
  }
}

Now we have our whole process:

downloading the precompiled packages
listing the dir tree in Node
bringing everything into the webR filesystem

🚀 Yeay (bis) 🚀

To make things simple, I have put these into a node package, at npmjs.com/package/webrtools.

Bringing everything together

Workflow for building the app:

cd webr-example
mkdir webr-preloadr
cd webr-preloadr
npm init -y
npm install express webr webrtools
Rscript ./node_modules/webrtools/r/install.R dplyr
touch index.js

Then :

const app = require('express')()
const path = require('path');
const { loadPackages } = require('webrtools');
const { WebR } = require('webr');

(async () => {
  globalThis.webR = new WebR();
  await globalThis.webR.init();

  console.log("🚀 webR is ready 🚀");

  await loadPackages(
    globalThis.webR,
    path.join(__dirname, 'webr_packages')
  )

  console.log("📦 Packages written to webR 📦");

  const res = await globalThis.webR.FS.lookupPath("/usr/lib/R/library");
  console.log(res)

  await globalThis.webR.evalR("library(dplyr)");
  app.listen(3000, '0.0.0.0', () => {
    console.log('http://localhost:3000')
  })
})();

app.get('/', async (req, res) => {
  let result = await globalThis.webR.evalR('sample_n(mtcars, 10)');
  let output = await result.toJs();
  res.send(output.values)
});


node index.js

And in another console:

$ curl http://localhost:3000
[{"type":"double","names":null,"values":[21,24.4,14.3,32.4,18.7,17.8,22.8,10.4,30.4,21.4]},{"type":"double","names":null,"values":[6,4,8,4,8,6,4,8,4,4]},{"type":"double","names":null,"values":[160,146.7,360,78.7,360,167.6,108,460,75.7,121]},{"type":"double","names":null,"values":[110,62,245,66,175,123,93,215,52,109]},{"type":"double","names":null,"values":[3.9,3.69,3.21,4.08,3.15,3.92,3.85,3,4.93,4.11]},{"type":"double","names":null,"values":[2.62,3.19,3.57,2.2,3.44,3.44,2.32,5.424,1.615,2.78]},{"type":"double","names":null,"values":[16.46,20,15.84,19.47,17.02,18.9,18.61,17.82,18.52,18.6]},{"type":"double","names":null,"values":[0,1,0,1,0,1,1,0,1,1]},{"type":"double","names":null,"values":[1,0,0,1,0,0,1,0,1,1]},{"type":"double","names":null,"values":[4,4,3,4,3,4,4,3,4,4]},{"type":"double","names":null,"values":[4,2,4,1,2,4,1,4,2,2]}]

🚀 Yeay (ter) 🚀

And now for the Dockerfile

FROM r-base:4.3.1

# Create app directory
WORKDIR /usr/src/app

RUN apt update && \
  apt install nodejs npm -y

COPY package*.json ./

RUN npm install

RUN Rscript ./node_modules/webrtools/r/install.R dplyr

COPY . .

EXPOSE 3000

CMD ["node", "index.js"]

You can find the code here, and see it live at srv.colinfay.me/webr-preloadr.

You can also try it with

docker run -it -p 3000:3000 colinfay/webr-preloadr

Final notes

Should I use it in prod?

Probably not — for now it’s more of an experiment than a real bullet-proof approach. But if you’re a reader from the future, many things may have happen, and maybe the webrtools node module is now prod ready. Who knows!

Further work

After some discussion on the issue on the webR repo, it seems that a promising path to do what I’d like to do is to use the FS.mount interface from WASM, but it needs to be expose by webR. Keep an eye on the repo!

Finally, I’m a bit displeased with writing the installR fun in R and having to install R in the Dockerfile to do it, so I should probably re-write it in full NodeJS. We could read the package tree in R with webR and then download/untar with Node.

Food for thoughts.

The Old Faithful Geyser Data shiny app with webR, Bootstrap & ExpressJS

Colin Fay — Tue, 25 Jul 2023 00:00:00 +0000

This post is the second one of a series of post about webR:

Using webR in an Express JS REST API
The Old Faithful Geyser Data shiny app with webR, Bootstrap & ExpressJS

Note: the first post of this series explaining roughly what webR is, I won’t introduce it again here.

In this post, I’ll attempt to recreate a version of the famous Old Faithful Geyser Data {shiny} app using webR, Bootstrap & ExpressJS. (If you really don’t know which app I’m talking about, it’s this one.)

Introductory notes

Before starting, here are two notes regarding the app built in this post:

this app could have been build in full “web mode” (no server required, just an HTML page), but this is not the approach I’m currently experimenting with. Going full browser-based doesn’t work for all cases, and most of the time with production apps you’ll need some part of your code to be computed by the server (because of resources, because you’ll connect to API with token, because you need access to DB with passwords, because you don’t want the full data to be available in the brower, or many other good reason…).
In order to recreate the app, my first approach was to try to draw the histogram in base R and send it back to the browser.boB has a great blogpost about how to do exactly that, but after a lot of bad code and good 4 letter words, I realized there was no sane reason for me to display a base R plot instead of a JavaScript based one. That’s why I chose to return the data for the barplot (using the cut() function from R) and draw with chart.js, instead of trying to display the “not so user friendly” base plot.

Project init

Let’s start by creating a new Express app:

mkdir express-webr-old-faithful
cd express-webr-old-faithful
npm init -y
# Installing the deps we'll need
npm i @r-wasm/webr express
# Creating a server, and the front page
touch index.js
touch index.html

Server

Let’s move to the server side first (index.js ). We’ll start by taking the file from the previous blog post, and modify it to:

serve index.html on /
create a route that returns the data of the bins for our histogram

Let’s start with our code to init webR.

'use strict';
const express = require("express")
// For serving the html
const path = require("path")
const app = express()
const { WebR } = require('@r-wasm/webr');

(async () => {
  globalThis.webR = new WebR();
  await globalThis.webR.init();
  // Given that we will reuse this value,
  // we assign it at launch
  await globalThis.webR.evalR('x <- faithful[, 2]')
  console.log("webR is ready");
  app.listen(3000, '0.0.0.0', () => {
    console.log('http://localhost:3000')
  })
})();

Then, an endpoint that serves index.html:

app.get("/", (req, res) => {
  res.sendFile(path.join(__dirname, "index.html"))
})

And an endpoint that sends the bins for our plot, using ExpressJS parameter notation (path/:n). The value is sent to webR via the env option of evalR

app.get("/hist-data/:n", async (req, res) => {
  let result = await globalThis.webR.evalR(
    'table(cut(x, seq(min(x), max(x), length.out = n + 1)))',
    { env: { n: parseInt(req.params.n) } }
  );
  let output = await result.toJs();
  res.send(output)
})

Now that our backend is ready, let’s check that we can now call it from the command line:

curl http://localhost:3000/hist-data/10
{"type":"integer","names":["(43,48.3]","(48.3,53.6]","(53.6,58.9]","(58.9,64.2]","(64.2,69.5]","(69.5,74.8]","(74.8,80.1]","(80.1,85.4]","(85.4,90.7]","(90.7,96]"],"values":[15,28,26,24,9,23,62,55,23,6]}

Front

Now, time to build the front.

We’ll start with the Boostrap boilerplate from https://getbootstrap.com/docs/5.3/getting-started/introduction/#quick-start.

Note also that I’ve chosen (for simplicity’s sake) to use the CDN version of the external deps, instead of installing them in my Node project, which would be what I would do in a normal context.

<!doctype html>
<html lang="en">
  <head>
    <meta charset="utf-8">
    <meta name="viewport" content="width=device-width, initial-scale=1">
    <title>Bootstrap demo</title>
    <link href="https://cdn.jsdelivr.net/npm/bootstrap@5.3.0/dist/css/bootstrap.min.css" rel="stylesheet" integrity="sha384-9ndCyUaIbzAi2FUVXJi0CjmCapSmO7SnpJef0486qhLnuZ2cdeRhO02iuK6FUUVM" crossorigin="anonymous">
  </head>
  <body>
    <h1>Hello, world!</h1>
    <script src="https://cdn.jsdelivr.net/npm/bootstrap@5.3.0/dist/js/bootstrap.bundle.min.js" integrity="sha384-geWF76RCwLtnZ8qwWowPQNguL3RmwHVBC9FhGdlKrxdiJJigb/j/68SIy3Te4Bkz" crossorigin="anonymous"></script>
  </body>
</html>

Let’s change the title, and add:

the range input
a div to receive the chart

<div class="container">
  <h1>Old Faithful Geyser Data</h1>
  <!-- Bootstrap grid system -->
  <div class="row align-items-start">
    <div class="col-4">
      <label for="customRange1" class="form-label">Number of bins:</label>
      <input type="range" class="form-range" id="customRange1" min=1 max=30 value=10>
      <div id="bins">Selected: 10</div>
    </div>
    <div class="col-8">
      <div>
        <!-- Where we'll get the chart drawn -->
        <canvas id="myChart"></canvas>
      </div>
    </div>
  </div>
</div>

In order to draw the graph, I’ll rely on a dead simple (yet powerful) JavaScript lib called Chart.js. It has a great bar chart graph that will work perfectly for our case.

Let’s start by adding the lib with <script src="https://cdn.jsdelivr.net/npm/chart.js"></script>.

We then need some JavaScript to:

Initiate the chart at launch

// https://colinfay.me/api-from-client-shiny/
// Default to 10 bins
fetch("hist-data/10")
      .then((data) =>{
        // Convert the data to json and
        // create a chart
        data.json().then((res) => {
          // Keeping a global object with the chart
          globalThis.chart = new Chart(
          // This is where the chart will go
          document.getElementById('myChart'), {
          type: 'bar',
          data: {
            // webR returns the names
            labels: res.names,
            datasets: [{
              label: "Histogram of waiting times",
              data: res.values
            }]
          }
        });
        })
        .catch((error) => {
          alert("Error catchin result from R")
        })
      })
      .catch((error) => {
        alert("Error catchin result from R")
      })

Update the chart when the slider is moved

    const update = function (n = 10) {
      fetch(`hist-data/${n}`).then((data) => {
        data.json().then((res) => {
          globalThis.chart.data.labels = res.names;
          globalThis.chart.data.datasets.forEach(dataset => {
            dataset.data = res.values;
          })
          globalThis.chart.update();
        })
      })
      document.querySelector('#bins').innerHTML = `Selected: ${n}`;

    }
document.querySelector('#customRange1').addEventListener(
  'change',
  function() { update(this.value); }
 );

And here it is! You can see the app live at srv.colinfay.me/express-webr-old-faithful. You can find the code here.

You can also try it with:

docker run -it -p 3000:3000 colinfay/express-webr-old-faithful

Using webR in an Express JS REST API

Colin Fay — Tue, 04 Jul 2023 00:00:00 +0000

webR? wat again?

As described in the doc:

WebR is a version of the statistical language R compiled for the browser and Node.js using WebAssembly, via Emscripten.

In this post, I won’t go into details into what webR is, as if you want to know more, you’ll probably get a better understanding by going through the documentation.

But if like a normal person you don’t read the doc, let’s sum up what webR is. Simply put, WebAssembly (shorten wasm) is a format that can be used to encode a programming language to something that can then be called from a JavaScript runtime (in the browser or in NodeJS). Schematically, you take one language, you translate it to wasm, and the output can be read and used from JavaScript.

If you still have no idea what wasm is but want to get a better grasp on what it is, I suggest reading this blogpost that really sums it well, especially this quote:

WASM is a way to use non-Javascript code and run it in your browser.

I also suggest reading the blogposts by boB on the subject.

The rest of this post assume that you understand what wasm & webR are.

Playing with webR

I’ve been interested with WebAssembly for a while now, and can remember talking about it late during the night at R conferences around 2018 and 2019… but that’s a story for another time 😅

Today’s story is about exploring how to build tools in JavaScript that will call webR.

I’ll try to document my experimentations, and this post is the first one of a series that I hope I’ll find time to continue over the summer.

Hello World! - Creating an API where we call webR

I’ll start with a simple Express JS REST API that uses webR to run R code.

# Initiating a projet and opening it with VSCode
mkdir webrexpresshelloworld
code webrexpresshelloworld

# Quick init of a node project
npm init -y
touch index.js
npm install express

We’ll now install the webR nodejs package

npm i @r-wasm/webr

And now, let’s move to our index.js file. We’ll start with the basics of Express JS: creating the app object.

// Creating the express app
const app = require('express')()

Then, we’ll import WebR from @r-wasm/webr.

// importing the `WebR` class inside the runtime
const { WebR } = require('@r-wasm/webr');

We’ll now be initiating the webR session by:

creating an instance of webR
binding it to globalThis so that it’s available everywhere
calling the init method
Once webR is ready, we’ll launch the app

As webR communicates via the worker thread asynchronously, we need to use async/await.

(async () => {
  globalThis.webR = new WebR();
  await globalThis.webR.init();
  console.log("webR is ready");
  // Starting the express app
  // only after webR is ready
  app.listen(3000, () => {
    console.log('http://localhost:3000')
  })
})();

Finally, we add a route that returns an hello world:

// Creating a route for the express app
// that will return a titleCase hello world from r
app.get('/', async (req, res) => {
  // Evaluating the R code inside the R worker
  let result = await globalThis.webR.evalR('tools::toTitleCase("hello from r!")');
  // Converting the result to a JS object
  let output = await result.toJs();
  // Sending the result back to the client,
  // (webR outputs an object with types/names/values
  // and here we only want the values)
  res.send(output.values)
});

And now, let’s run our app!

node index.js

You can find the code here, and see it live at srv.colinfay.me/express-webr-hello-world.

You can also try it with

docker run -it -p 3000:3000 colinfay/express-webr-hello-world

The ‘Engineering Production-Grade Shiny Apps’ book is available in print!

Colin Fay — Mon, 04 Oct 2021 00:00:00 +0000

I’m so, so excited to announce that the ‘Engineering Production-Grade Shiny Apps’ book is now available in print! This book is the result of two years of work, and I couldn’t be more happy to see it available in print.

As its title suggests, ‘Engineering Production-Grade Shiny Apps’ is a book about building {shiny} apps that will be sent to production. It goes deep into {golem}, of course, but it’s not a book about {golem} only — it tries to cover a large panel of topics, and I’m pretty sure that if you are a {shiny} developer, there is something for you in this book: project management, code organization, team work, best practices for front end development, CI and CD, deployment, code optimization…

This book will remain available online for free at https://engineering-shiny.org/ (and I want to thank Chapman and Hall/CRC for this), but if you want to grab a copy, you can go on routledge.com and order one. I know it can feel counter-intuitive to buy a print copy of a book which is available online, but if you want this to continue to happen, please continue to support the publisher by buying a copy of this book, or of any other from Chapman and Hall/CRC — they have been doing a massive work when it comes to organizing, editing and reviewing.

I want to personally thank everybody that has been contributing to this book, with some special thanks to Eric Nantz which has been a {golem} early adapter and an all time supporter of everything from the golemverse, to David Granjon for all his feedback on the book, and to Christophe Dervieux for his precious help during my battles with Markdown. Thanks also to David Grubbs from Chapman and Hall/CRC for making this happen.

That being said, I still can’t believe I’ve added my name to the list of authors in “The R Series” books 😱

I hope you’ll enjoy it!

Multi-page {shiny} Applications with {brochure}

Colin Fay — Thu, 18 Feb 2021 00:00:00 +0000

[Disclaimer] The package presented in this blog post is still experimental at the time of writing these lines (2021-02-18), so it might face some API changes in the future.

Multi-page in {shiny}, and random thoughts about designing web applications

Something that has been bugging me for a while is the inability to really share a specific part of a {shiny} dashboard, at least natively, using a specific path. In other words, I’ve always wanted to be able to do something like my-uberapp.io/contact to share with someone a specific part of the app. And only this part.

Another need I was facing, probably born out of building web applications using NodeJS, is a natural way to manipulate endpoints, http requests and responses , so that I could do something pretty common in web application: a home page, and next to it a login page that verifies your identity, and redirects you to another page after setting a cookie. That of course, leads to the necessity of having the ability to define various endpoints with a specific behavior for each : a unique UI, and a 18 server response. Pure, native, multi-endpoint applications, not one big ball of UI with parts hidden using JavaScript & CSS, and a global server function that might launch computation you don’t need for your page.

A few weeks back, I’ve decided to focus on this question. Multi-page in{shiny} is not new: I’ve found both{shiny.router}, and{blaze} that already implement a form of “multi-page”.

Note: both these packages work perfectly, and they definitely do what they are designed for. There are the product of tremendous work with smart implementations and my goal is in no way to undermine these packages. They just don’t answer the need I was having, hence my new approach to this question. {brochure} will answer the needs for more large and complex applications built with {shiny}, while both the{shiny.router} & {blaze} should be easier to get started with.

As far as I saw it, these two packages didn’t answer what I was looking for: “real” multi-page. Both {shiny.router} & {blaze} still produce a form of Single Page Application , but plays with the URL to make you feel like you’re on a multi-page application.

For example, {shiny.router} hides the parts of the app via JS & CSS – if you browse the source code, you’ll still see the hidden HTML. That also implies that they are potential conflicts between ids: you have to be sure that all your pages have unique id on their ouputs/inputs, otherwise the app doesn’t render, meaning that a “page” doesn’t have its own server function. And of course, if you look at the Networktab of your browser developer tool, you’ll see that there is no newGET request made whenever a new “page” is loaded: this new page is still the same application, with parts hidden through JavaScript.

Hiding UI parts is a practice that has always been bothering me in term of performance – on a large {shinydashboard}, for example, you’ll be transferring the full HTML, CSS and JavaScript for the whole application when the app launches, even if the user never visits all the tabs.{shiny} is very smart when it comes to transferring CSS and JS files: for example, it will only serve the external resources for thesliderInput() if there is one in the app. But if you’ve got a{shinydashboard} dashboard with 19 pages, and the sliderInput() is on page 19 and will only be seen by 0.1% of the visitors, it will still be transferred to every user, regardless of whether or not they need it. This might result in lowering the performance of the app, as it might be raising the time to ‘First Meaningful Paint’, and of course it is a waste of resources, especially if your application is served to people with low bandwidth, and/or browsing your app using cellular data. I know this is a question you’ve probably never asked yourself before ;) - but performance is a pretty common question in the web development world, and something to be aware if you’re serving your app at scale. If this is something you’re interested in, I suggest reading Why does speed matter? from the Google dev center, and Web Performancefrom Mozilla Web Docs. See also 14.2.2 Profiling {shiny}from the Engineering Shiny book.

And, another final thing I wanted is a way to “flush” objects when you change page: in the current implementation of {shinydashboard}, if you create something on tab1, it will still be there in tab2, taking some space in the RAM, even if you don’t need it. It’s rather convenient because you don’t have to think about these things: you start a{shiny} session, and all the objects will still be there as long as the session lives. But that also means that as soon as the session stops, there is no way to get the values back as they only live in the RAM, inside the session.

On the other hand, with an implementation where every page gets its own{shiny} session, you need to find a way to identify the user with a form of id, save the relevant values (potentially write in a DB), then fetch these values again using the id stored in the browser (typically, you have a session cookie in the browser that is used as a key to search the DB). A process forcing you to think more carefully about the data flow of your app. What that also means is that as theid is a cookie in the browser, whenever the app crashes, it will be able to reload to the exact same place as long as the cookie is still in the browser (well, not exactly but you understand the idea).

So, back to our first topic – what was I looking for? My goal was to find a way to build {shiny} applications that are natively multi-page. In other words, an application that answers to a request on /page2, not an app that silently redirect to /, nor an application that requires playing with /#!/page2 in the url. (Reminder, # is traditionally used as an anchor tag in a web page). And, of course, I wanted each page to have its own shiny session, its own UI and server functions.

I was also looking for a way to manipulate both the GET request from the server and the httpResponse that is sent back, for example in order to set httpOnly cookies in the header of the HTTP Response or change the HTTP status code , the same way you can manipulate these when building an application with NodeJS for example. This is something that you’d find in the{ambiorix} project by the amazing John Coene (who will get some SEO juice thanks to this backlink). It’s an amazing approach that definitely resonated with me as I’ve been a NodeJS & express.js user for some time now, but that’s still far from the way you’d build things in{shiny}, and will probably require a lot of coding to get an application up and running (which is good, because I love coding, but let’s use our {shiny} knowledge for now :) ).

To sum up, I wanted a way to get closer to how you build applications in other languages, but will as little deviation as possible from the{shiny} way of building apps.

Here comes `{brochure}`

As said on the top of the article, this package is still a work in progress, so you might see some changes in the near future :)

[Disclaimer, again] The way you will build applications with{brochure} is different from the way you usually build {shiny} apps, as we no longer operate under the single page app paradigm. Please keep this in mind and everything should be fine.

Installation

You can install the development version of {brochure} with:

remotes::install_github("ColinFay/brochure")

Minimal `{brochure}` App

library(shiny)
library(brochure)


## 
## Attaching package: 'brochure'

## The following object is masked from 'package:utils':
## 
## page

`page()`

A brochureApp is a series of pages that are defined by an href(the path/endpoint where the page is available), a UI and a serverfunction. This is conceptually important: each page has its own shiny session, its own UI, and its own server.

brochureApp(
  # First page
  page(
    href = "/",
    ui = fluidPage(
      h1("This is my first page"), 
      plotOutput("plot")
    ),
    server = function(input, output, session){
      output$plot <- renderPlot({
        plot(iris)
      })
    }
  ), 
  # Second page, without any server-side function
  page(
    href = "/page2", 
    ui = fluidPage(
      h1("This is my second page"), 
      tags$p("There is no server function in this one")
    )
  )
)

-> In your browser, you can now navigate to /, and to /page2.

`redirect()`

Redirections can be used to redirect from one endpoint to the other:

brochureApp(
  page(
    href = "/",
    ui = tagList(
      h1("This is my first page")
    )
  ),
  redirect(
    from = "/nothere",
    to = "/"
  ),
  redirect(
    from = "/colinfay",
    to = "https://colinfay.me"
  )
)

-> In your browser, if you got to /nothere, and you’ll be redirected to / .

`req_handlers` & `res_handlers`

Sorry what?

This is where things get more interesting.

Each page, and the global app, have a req_handlers and res_handlersparameters, that can take a list of functions.

An *_handler is a function that takes as parameter(s):

For req_handlers, req, which is the request object (see below for when these objects are created). For example function(req){ print(req$PATH_INFO); return(req)}.

For res_handlers , res, the response object, & req , the request object. For example function(res, req){ print(res$content); return(res)}.

req_handlers must return req & res_handlers must returnres.

They can be used to register log, or to modify the objects, or any kind of things you can think of. If you are familiar with express.js, you can think of req_handlers as what express.js calls “middleware”. These functions are run when R is building the HTTP response to send to the browser (i.e, no server code has been run yet), following this process:

R receives a GET request from the browser, creating a request object, called req
The req_handlers are run using this req object, potentially modifying it
R creates an httpResponse, using this req and how you defined the UI
The res_handlers are run on this httpResponse (first app levelres_handlers, then page level res_handlers), , potentially modifying it
The httpResponse is sent back to the browser

Note that if any req_handlers returns an httpResponse object, it will be returned to the browser immediately, without any further computation. This early httpResponse will not be passed to theres_handlers of the app or the page. This process can for example be used to send custom httpResponse, as shown below with thehealthcheck endpoint.

You can use formulas inside your handlers. .x and ..1 will be reqfor req_handlers, .x and ..1 will be res & .y and ..2 will be req for res_handlers.

Design pattern side-note: you’d probably want to define the handlers outside of the app, for better code organization (as with log_wherebelow).

Example: Logging with `req_handlers()`, and building a healthcheck point

In this app, we’ll log to the console every page and the time it is called, using the log_where() function.

log_where <- function(req){
  cli::cat_rule(
    sprintf(
      "%s - %s", 
      Sys.time(), 
      req$PATH_INFO
    )
  )
  req
}

For the sake of organization, we’ll create functions that return pages:

nav_links <- tags$ul(
  tags$li(
    tags$a(href = "/", "home"), 
  ),
  tags$li(
    tags$a(href = "/page2", "page2"), 
  ),
  tags$li(
    tags$a(href = "/contact", "contact"), 
  )
)

page_1 <- function(){
  page(
    href = "/",
    ui = function(request){
      tagList(
        h1("This is my first page"),
        nav_links,
        plotOutput("plot")
      )
    },
    server = function(input, output, session){
      output$plot <- renderPlot({
        plot(mtcars)
      })
    }
  )
}

page_2 <- function(){
  page(
    href = "/page2",
    ui = function(request){
      tagList(
        h1("This is my second page"),
        nav_links,
        plotOutput("plot")
      )
    }, 
    server = function(input, output, session){
      output$plot <- renderPlot({
        plot(mtcars)
      })
    }
  )
}

page_contact <- function(){
  page(
    href = "/contact",
    ui = tagList(
      h1("Contact us"),
      nav_links,
      tags$ul(
        tags$li("Here"),
        tags$li("There")
      )
    )
  )
}

We’ll also build an healthcheck endpoint that simply returns ahttpResponse with the 200 HTTP code.

# Reusing the pages from before
brochureApp(
  req_handlers = list(
    log_where
  ),
  # Pages
  page_1(),
  page_2(),
  page_contact(),
  page(
    href = "/healthcheck",
    # As this is a pure backend exchange, 
    # We don't need a UI
    ui = tagList(), 
    # As this req_handler returns an httpResponse,
    # This response will be returned directly to the browser, 
    # without passing through the usual dance
    req_handlers = list(
      # If you have shiny < 1.6.0, you'll need to 
      # do shiny:::httpResponse (triple `:`) 
      # as it is not exported until 1.6.0.
      # Otherwise, see ?shiny::httpResponse
      ~ shiny::httpResponse( 200, content = "OK")
    )
  )
)

If you navigate to each page, you’ll see this in the console:

Listening on http://127.0.0.1:4879
── 2021-02-17 21:52:16 - / ──────────────────────────────
── 2021-02-17 21:52:17 - /page2 ─────────────────────────
── 2021-02-17 21:52:19 - /contact ───────────────────────

If you go to another R session, you can check that you’ve got a 200 onhealthcheck :

> httr::GET("http://127.0.0.1:4879/healthcheck")
Response [http://127.0.0.1:4879/healthcheck]
  Date: 2021-02-17 21:55
  Status: 200
  Content-Type: text/html; charset=UTF-8
  Size: 2 B

Handling cookies using `res_handlers`

res_handlers can be used to set cookies, by adding a Set-Cookieheader.

Note that you can parse the cookie using parse_cookie_string.

brochure::parse_cookie_string( "a=12;session=blabla" )


## a session 
## "12" "blabla"

In the example, we’ll also use brochure::server_redirect("/"), from the server-side to redirect the user after login.

# Creating a navlink
nav_links <- tags$ul(
  tags$li(
    tags$a(href = "/", "home"), 
  ),
  tags$li(
    tags$a(href = "/login", "login"), 
  ),
  tags$li(
    tags$a(href = "/logout", "logout"), 
  )
)

home <- function(){
  page(
    href = "/",
    ui = tagList(
      h1("This is my first page"), 
      tags$p("It will contain BROCHURECOOKIE depending on the last page you've visited (/login or /logout)"),
      verbatimTextOutput("cookie"),
      nav_links
    ),
    server = function(input, output, session){
      output$cookie <- renderPrint({
        parse_cookie_string(
          session$request$HTTP_COOKIE
        )
      })
    }
  )
}

login <- function(){
  page(
    href = "/login",
    ui = tagList(
      h1("You've just logged!"),
      verbatimTextOutput("cookie"),
      actionButton("redirect", "Redirect to the home page"),
      nav_links
    ), 
    server = function(input, output, session){
      output$cookie <- renderPrint({
        parse_cookie_string(
          session$request$HTTP_COOKIE
        )
      })
      observeEvent( input$redirect , {
        # Using brochure to redirect to another page
        server_redirect("/")
      })

    },
    res_handlers = list(
      # We'll add a cookie here
      function(res, req){
        res$headers$`Set-Cookie` <- "BROCHURECOOKIE=12; HttpOnly;"
        res
      }
    )
  )
}

logout <- function(){
  page(
    href = "/logout",
    ui = tagList(
      h1("You've logged out"),
      nav_links,
      verbatimTextOutput("cookie"),
      actionButton("redirect", "Redirect to the home page"),
    ), 
    server = function(input, output, session){
      output$cookie <- renderPrint({
        parse_cookie_string(
          session$request$HTTP_COOKIE
        )
      })
      observeEvent( input$redirect , {
        # Using brochure to redirect to another page
        server_redirect("/")
      })
    },
    res_handlers = list(
      # We'll add a cookie here
      function(res, req){
        res$headers$`Set-Cookie` <- "BROCHURECOOKIE=12; Expires=Wed, 21 Oct 1950 07:28:00 GMT"
        res
      }
    )
  )
}

brochureApp(
  # Pages
  home(),
  login(),
  logout()
)

Random notes about design patterns

Data persistence

Every time you open a new page, a new shiny session is launched. This is different from what you usually do when you are building a{shiny} app that works as a single page application. This is no longer the case in {brochure}.

What that means is that there is no data persistence in R when navigating from one page to the other. That might seem like a downside, but I believe that it will actually be for the best: it will make developers think more carefully about the data flow of their application, and allow a faster flush of R sessions.

That being said, how do we keep track of a user though pages, so that if they do something in a page, it’s available on another page?

To do that, you’d need to add a form of session identifier, like a cookie: this can for example be done using the{glouton} package if you want to manage it with JS only. You can also use the cookie example from before.

You’ll also need a form of backend storage (for example with{cachem} here in the example, but you can also use an external DB like SQLite or MongoDB).

library(glouton)
# Creating a storage system
cache_system <- cachem::cache_disk(tempdir())

nav_links <- tags$ul(
  tags$li(
    tags$a(href = "/", "home"), 
  ),
  tags$li(
    tags$a(href = "/page2", "page2"), 
  )
)

cookie_set <- function(){
  r <- reactiveValues()

  observeEvent(TRUE, {
    # Fetch the cookies using {glouton}
    r$cook <- fetch_cookies()

    # If there is no stored cookie for {brochure}, we generate it
    if (is.null(r$cook$brochure_cookie)){
      # Generate a random id
      session_id <- digest::sha1(paste(Sys.time(), sample(letters, 16)))
      # Add this id as a cookie
      add_cookie("brochure_cookie", session_id)
      # Store in in the reactiveValues list
      r$cook$brochure_cookie <- session_id
    }
    # For debugging purpose
    print(r$cook$brochure_cookie )
  }, once = TRUE)
  return(r)
}

page_1 <- function(){
  page(
    href = "/",
    ui = tagList(
      h1("This is my first page"),
      nav_links,
      # The text enter on page 1 will be available on page 2, using
      # a session cookie and a storage system
      textInput("textenter", "Enter a text"), 
      actionButton("save", "Save my text and go to page2")
    ),
    server = function(input, output, session){
      r <- cookie_set()
      observeEvent( input$save , {
        # Use the session id to save on the cache system
        cache_system$set(
          paste0(
            r$cook$brochure_cookie,
            "text"
          ),
          input$textenter
        )
        server_redirect("/page2")
      })
    }
  )
}

page_2 <- function(){
  page(
    href = "/page2",
    ui = tagList(
      h1("This is my second page"),
      nav_links,
      # The text enter on page 1 will be available here, reading
      # the storage system
      verbatimTextOutput("textdisplay")
    ), 
    server = function(input, output, session){
      r <- cookie_set()
      output$textdisplay <- renderPrint({
        # Getting the content value based on the session cookie
        cache_system$get(
          paste0(
            r$cook$brochure_cookie,
            "text"
          )
        )
      })
    }
  )
}

brochureApp(
  # Setting {glouton} globally
  use_glouton(),
  # Pages
  page_1(),
  page_2()
  # Redirections
)

Working with golem

{golem} is a small project that is starting to get some traction in the {shiny} world. I’m still not sure of its relevance, but I’ve heard that some of you are using it for some reasons, so I suppose you might want to make {brochure} work inside a {golem} based app.

To adapt your {golem} based application to {brochure}, here are the two steps to follow:

Remove the app_server.R file, and the top of app_ui => You’ll still need golem_add_external_resources().

Build the pages inside separate R scripts, following the example from this README.

.
├── DESCRIPTION
├── NAMESPACE
├── R
│ ├── app_config.R
│ ├── home.R ### YOUR PAGE 
│ ├── login.R ### YOUR PAGE 
│ ├── logout.R ### YOUR PAGE 
│ └── run_app.R ### YOUR PAGE 
├── dev
│ ├── 01_start.R
│ ├── 02_dev.R
│ ├── 03_deploy.R
│ └── run_dev.R
├── inst
│ ├── app
│ │ └── www
│ │ ├── favicon.ico
│ └── golem-config.yml
├── man
│ └── run_app.Rd

Replace shinyApp with brochureApp in run_app(), add the external resources, then your pages.

run_app <- function(
  onStart = NULL,
  options = list(), 
  enableBookmarking = NULL,
  ...
) {
  with_golem_options(
    app = brochureApp(
      # Putting the resources here
      golem_add_external_resources(),
      home(),
      login(),
      logout(),
      onStart = onStart,
      options = options, 
      enableBookmarking = enableBookmarking
    ), 
    golem_opts = list(...)
  )
}

Improvement & further work

Testers and apps. If you want to give {brochure} a shot, please do! It’s still a young project that needs more testing, so if you feel like you want to build an app with it, please do. I’d be happy to help you if you have some questions so feel free to reach me onTwitter, or comment on the post below.

Functions to manipulate cookies (create and delete them).

Built-in handlers for req & res that will answer common use cases.

Allow a way to implement :id: in the URL, the way express.jsdoes: you can set a route as /profile/:id: and the app will answers to /profile/colin, /profile/jean, /profile/whatever, setting the id as a variable that can be reused in the response. In the context of {shiny}, this would of course be available the server.

Auto build a proxy based on the application. This will allow to create one R session by page, then create a proxy that will plug each endpoint to an R session, allowing a little bit of scalability “for free”.

If you have any idea or question, please open an issue on the GitHub repo!

DEV Community: Colin Fay

Rlinguo — Why Did We Build It?

1. Field Data Collection and Analysis for Environmental Researchers

Who?

The Problem

How R on Mobile Helps

2. A Portable Learning Tool for Statistics Students

Who?

The Problem

How R on mobile Helps

3. Real-Time Quality Control in Manufacturing

Who?

The Problem

How R on Mobile Helps

4. Supply Chain Optimization for Logistics

Who?

The Problem

How R on Mobile Helps

5. Healthcare Data Analysis for Field Medics

Who?

The Problem

How R on mobile Helps

Why R on Mobile?

Setting values in R6 classes, and testing with shiny::MockShinySession

Context

Testing headers

Overriding session$request

Writing the Test

Cleaning Up After Tests

Do you need help with testing your apps?

Introducing Rlinguo, a native mobile app that runs R

Once upon a time…

{shiny} on my mobile

The “What If” Moment

The Shape of R to Come

You've got a use case?

webrcli & spidyr: What’s new

What webrcli & spidy wants to achieve

An important note

Making the template code smaller

new Library() is now library()

Better installation from package.json & DESCRIPTION

Other tools

Workflow summary

Please do give it a shot

Future works

webrcli & spidyr: A starter pack for building NodeJS projects with webR inside

Again, a new project?

webrcli & spidyr

Project init

With a CRAN package

Some random notes

About the current implementation

Future

Rethinking packages & functions preloading in webR 0.2.2

Using my own R functions in webR in an Express JS API, and thoughts on building web apps with Node & webR

The problem

The how

The end app

Preloading your R packages in webR in an Express JS API

The problem

Preloading R packages in my ExpressJS/webR API

Building the R lib

Moving everything to the webR lib when the API is launched

Creating the directory tree

Bringing everything together

And now for the Dockerfile

Final notes

Should I use it in prod?

Further work

The Old Faithful Geyser Data shiny app with webR, Bootstrap & ExpressJS

Introductory notes

Project init

Server

Front

Using webR in an Express JS REST API

webR? wat again?

Playing with webR

Hello World! - Creating an API where we call webR

The ‘Engineering Production-Grade Shiny Apps’ book is available in print!

Overriding `session$request`

`{shiny}` on my mobile

`new Library()` is now `library()`

Better installation from package.json & `DESCRIPTION`

Here comes `{brochure}`

Minimal `{brochure}` App

`page()`

`redirect()`

`req_handlers` & `res_handlers`

Example: Logging with `req_handlers()`, and building a healthcheck point

Handling cookies using `res_handlers`