DEV Community: Peter

Application Design for Multi-Tenancy Apps

Peter — Fri, 05 Mar 2021 21:29:44 +0000

This is mostly a question of design and implementation, with regards to a quasi-multitenant application.

Let me try to explain the current system, and what the goals are with how the application should work.

I'm creating an application for a client, who in part as their own clients (we'll call these sub-clients going forward). The sub-clients do not have access to the system. The core of the application is to store sub-client data in separate databases and be able to download exported versions (filtered, cropped down etc.) of the data in the tables. Basically the analytical staff need a quick way to export database data for use in other applications.

The frontend of the application is the same regardless of which sub-client a users selects to see data from, the only difference between them is the data that is being stored.

Here is where it gets interesting, and while I have implemented a solution, I'm not sure if it is a good one.

Each sub-client has a potentially unique database schema. While this is fairly easy to get around using node + knex.js, using what seems like any other setup (like Go, or Rust), not knowing your SQL schema is a major pain. This becomes increasingly difficult when trying to add in automated data insertion, where users can upload data files to be added to the database. Without knowing the structure of the tables it is very hard to verify what they are trying to upload fits.

Not using an ORM makes this a bit easier, as with knex I can just query for each table's columns prior to continuing on, or I can just use SELECT * in my queries and just get everything, which for displaying the tables in the browser is what is needed. Data is currently uploaded/updated semi-manually using some python scripts with pandas.

The other potentially hard part of this is that new sub-clients may need to get up and running quickly, and they may have a fair number of tables (10-20 at the start). With just using pandas to quickly dump the data into the database you can get going quite quickly, but adding to the data then becomes a bit more of a task. On the other hand, if we used an ORM, we would need to create new models for each table for new sub-clients, prior to that sub-clients data being accessible and updateable.

Being able to take the data insertion capabilities out of my sole responsibility would be a huge asset to this project, as then I am not the bottleneck in getting new data into the application while also running the devops on it as well.

I have been reading up on multi-tenancy, but I'm not totally sure that is what this application is. It both is and isn't. There is only one version of the application, and it is only available to one group of people. The data that the application uses, pertains to various different groups, who have different data structures.

I keep feeling like there must be a better way to handle the database. With each sub-client having a different schema, it makes using something like an ORM almost impossible from a maintenance perspective, but not using an ORM could lead to improper data being inserted into a table.

I think a part of this is when using python for data extraction you don't have to worry about types as pandas + sqlalchemy do a good job of fitting your data into the tables. When using more strongly typed languages, without knowing what data is coming in and the structure, everything ends up being strings, when most of the data is numerical. While I don't really like python, the flexibility in data management does make it easier.

Does anyone have suggestions of how I may better approach a nice solution?

Using existing regression model for statistics

Peter — Fri, 30 Oct 2020 17:24:32 +0000

Looking for some direction on this situation.

Previously, I have created OLS models using statsmodels. This works great and is easy to use when creating a new model from scratch.

Now, we have a new requirement to allow users to submit an existing equation (from a pre-existing model), to allow for comparison between the user submitted regression model and the statsmodels generated model. This comparison would ideally be able to be done looking at each models resulting ANOVA table.

I have looked at using r-style formula to set the existing equation, like so.

eq = 'y ~ 2.3(x) + 0.86(y) + 0.85(z) - 1786'

This causes some issues with how statsmodels implemented the equation parsing, which uses patsy.

Basically I'm looking for a way to set a regression formula, and pass in data to generate stats about the model.

Possible solutions don't have to be in python.

Thanks in advance!

wasm-pack access denied

Peter — Thu, 04 Jun 2020 18:22:16 +0000

I am wanting to start using rust as wasm in a preact application.

I am running on a Windows 10 machine with the latest versions of both rust and wasm-pack installed.

When I run wasm-pack --version, I get the version print out as expected.

When I try and run any other wasm-pack (i.e. wasm-pack build) command I get OS Error 5 Access Denied.

Related GitHub issue

Has anyone else encountered a similar problem or have some places I could look for a solution?

I have tried re-installing wasm-pack (both using the windows installer found here and using cargo install wasm-pack). I have also tried changing the permissions on both the cargo/bin folder and the application directory.

Web Workers - Introduction

Peter — Thu, 20 Jun 2019 17:57:23 +0000

Some pre-reading if you wish!

After reading the above post I started thinking more about web workers. Why are more people not talking about them. So many people seem to be worried about performance, so why is the worker discussion not more prevalent?

The MDN Docs for workers are quite extensive, discussing every aspect from API Specs to usage.

My initial reasoning for not using them was that I did not think that there really was a need for them. Second, the initialization methodology

const worker = new Worker('/path/to/worker.js')

does not jive nicely with a full TypeScript environment. To use the base call method, you would need to create a separate worker.ts file to transpile over, and it would need to be in the correct location for the final application (which may be quite different from where app src is). I found this frustrating, so I decided at the time to not bother with them.

Then I saw this post from Surma, and I started thinking more about them again. With a data vis dashboard as our primary app, we could definitely benefit from moving calculations from the main thread into a worker.

So I started looking through some of the worker wrappers mentioned the above post (Comlink & Workerize). I started with workerize, then quickly decided I wanted a more in depth approach with multiple methods, so I settled on using Comlink! Using webpack, I also found webpack worker-loader.

Comlink does a good job at making you feel like working with your worker is working with a class. Methods are easy to use, and setup is pretty straight forward! If you are on the fence about adding workers to your app, I would suggest taking a look at Comlink, it makes the process of using a worker way easier, and pairing it with worker-loader takes the awkwardness of defining things as workers away.

The next post in the Workers series will be about my implementation!

The Beginning

Peter — Sat, 15 Jun 2019 05:13:09 +0000

For my first post I decided I should give some background as to how I got to where I am today!

The Start

I started at a small engineering firm about 3 years ago, with only some minor coding knowledge. While coding was supposed to be a portion of my position, I became more interested in moving our data visualization dashboard forward.

The initial stack was pretty simple, data fetching from .csv, some minor database interaction through a small php backend. The frontend was initially built using DC.js and jQuery, with no webpack or build/toolchain.

We quickly began to stress the limits of what DC could do perfomantly. Thus began my first major project: writing a custom charting library.

The First Project

First I needed to learn how to create a library that could be referenced under a common namespace. So I read the d3.js source code! Went in and copied the umd header section and started going.

Then it was callbacks, how to get functions to update something in a constructed function (hindsight should have made it a class, but hindsight is 20/20, we're also not at the end).

The initial code was pretty ugly and not DRY even in the slightest. No base to extend from. The only upside is we got away from the limitations of DC.js, and cleared up quite a bit of rendering jank.

The downside: still importing jQuery(30kb), d3.js (70kb) and at it's most lean my library @ 70kb. And that was only the dependencies. The rest of our code was over 200kb.

This led to the next step: removing jQuery.

The removal of dependencies and the start of webpack

jQuery was relatively easy to remove. d3 does many of the same things jQ does. One dependency gone!

Now I had seen webpack mentioned before, but up to this point we only had minimal use for it, and as it complicated our build chain, it took a bit for us to adopt it. Once we got it figured out, it was life changing. Still not using npm at this point, but it was a start.

lithtml

We then moved to using lithtml. I had though of moving to a ui framework for a bit, at at the time this was the least daunting option.

From here I started my second project, a quasi-framework using lithtml as the render agent.

After struggling with implementation I decided, this has already been done, why not use an established library. So I started messing around with React. But not wanting to get bound up with its ties to Facebook, I opted for preact!

preact + TypeScript

Once we started to use preact, I opted to use ts along side, for proper prop checking.

This also came with properly using npm, and a fairly in depth webpack config.

At this point we had fully transitioned to preact, but the charting library had gone pretty much untouched, with some preact wrappers for proper integration.

Then came my most recent and now open source project preact-charts!

preact-charts and OSS contributing

This started out as creating a small library for us to use internally, but I open sources it as all the current chart libs were react based, with no pure preact equivalents. It's still a WIP, but its stable, and currently supports both preact 8 and the upcoming preact X.

Then I got into supporting the libraries I used. Started with some bug reports, that slowly turned into pull requests.

I now enjoy helping out with the preact typescript definitions where I can!

Hopefully in the future I will continue to engage more with the dev community, and help out on many more projects along the way.

Want to give a shout out to the preact team! All are super friendly and willing to help make your pull requests as awesome as possible and help out on slack. Without their support, I would not have the confidence to help out where I can!