DEV Community: Greggman

[Rant] Github needs to fix their connections to Japan

Greggman — Thu, 27 Aug 2020 09:11:35 +0000

In particular downloading release files.

Trying to install Electron it can often take > 10 minutes

That picture is after it said > 600 seconds and I thought to take a screenshot.

It's not just at my home. I've seen this issue all over Japan on different machines over the last several years. I have even seen it using the free wifi at the Tokyo Amazon co-working space in the Amazon Tokyo headquarters (github release files are served from AWS!)

When I bring it up with github they brush me off as "not our problem" but clearly it is their problem.

I have a fast connection. I'm using NTT (the AT&T of Japan) with a 1gig fiber optic connection and So-Net (Sony's internet service). I can download games from Steam for example in quick. For example I just picked to install GTA5 in steam.

You can see above it's going at 37.1 MB/s. Electron is only 72meg. If things were working correctly it should only take a few seconds, not > 10 minutes to download

It's not just Steam. Most services are fast. Fast.com(netflix) reported 300Mbps last night. Right now it's at 160Mbps.

Google's speedtest reports 198.27Mbps

Google's Fiber test shows 30Mbps down, 199Mbps up from Tokyo to Los Angeles

I'm bringing this here because it's a github related issue and github is a cornerstone of software development. The nice people staffed at github seem to be customer service reps reading from a customer service database instead of considering their impact on the devs around the world. It would be great if someone with the power to make stuff happen at github (or Microsoft) would give this issue more consideration.

Ideally if they are going to be a backbone of development they need to mirror the files across the world instead of only serving from Seattle AWS (which is apparently what they are doing)

Github claims github pages are served by a "global content delivery network". Can that same network be used to speed up release files? Here's crossing my fingers. 🤞

Forgetting to "npm install" solved...

Greggman — Wed, 15 Jul 2020 20:51:03 +0000

Does this sound familiar?

$ git pull upstream master
$ npm run build

ERROR! ERROR! ERROR! ERROR!

5 to 15 minutes of searching and finally

$ npm install
$ npm run build
build complete

The dependencies changed, you didn't notice, and doh! Okay, maybe that only happens to me? 😊

Well, you can fix that with ld-check-dependencies. Just add it to your build script in your package.json as in

  ...
  "scripts": {
    "build": "ld-check-dependencies && rollup ..."
  },
  ...

Now when you build if your dependencies are out of date you'll get told to run npm install

The best part is ld-check-dependencies is tiny. It's only 9 lines of code, 0 dependencies.

photo by: Rikki's Refuge

Reduce Your Dependencies

Greggman — Sat, 23 Nov 2019 03:47:29 +0000

I recently wanted to add colored output to a terminal/command line program. I checked some other project that was outputting color and saw they were using a library called chalk.

All else being equal I prefer smaller libraries to larger ones and I prefer to glue libraries together rather than take a library that tries to combine them for me. So, looking around I found chalk, colors, and ansi-colors. All popular libraries to provide colors in the terminal.

chalk is by far the largest with 5 dependencies totaling 3600 lines of code.

Things it combines

It combines checking whether or not the your output stream supports colors. Because of this it has to add way to tell it don't check for me because I'll do the checking myself
It peeks into your application's command line arguments magically looking for --color or --no-color so without modifying your app or documenting what arguments are valid it will look at these arguments. If your app uses those arguments for something else you lose.
It combines all the named colors from HTML even though they are of questionable usefulness in a terminal.
It includes 800 lines of code for color conversions so you can use rgb or hsl or lab or cmyk etc.

Next up is colors. It's about 1500 lines of code.

It hacks the string prototype. The author seems to think this is not an issue.
It has a theme generator which works something like this

colors.setTheme({
    cool: 'green',
    cold: 'blue',
    hot: 'red',
  });

And you can now do

colors.hot('the sun');

Like chalk it also spies on your command line arguments.

Next up is ansi-color. It's about 900 lines of code. It claims to be a clone of colors without the excess parts. No auto detecting support. No spying on your command line. It does include the theme function if only to try to match colors API.

Why all these hacks and integrations?

Themes

Starting with themes. chalk gets this one correct. They don't do anything. They just show you that it's trivial to do it yourself.

const theme = {
  cool: chalk.green,
  cold: chalk.blue,
  hot: chalk.red,
};

console.log(theme.hot('on fire'));

Why add a function setTheme just to do that? What happens if I go

colors.theme({
  red: 'green',
  green: 'red',
});

Yes you'd never do that but an API shouldn't be designed to fail. What was the point of cluttering this code with this feature when it's so trivial to do yourself?

It gets worse though because a new user who sees console.log(colors.hot(someMsg)) will be effectively be taught that colors.hot is an official API of colors. Then'll then copy that to some other project and learn that in fact no, it's an app specific hack. If they had used more direct way it arguably becomes clear. I've had to help hundreds of users on stackoverflow where some example they saw had monkey patched on a non-standard feature to some object and then when they tried to use it in their own code they got an error and didn't understand why because it looked like part of the official API but wasn't.

Color Names

It would arguably be better to just have them as separate libraries. Let's assume the color libraries have a function rgb that takes an array of 3 values. Then you can do this:

const pencil = require('pencil');
const webColors = require('color-name');

pencil.rgb(webColors.burlywood)('some string');

const chalk = require('chalk');

chalk.keyword('burlywood')('some-string');

In exchange for breaking the dependency you gain the ability to take the newest color set anytime color-name is updated rather than have to wait for chalk to update its deps. You also don't have 150 lines of unused JavaScript in your code if you're not using the feature which you weren't.

Color Conversion

As above the same is true of color conversions

const pencil = require('pencil');
const hsl = require('color-convert').rgb.hsl;

pencil.rgb(hsl(30, 100, 50))('some-string');

const chalk = require('chalk');

chalk.hsl(30, 100, 50)('some-string');

Breaking the dependency 1500 lines are removed from the library that you
probably weren't using anyway. You can update the conversion library if there are bugs or new features you want. You can also use other conversions and they won't have a different coding style.

Command Line hacks

As mentioned above chalk looks at your command line behind the scenes. I don't know how to even describe how horrible that is.

A library peeking at your command line behind the scenes seems like a really bad idea. To do this not only is it looking at your command line it's including another library to parse your command line. It has no idea how your command line works. Maybe you're shelling to another program and you have a —- to separate arguments to your program from arguments meant for the program you spawn like Electron and npm. How would chalk know this? To fix this you have to hack around chalk using environment variables. But of course if the program you're shelling to also uses chalk it will inherit the environment variables requiring yet more workarounds. It's just simply a bad idea.

Like the other examples, if your program takes command line arguments it's literally going to be 2 lines to do this yourself. One line to add --color to your list of arguments and one line to use it to configure the color library. Bonus, your command line argument is now documented for your users instead of being some hidden secret.

Detecting a Color Terminal

This is another one where the added dependency only detracts, not adds.

We could just do this:

const colorSupport = require('color-support');
const pencil = require('pencil');

pencil.enabled = colorSupport.hasBasic;

Was that so hard? Instead it chalk tries to guess on its own. There are plenty of situations where it will guess wrong which is why making the user add 2 lines of code is arguably a better design. Only they know when it's appropriate to auto detect. (PS: you may want to detect stderr separate from stdout via something like colorSupport({stream: process.stderr}).hasBasic).

Issues with Dependencies

There are more issues with dependencies than just aesthetics and bloat though.

Dependencies = Less Flexible

The library has chosen specific solutions. If you need different solutions you now have to work around the hard coded ones

Dependencies = More Risk

Every dependency adds risks.

Risk there will be a security vulnerability
Risk a dependency will be abandoned
Risk the library you want to use will depend on a old outdated version of one of its dependencies
Risk a malicious actor will compromise one of the dependencies
Risk by expanding the number of people you have to trust.

You need to trust every contributor of every dependencies. A library with 5 dependencies probably has between 5 and 25 contributors. Assuming the high end that's 25 people you're trusting to always do the right thing each time the library is updated. Maybe they got angry today and decided to take their ball home or burn the world. Maybe they got offered $$$$$$$ to help hack someone and needed the money for their sick mom. Maybe they introduced a bug by accident or wrote a vulnerability by accident. Each dependency you add adds a larger surface area for these issues.

Dependencies = More Work for You

Every dependency a library uses is one more you have to deal with. Library A gets discontinued. Library B has a security bug. Library C has a data leak. Library D doesn't run in the newest version of node, etc…

If the library you were using didn't depend on A, B, C, and D all of those issues disappear. Less work for you. Less things to monitor. Less notifications of issues.

Lower your Dependencies

I picked on chalk and colors here because they're perfect examples of a poor tradeoffs. Their dependencies take at most 2 lines of code to provide the same functionality with out the dependencies so including them did nothing but add all the issues and risks listed above.

It made more work for every user of chalk since they have to deal with the issues above. It even made more work for the developers of chalk who have to keep the dependencies up to date.

For chalk, just like they have a small blurb in their readme on how to implement themes they could have just as easily shown how to do all the other things without the dependencies using just 2 lines of code!

I'm not saying you should never have dependencies. The point is you should evaluate if they are really needed. In the case of chalk it's abundantly clear they were not. If you're adding a library to npm please reduce your dependencies. If it only takes 1 to 3 lines to reproduce the feature without the dependency then just document what to do instead of adding a dep. Your library will be more flexible. You'll expose your users to less risks. You'll make less work for yourself because you won't have to keep updating your deps. You'll make less work for your users because they won't have to keep updating your library just to get new deps.

Less dependencies = Everyone wins!

Heavy Resource References in React

Greggman — Tue, 26 Mar 2019 00:38:33 +0000

I'm curious if there are any good examples or best practices of managing heavy resources with react. I'm not sure if I can give any good examples but I'll try.

Typical heavy resources are images. Images seem to generally not be managed directly. They are referenced by URI and managed by magic behind the scenes.

const MyComponent = () => { return (
  <div class="recipe">
    <div class="title">Fruit Salad</div>
    <div>
      <img src="images/banana.jpg" />
      <img src="images/strawberry.jpg" />
      <img src="images/grape.jpg" />
    </div>
  </div>
  <div class="recipe">
    <div class="title">Banana Split</div>
    <div>
      <img src="images/ice-cream.jpg" />
      <img src="images/banana.jpg" />
      <img src="images/whipped-cream.jpg" />
    </div>
  </div>
)};

In the example above react declares <img> tags but the resources themselves are managed by the browser. The browser magically looks at the src property and loads the corresponding image. Even though both recipes refer to banana.jpg the actual data for the image will only be loaded once. If this part of the tree of nodes is not part of the active DOM the browser is free to release all those images from memory.

So, let's say we want to do something else just as resource intensive. I'm not sure what a good example is. Let's imagine it's a graph of sales.

I see lots of libraries that do something like this

const MyComponent = () => { return (
  <MyGraphComponent xaxis="date" yaxis="amount">
    <data>
      <datum xaxis="2012/01" yaxis="145" />
      <datum xaxis="2012/02" yaxis="121" />
      <datum xaxis="2012/03" yaxis="131" />
      <datum xaxis="2012/04" yaxis="152" />
      ... 2000 items ...
    </data>
  </MyGraphComponent>
)};

That seems wrong compared to the <img> example. Following the <img> example the data for the component should be external (orthogonal) to the <MyGraphComponent>. Like <img> the data is not children in the graph rather it's something that should be referenced somehow so that multiple components can access the same data.

Another example might be a 3D scene

const MyComponent = () => { return (
  <scene>
    <node transform="translate(10, 20, 30)">
      <sphere radius="1" radialDivisions="100" verticalDivision="50" />
      <material color="red">
    </node>
  </scene>
)};

Here again it looks innocent enough but again it's not following the example set by <img>.

We can imagine a scene with multiple versions of the same sphere. Following the <img> example we'd imagine something more like

const MyComponent = () => { return (
  <scene>
    <node transform="translateX(-5)" geometry="100-50-unit-sphere.geo" material="red-material" />
    <node transform="translateX(0)"  geometry="100-50-unit-sphere.geo" material="red-material" />
    <node transform="translateX(5)"  geometry="100-50-unit-sphere.geo" material="red-material" />
  </scene>
)};

making it clearer the example above this one was probably the wrong model.

We could try

const MyComponent = () => { return (
  const geometry = createSphere(100, 500);
  const material = createMaterial({color: 'red'});

  <scene>
    <node transform="translateX(-5)" geometry={geometry} material={material} />
    <node transform="translateX(0)"  geometry={geometry} material={material} />
    <node transform="translateX(5)"  geometry={geometry} material={material} />
  </scene>
)};

It's not clear how these would work. For <img> they are magically referenced by URIs but for other heavy resources it's much less clear what should happen and how they should be managed.

Maybe by naming things?

const MyComponent = () => { return (
  // only creates resource if resource for that name doesn't already exist
  createGeometry('100by500sphere', () => { return new Sphere(100, 500); });
  createMaterial('redMaterial', () => { return new Material({color: 'red'}); });

  <scene>
    <node transform="translateX(-5)" geometry="100by500sphere" material="redMaterial" />
    <node transform="translateX(0)"  geometry="100by500sphere" material="redMaterial"  />
    <node transform="translateX(5)"  geometry="100by500sphere" material="redMaterial"  />
  </scene>
)};

The above has the strange property that we're making things by name and assuming no name conflicts. But, it does follow the <img> example and works just like <img> in that names (including path) have to be unique.

Maybe another way would be to require declaring resources early like resource factories

resourceManager.registerResources({
  '100by500sphere': () => { return new Sphere(100, 500); },
  'redMaterial': () => { return new Material({color: 'red'}); },
});

then elsewhere

const MyComponent = () => { return (
  <scene>
    <node transform="translateX(-5)" geometry="100by500sphere" material="redMaterial" />
    <node transform="translateX(0)"  geometry="100by500sphere" material="redMaterial"  />
    <node transform="translateX(5)"  geometry="100by500sphere" material="redMaterial"  />
  </scene>
)};

Assuming the registering of resources happens in one place the naming conflict issue disappears and like images resources are defined external to the part react is managing. Of course some would argue it's no fun to do it this way where you can't just declare stuff inline like the first 3D example

And of course there is nothing saying the <img> way is the correct way. It's only the observation that the resources are not truly part of the tree of nodes being managed by React. They aren't children, they are orthogonal resources. And, that the only common similar examples in react are images, audio, and video, all of which are usually referenced by URI, not as children.

Are there any good references for doing this in the true spirit of React?