DEV Community: Chee Aun 🦄

Replicating 3D Trees from Apple Maps

Chee Aun 🦄 — Fri, 05 Nov 2021 00:00:00 +0000

On June 2021, Apple’s WWDC21 event announced the all-new Apple Maps, introducing an amazing new three-dimensional city experience with unprecedented detail for elevation, neighbourhoods, buildings, trees, and road features.

It looks like this.

There’s a lot going on here, but I’m going to focus on only one.

Trees.

Two years ago, I wrote “Next-level visualizations with ExploreTrees.SG”. It has details on how I try to render three-dimensional trees on a map and my experience along the way. I’m not an expert in this topic but I had my fair share of trials and errors while working on it.

Here’s a comparison of my attempt and Apple’s:

They look different but I have to admit, Apple’s one looks so good.

Let me recap on how I came up with my attempt in the first place:

The trees have to look semi-realistic. They cannot be too realistic because they’ll never look like the real thing anyway. Real-world tree measurement is actually very complicated.
There are way too many species of trees with different crown shapes and sizes. I settled on a simple sphere geometry with leaves texture. For some reason, I was keen on making the crown see-through (cracks between leaves) because I feel that’s how trees should look like.

Looking at Apple Maps’ trees, I was kind of surprised they also use a sphere but with random “bumps”, which gives some form of irregularity. The trunk is also slightly curved and they even add a drop shadow.

I’m so fascinated by this level of detail and decided to study how Apple is designing these trees.

So far, there are three types of trees/crowns.

Obviously Apple didn’t tell us exactly how many and why there are three types. I kind of looked around and deduced that on my own. I could be wrong.

For weeks, I’ve been thinking really hard why, and one day I realised that these are analogous to the trees in the emoji list:

For those emoji fanatics, it’s also known that there are two more trees; 🎄 Christmas tree and 🎋 Tanabata tree. But we all know they are different 😉.

Wow, I find this pretty smart. In my own attempt, I was scratching my head on how to diversify the crowns based on the tree families or species. Perhaps I was thinking too much, looking at this diagram from “Structure of Forest: Crown of Tree, Types, Canopy, Videos and Examples”.

Now that I see Apple Maps showing only three types of trees, things seem easier.

I mean, why are there only three types of trees in the emoji list in the first place, right? Hmm, I think I’ll leave that question for another day 😜.

Revisiting the code

I don’t update my ExploreTrees.SG project frequently. I kind of revisit it once a year, while juggling with my other projects, mainly to update the dataset and code dependencies.

I looked at the repository, created a separate folder and started working on it.

On 3 October, I showed a sneak peek:

I made a new map style on Mapbox Studio that matches Apple Maps’ looks. It’s pretty close but not there yet. Map enthusiasts would notice the change for the roads. It’s grey instead of white, yellow or orange 😉.

I re-used the code and styled the tree crown with green colour, instead of the leaves texture.

The lighting and shadows look a little odd, so I made some adjustments.

Slightly better.

I wanted to try making the sphere “bumpy”, like the one in Apple Maps.

Using Spline, I manage to make a 3D model of a sphere with displacements.

Unfortunately on Spline, when exporting this to a GLTF model file, it becomes a sphere without displacements 😑. Maybe these displacements are just “decorations” on top of the model and not really part of it. I’m not sure since I’m not a 3D designer 😅.

Later on, I figured out that I could make my own displacements by modifying the model itself. Spline has a feature where once a 3D geometry is added, pressing "Smooth & Edit" will convert it into a subdivision surface. It’s pretty darn powerful because I can edit a surface, line or even a point of any surface, drag them, and split them into multiple segments! 🤯

The end result looks amazing.

This gives me goosebumps 😬.

To avoid the trees looking too similar, the crowns are initially rotated randomly with JavaScript Math.random, but it wasn’t a good solution because every time it’s re-rendered, the crowns’ rotation change.

So, the solution is that they are deterministically random, rotated based on the tree ID (string of integers). It’s not complicated; the rotation in degrees is calculated as (id.slice(-1) / 9) * 180.

I compared them with Apple Maps, side-by-side.

Left is my attempt on Mapbox tiles. Right is Apple Maps.

It’s pretty close. Maybe the sphere should have more bumps, but I don’t plan to make them too similar one-to-one.

Launching a mini-site

After spending time on this, I wanted to share my work to the public.

But I have a dilemma here.

ExploreTrees.SG was initially built to be an immersive 2D-based visualisation site for all trees in Singapore. 3D-based trees were an afterthought that I fancied and sort of retrofitted in it.

Having both 2D and 3D mode on a single map or page proves to be quite confusing.

2D is great for visualisations. It’s easier to navigate and understand. 3D negates that advantage due much more complicated panning and rotation gestures with more overwhelming visuals.
3D mode looks really cool and gives a spatial overview of how the trees look like in real life. However, it’ll only work best when zoomed in. From a visualisation perspective, it’s harder to grasp the full picture. It’s also more taxing on the visitor’s computer resources.
Discovery of the 3D mode feature hasn’t been great. There’s a ‘3D’ button on the right side of the map but I guess most people don’t click it. Users can trigger 3D mode by zooming in and tilting the map but that requires quite a few gestures instead of simple clicks. 3D mode is also only available on more powerful desktop browsers, mainly due to the way it is coded (loading all 500K trees at once).

I decided to create a separate mini-site that strives to do only one thing and do it well.

On 5 October, I launched ExploreTrees.sg/3D.

The launch gathered quite a few attention 🤩.

The site is also featured on Maps Mania: Mapping Trees in 3D!

Post-launch improvements

The amazing part here is that I manage to make it work on a mobile browser.

Previously, this was deemed impossible because the site is loading 500+K trees in the browser and causing memory pressure issues.

Here’s a quote from deck.gl’s documentation:

Modern phones (recent iPhones and higher-end Android phones) are surprisingly capable in terms of rendering performance, but are considerably more sensitive to memory pressure than laptops, resulting in browser restarts or page reloads. They also tend to load data significantly slower than desktop computers, so some tuning is usually needed to ensure a good overall user experience on mobile.

For this mini-site, the page doesn't load all 500+K trees 😉.

The trees data are loaded via Mapbox tile sets. Technically they are vector tiles, broken up into a uniform grid of square tiles at multiple zoom levels. They can be created by converting a GeoJSON file into a mbtiles file, using tippiecanoe.

After that, the data is filtered with Mapbox GL JS’s queryRenderedFeatures function.

Returns an array of GeoJSON Feature objects representing visible features that satisfy the query parameters.

This can be used to get a list of trees that’s only visible in the current viewport. Think of it like the Intersection Observer API but for maps.

However, loading a number of trees based on viewport boundary still depends on the map zoom level. When zoomed-in, number of trees could be around 1K and that’s fine. When zoomed-out, number of trees could be over 5K+ and will cause performance issues.

It needs a form of gradual appearance of trees when zooming in.

Observing how Apple Maps work, I came up with a solution — align the zoom levels with the height of trees. The logic goes like this:

Zoom level 0 to 15: don’t show trees.
Zoom level 15: start showing trees with height more than 24 meters.
Zoom level 19: show all trees (more than 0 meters).

Transitioning between zoom level 15 and 19, the amount of trees shown will be based on height value of 24 meters to 0 meters. In other words, as the user zooms in, the taller trees will appear first and gradually surface the shorter trees. This makes sense because there’s no point rendering short trees when they’re barely visible on lower zoom levels.

The math and code look like this:

const minZoom = 15;
const maxZoom = 19;
const minHeight = 24;
const maxHeight = 0;

const height = Math.max(0, ((zoom - minZoom) / (maxZoom - minZoom)) * (maxHeight - minHeight) + minHeight);

const trees = map.queryRenderedFeatures(geometry, {
  filter: ['>', 'height_est', height],
  layers: ['trees'],
});

The minHeight value is hardcoded to 24 because I already know the highest value from the dataset, so this may not work for other datasets. The geometry variable is the viewport boundary as mentioned above.

Here’s a video of how the experience looks like:

What about the other crowns?

The map was showing only deciduous trees. There are still palm trees and evergreen trees left to be done.

Actually, how do I know if a tree is palm tree or evergreen tree?

I researched and figured out that the family for palm trees is Arecaceae (Palmae). For Evergreen trees, there are 14 families; Araucariaceae, Cupressaceae, Pinaceae, Podocarpaceae, Taxaceae, Cyatheaceae, Aquifoliaceae, Fagaceae, Oleaceae, Myrtaceae, Arecaceae, Lauraceae, Magnoliaceae, and Cycadaceae. 😵

At this point I realised I’m using these terms differently. For me, they are meant to differentiate the crown shapes, because that’s all I care about for this 3D rendering.

Quoted from Wikipedia: Deciduous:

In the fields of horticulture and botany, the term deciduous (/dɪˈsɪdjuːəs/; US: /dɪˈsɪdʒuəs/)[1] means "falling off at maturity"[2] and "tending to fall off",[3] in reference to trees and shrubs that seasonally shed leaves, usually in the autumn; to the shedding of petals, after flowering; and to the shedding of ripe fruit. The antonym of deciduous in the botanical sense is evergreen.

The last sentence caught my attention for some reason.

Okay, the terms don’t relate to the crown shapes at all. In fact, some evergreen trees’s crowns don’t even look like a Christmas tree 🌲🎄. Even more surprising to me, one of the families of Evergreen trees is Arecaceae, which is a 🌴 palm tree! 🤯

To be clear, here’s a Venn diagram of how I understand this:

Anyway, I decided to simplify the scope. According to Emojipedia, the meaning of the 🌲 Evergreen tree emoji is:

An evergreen tree, which keeps green leaves all year round. Depicted as a tall, dark green, cone-shaped tree with shaggy, layered leaves, as a pine or fir, showing a brown trunk.

Looking at Wikipedia pages of Pine and Fir, both of them falls under the Pinaceae family. So let’s do this:

Evergreen trees = Pinaceae
Palm trees = Arecaceae (Palmae)

That's it.

Next step would be the 3D models 😅.

Initially, I contemplated a lot on whether I should continue doing this, but after a few practices with Spline, I actually did it!

This is one of the most exciting moments of this project 🤩.

I tried rendering the palm tree first, just to see how it’ll look like in the “real world”.

Few points:

Obviously not all trees are palm trees in Fort Canning 😂
The palm trees all look the “same”, despite in different sizes, because they’re not randomly rotated yet.
The colours turn out looking pretty good!

Sprinkled with additional conditions based on the tree family, I managed to make it all work together 🌳🌴🌲.

Note that the evergreen-crown trees are at the bottom in the image.

I was curious, do they actually look similar to the real real world? 🤔 Let’s compare it to Google StreetView!

First location check: Palm trees at Marina Bay Sands.

Correct. 🤩

Second location check: Evergreen trees at Japanese Garden.

Kind of… correct. 😜

Finally, everything rendered in a single beautiful masterpiece. 💎

Applying the finishing touches

After the 3D mini-site is live, I made some updates to the main site:

Fetch the latest dataset
- June 2020: 578,770 trees → Oct 2021: 696,228 trees
- Moved to a new separate repository: cheeaun/sgtreesdata
Upgraded to Parcel 2.
Changed from using MsgPack to plain CSV for the dataset. It’ll be compressed with Brotli and parsed with PapaParse on a web worker.
New custom map styles (probably won’t be noticeable by most people)
Removed “Flowering” category because I realised that this tree characteristic changes often and is actually user-contributed on Trees.sg.
Added the ability to filter by tree family!

After making so many screen recordings and videos, I eventually made a more professional-looking video to show what’s new on ExploreTrees.SG.

Feedback was pretty good as I slowly dive into this new way of writing changelogs 😂.

In fact, I had so much fun browsing around the mini-site, looking at all the 3D trees, that I took some inspiration from Apple Maps web site and made another nice video that feels like flying drone shots around some areas.

Honestly I really enjoyed making these videos 😍.

Wrapping up

My attempt to replicate Apple Maps’ trees has lead me to learn a variety of new things, from 3D design and modelling to various tree terminologies. The final result may not look exactly like it, but it’s pretty close.

Along the journey, I got distracted and accidentally did a side quest though I manage to return back to this project and finish it 😌.

I could be wrong in certain parts regarding tree families and stuff. There’s probably more things that I could improve too. I think I’ll revisit this one day when I feel bored or inspired. 😉

I have to say, ExploreTrees.SG 3D is officially the most satisfying and coolest project I've done this year.

Next-level visualizations with ExploreTrees.SG

Chee Aun 🦄 — Wed, 10 Jul 2019 00:00:00 +0000

Last year, I wrote about one of my most ambitious side project ever, ExploreTrees.SG. It was simply breath-taking.

Revisiting the masterpiece

I haven’t touch it since then. March this year, I suddenly had an itch to update the dataset and see if anything’s changed. I ran the script to fetch the latest data and got this:

It’s a script that scrapes data from National Parks Board’s Trees.sg, showing the total count of trees and species, and generates a GeoJSON file. From the count, I can compare the previous year’s number of trees to this year’s.

March last year, 564,266 trees. March this year, 564,678 trees. It increased by few hundreds!

Looking back previously, I attempted to render all trees on the map in the web browser, but failed due to exceedingly large file size and slow performance. I ended up uploading the data to Mapbox Studio as vector tileset, to be served back on the map. It’s not a pure client-side solution, but a back-end supported one, which makes it no different from Trees.sg except it’s faster 🤷‍♂️.

Ultimately, I still want to achieve this pure client-side solution because I love to push the limits 😉.

A year has passed, technologies have improved, right?

I took a hard look at trees-everything.geojson which contains all trees data. It’s 197.6 MB in size, which is insane for any web sites.

I came up with two ideas:

Give up on the GeoJSON format. Embrace normal JSON, remove all the keys and only store values. Convert, for example {"id": "123", "height": 200} into ["123", 200]. Keys will be hardcoded somewhere else in the code.
Group all coordinates of trees into an array, technically like a line, and convert into Encoded Polyline Algorithm Format. It’s a lossy compression algorithm that allows you to store a series of coordinates as a single string. It’s lossy, with a precision of 5 decimal places, roughly 1 m in distance near equator. For example, [[1.27612,103.84744], [1.28333,103.85945]] will be encoded into a shorter string: wfxFouyxRal@ajA.

Here’s the code:

const data = JSON.parse(fs.readFileSync('data/trees-everything.geojson'));

const props = data.features.map(f => Object.values(f.properties).map(v => v === null ? '' : v));

const points = data.features.map(f => f.geometry.coordinates);
const line = polyline.encode(points);

const finalData = { props, line };

One is the props variable storing all the values and one more line variable for the encoded polyline string.

The final result is a 37.7 MB JSON file. That’s 524% smaller than the GeoJSON file! 😱 After I compress the file with gzip, it becomes 5.7 MB, 3,466% smaller! 😱😱

I think 5.7 MB is not bad. According to this 2017 article on SpeedCurve, the average web page size was 3 MB and it predicted that by 2019, it will be 4 MB.

According to the HTTP Archive, almost 16% of pages today – in other words, about 1 out of 6 pages – are 4 MB or greater in size.

That “today” was in 2017.

Compared to those sites that load 4 MB of junk plus few bytes of real content, I’m building an awesome site with 5 MB of useful data plus few hundred kilobytes of map tiles and JavaScript files.

Obviously at this point, I’m trying really hard to justify my actions 😅. I’m very excited that I manage to squeeze the bytes out of the dataset, but it’s still not small enough to be lower than the average web page size, so I try to deceive myself that I can do this for a good reason 😂.

As I look at the dataset, I noticed a few changes. I investigated and found that:

Some flowering trees are gone, which I have no idea what happened until now.
The API URL has changed from https://imaven.nparks.gov.sg/arcgis/rest/services/maven/PTMap/FeatureServer/2/query to https://imaven.nparks.gov.sg/arcgis/rest/services/maven/Hashing_UAT/FeatureServer/0/query.

The returned response has changed too. The most significant one is the tree girth data. It used to be precise measurements like 1.1 meters, but it became ranges like 0.0 - 0.5 and > 1.5. I don’t know the exact reason but I guess tree girths are pretty complicated stuff.

While re-fetching the data from scratch, I decided to restructure the grids. The fetching works by constructing a list of grids that will be passed as boundaries for the API calls. In other words, every box is equivalent to one API request. The previous one looks like this:

60 boxes, 60 API requests. The right and bottom side are sort of neglected, and luckily there are no trees data in those areas.

The new grid looks like this:

630 boxes, 630 API requests. Higher coverage over the whole Singapore but not the further southern parts. Fortunately, most trees are covered, as in the trees in other areas are not recorded by National Parks yet.

So, I got all the new data. I plotted them on the map with Deck.gl, a large-scale WebGL-powered data visualization library by Uber. It has better performance when it comes to large quantities of map features as I’ve tried in my previous side project.

It was… slow. 😰

Nope, it’s not the fault of the library or Mapbox. It’s Chrome. My Chrome desktop browser was taking a long time decoding the JSON response. Turns out JSON parsing is pretty darn slow for super large files. It’s also a synchronous blocking operation. It took more than 10 seconds on my Macbook Air (2019). 😰

As I start to switch from my home-grown build scripts to Parcel, even the Parcel build step fails with “Allocation failed - JavaScript heap out of memory” errors. I guess it tries to read the JSON file, probably doing something to it and Node keeps running out of memory 😅. I could fix the build step but let’s not get sidetracked here.

Maybe I need a faster JSON.parse. Maybe I could run it in a web worker, but it could be even slower due to the huge payload in message passing.

I had a different idea in mind, that is to use a different data format. Mapbox uses Protocol buffers for the map tiles. Deck.gl supports some form of binary data with an upcoming RFC, which looks… quite complicated for me.

I ended up using MessagePack because:

Protocol buffers need types (double, int64, etc) and it’s quite troublesome to do a quick conversion from JSON. I’ve tried converting the GeoJSON file using Geobuf and the file size still seems bigger than MessagePack’s (with my combo compression ideas mentioned above).
Deck.gl’s binary data thing doesn’t seem to be stable yet and needs manual manipulation on the data.
MessagePack just works™.

There are two libraries available that can perform encoding and decoding of the MessagePack format:

msgpack-lite (listed on the official site)
@ygoe/msgpack

I tried both and chose the latter because it’s smaller in bundle size, according to Bundlephobia. I’m not choosing based on performance because I’ve confirmed that both are way faster than JSON.parse. Instead of more than 10 seconds, the data is decoded in about 3 seconds or less. The file size is smaller too, at around 30 MB but after gzipped, it becomes the same as the gzipped JSON file at roughly 5 MB. Too bad, I was hoping it'll be even smaller 😅.

In the previous dataset, I noticed a few discrepancies such as two or more trees, with different IDs, located at the exact same coordinates. This time, I try to clean it up and remove duplicates, partially with the hopes of reducing the file size.

It might sound silly but I’m actually performing strict comparison of exact coordinates. I’m not kidding that there's a few trees with exactly the same coordinates up to every single decimal place. Imagine two trees with the coordinates of 103.702059 longitude and 1.406719 latitude, with not a single difference in the decimals. 😅

Someone told me before that there could be a tree growing on top of another tree, but I quite… doubt it. 🤨

I checked the data, and could find other similarities in the name and species name. Okay, I could be wrong but I decided to remove the potential duplicates anyway, since this affects the map user interface. Two overlapping tree dots on a map poses quite a challenge, especially for visualization and user interactions. I have high suspicion that the original dataset actually comes from multiple datasets, via different agencies, which could explain this phenomenon.

The final gzipped file size remains the same 😅.

Pushing the limits, again

As the dataset is finalised with decent loading and decoding performance, I proceed to reimplement most of what I did in the first version using Mapbox GL JS, with Deck.gl instead.

In the original implementation with Mapbox GL JS:

Every tree dot is styled via the circle layer.
Pure client-side solution is impossible if there are over 500,000 data points, thus the server-side solution via Mapbox Studio.
Not all 500,000+ trees are rendered on the map in higher zoom levels because that’s how the vector tiles work; dropping or coalescing features on every zoom level if the limits on the tiles are exceeded.

Here’s a code example:

map.addLayer({
  id: 'trees',
  type: 'circle',
  source: 'trees-source',
  'source-layer': 'trees',
  paint: {
    'circle-color': [
      'case',
      ['all', ['to-boolean', ['get', 'flowering']], ['to-boolean', ['get', 'heritage']]], 'magenta',
      ['to-boolean', ['get', 'flowering']], 'orangered',
      ['to-boolean', ['get', 'heritage']], 'aqua',
      'limegreen'
    ],
    'circle-opacity': [
      'case',
      ['to-boolean', ['get', 'flowering']], 1,
      ['to-boolean', ['get', 'heritage']], 1,
      .5
    ],
    'circle-radius': [
      'interpolate', ['linear'], ['zoom'],
      8, .75,
      14, ['case',
        ['to-boolean', ['get', 'flowering']], 3,
        ['to-boolean', ['get', 'heritage']], 3,
        1.25
      ],
      20, ['case',
        ['to-boolean', ['get', 'flowering']], 10,
        ['to-boolean', ['get', 'heritage']], 10,
        6
      ]
    ],
    'circle-stroke-width': [
      'interpolate', ['linear'], ['zoom'],
      11, 0,
      14, 1,
    ],
    'circle-stroke-color': 'rgba(0,0,0,.25)',
  },
});

For the new implementation with Deck.gl:

The dots are rendered with ScatterPlotLayer. May look the same as the former but the styles are written differently.
Pure client-side solution becomes possible. The documentation quotes:

On 2015 MacBook Pros with dual graphics cards, most basic layers (like ScatterplotLayer) renders fluidly at 60 FPS during pan and zoom operations up to about 1M (one million) data items, with framerates dropping into low double digits (10-20FPS) when the data sets approach 10M items.

All trees are rendered in all zoom levels.

The new code example:

new MapboxLayer({
  id: 'trees',
  type: ScatterplotLayer,
  opacity: 1,
  radiusMinPixels: .1,
  radiusMaxPixels: 5,
  lineWidthUnits: 'pixels',
  getLineWidth: 1,
  getLineColor: [0,0,0,200],
  getRadius: (d) => (d.flowering || d.heritage) ? 100 : 3,
  getFillColor: (d) => {
    if (d.flowering && d.heritage) return colorName2RGB('magenta');
    if (d.flowering) return colorName2RGB('orangered');
    if (d.heritage) return colorName2RGB('aqua');
    return colorName2RGB('green');
  },
});

Yeap, shorter.

I have to create a new function called colorName2RGB to convert color names (orangered, aqua, etc) to RGB values in array form ([R,G,B]), because Deck.gl doesn’t support them. This function is surprisingly simple because it uses canvas’s magic fillStyle property to convert a CSS color, instead of a lookup table, thanks to this StackOverflow answer by JayB.

Everything became crazy smooth.

The reimplementation didn’t take long. There were some tiny differences in how the circle scales for different zoom levels but not significant enough for anyone to notice.

Deck.gl has yet again amazed me with its powerful features and performance. Honestly, I always get wowed every single time by how much I can achieve with this library! And I didn't stop there 😉.

This was my previous failed attempt:

It’s a failed attempt mainly due to performance. It was too darn laggy rendering all the 3D tree trunks! I also suspect that some girth measurements were wrong, which you could see that one huge hexagon in the image above, at Fort Canning Park. This was done with Mapbox GL JS’s 3D extrusion features.

This is my second attempt, with Deck.gl:

Those orange dots are the tree trunks in 3D. Lets zoom in.

This attempt is about 3 to 4 times faster in rendering everything 😱 (based on my perception). But, it still lags when panning around in higher levels 😢.

Ugh, I feel like I’m on roller coaster ride when things become fast, super fast, then slow again, then fast again, and then end up slow again 😖. Technically it’s my own fault. Repeatedly, I did try to make it fast and then purposely make it slow again. I have no idea why I keep doing this 😂

Anyway, I had an idea on how to make it fast 😏. I limit this 3D mode to higher zoom levels and then filter the list of trees based on the map’s geographical bounds. Like this:

const bounds = map.getBounds();

const results = ... // magically filter the list of trees based on bounds

trees3DLayer.setProps({
  data: trees3Dify(results),
});

The returned value of map.getBounds() is the smallest bounds that encompasses the visible region of the map. Instead of rendering 500,000+ trees, it can be made to only render few thousands instead.

Basically this not only solves the performance problem, but also makes sense. There’s no point rendering 3D trees on lower zoom levels anyway since they’ll all look like small dots. Users still have to zoom in to see the 3D trunks in detail, which is the same reason why 3D buildings are only visible in higher zoom levels on maps like Google Maps and Apple Maps.

Thanks to this, it’s fast again 😬.

The right side is the Developer Tools Console, logging the number of 3D trees rendered every time the map is zoomed, panned or rotated. It’s quite noticeable that there’s completely no lag at all when panning around. The 3D trees outside of the bounds will only start rendering after the pan, zoom or rotate ends.

Perfect. What’s left for me is to finish up the remaining reimplementation, remove all the old code, choose a better color for these tree trunks and wrap up!

At first, I wanted to color the trunks as brown but it’s too dark and doesn’t contrast well with the dark map tiles. This explains why I choose brighter brown or orange as my first attempt. And then I changed the color to white because it’s still doesn’t feel right somehow…

Yes… 3D tree trunks look kind of weird, right?

I thought to myself, what was I trying to do again? What’s the purpose of the 3D renderings? Isn’t 2D enough for this visualization? I guess that drawing 3D trees on a map would be cool but didn’t really think much after that.

But what makes them look weird?

Oh! The tree crowns! It looks weird because it’s incomplete. Wait, the problem is that I don’t have the crown data, so how am I going to draw the tree crowns? Depending on the tree species or families, I might need to draw different shapes of crowns, which can be a lot of work 😅.

So I start to think, what’s the simplest form of a tree crown that is achievable? Of course, another cylinder.

I did a few quick sketches during my free time:

I have the height information of the trees, but it doesn’t necessarily mean the height of the trunks themselves. I could “chop off” the trunk to about 75% and leave the rest for the crown. The crown could take up 50% of the height so that it looks like it’s covering the trunk. As for the radius of the crown, I can roughly measure it based on the height as well. From my trials and errors, I found the sweet spot for the radius to be roughly 40% of the height.

Sounds like complicated math, but… voilà!

This is the moment when I felt that my efforts have finally becoming fruitful. Little did I know that the crowns actually make such a huge difference!

To be honest, I got super excited when this actually works. I tried to finish up the work and ensuring feature parity with the old implementation. It reached a point where it's almost complete.

I’ve also enabled 3D buildings, for a more complete picture.

…and more photos!

Look at those cute little trees! 😍

Nearby 3D buildings tend to give a pretty realistic context to the 3D trees around them. In a way, it feels like there’s a pattern on how these trees are planted based on the surrounding geography 🤔.

This image above shows one of the data mismatch cases, near Cecil Street, where the trees are no longer there and an interim food centre is built in this area.

Alright, I’m almost done. The 2D tree markers are working. Visualization filters work. 3D trees perform really well.

Just one thing left: laggy blue marker highlighter and tree information hover cards, as rendered on the videos above. On desktop browsers, when the mouse cursor hovers over a tree, the blue marker highlighter will appear around it, with the hover card popping out from the bottom right of the page. There’s a significant lag when the cursor moves across multiple trees on the map as the user interface is trying to keep up.

I tried three attempts.

My first piece of code was using the mousemove event from Mapbox GL JS. When the event is fired, a specific layer from the map can be queried and the marker or feature information can be extracted to be displayed as part of the hover card. The query is done by converting the pixel coordinates of the cursor to the actual map coordinates in latitude & longitude, and then find the nearest map feature from the coordinates. This operation is quite tedious — the mousemove event fires too often, there are way too many features under the cursor and every single call is blocking the UI thread, thus affecting the rendering speed of the marker highlighter and the hover card.

My second attempt was using Deck.gl’s powerful picking engine, which uses something called the Color Picking Technique:

Rather than doing traditional ray-casting or building octrees etc in JavaScript, deck.gl implements picking on the GPU using a technique we refer to as "color picking". When deck.gl needs to determine what is under the mouse (e.g. when the user moves or clicks the pointer over the deck.gl canvas), all pickable layers are rendered into an off-screen buffer, but in a special mode activated by a GLSL uniform. In this mode, the shaders of the core layers render picking colors instead of their normal visual colors.

Honestly, I have no idea how this works. I roughly get it that the engine tries to pick the color under the cursor and somehow manage to find the relevant feature which matches the color?!? But how?

Anyway, I’ve tried it and this method doesn’t work too. Everything’s still slow 😭.

So, lo and behold my third attempt using a spatial index for points with a library called geokdbush:

A geographic extension for kdbush, the fastest static spatial index for points in JavaScript.

It implements fast nearest neighbors queries for locations on Earth, taking Earth curvature and date line wrapping into account. Inspired by sphere-knn, but uses a different algorithm.

I think it’s pretty cool because it’s written by Vladimir Agafonkin, the creator of Leaflet.

Here’s a rough code snippet:

const index = new KDBush(data, (p) => p.position[0], (p) => p.position[1]);
...
const nearestPoints = geokdbush.around(index, point.lng, point.lat, 1, radius);

The geokdbush.around method returns an array of the closest points from a given location in order of increasing distance. It has optional arguments to set custom maximum results, maximum distance in kilometers to search within, and even an additional function to further filter the results! 🤯

In fact, these methods are so useful that I also use it for filtering the rendering of the 3D trees based on the map bounds (in the beginning, for the trees3Dify call).

The result is insanely fast. ⚡️

Even works in 3D mode:

I’m quite satisfied with this.

But I think I can do more. 💪

Plus ultra

Initially I have an idea for getting a listing of all tree species mapped to their own crown shape patterns. I couldn’t find such list unfortunately 😭.

My research led me to few subjects like tree crown classes and I especially like this diagram from Structures of Forest:

These are the various crown shapes of trees and I think I only covered the “full-crowned” type, with blocky cylinders. The other crowns would be a bit difficult to model in 3D…

Similar to how the tree trunks are rendered, the tree crowns are rendered with deck.gl’s SolidPolygonLayer, which works like PolygonLayer but without strokes. Extrusion is enabled from the extruded option combined with z-indices of the coordinates. The circle shape is formed with @turf/circle. Elevation is performed via the getElevation accessor.

This is the basics.

Creating complex polygons for all the different tree crowns would be too complicated for me, both computationally and mathematically 😵.

While developing this project, I’ve been following the updates on the then-upcoming version 7.0 of Deck.gl. The changes that caught my attention was the (re-)introduction of SimpleMeshLayer and ScenegraphLayer, which allows the rendering of actual 3D models in formats such as OBJ, PLY and glTF. This means… I can put real tree models on the map! 😱

Wait the minute, I don’t have a species-to-crown-type mapping yet, so rendering a super realistic full-crowned tree for every single tree would be kind of weird, especially for those tiny little trees. Even if I have the mapping, it would take a gargantuan effort for me to code the species-specific crowns for all 500,000+ trees, manage all the 3D model files, and tune the asset loading & map rendering performance at the same time!

There has to be some compromise here. I want to replace the cylinder tree crowns with something more realistic but cannot be too realistic. So it should be semi-realistic, right?

I read through the documentation and noticed this interesting piece of code from the example:

import {CubeGeometry} from 'luma.gl'

That’s one of scenegraph model nodes provided by luma.gl. Pretty neat.

They are like pre-made 3D objects that can be used like, as what it described, WebGL components. One of the geometries that I found is the SphereGeometry, which looks like what I need. Or perhaps, a sphere is what I think would be a better alternative than a cylinder, right? 🤔

Not only that I can create a sphere object with this, I can also apply a texture on it with the texture property for SimpleMeshLayer.

Here’s a code snippet:

const treesCrownLayer = new MapboxLayer({
  id: 'trees-crown',
  type: SimpleMeshLayer,
  texture: document.getElementById('leaves'),
  mesh: new SphereGeometry(),
  ...
});

The texture property accepts one of these 3 types of value:

A luma.gl Texture2D instance
An HTMLImageElement
A URL string to the texture image

So the document.getElementById('leaves') code above is a reference to an img element in the HTML page. I use this method to preload the image instead of lazy-load it during runtime.

As for the image, I googled and managed to sift through hundreds of images to find one “leaves” texture image from Dim's environmental and architectural textures. Honestly, it’s not easy to find a good texture image at all 😂, but luckily there are existing available resources mostly for game development 😀.

Here’s the moment of truth 🤞:

Oh my god, It works!! 😱😱😱

For this image, I tried playing around with the number of vertices. If it’s reduced, the sphere will have less triangular faces and look less spherical. I kind of suspect that reducing the number of vertices might improve performance but doesn’t seem to affect much.

Anyway, before I get too happy with this result, there are few problems:

The trees look a bit too dark, so might need some sort of lighting. Maybe use brighter colors too.
The tree crown need to be “see-through”, to simulate the empty spaces between the leaves.
The leaves on the crown are too big, especially when zoomed in. I’ll need to make them smaller.

Using Affinity Photo and Affinity Designer, I made some adjustments from the original texture image (top left):

First, I select the dark areas between the leaves and make them alpha-transparent. To reduce the size of the leaves, I make the image bigger instead, by flipping it 3 times, creating a repeatable leaves image pattern.

The results:

Not bad. The leaves are smaller and it’s possible to see the tree trunk hiding inside the crown. 👀

As for the lighting, I use the new lighting effects system introduced in deck.gl v7.0, particularly the AmbientLight source and the new experimental SunLight source. SunLight is a variation of DirectionalLight which is automatically set based on a UTC time and the current viewport. In other words, it simulates the sun by calculating the sun position with a JavaScript library called SunCalc (again, created by Vladimir Agafonkin, creator of Leaflet 🤩).

Coincidentally as I was researching on how the code and formula work, I saw Vladimir tweeting about his tiny 900-byte function to calculate the position of the Sun! 😱 If I’m not mistaken, it’s like a simplified version of SunCalc.

Without further ado, I implemented them all, this way:

const phaseColor = getPhaseColor(timestamp);
const ambientLight = new AmbientLight({
  intensity: phaseColor === 'dark' ? 1 : 1.5,
});
const sunLight = new SunLight({
  timestamp,
  intensity: phaseColor === 'dark' ? .5 : 2,
});;
const lightingEffect = new LightingEffect({
  ambientLight,
  sunLight,
});
treesCrownLayer.deck.setProps({
  effects: [lightingEffect],
});

The light intensities reduce when the sun phase is “dark” to simulate night time.

Before 🔅:

After 🔆:

Looks much better!

Here’s a time-lapse of the sun lighting on the trees, to show how the (sun) light direction changes based on time. Also, it’s not affected by user's local time and will always be in actual Singapore time, since the formula depends on location coordinates instead of time.

Unfortunately, the lighting transition from night to day and day to night is kind of abrupt for now. Just in case someone stays on the site for too long, I’ve also added a time interval to update the lighting every 10 minutes.

Along the way, I’ve added a few useful POIs (Points of Interest) that are listed on NParks, which include these categories of places:

Parks
Community gardens
Heritage roads
Skyrise greenery

On their site, it looks like this:

On ExploreTrees.SG:

Besides the POIs, I’ve also made the train stations and bus stops more prominent to guide navigation and exploration. These markers only appear on higher zoom levels as they are only useful in those levels and to prevent clutter on the map.

As a side quest, I took this chance to try out lit-html for the hover card, replacing the constant innerHTML layout-trashing (destroying and rebuilding the content on every hover event).

The final boss

Despite the fact that I’ve ticked off all the checkboxes in my todos for this project, there’s one problem that I ignore since the beginning of this exercise.

It doesn’t work well on mobiles. 😞

There are a few issues:

MessagePack decoding crashes on Mobile Safari. When I try switch it to JSON format, the browser didn’t crash but it takes a significant amount of time to load. Either way, it’s a pretty bad user experience.
Even after everything is loaded successfully, Mobile Safari will randomly crash after few minutes of panning and zooming on the map. I’ve tried asking a friend to try the site on an iPad, which I suspect has more memory and processing power than iPhone, yet the site still crashes 😭.
The site doesn’t crash on Chrome Android but still kind of laggy. 🐌

Most-likely suspicion could be the memory pressure, as highlighted by Deck.gl’s documentation on performance:

Modern phones (recent iPhones and higher-end Android phones) are surprisingly capable in terms of rendering performance, but are considerably more sensitive to memory pressure than laptops, resulting in browser restarts or page reloads. They also tend to load data significantly slower than desktop computers, so some tuning is usually needed to ensure a good overall user experience on mobile.

I guess loading a 5MB compressed file, which uncompresses into a 30MB file, that loads 500,000+ trees on a WebGL-powered 3D-rendered map is just… too much? 😝

I’ve tried dozens of ways to fix this, like:

Reduce the number of layers on the map.
Reduce the number of polygons on the map.
Remove some features and code that might use a lot of memory.
Apply micro-optimisations that I thought would help.
Reduce number of trees by using cluster mode with supercluster, which kind of defeats the purpose why I build this in the first place.

Nothing works. It’s pretty daunting 😩.

This is the only thing that blocks me from launching this new version of ExploreTrees.SG and I have to make the call.

After days of contemplating, I decided to not replace the version 1.0 of ExploreTrees.SG and instead, slot in the version 2.0 together with it. The old version 1.0 works fine and seems more stable on mobile browsers. The new version works fine on desktop browsers but not the iPad. I’m not sure about Android tablets so I’ll assume they’re slightly better or worse than iPad. 🤷‍♂️

On version 1.0, all the tree data are served from Mapbox Studio, as vector tilesets. They are partially loaded based on zoom levels, so not everything is shown at once. This probably helps in reducing memory pressure, using less bandwidth, and having better performance.

On version 2.0, which I kept pushing the limits, all the trees data are loaded in the web browser, so there’s no more round trip to the server. Technically it’s pure client-side and uses a lot of bandwidth and all the power from the user’s machine to render everything nicely.

I needed a way to switch between the versions based on certain device capabilities.

There’s no way for me to detect the device's or browser’s ability to handle large memory pressure. There’s also no way to detect possible crashes on a web app before it could crash. User agent string detection won’t work here either.

Since there’s no reliable way to detect these conditions, I applied touch detection instead:

const isTouch = 'ontouchstart' in window || navigator.msMaxTouchPoints;
const hqHash = /#hq/.test(location.hash);
const renderingMode = !hqHash && isTouch ? 'low' : 'high';

I made a few assumptions:

Most modern mobile phones have touch screens.
Older phones won’t be able to load the site anyway since it uses WebGL (sorry 🙇‍♂️).
Even though iPad is quite powerful, it also has a touch screen so this detection will rule it out.
The only exception would be desktop computers with touch screens.

I call version 2 as “High-quality mode”, and provide an option for users who are routed to version 1 to switch to this mode.

This option will reload the page and switch to version 2 by appending #hq to the URL. After that if the site crashes, the user can still go back to version 1.

To further optimise the performance of the site, I make good use of the renderingMode variable to conditionally enable or disable certain features on the map. I took one step ahead by applying code splitting with dynamic imports. I extract all version 2 related dependencies into a separate bundle and load it asynchronously on the page.

const {
  MapboxLayer,
  AmbientLight,
  SunLight,
  LightingEffect,
  ScatterplotLayer,
  SolidPolygonLayer,
  SimpleMeshLayer,
  SphereGeometry,
  msgpack,
  polyline,
  KDBush,
  geokdbush,
  circle,
  throttle,
} = await import('./hq.bundle');

Besides that, I did an interesting trick for the MessagePack data file. I realised that MessagePack doesn’t have an official MIME type. Even though I started using Parcel for bundling files, the site is still deployed to GitHub Pages via the GitHub Pages Deploy Action. This means gzip compression could only be available for certain file extensions.

For example, .js files will be served with gzip compression but not .png files. I tried using .mp extension for the MessagePack file and… it’s not compressed unfortunately 😟. Since the site traffic is also handled by Cloudflare, I looked through the list of content types that Cloudflare will compress.

I needed a file format that…

Is whitelisted in the server's content type compression list, from either GitHub Pages or Cloudflare. I’m aware that Netlify allows custom headers which may be able to set gzip headers, but I’m not moving the site there for now.
Bypasses Parcel’s intelligent asset handling. For example, if .json is used, Parcel will include the file content into the JavaScript bundle.
Is not confusing for me to revisit in the future. If I use .png, which could work, the future me would be confused and wonder why it’s used for a non-image file 😅.

I tried pre-compressing the file into .gz but weird things just happen when trying to read the file in JS. 😅

In the end, I chose .ico because it’s one of the formats that is a (image) binary and can be gzip compressed at the same time. It is also rarely used, unlike normal image formats, which could prevent conflicts with other files. I could use .ttf or .otf extensions too but it could be confused with actual font files.

I named the file as trees.min.mp.ico, tested it and it works. 🤩

The launch and the aftermath

On 30 April 2019, I finally relaunched ExploreTrees.SG V2.

Despite all my effort rewriting the code and pushing the limits, I’m especially proud of the 3D trees. It feels like an achievement-unlocked moment for me. 🤩

It got featured on Reddit /r/singapore, thanks to a friend 😉.

Feedback has been positive so far. I’m surprised that the conditional routing for Version 1 and 2 seems smooth enough that no one noticed it. 😝

A day after the launch, I received a pretty interesting email at 2 AM.

So this person asked me for my advice on creating a map for all durian trees in Singapore! I was reading this in the morning, had a good laugh and ignored it. 😂

It was Labor Day and there’s no work. In the afternoon, I had a hunch that this person might not be joking after all.

So, I did some research on durians, found a Wikipedia page on List of Durio species, looked through the dataset and found 286 Durio species! 😱

I didn’t know that there are durian trees in Singapore! Even more surprising, durian trees have flowers! 😱

The next logical step is to quickly load them up on a map.

Looking at this, I was thinking to myself. Are there any other fruit trees in Singapore? I searched through the Internet and found a list of fruit tree species names. I was thinking of how to visualise the fruits on a map, maybe still colorful circles or fruit icons instead? There could be too many exotic fruits to be shown so maybe need to be filtered to only those “popular” ones?

Hmm, yeah… Lots of ideas, yet so little time.

From my experience in building side projects, it’s always good to hold on to an idea for some time before executing them. This time, I’ve already had so much fun rebuilding everything, squeezing every inch of the performance, and even modelling 3D trees on a map. Most importantly I manage to launch my open-source work to the public before I got bored of it. 😆

This has been fun and I’ve learnt a lot.

So, till next time then. 😉

What I learned from printing custom swags

Chee Aun 🦄 — Sun, 02 Jun 2019 00:00:00 +0000

Apparently, I’ve been called the “swag king” of Singapore 😅, mainly due to my voluntary work on printing swags like stickers and t-shirts for the past few years. My contributions have made a significant impact towards the lives of all meetup goers and conference attendees in Singapore, and perhaps other countries as well.

Before I begin, a few disclaimers:

I don’t print swags literally. I simply order them from custom-printing shops and suppliers.
I’m not an expert in custom-printing swags and still have a lot to learn, compared to those who have done this work for many years.
I do this purely out of curiosity. Maybe as a hobby but it’s a rather expensive one 💸.
What I’ve learned may not be applicable to other situations, places or countries. Things might have changed over time. It’s still best to self-experiment before trusting everything written here.

The beginnings

I go to a lot of meetups and conferences. Sometimes these events, usually conferences, give free swags and I’m always very happy to get them. They’ve become a reason for me to attend, and swags are always good for ice-breakers to know new like-minded people.

After years of getting free swags, I start wondering to myself, can I create my own swags? Instead of doing the taking, maybe it’s time for giving? 🎅

I had no clue how this swag stuff works and honestly, been taking them for granted. I used to be quite arrogant and kept asking the organisers for free swags, without realising how difficult it is to produce them in the first place.

It all started with shirts.

Before 2014, most t-shirt swags from the conferences that I attended, in Malaysia and Singapore, were not… super great. I’m not picky about shirts but the quality for conference shirts were not very high-quality, and I don’t usually complaint much since they are free. Maybe due to this fact, I didn’t pay much attention to them, including all other swags like stickers. I just like them because… they are free, not because they’re awesome. Material-wise and design-wise were sub-par and didn’t leave much impression on me.

In 2014, I traveled to San Francisco, attended a few meetups and conferences, and found one very important difference: the swags were much, much better. They felt better somehow and at that time, I don’t really know why. I don’t know what makes it better or how it can be better at all. I thought all stickers and shirts are the same, right?

Right…?

I decided to find out why and did a few experiments.

Shirts 👕

I printed not one, but three t-shirts for my own ‘CA’ logo. They were completely experimental and at that time, I have no idea what I’m doing.

On 31 March 2014, I ordered a t-shirt.

This is my first ever t-shirt with my own logo on it. Yes, I order only one t-shirt from CustomInk and it costs US$24.75. It’s American Apparel Jersey, 100% combed ringspun cotton, size S, and white.

On 13 April 2014, I ordered a second t-shirt from CustomInk.

This time, it’s a grey logo on black background. Gildan Softstyle Jersey, 100% pre-shrunk ringspun cotton, size S and costs US$25.75.

It’s pretty neat that CustomInk allows me to order only one shirt as most vendors enforce a minimum order. This allows me to experiment and see how the shirt looks like.

I personally don’t quite like the white shirt as the base, because it looks more crumpled. The orange logo was too big because I was trying to measure it in inches, a measurement unit that I’m not familiar with (including the rest of the world 😏).

It does match my personal branding, orange and white, but I realise that I have to change it when it comes to shirts, since it’s something that I would be wearing, unlike a web site or a magazine.

Black (including other darker colors) is always the safest bet because:

It’s hard to decide on a base color that suits everyone’s taste.
Black is versatile enough to go well with anything to match your wardrobe.
Black is always mysteriously stylish.

With my second attempt, I use black as the base color with a grey version of my logo. I’ve also made the logo smaller as I try to match the measurements from Iron Man’s arc reactor 😎.

It was good… but the shirt material and printing were not great. The material is 100% cotton and feels thick. The logo print feels like it sticks on top of the shirt and not part of the shirt. I was comparing it to some conference t-shirt swags and the level of quality is completely different.

I did some research and found an interesting material called Tri-blend. Googling it will reveal a few resources like:

Tri-blend is a combination of 3 fibers:

Cotton — Breathable, doesn’t trap heat.
Polyester — Wrinkle-resistant.
Rayon — Fit, durability, and crazy softness.

Sounds amazing, right? It is! Ever since I tried them, I’ve been slowly updating my wardrobe over the years, replacing all my 100%-cotton shirts to this.

I wanted to print my logo on this material but found that the price was too costly to print one shirt so I kind of gave up, until…

On 8 December 2014, I ordered 3 t-shirts with my Kopi.JS logo.

It’s American Apparel Tri-Blend, in S, M and L sizes, and costs me US$105.30 (Yes, still expensive 😂).

I got size S for myself and gave the rest to my other Kopi.JS co-founders ☕️.

On 28 November 2015, I ordered another one (again).

This is the 3rd, and perhaps the last, t-shirt with my logo. I ordered it from ooShirts. It’s Next Level Tri-Blend, DTG-printed, size S and costs US$31.77 (getting more expensive, I know 💸).

I printed this one because I like tri-blend and I want my logo on it 😂. I also wanted to try other brands that produce this material and see if there are any differences.

Turns out it’s not the brand, it’s the fabrics themselves! I don’t know about American Apparel but this Next Level shirt contains 50% polyester, 25% cotton and 25% rayon. My research revealed that there are also the 50/50 blends (50% cotton, 50% polyester) and 65/35 (65% cotton, 35% polyester), which are also known as polycotton. And yeah, there are also other percentage ratios for the tri-blends besides the 50/25/25.

In short, this is how I naively see them:

Cotton is the most popular fabric for normal wear. Best for vibrant printing and also the cheapest option.
Polyester is mostly used for sportswear. Lighter and silkier.
Rayon is super soft and perfect for women dresses, skirts, bed sheets, etc.

At first I don’t quite get rayon, since cotton and polyester already feels soft for me. So I looked into my wardrobe and found that I already have a few polycotton shirts (I didn’t realise it until my research 😅).

One of them is my most favourite so far, the Dribbble t-shirt (with stickers 🤩).

It’s a 50/50 heather black.

Pretty soft. But not as soft as tri-blend. It feels slightly heavier too. I don’t know how to explain it properly but it just felt that way 🤷‍♂️. Maybe I just like the word “tri-blend” because it sounds cool 😂.

So the problem here is that most, if not all, local printers in Singapore don’t have tri-blend t-shirts. 100% cotton is like everywhere and the alternatives are usually dri-fit (100% polyester).

After months of research and googling, I finally found a local custom t-shirt printer in Singapore called Cottony that supplies tri-blend!

So heck yeah, I went in full force.

On 16 August 2016, I ordered 35 t-shirts (12 S, 13 M, 8 L, 2 XL) with the pixelated Kopi.JS logo. They are Next Level Tri-blend Crew, vintage black, DTG-printed, and cost SG$770 (free shipping, SG$22 per shirt).

I gave them away for free, mostly to those who are regulars to my Kopi.JS meetups. It’s also the first time I order so many shirts, all packed inside a huge plastic bag.

Folks who received the shirts, seem to like it a lot, which makes me feel really glad. 🤗

On 19 September 2016, I ordered another batch, 30 t-shirts (12 S, 15 M, 1 L, 2 XL) with the Milo Dinosaur logo, from Cottony. It’s the same Next Level Tri-blend Crew, costs SG$606.60 (free shipping, SG$20.22 per shirt), but with screen printing instead.

Okay now, I mentioned DTG (Direct to Garment) and screen printing. What’s the difference?

DTG
- Ideal for small batches, thus affects the minimum order 😏.
- Good for detailed designs and photos, with lots of colors.
- Gets more expensive for large batches.
Screen printing
- Ideal for simple designs with limited colors (I had to specify the number of colors 😅)
- Cost-effective for large batches (e.g. above 30 pieces).

Here is when it gets a little difficult. If I have a design that’s complicated, I would choose DTG but it’ll be more expensive. If I’m budget-conscious, I would have to tweak the artwork to contain lesser colors so that it would be screen-printed instead.

Screen-printed artwork does look more vibrant, but also feels more sticky (My personal description, some people described it as thicker). DTG-printed artwork feels a little rough but not sticky.

After some conversation with the Cottony folks, the first batch that I ordered was DTG-printed with water-based ink while this second batch is screen-printed with "rubbered-dyed water-based ink", which I think they call it Plastisol(?).

According to my research, there are three types of ink that can be used for screen printing; plastisol ink, water-based ink and discharge ink. 🤯 Oh my god, not only that I need to learn about the various printing methods, I also have to learn about the ink used in each printing method?!? Not to mention that these inks and printing methods also depend on the used fabric(s) for the shirts! 😱

On November 2017, my co-organizers and I printed 35 tri-blend t-shirts for Super Silly Hackathon. There were 26 BELLA+CANVAS unisex tri-blend t-shirts (10 S, 5 M, 4 L, 1 XXL) for the guys and 9 Tultex ladies-cut polycotton t-shirts (5 S, 2 M, 2 L). There were no stock for BELLA+CANVAS ladies-cut so we fallback to Tultex.

Interestingly, there were concerns about this artwork as we were told that DTG printing method is unable achieve the neon-type colors of the design and will make them less vibrant. Heat transfer printing was suggested instead, which is also cheaper than DTG 🤔.

At that time, I decided it’s fine to be less vibrant so we proceeded with DTG.

This led me thinking, what is heat transfer? As I researched, I also found out about dye sublimation which is also a form of heat transfer! They are kind of the same but different 😵. What’s the difference?

Heat transfer
- Works on light and dark-colored cotton, polyester and polycotton blends.
- Adds a layer on top of the garment.
- More affordable, not as durable.
Dye sublimation
- Only works on polyester fabrics, no 100% cotton.
- Said to only work for white or light-colored fabrics.
- Ink becomes part of the fabric.
- More durable, more expensive.

Besides that, I also found that the shirt cuttings and sizes are different between brands as well, even when they’re all unisex. I think this is rather expected since there's really no standard followed by all these providers. 🤷‍♂️

Here’s a list of things that I’ve learned:

Fabric — cotton, polyester & rayon.
Printing method — DTG, screen printing, heat transfer & dye sublimation.
Ink — plastisol, water-based & discharge.
Sizing & cutting — unisex & ladies-cut, differs by brand.

All the above affects or can be affected by these requirements:

Number of shirt pieces (minimum order or large batch orders)
Costs
Number of colors
Quality of print on the shirt (sticky, rough, smooth, etc)
Quality of shirt (durability, breathability, softness, number of washes, etc)
Production lead time and delivery time

It’s like a balancing act. If I want the print to be vibrant, it could be printed on top of fabric and feels sticky or rough. If I have a low order count, it could only be limited to DTG since it’s cheaper. If I want a high-quality shirt with blended fabrics, some printing methods might not work. If the design is complex, it might take a longer time for production and I might not get the shirts in time.

There’s also no way to measure the ROI (Return on Investment) for shirts, except based on my observations and intuition. If I try to save cost and produce low-quality shirts for a conference, people might wear them once and never wear again. If the shirt sizes don’t match well, especially for women who requires ladies cut, they might just end up as pyjamas or get KonMari-ed.

Most companies or organisations want their logos to pop out by making them big, vibrant and even on both sides of the shirt. I personally find them too much. Catchy logos do grab attention, which serves the purpose, but it’s weird when people keep seeing my shirt instead of my face while walking down the street or having a conversation. It's a typical “my eyes are up here” scenario for women, which atypically happens to me as a man 😅. If I wear a Google t-shirt, people assume that I work there and I have to keep explaining that I’m not, over and over again 😅.

For the past years of collecting t-shirts, wearing them and printing them, I’ve formulated the ultimate ingredients for the best t-shirt swag ever:

Tri-blend fabric, just because. Polycotton, if budget-conscious.
Screen printing with discharge ink. Or any printing method that lets the ink becoming part of the fabric. DTG as the last reliable fallback.
Single-colored gray logo on black shirts or dark-colored shirts. No catchy colors on the artwork. Let it blend naturally into the shirt, thanks to the ink.
Go an extra mile and provide fitted-cut shirts.

Personally I haven’t manage to achieve these goals yet, but I’ll be experimenting more in near future.

Stickers 👾

Stickers are easier to print than shirts. As a swag option, it takes up less space and weight to carry around. Most people, if not all, generally like stickers, regardless of gender or background.

I collect stickers from conferences and meetups. Sometimes, I even buy them if the designs are super nice or when there are discounts. I don’t stick them on my laptop and I don’t keep them either. I just give them away for free, regularly at events like Geek Brunch SG.

I started my sticker printing experiments in 2014. I ordered stickers of my own ‘CA’ logo from Sticker Mule on 10 April. 50 die-cut stickers, 51×51mm and costs US$53 (free shipping, US$1.06 per sticker).

It’s pretty neat and nicely shrink-wrapped. The free t-shirt was part of a promotion at that time. I think it's 100% cotton 😏.

On 29 May 2014, I ordered a second batch from a different provider called MOO. 52 kiss-cut stickers and costs US$12.74 (US$1.75 shipping with US$10 promo code discount, US$0.25 per sticker).

This obviously looks different from the first batch 😅. The stickers are kind of grouped together in a 4×4 grid instead of separated individually. It’s not quite visible in the photos above but this batch is actually glossier than the first one. At that time, I really did not expect this kind of differences at all.

On 12 November 2014, I printed the KopiJS stickers from a local vendor in Singapore called Touch & Print. 70 pieces, 5×5cm and costs SG$21.40 (self collection, SG$0.31 per sticker). On 19 November 2014, I tried again printing the KopiJS stickers from another local vendor called Onedayprint. 50 pieces, 5×5cm and costs SG$32.35 (self collection, SG$12 processing fee, SG$0.65 per sticker).

It’s… huge. Not the artwork but the whole sticker sheets themselves! Also, quite evidently on this photo, they look very glossy.

It’s pretty troublesome to bring these sheets around. During KopiJS #7 meetup, I tried encouraging folks to grab these stickers and have to pass a pair of scissors around for them to cut the individual stickers out 😅. Compared to my first batch (CA logo from Sticker Mule), the sticker distribution flow and sticker acquisition experience are… not good enough 😓.

To double-confirm my judgement, I ordered another batch of KopiJS stickers on 27 November 2014, from Sticker Mule.

100 die-cut stickers, 51×49mm and costs US$68 (US$0.68 per sticker). They are not glossy, not reflective, slightly thicker, cut into individual pieces instead of sheets, and again, shrink-wrapped. So good!

Okay, let’s step back a bit and do a little research here.

There are two terms for sticker cuttings:

Die-cut — cuts through the vinyl and paper backing.
Kiss-cut — cuts through the vinyl but not the paper backing.

Personally I think die-cut looks cooler because the edges are cut along the shape of the artwork or logo. The only exception that die-cut won’t make sense would be square-shaped or rectangle-shaped artworks.

Die-cut also makes it easy to preview how the sticker would look like on any surface without sticking it first.

From my observations, kiss-cut is only useful if there are multiple artworks to be printed at once. So it can be roughly 5 to 6 artworks printed on a sheet.

Here’s an example of a good kiss-cut sticker from RedDotRubyConf:

Unfortunately in Singapore, there’s quite a number of local print shops that neither does die-cut or even understand what it means sometimes. To speed up production and reduce cost, it’s easier and more efficient to print an artwork in grids on a large piece of sheet, and later cut it out with a sheet cutter machine. And… that is if the print shop provide the cutting service in the first place 😅.

Besides the cutting, there’s one more thing which is the types of lamination:

Glossy lamination — reflective, shiny, and more vibrant colors.
Matte lamination — nice texture feel, slightly muted palette and lower contrast.

Either one is fine, though I personally prefer matte lamination just because of the feel and texture 😊.

On January 2016, as I was desperately looking for cheaper-yet-good alternatives to Sticker Mule (US-based), I found a new vendor called StickerHD, based in Taiwan. It looks pretty new at that time and I couldn’t find any information if they could ship to Singapore, so I emailed them.

Turns out I could be the first one in Singapore, or outside Taiwan, to order from them 😆.

My first order from StickerHD was 50 die-cut 5×5cm stickers of my pixelated KopiJS artwork. It costs NT$620 (NT$70 for shipping, NT$11 per sticker; roughly SG$0.47 or US$0.33 per sticker).

Not only that it’s shrink-wrapped, it’s also packed inside a sealed bag! This is more convenient than Sticker Mule because whenever I open up the shrink wrap for a deck of stickers, they might fall out so I always need another storage space to keep them.

The quality is also on-par with Sticker Mule, at a much cheaper price. Shipping took maximum two weeks from Taiwan to Singapore.

I also quite like this detailed preview confirmation step before they start printing:

It shows how your stickers will look like, with exact measurements! Since the size affects the pricing and is based on the actual dimensions of the artwork, the vendor has to readjust the width and height ratio instead of strictly limiting them as a maximum length. If I choose to print a non-rectangle artwork on a 5×5cm sticker, it should be readjusted accordingly, for example 4.5×5.5cm, instead of maybe 4×5cm.

Ever since then, I printed a whole lot of stickers from StickerHD 🤩.

3 March 2016 — 100 die-cut SingaporeCSS stickers and 100 rectangle SingaporeJS stickers, costs NT$2,200 (NT$200 for shipping, NT$10 per sticker; roughly SG$0.42 or US$0.30 per sticker).

10 April 2016 — 100 KopiJS, 100 pixelated KopiJS and 100 Milo Dinosaur stickers, costs NTD$3,200 (NT$200 for shipping, NT$10 per sticker).

And way, way more… 💸💸💸

18 May 2016 — 100 stickers, NT$1,240.
28 July 2016 — 150 stickers, NT$1,760.
6 August 2016 — 400 stickers, NT$3,880.
8 September 2016 — 200 stickers, NT$2,020.
3 October 2016 — 120 stickers, NT$1,580.
27 October 2016 — 600 stickers, NT$5,540.
1 March 2017 — 350 stickers, NT$3,870.
9 June 2017 — 700 stickers, NT$6,580.
27 October 2017 — 450 stickers (300 transparent), NT$4,700.
18 November 2017 — 450 stickers (200 transparent), NT$5,370.
17 December 2017 — 600 stickers (400 transparent), NT$6,310.
4 February 2018 — 100 transparent stickers, NT$1,460.

Note that transparent stickers are die-cut but with transparent edges instead of white. They are slightly more expensive, but look better.

That is a lot. And… there’s more 😆.

Even though I’ve been loyally using StickerHD, I’m also constantly on the lookout for other vendors. I mean, it’s always good to check out the competitors and see who can do better, right?

On 18 February 2017, I found a new vendor based in India called Just Stickers. I ordered 200 2.4×2.4" reusable stickers, which costs US$101.80 (US$8.50 shipping with 15% discount, US$0.51 per sticker).

This is the so-called reusable stickers which can be removed and reapplied multiple times, up to almost 99 times! I never actually try stress testing them but I think it’s a pretty nice-to-have feature for stickers 🙂. Despite that, the stickers are kind of thinner and the colors are a bit faded out compared to my other KopiJS stickers from other vendors 🤷‍♂️.

On 21 April 2017, I found another new vendor based in China, called Zigpac on Alibaba, from a tweet by Wes Bos. I had to order the stickers on the Alibaba web site, which is my first time trying it out. The site can get quite overwhelming and feels like an online marketplace for factories and suppliers. It took me a while to understand some parts like the Message Center and stuff. Some of the vendors on Alibaba not only can produce stickers but also a whole wide variety of things!

At first, I ordered 200 5×5cm KopiJS stickers. They're matte-laminated, die-cut and cost US$80 including shipping (US$0.40 per sticker). After that, on 8 May 2017, they emailed me that their worker had mistakenly printed 1,500 stickers(?!) and asked me if I want to take the rest of them at US$200, which totals up to US$280 (US$0.19 per sticker! So cheap?!? 😱).

I know, it sounded fishy, but I took the deal anyway 😂. For the first time in my whole life, I received 1,500 stickers 💥💥💥.

Surprisingly, the quality of the stickers is pretty decent 😅 (I was expecting it to be really bad). The stickers are shrink-wrapped and the price is very cheap. It’s hard to say if it’s cheaper than the other vendors, because for most vendors, it gets cheaper as the quantity increases.

Ordering from China or India is still geographically too far away from Singapore. I wanted a vendor that’s nearer to reduce shipping time and cost. There are probably some local nearby vendors who might be able to produce high-quality stickers (die-cut, matte laminated) but it’s really hard to find them.

On July 2018, I finally found yet another vendor, based in Indonesia (and Singapore?), called GoodieSwag. I sent over a quick email enquiry to ask about the pricing and quality. After some back and forth, turns out the co-founder sort of knows me and have seen my stickers 😱.

So I ordered 200 die-cut matte-laminated 4.5×5.5cm matte-vinyl Pixelated KopiJS stickers, which costs SG$72 (SG$20 shipping, SG$0.36 per sticker).

On 9 October 2018, I ordered stickers again from GoodieSwag. 300 pieces of the Unicat artwork, same quality, at SG$95 (SG$20 shipping, SG$0.32 per sticker).

Obviously, I’m getting better at taking close-up photos of the stickers 😉.

This is a full-bleed, where the printing extends to the edge of the sticker. The alternative is the white borders around the edges. Some stickers have white borders, some don’t (full-bleed). Why is that?

There are a few resources on how to set up for full bleed printing:

Most of the time, clients (including myself) don’t really do this extra work on “bleeding” the artwork. Some vendors will proactively ask if the stickers require a white border or not. If no, then they will do the bleeding work and send a preview before start printing. Depending on the design, full-bleed may not make it look nicer and sometimes might not even be possible. It’s kind of like “extending” or “extrapolating” the edges of the artwork with matching backgrounds, which looks like this:

The red line is where they’ll cut the edges of the stickers.

After 5 years of custom-printing over 7,200 stickers, I start to feel like a "sticker god” 😂.

Here are the things to consider when printing stickers:

Die-cut or kiss-cut — I prefer the former because it looks better.
Matte or glossy lamination — I’ve tried both, usually prefer matte but sometimes, glossy looks cool.
Full-bleed or white borders — again, depends on the design.
Reusability — A nice-to-have but not essential, in my opinion.
Base location of vendor — whatever it takes to reduce shipping time and cost.
Printing flow — ordering via web site or email, preview confirmation before printing, and customer support.

The final most-ignored consideration would be design itself. There's no limits in terms of shapes, sizes and colors. Literally anything can be printed as stickers, including even a cat’s face (photographs instead of computer-generated art). But as a swag, it doesn’t mean that they would work and people would want them. Even if people grab the stickers, it doesn’t mean they would stick them. And if no one sticks them, there’s no point printing stickers in the first place 💥.

So, here’s my opinionated formula for the best sticker designs ever:

Cute animals, mascots or logos. Boring company or project logos are fine but very limited to those who are fans of them.
No words. Unless it’s catchy or very meaningful in some ways.
Colorful and eye-catching. Because who wants to stick a plain-looking sticker right?

For example, GitHub does a pretty darn good job at this:

Magnets 🧲

I used to believe that magnets will be the next logical step of my experiments. I mean, both stickers and magnets stick to a surface, so they are not much different, right?

Anyway… it’s too tempting for me to resist trying it. On 14 April 2014, I ordered 10 2.25" round magnet of my ‘CA’ logo, from Zazzle. It costs US$35.06 (US$6.89 shipping, US$2.27 tax, US$8.60 discount, US$3.51 per magnet).

These were huge and thick! The magnets were pretty strong too, and really sticks to the refrigerator (Yes, I was skeptical). Clearly, they are way more expensive than stickers 💸.

On 7 September 2016, I ordered 30 51×49mm KopiJS magnets from Sticker Mule which costs US$51 (free shipping, US$1.70 per magnet).

These are much, much thinner than the first one. It feels like a sticker being glued to a thin magnet at the back. It’s still thicker than a normal sticker but feels unexpectedly bendable. I thought magnets are supposed to be pure solids, but these are called flexible magnets, which can be bought as sheets or tapes. 🤔

The ‘CA’ magnets were quite a hit. These KopiJS magnets… not so much as most people were confused whether they’re stickers or magnets. Even then, it’s hard to find places to stick them, except… refrigerators 😅.

It’s cool at first but the novelty wore off pretty quickly 😐.

Embroidered stickers 🧶

After printing so many stickers, I wanted more. Magnets didn’t quite make it. I have to go back to basics and extend my ideas from there.

I don’t remember exactly how I got the idea for embroidered stickers but it was a few occasions when:

I randomly saw embroidered iron-on patches in some hipster stores.
I saw actual embroidered stickers in some conferences.
I realised that those cute animal stickers need to be cuter by having some texture or something.

My first action was to find a local vendor that can print embroidered stickers. My initial encounter was with Pew Pew Patches based in Singapore. I emailed them for the pricing and it was roughly SG$4 per sticker with minimum order of 100. I’m not sure if they are standard market prices, but I decided that I can’t afford to spend so much on such experimental swags that are not proven or mainstream yet, like the shirts and normal stickers.

Since I had my virgin experience printing stickers with Alibaba, I decided to try it again for embroidered stickers. Heck it took me a long time to browse through all the vendors, comparing them, reading the reviews, checking the prices, judging the quality from their previous productions, and doing background checks on the vendors.

At the same time, I was also learning a lot of new terms:

Type — Embroidery or woven.
Fabrics — twill, gingham, satin, felt, canvas or linen.
Border — stitched/merrow, heat-cut, or laser-cut/die-cut.
Backing — none, iron-on, fabric, hook-and-loop, pin, or self-adhesive (the one I want).

In all seriousness, I googled all the terms and note down the pros and cons.

Initially I don’t get the difference between embroidery and woven. I have no idea why I settled down on embroidery but it’s pretty cool to know there’s an alternative to it 🤔. Let’s get into the details:

Embroidery
- Thicker and messier. Looks classic or vintage.
- More expensive.
Woven
- Thinner threads, finer details, smoother surface, brighter colors, and looks flatter.
- Cheaper.

Naively speaking, woven is like an evolved, modern version of embroidery. It’s better, but looks pretty boring as I don’t see the pros against actual printed stickers. Embroidery is thicker, less-refined and gives that old-school feeling. Since it’s thicker, it’s also nicer to touch it and feel the texture. Its imperfections are its charm.

After doing much extensive research, I finally boiled down to two potential vendors.

On 18 March 2018, I ordered 100 embroidered stickers of the Gophercon SG mascot, from Shenzhen Xinbaoyuan Weaving Co., Ltd.. 100% embroidery, 5×5cm, twill fabric, heat-cut border, adhesive backing, and costs US$101 (US$26 DHL shipping, US$1.01 per sticker).

I have a feeling that this is actually woven and maybe… partially embroidered? 🤷‍♂️

On 21 March 2018, I ordered a similar batch of 100 embroidered stickers from Shenzen Awells Gift Co., Ltd.. 100% embroidery, 5×5cm, heat-cut border, adhesive backing and costs US$69.44 (US$26.94 shipping, US$0.69 per sticker).

Looks better, though it’s not 100% die-cut, with the adhesive paper at the back.

Here’s a comparison shot:

Pretty neat, huh?

I don’t know if I’m doing this right but I really like the feel of the stickers. I think this is great for cute mascots or animals, as it gives that soft cuddly feeling to them 😊.

Socks 🧦

I’m not really a socks person. As I’ve tried printing multiple types of swags, I wanted a new challenge.

Socks as conference swags are not really new. Slack socks are pretty sweet. Socks are becoming a fashion statement, thanks to Canadian prime minister, Justin Trudeau.

There’s also a tweet by Sarah Mei:

Petition to replace all conference tshirts with socks.

React Rally also has this pretty smart idea on combining socks and sandals:

regular brain: give out socks as conference swag

galaxy brain: give out socks and sandals as conference swag so everyone can see the socks

Yet it’s quite rare to see conferences in Singapore that give out socks. Maybe it’s not trendy yet? I don’t know. Why not give a try?

I began my research by finding out how Slack socks are made. They were produced by Sock Club, a custom socks vendor based in Austin, Texas, US. I learnt that there are many types of socks, with variety of fabrics, offered in a full range of sizes. Again, this is yet another recurring theme of me spending days learning all these stuff and trying to find out what’s the best among them 🕵️‍♂️.

However, the very first thing that caught my attention is the socks sizing. I don’t remember having to choose my size when I grab the socks from conferences. Conference registrations usually ask for t-shirt size but not sock size. According to Sock Club, sock sizing is based on shoe sizing, which leads to varied sizes made for men, women, kids and toddlers. It also emphasises the “One Size Fits Most” which makes it more confusing 😵.

I dug though the FAQ and found this:

Most of our clients order our one-size-fits-most, size Adult Medium. That size fits about 90% of our customers.

It doesn’t mention whether it's Adult size for men or women 🤷‍♂️. Also, they enforce a minimum order quantity (120) per design and per size. So I can’t order like 10 S, 20 M and 5 L, instead I have to order 120 S, 120 M and 120 L, for a single design!?! 😱 This basically leaves people no choice but to settle with one sizing, right? I don’t understand why it’s done this way but suspect could be some production or manufacturing challenges that seems more limited than t-shirts 🤔.

Anyway, the second thing that caught my attention is that some socks have certain “unprintable” areas. I found some guidelines from Eversox.

In short, the cuff, heel and toe areas are “prohibited” because they are prone to stretching or have knitting difficulties. This is also why Sock Club’s design mockup looks like this:

Those purple areas are “prohibited”.

On 26 June 2018, I asked Charis about the vendor she used for the Webconf.asia conference, which I attended twice. It’s the Fuzhou Firebird Sporting Goods Co., Ltd.. Two days later, I messaged them for a quotation and gave them this design; a repeatable image and how it should potentially look like on the sock:

This is the “Elephant” mascot artwork designed by Lim Min Li for PHPConf.Asia 2018. I felt that a repeated pattern of randomly rotated Elephants would be cool. Since this is the first time I’m trying this out, I wanted to make the design as simple as possible to prevent unexpected results.

I hoped to get a sample delivered to reduce risk and the vendor told me that it would cost US$38 per design plus shipping cost of US$28 😅💸. So I decided to bite the bullet, made dozens of design amendments and color corrections with them, and finally ordered 200 23×23cm 6-color (one for base, 5 for the mascot) socks (plus one sample), delivered in a huge box.

Yeah, I totally miscalculated the amount of space needed for 200 socks 😅. It costs US$573 (US$139 shipping, US$24 PayPal fee, US$2.05 per pair) and additional US$30 for material fee due to late change of one fabric color.

The socks are packed in these neat plastic bags.

I’m not super picky about this, but honestly very satisfied with the socks! The colors are still a bit off but seriously, colors are generally very intricate in the real world 😣.

This time, I didn’t print a second batch mainly due to time and delivery constraints. Taking this huge box to the conference venue is no easy task 😅. I do get some feedback that people liked it but it’s pretty hard to know if anyone wears them 🤷‍♂️. Unlike t-shirts or even stickers, it’s more difficult to spot people wearing your custom-designed socks.

Nevertheless, this is really fun for me 😊.

The journey

Throughout my whole journey of experiments with printing custom swags, it’s clear that I’ve changed and improved my procedures whenever I start exploring into new territories. In the beginning, I thought that this task is easy-peasy and underestimated the complexity behind all the free swags that I use to snag mindlessly. I did not do proper research and simply ordered without much considerations.

Along the journey, I slowly take baby steps to learn every single fabric, every single material, every single manufacturing techniques, and every single detail. I try to reverse-engineer some of the swags I got and try to find out how they were made and where they're manufactured. I spent countless hours researching and always play the balancing game between fast, good and cheap (and many more parameters). After that I check the worthiness of the production as my goal is to delight people with swags and make sure they actually use or wear them.

I’ve talked to a bunch of people and been misunderstood a few times on why I’m so meticulous, when they only care about minimizing costs and not maximizing the swag experience.

The design matters a lot, as I found some designers were designing the materials without understanding them in the first place. It’s like print designers who design for the web without any comprehension of the medium. It’s like artists who portray an artwork without understanding the canvas, the brushes, and the paints used. I personally believe that understanding the medium is very important because designs will always have certain challenges and constraints in the real world. If I could grasp the whole idea realistically, I’ll be able to produce better work that I’ll be proud of.

At this point, I’m not really done with my experiments yet. I always feel that there are more improvements to be done and more options to try out. As I mentioned, this is a pretty expensive “hobby”. However from these experiments, I’ve learned so much, had lots of fun and have thoroughly helped others in creating their own swags for their own meetups, conferences and even companies.

I do hope that my learnings could help more people, and hopefully encourage everyone to dive further into the world of awesome swags 😎.

Building side projects

Chee Aun 🦄 — Sat, 25 May 2019 17:23:41 +0000

On 3rd January 2016, I started my first side project of the year, despite having few more side projects up my sleeve. It's called RailRouter SG. It's a web app for people to explore MRT and LRT routes in Singapore. On 20th January 2012, I built BusRouter SG which is initially known as 'Singapore Bus Routes Explorer'. It's a web app for people to explore bus stops and routes for all bus services in Singapore. Quite similar to RailRouter SG but for buses instead.

I've built a lot of things. From small web applications to desktop applications. From small libraries to browser add-ons. From basic demo projects to web-based multiplayer games. Even then, I still find myself not that great compared to other people with more than hundreds of projects, like TJ Holowaychuk and Sindre Sorhus.

So asking myself few questions. Why do I build it? What motivates me to build it? Is there a need for this kind of app? Do people request for this? Am I doing this for fame and fortune? Well, no. There's only one reason.

Curiosity

Some people call it an 'idea'. I prefer to call it 'curiosity'. Sayanee asked a question in my AMA, How do you get your next idea to work on? and my brief answer is:

I usually don't find for ideas. For me, ideas come from being aware of things around you.

And being aware leads to being curious:

First is being aware of things around you, be curious about them, think about them, and things just naturally flow from that point.

More than 10 years ago, I used to take buses all the time. At first, it's quite exhilarating because there were no apps or smart phones at that time. No Google Maps, no directions, no GPS. Every time I take a bus, I have to board the bus and ask the driver if the bus goes to destination X. And once I tried it few times, it slowly becomes a routine for me to take bus from point A to point B and vice versa. I have to memorise bus numbers and their routes. Whenever I look at bus stops, I'll observe every bus that stops there and note to myself, "Ah okay, these buses stop here, I'll have to memorise them in case I need to come here one day." A question always pops into my mind, what if one day, I missed my stop? Where would I end up at? In fact, where does the bus end up anyway? I know that every bus has a route but I don't know how the route looks like.

That curiosity has stuck with me for a long time.

From my experience, there's a difference between wanting to build something and starting to build something. Curiosity is the first trigger to make me feel like I want to build something. Or perhaps in other words, I need to build something. It's quite similar to what people say that they have an idea and they want to materialize it into something more solid. So now that I have the curiosity to ignite my project, how do I start? When do I start?

Spark

Spark is the second trigger. For some people, this trigger happens in seconds right after the first trigger. For others, it happens in few weeks, months or even years. For me and BusRouter SG, it happened on 7th January 2012 when I read this tweet:

if i could view bus routes on a map the way you can view subway routes on a map, i would take them a whole lot more.

—@mengwong

9 days later, I tweeted:

If I'm not mistaken, there are 318 bus services and 4617 bus stops in Singapore.

4 days after that, I launched BusRouter SG to the public.

Perhaps there's another term for this. It's kind of like a magical moment for me to start building it. It feels like good timing for me to start now rather than later. This kind of spark doesn't have to come from other people, places or any external inspirations. Sometimes it comes from within, for example when I'm taking a shower, hiking up a hill, or talking to a friend.

Obviously, this is not all that magical. Things don't just suddenly happen in the timing between the first trigger and the second trigger. For years, I subconsciously kept my curiosity somewhere at the back of my mind, and constantly adding small bits and pieces of information to slowly grow the idea. I need to know where to get the data and which sites have it. I need to make sure there's a feature to detect my current location to instantly find nearby bus stops. I know that I can't draw thousands of bus stops and routes on the map because it'll be too overwhelming for the user, so I need to think of a better UI for this. I kind of draw mockups in my head and learn from how other apps present huge amount of data with a simple user interface. I probably need a better name for the app. I need to market it so that users would learn about it and use it, because I know there's no point launching an app that no one uses. I kind of want to display a realtime bus arrival information for every bus stop but the API was not perfect at that time, so I'll probably do that in the next phase.

Some people call this 'Product planning', as an ongoing process to define a product's feature set. And it has to be simple and feasible in terms of time and money spent for development. That's when the term 'Minimum Viable Product' (MVP) came into the picture.

From my experience in building apps, I learnt this the hard way. In fact, I don't even have or know the term until I read some articles that mention them. And I'm like, "Oh, there's a term for this? Okay, cool." So how minimum does the product need to be? For my own side projects, I always try to estimate if I can finish it in a day. How about a week? Two weeks? A month? I usually avoid side projects that last more than a month because I always feel bored looking at the same thing over and over again. Basically maximum one month. Within a month, I have to launch it publicly. If somehow I'm unable to publish it, I have to at least make it open-source and let the world see my code.

Launch

This is the ultimate third trigger. I call this a trigger because it's like a game-changer, well, depending on how you launch it. I've seen some developers didn't even manage to cross this line because of lame reasons like the app is not perfect yet or the code is too ugly. I've also seen some developers did a soft-launch for their side projects, which I think is pretty lame too. When I launch a product, I tweet it. I tell people. I try to blog about it. I post it on Hacker News. I want many, many people to look at my imperfect app or ugly code.

The moment when I launch a project, my senses are heightened. Suddenly I become aware of bugs that I didn't notice while coding. I start writing good READMEs so that developers understand what I did. I start writing good copy for non-technical users to understand what the product does. I check for grammatical errors, broken links and broken flows.

It's addictive. After the project is launched, I start to learn how to market it, experiment with advertising, play around with SEO, think about monetisation, and even pitch it to other people. It's surprisingly a lot of work.

Every single time when I do all these work, I start to understand the situation and difficulties behind all the non-engineering work. In a typical company, these are done by other departments; marketing, sales, advertising, social media, designers, bloggers and copywriters. There are times when I find it funny that I could understand what other (non-engineering) people do, but they don't understand what I do 😥

Feedback

When the project is public, I start to get many types of feedback from people. Positive feedback makes me happy. Negative feedback makes me try harder. Unconstructive feedback makes me learn how to ignore them. No feedback? Well... either try harder or be patient. It's always surprising when the project that you thought no one is going to care, actually cares. If there's still no feedback, just move on, start another project, go though the whole cycle again starting from the first trigger!

Sometimes I'll get one more type of feedback, which I think is worse than negative ones. It's the kind of feedback that makes me feel really bad and start to lose my love on the project, which ultimately lose my passion in building things. Obviously when we build stuff for the public, things don't always go your way. I'm not sure how to describe this but it's kind of like a moment when I ask myself, "Oh god, what have I done?". I've had my bad moments. I've done a lot of mistakes. I also know that some people have had it worse than me. One of them is Remy Sharp, who has the courage to even write a 5-part story on the dark side of JS Bin, especially part 5.

It's like a test for mental strength. A test to see if I'll give up and quit. As what most people think, it's just a side project, right? How bad could it be? Well, you'll never know until you launch it.

Maintenance

I call this the maintenance phase, the period of time when I have to continuously handle user feedback, do customer support, fix bugs, add features and improve the project. It's a bit like an endurance test to see how long I can last maintaining everything and stay focused on them. At this stage, many things could happen. Some people end up stop maintaining and starting something else. Some people manage to monetize their own project and let it self-sustaining. Some people manage to turn a side project into a startup, for example, Lanyrd.

This 'maintenance' phase doesn't always work with the first trigger, curiosity, also known as the 'starter' phase. It's very difficult to start something new while maintaining the old stuff. Vice versa, it's hard to dedicate the time on maintenance when you keep creating new things.

James Long mentioned:

There are two roles for any project: starters and maintainers. People may play both roles in their lives, but for some reason I’ve found that for a single project it’s usually different people. Starters are good at taking a big step in a different direction, and maintainers are good at being dedicated to keeping the code alive.

As for me, I try my best to play both roles. It's challenging. Sometimes I fail. Sometimes I got bored. Sometimes the projects I built became deprecated.

Few weeks ago, I got a new GitHub issue reported on an old project that I'm no longer using. The project is called mangafeeder, built in 2012, and never had any issues or pull requests until that very day. It was hosted on AppFog V1 which became retired since December 15th last year, and I didn't even realise it. When I read the reported issue, I was very surprised that someone is actually using it. Despite the fact that I've built a lot of things, this shouldn't be that surprising anymore but I can't help it. I tried to redeploy the app to the new AppFog V2, found it quite troublesome so I decided to deploy it to Heroku instead. The site is up and I resolved the issue with a reply.

The response I got is this:

awesome 👍

Somehow it reminded me of the WWDC 2012 developer appreciation video, especially the part where Emil Ovemar said:

It's really fantastic to see what we have created has made a difference in people's lives. Those reactions worth way more than any downloads.

From time to time, I always see people getting excited or obsessed over the numbers. Number of downloads, likes, loves, retweets, reposts, reshares, stars, followers, fans, visits, impressions and clicks. For me, those are just nice-to-haves. They are not really the most important thing in the world.

The number one reason why I don't give up and will continue building things is the positive response from other people. The thank you's. The "Oh my god, you saved my life" kind of feedback. Sometimes when I get 9 positive feedback and suddenly one negative feedback, it will make me feel bad for the rest of the day. I personally learnt that if one day, I get 9 negative feedbacks and one positive feedback, I should be happy instead, thinking that I helped at least one person.

Beyond side projects

Few days ago, on January 21st, I gave a talk on this topic, at the time of writing, to a group of students in General Assembly Singapore. I drew this simple diagram as a visual way to describe the complete 'side project timeline'. It's a rough sketch of how I visualize it in my mind.

I mentioned that if you look at this timeline, it's a bit like a startup, but without the business strategy, hiring process, VC funding and money.

I took a third look at this and realise that it's a bit like life itself. 'Curiosity' begins when you're born into this world. 'Spark' probably begins when you start studying in school. 'Launch' happens when you graduate. 'Maintenance' is when you start working, buying a house, having a family and kids. Okay, pardon me, probably I got too far with this 😅

Nevertheless, it's a journey.

A journey that I wish that more people would be able to experience and tell the story.

Originally published at cheeaun.com.