DEV Community: Martin Capodici

No local GPU? No Problem! Running Andrej Karpathy’s NanoGPT on Modal.com

Martin Capodici — Sun, 16 Jul 2023 02:07:00 +0000

Andrej Karpathy released a series of timeless lectures teaching us mortal 9-5 programmers from scratch how to train an “AI” language model, a bit like that GPT4 or ChatGPT you may have heard of.

He goes into a deep dive that includes building your own tiny Pytorch from scratch, setting up bigram models, and simple neural nets, before moving over to use the real Pytorch later. He then explains how transformers (the T in GPT) work, and codes one up to generate some dubious Shakespeare. This final model he calls “NanoGPT”, because of the similarity between it’s model and that of the early GPT models that lead to ChatGPT.

So why this post?

Well, while I absolutely loved the series, I don’t enjoy working with Colab or Jupyter Notebooks. It is easy to forget what code blocks have run, and I am forever scrolling up and down because the code is mixed up with results in one giant page. Not only that but if you are using Google Colab it will time out fairly quicky so you need to waste time running everything again.

I’d run it on my machine instead, but…

I want to run NanoGPT locally but I don’t have a good GPU. To save buying one for $2000+, I would like to rent one in the cloud if possible. If I use cloud GPUs I can experiment quickly with different chips as needed. An A100 GPU for example costs maybe $7000 – $15000 USD, but grabbing one for an hour for $4 is much more in my budget.

modal.com provides this service, and they take care of all of the “devops” as we will see soon. There is some housekeeping Python code to write, but no bash, Terraform or Ansible, which is great because I don’t want to do that.

Their GPU prices are not the cheapest. I would say they charge fair (average) prices though. And they charge for the milliseconds of actual usage and nothing else. That means I don’t pay extra because I forgot to shut down a server. Also they include $40/month credit for free anyway so it is costing me nothing to learn.

In this post I will show you how I used Modal to quickly train and run the NanoGPT model, while having the creature comforts of developing in VSCode.

What is NanoGPT anyway?

NanoGPT is nothing but a text producing bot!

When trained on some text it will learn how to predict the next character. So for example if you feed it “Hello ” it might predict W. You then feed it “Hello W” and it might predict o and so on. By repeating this you get text generation.

When trained on Shakespeare it makes muddled text that is quite a bit Shakespeare-looking.

Example of NanoGPT generated text:

FlY BOLINGLO: Them thrumply towiter arts the muscue rike begatt the sea it What satell in rowers that some than othis Marrity.

LUCENTVO: But userman these that, where can is not diesty rege; What and see to not. But’s eyes. What?

JOHN MARGARET: Than up I wark, what out, I ever of and love, one these do sponce, vois I me; But my pray sape to ries all to the not erralied in may.

If you want to know more, you can check out:

Video (with links to resources)
Google Colab code
Github code

Now let’s get started, and get NanoGPT trained and running with local code, and a cloud GPU from Modal.

Step 1: Learn how to run code on Modal

I won’t parrot too much what Modal have in their tutorials, as that is the best place to go, but in a nutshell you can decorate functions in Python that you want to run on their servers.

For example you have a function you want to run in their cloud:

@stub.function()
def f(i):
    if i % 2 == 0:
        print("hello", i)
    else:
        print("world", i, file=sys.stderr)

    return i * i

And then you can call this from a local function either as-is (to run locally) or with .call (to run on the server):

@stub.local_entrypoint()
def main():
    # Call the function locally.
    print(f(1000))

    # Call the function remotely.
    print(f.call(1000))

To run this from the command line:

modal deploy example.py

Step 2: Fork the NanoGPT repo, and check it works on local computer

The next step is to make a fork of https://github.com/karpathy/nanoGPT and clone that fork to my computer, so that I can make some changes to adapt it to use Modal.

Note: If using Windows, you will need to use a Linux distribution installed to WSL2 to do this successfully as Windows is not supported for torch.compile

It is a good idea to check that we can get it to run locally. I just want to check the code works fast so I will reduce the number of iterations in train_shakespeare_char.py to 5, and dumb down the model size to ridiculously small so it completes in a few seconds on a crap laptop. Here are the changed lines in train_shakespeare_char.py:

...
max_iters = 5
...
# baby GPT model :)
n_layer = 2
n_head = 4
n_embd = 16
dropout = 0.2
...

In addition, I uncomment these 2 lines in the same file (train_shakespeare_char.py) to make it possible to run on an average laptop with no GPU:

# on macbook also add
device = 'cpu'  # run on cpu only
compile = False # do not torch compile the model

To check that it works, I set up a Python environment, and run similar commands as shown in the NanoGPT README.md:

python -m venv .
source bin/activate
pip install torch numpy transformers datasets tiktoken wandb tqdm
python data/shakespeare_char/prepare.py
python train.py config/train_shakespeare_char.py

From this we get a confirmation that this training loop is running correctly:

step 0: train loss 4.1783, val loss 4.1771
iter 0: loss 4.1791, time 47896.67ms, mfu -100.00%

Knowing that it works on my computer makes me more confident to try and getting it working on Modal.

Step 3: Upload the training data to modal

3.1 Authenticate with modal

First, lets do the basic setup for Modal and get authenticated:

pip install modal-client
modal token new

3.2 Change the prepare.py to upload to Modal

Now edit data/shakespeare_char/prepare.py, and add these lines so that once the prep is done, the data is uploaded to Modal

Next, nest the existing code inside a main function, adding the entry point stub, so that Modal knows to run this locally only. Otherwise it will try to do the imports on the server, which won’t work as those packages are not installed.

@stub.local_entrypoint()
def main():
    """     
    Prepare the Shakespeare dataset for character-level language modeling.
    So instead of encoding with GPT-2 BPoE tokens, we just map characters to ints.
    Will save train.bin, val.bin containing the ids, and meta.pkl containing the
    encoder and decoder and some other related info.
    """
    import os
    import pickle
    ...

Add the following lines at the top of the file to define the volume and app name:

import modal

volume = modal.NetworkFileSystem.new().persisted("nano-gpt-volume")
stub = modal.Stub("nano-gpt-code")

Now add this function at the bottom, which will run on the remote server. All it does is copies the files over with some prints to check if it was successful. It keeps the folder structure on the server the same (the working directory is /root there) so that there is less code to change in train.py when we get to it.

dataset = "shakespeare_char"

@stub.function(
        mounts=[modal.Mount.from_local_dir("data", remote_path="/source/data")],
        network_file_systems={"/root/data": volume})
def copy():
    import shutil          
    import os


    source_dataset_path = os.path.join("/source/data", dataset)
    dest_dataset_path = os.path.join("/root/data", dataset)

    def check():        
        if os.path.exists(dest_dataset_path):
            files = os.listdir(dest_dataset_path)
            print(f"Files: {str.join(', ', files)}")
        else:
            print(f"Path doesn't exist")

    check()
    shutil.copytree(source_dataset_path, dest_dataset_path, dirs_exist_ok=True)
    print("files copied")
    check()

Now make the call to copy from main:

    # length of dataset in characters:  1115394
    # all the unique characters:
    #  !$&',-.3:;?ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz
    # vocab size: 65
    # train has 1003854 tokens
    # val has 111540 tokens

    copy.call()

3.3 Run the upload

You can now run this to perform the upload:

modal run data/shakespeare_char/prepare.py

You should get an output like this:

Path doesn't exist
files copied
Files: meta.pkl, val.bin, prepare.py, input.txt, __pycache__, train.bin, readme.md

If you run it again, it should show that the files exist before it is copied, proving that the data was persisted. Now the remote machine has access to the training data.

Step 4: Adapt the training code to run on Modal

4.1 Make the training code into a Python package

As far as I can tell, in order for Modal to see all of your Python code it must be organised in a package.

To make the code into a Python package those is quite simple, first move the python files for the model training and inference into a new folder:

mkdir nanogpt
mv config *.py nanogpt

Find all instances of from model in these files, and replace with from .model (Add a period). For example in train.py:

from .model import GPTConfig, GPT

Adding a period to these local imports says “this is from the current directory’s package”. This allows the code to work when called from another package or location, which will be doing when using Modal.

4.2 Remove the configurator

There is a line in train.py that needs to be commented out because it won’t work in Modal (because it doesn’t have the source files in the same place), so comment this out, and add a hard-coded line that does the equivalent thing for the Shakespeare model.

# exec(open('nanogpt/configurator.py').read()) # overrides from command line or config file
from .config.train_shakespeare_char import *

This is perhaps not the ideal way to do it, but a quick change for the purposes of making this blog post not too long.

4.3 Add a python script to run the code in Modal

Create a new file called train.modal.py in the root of the project (so one up from nanogpt folder) and add the code below. I have put some comments in there to explain it.

import modal

# Make sure we have access to the data we prepared earlier:
volume = modal.NetworkFileSystem.new().persisted("nano-gpt-volume")

# Set up the container for running the training, and make sure it has the necessary
# python pacakages installed.
stub = modal.Stub("nano-gpt-train",
    image=modal.Image.debian_slim().pip_install(
        ["torch", "numpy", "transformers", "datasets", "tiktoken", "wandb", "tqdm"]
    )
)

# This stub.function allows train_modal to be called remotely on their servers. We will
# now specify how we want that set up...
@stub.function(
        # Ensure that the function runs with a GPU, I have picked out a cheap one, but you can replace
        # this with "any" in the future if this GPU is no longer available.
        gpu=modal.gpu.T4(), 

        # Increase the timeout to allow long training times.
        timeout=3600, 

        # This tells modal to upload the entire nanogpt package we created. Without doing
        # this it won't be able to locate train.py, model.py etc.
        mounts=[modal.Mount.from_local_python_packages("nanogpt")],

        # Mount the data we prepared earlier
        network_file_systems={"/root/data": volume}
        )
def train_modal():
    # This import is a cheeky and quick way to run nanogpt with minimal changes to Andrej's code. Ideally we would change
    # the `train`` module to expose a function. Then import `train` and call that function.ction and call that.
    import nanogpt.train

# This is what gets called locally when running `modal run train_modal.py`, and it just calls the 
# remote function.
@stub.local_entrypoint()
def main():
    train_modal.call()

With a GPU available, we can comment these 2 lines back out in train_shakespeare_char.py:

# on macbook also add
# device = 'cpu'  # run on cpu only
# compile = False # do not torch compile the model

4.4 Run the script

Now we can run this from the command line: modal run train_modal.py, and here is the result:

(nanoGPTonModal) [email protected]:~/nanoGPTonModal$ modal run train_modal.py
✓ Initialized. View app at https://modal.com/apps/ap-k9Oehw5IpXCxmt3yNBUNds
✓ Created objects.
├── 🔨 Created train_modal.
├── 🔨 Created mount /home/martin/nanoGPTonModal/nanogpt
└── 🔨 Created mount /home/martin/nanoGPTonModal/train_modal.py
tokens per iteration will be: 16,384
found vocab_size = 65 (inside data/shakespeare_char/meta.pkl)
Initializing a new model from scratch
number of parameters: 0.01M
num decayed parameter tensors: 10, with 11,280 parameters
num non-decayed parameter tensors: 5, with 80 parameters
using fused AdamW: True
step 0: train loss 4.1783, val loss 4.1771
iter 0: loss 4.1791, time 3620.00ms, mfu -100.00%
✓ App completed.

4.5. Revert to the proper sized hyper-parameters

Revert the values in train_shakespeare_char.py to the bigger model values, with more iterations. Now we are using Modal, this will be able to run in a reasonable time.

...
max_iters = 5000
...
# baby GPT model :)
n_layer = 6
n_head = 6
n_embd = 384
dropout = 0.2
...

Running modal run train_modal.py again:

(nanoGPTonModal) [email protected]:~/nanoGPTonModal$modal run train_modal.py

✓ Initialized. View app at https://modal.com/apps/ap-HU6D2SRnxOv1OsJpmlb3Fj

✓ Created objects.

├── 🔨 Created train_modal.

├── 🔨 Created mount /home/martin/nanoGPTonModal/nanogpt

└── 🔨 Created mount /home/martin/nanoGPTonModal/train_modal.py

tokens per iteration will be: 16,384

found vocab_size = 65 (inside data/shakespeare_char/meta.pkl)

Initializing a new model from scratch

number of parameters: 10.65M

num decayed parameter tensors: 26, with 10,740,096 parameters

num non-decayed parameter tensors: 13, with 4,992 parameters

using fused AdamW: True

compiling the model... (takes a ~minute)

step 0: train loss 4.2874, val loss 4.2823

iter 0: loss 4.2649, time 29573.95ms, mfu -100.00%

iter 10: loss 3.2438, time 101.76ms, mfu 3.66%

iter 20: loss 2.7899, time 103.62ms, mfu 3.66%

iter 30: loss 2.6383, time 104.10ms, mfu 3.65%

iter 40: loss 2.5763, time 101.83ms, mfu 3.65%

iter 50: loss 2.5261, time 104.54ms, mfu 3.64%

iter 60: loss 2.5136, time 103.90ms, mfu 3.64%

...

iter 4980: loss 1.2050, time 117.62ms, mfu 3.16%

iter 4990: loss 1.2493, time 114.90ms, mfu 3.17%

step 5000: train loss 1.1405, val loss 1.4969

iter 5000: loss 1.2446, time 12044.48ms, mfu 2.86%

✓ App completed.

Costs

It took about 14 minutes and cost $0.21 to train the model. I think $0.14 was for the GPU and the rest was for CPU/memory.

Conclusion

First, this took a little more work than expected to get some local python code running on Modal.

The combination of design choices in the nanoGPT repo, and the fairly narrow happy path to get code to run in Modal meant that a lot of changes had to be made. To summarize these things meant code changes were needed:

Modal will only upload a bunch of Python files if specified as a package. NanoGPT didn’t do this.
Modal will put the files “somewhere”, so using exec() on relative paths to local scripts like NanoGPT does won’t work.
Modal requires additional functions and decorations, so a new file is needed.
Modal requires specification of mounts etc. so this new file has quite a bit to it.

I think if you build a Python project with Modal in mind, then the experience will be easier. You will know how to organize files, what not to do, etc. So there will be less work to do.

Next, it is worth saying that once you get this working, it works really well. Running modal run train_modal.py it gets going and chugs along, you almost forget this is doing a whole bunch of ops stuff in the cloud for you. Then you can iterate and change things up and Modal gets out of your way a bit.

With Modal set up, I can now code with an IDE, IDE Plugins, file structure, git, etc. It is more what I am used to than the Jupyter experience where you have to remember what state things are in, there is effectively one big file, and output and code are all mixed up. This is much better.

Therefore overall I think Modal is worth learning and experimenting with, and putting that initial effort to get set up. Or if money is no object, just go buy a big GPU :-).

In the next blog post I will run the inference to see what kind of Shakespeare this model can produce. This will require some code changes to get that to work on Modal, but I expect it to be a lot less as much of the work has been done.

I will also explore what other features are in NanoGPT and try them out using Modal too.

(originally posted at https://martincapodici.com/2023/07/15/no-local-gpu-no-problem-running-andrej-karpathys-nanogpt-on-modal-com/)

Going The Other Way

Martin Capodici — Sun, 09 Aug 2020 09:01:18 +0000

Have you ever "gone the other way", by which I mean where most people use product A then usually "grow up" or "evolve" and use product B, and you went the other way! B to A. A made up example of B to A would be ridesharing apps -> calling a taxi service on the phone.

In the software world, when you go the other way you notice how all the blog posts, online help is not really helpful at all. They assume you'd only ever go A -> B.

Examples of this I have found are: moving from AWS to Heroku for hosting or from Ledger-CLI to YNAB4 for budgeting.

I'll cover the first of those examples in this article. I will talk about going from AWS to Heroku for the tryjobalerts.com site hosting and data processing.

William Shock. Yes... Bill Shock.

I'm guessing many of you have tried the generous free year of AWS services, and then got your first bill on the 13th month and said "errr. OK". That happened to me and prompted me to cull or move out a load of stuff running in AWS. On the flip side, I was amazed at how cheap SES is - I think my bill was $0.10, and how expensive Elastic Search is, where my bill was $50 for a hardly used cluster. I would have not been suprised to see those numbers reversed.

However before the bill, I had started off moving EC2 servers to Heroku. The reason wasn't cost - Heroku is typically a little more expensive as it provides additional wrapping and services over cloud infrastructre. I moved for the obvious reason that I'm lazy. Which is a codeword for good time prioritisation. I don't want to manage a operating system, server or CI pipeline. Instead I want an expert to take care of that for me for an extra $5 a month.

The point where I switched from EC2 to Heroku wasn't a well thought out logical decision, based on sitting on a leather couch, musing about software architecture. It was an emotional decision based on my node app crashing, and all my services for tryjobalerts.com being out of action for 5 minutes and counting. What if someone tries to edit their settings and they get an error page? how embarassing. Although not that unusual these days, I didn't want to be the one.

What to do next? Well, in the time honoured solution of "turning it off and on again", I restarted the EC2 instance and BaaM! the node app crashed again. Whatever it did, it took out the machine in such a way I couldn't connect to it via SSH to fix the problem. By this point I rashly decided to move to Heroku. I just didn't have the heart to fix this shit when I wanted to be coding. And I knew this would likely happen again and again, and I'd probably end up on the slow march towards Kubernetes.

Was that a hasty decision? Well maybe it was, but it had been a subconcious thought for quite a while. Setting up nginx, let's encyrpt, PM2 etc. etc. was mildly intersting but feels like a distraction from what I am really trying to do. I am happy for someone else to deal with that, and if everything dies, build them up again. I'll concentrated on the code, thank you very much.

I gave Heroku a try. I thought, let's just see how easy it is. If it "just works" I'll switch over. Otherwise I'm downloading k8s (joking!). I was working on this while the site was out like a light, no one could access their settings, sign up or receive an email - so no pressure.

Moving to Heroku was redicuously easy. I just had to create a file called Procfile with this in it:

web: node distjs/node/index.js
worker: node distjs/bot/index.js

And set up a few things in the Heroku UI - secrets and such like. I also had to change the port express run on from a fixed port 3001 to the number Heroku provides as an environment variable. Within 15 minutes I say I had it working, sans DNS entries.

I kept SES and PSQL on AWS, and everything worked lovely.

I kept things this way for a few weeks. I then got that bigger than expected bill from AWS, including a fair amount for the PSQL. Nothing too catastophic but it didn't seem like good value for money.

Heroku has a weird limit for PSQL Hobby Basic

I noticed Heroku offers a "Hobby Basic" $9/month PSQL with up to 10M rows. It turns out I use a decent amount of space in TEXT fields, but not too many rows. So I migrated to that. Yes unbelievably I am migrating something from AWS to Heroku to save money.

The limit is weird, being based on number of rows rather than storage size. I wonder how much excellent data you can store as blobs in 10M rows - maybe I should keep quiet as the Heroku Product Owner could be reading.

I searched online for the best way to move the data over. I like Stack Overflow because if someone has had a war story you hear about it instead of experiencing it first hand. However most of the advice online is how to get your data out of Heroku into AWS. Going the other way.

Therefore I tried a couple of ways. It turned out it was easy enough to use psql command line tools pg_dump and pg_restore. It worked first time and took about 30 minutes to do - most of the time was me doing housework while waiting for pg_restore to do it's work.

(Incase it's helpful to anyone, the tip here is don't use the Heroku CLI, get the connection details from the Heroku portal, and use the psql standard tools.)

It is quite amazing to me that Heroku turns out to be cheaper on the PSQL database. It might be because they don't offer as much power as RDS, although they are using AWS in some capacity - perhaps they manage the VM themselves.

It could be that Heroku is happy to lose a bit of money on that Hobby Basic tier, or they more aggressively share resources than AWS does in RDS, so under load it would crack sooner.

Heroku's Hobby Basic PSQL doesn't offer rollbacks, and that could be another reason it's cheap. It means I need to take care of backups myself, however this can be automated quite easily with a periodic dump of the DB.

Don't be too cheap though

The big mistake I made when moving to Heroku was to try and use the free dynos. Only do this for non-production MVPs. By using the free ones, I had to do a lot of work setting up SSL with Cloudflare, and I never quite got it to work.

I then realised I needed the paid dynos anyway because the free ones cannot be restarted (!) without a deployment. So I upgraded, and then the SSL was sorted. No Cloudflare needed. Had I done that in the first place I'd have saved a couple of hours of frustrating DNS hacking.

It's always a fine line. I mean you try to save money where you can, but sometimes you don't know in advance what the free-tier traps are. Maybe Heroku could make this more obvious, but they did warn that free dynos are mainly for hobby use, so fair enough.

Herokus! Herokus! £2 a lb!

I sound like a Heroku salesman at this point. There are other "Platform as a Service" offerings out there. I had just used Heroku before in 2014, and liked that it still existed, so I used that. However there could be better options for you, check them out. Or maybe a regular old cloud like AWS is what you needed all along.

Regardless, it's OK to go the other way, choose whatever makes sense for the project you are doing.

Join JobAlerts to get free daily emails with the latest tech jobs from a curated list of excellent sources. Click here to learn more.

Love remote working in 2020? Then ensure your permanent transition to remote work in the post-COVID world.

Martin Capodici — Thu, 16 Jul 2020 10:23:44 +0000

Not many people called it, but a weird upside to the Coronavirus "New Normal" is that companies that are fiercely against working from home have now been forced to let their staff do it because of health laws, or staff refusal to come in and risk catching the virus.

Many tech workers have been enjoying this perk, and possibly thinking to themselves "can I work like this forever?". I believe the answer is yes, but for most, it won't be for zero effort. When the pandemic is over, old ways of thinking might creep back in.

With a bit of thought on your part, now is the best time ever to transition to remote work, so that you can travel more, commute less, spend less on rent and spend more time with your family and mates.

Here are some tips for doing just that:

1. Look out for remote-first companies.

Remote first companies such as Gitlab, are set up so that all employees work remotely, from anywhere in the world. As such they are almost certainly never going to transition you to an office (because they don't have any!) or look down on you as one of "those remote employees".

They are also more likely to have the tools and processes set up the make remote work smooth because everyone has to do it. They probably have a smoother developer onboarding process than "get Bob to come over and help you install everything", and have ironed over kinks to do with standups, meetings and agile ceremonies to adjust for the challenges of not being in the same physical room.

2. Market yourself as an experienced remote employee.

With the Coronavirus pandemic now an almost permanent fixture in our lives, I think most of us now have real remote-working experience. Use this to present yourself as remote-ready. In your CV or cover letter, you can emphasise this.

For the interview prepare: How will you explain how you are great at remote work? What challenges have you overcome? How do you stay motivated and productive when working from home? If you are a team leader, what initiatives have you put in place to help make remote working more effective? How do you ensure that your team members are doing great work and not getting stuck?

Show them that you can be trusted and that you communicate well and are easy to manage, and if applicable, can manage others well.

3. Move into a more permanent remote relationship with your current company.

The current working situations would have proved that yes, you can have a team working remotely, regardless of your company size. If your company has done it they will have this first-hand experience. If not you can point to myriad other examples such as Facebook, Amazon, Slack and Twitter who have let their employees work from home unless they have to be in the office for their specific job duties.

This is the time to negotiate what you want in terms of working from home, be it part-time WFH or full time. If you love where you work this is probably a better option than leaving. If you decide to move to another company in the future you can use point #2 with more force - marketing yourself as an experienced remote worker.

4. Market yourself through open source contributions.

Open-source contributions, to strategically chosen projects, will not only show you can code but help you build relationships with people working at remote-first or heavily remote-oriented companies. It might get you a foot in the door or even bypass early interviews. You don't need to build an entire project and be a "BDFL" of a major framework. Just find a good project in a programming language you are familiar with, and start fixing bugs. Be respectful of the rules, protocol and culture of that project so as not to annoy the maintainers, but make them enjoy working with you.

5. Use us!

Keep your eye on the market, even if you are not actively looking. Visit https://tryjobalerts.com and sign up for free to get daily job alerts from a ridiculous number of great sources.

Getting to know Webpack

Martin Capodici — Thu, 30 Jan 2020 19:53:26 +0000

One of Javascript's best features is you can start out really simple and learn it with just a text editor and a web browser. You can write a simple piece of HTML, add some Javascript, such as alert('hello'), and open that in your browser. You get immediate feedback about what you are doing. From there you could write code to do something when you click a button, submit a form and much more.

If you want to use other people's scripts to help build your application, you can include them via the src attribute of the script tag. You can use the script tag to pull in great libraries, like React, JQuery and many others to make your life easier developing certain kinds of UI. There are also tools for making Excel-like tables, or code editors and much more. You can reuse so much technology that you can quickly get an impressive app made in minutes.

However there at some point, when you make a complex enough app you might hit a few problems:

The app gets really big, and your Javascript takes too long to download, especially on mobile devices with slow connections.
You need libraries that depend on each other, and might use different versions of the same thing. For example you need to use something that itself requires JQuery or React.
Different libraries use different ways of referring to other libraries. Some will expect there to be a property on the window object set by another script, while others expect a module system, like RequireJS to be used.
The order of your <script> tags becomes important and needs to be maintained as you use lots of libraries.
You need to either copy the libraries to your computer and keep them up to date, or reference libraries on different website URLs, but you may not trust their load speed or even their content.

Many of these problems are addressed by using a couple of tools:

NodeJS - is a tool for running Javascript on a computer rather than in the browser. It can be used as either a "Server" that can be accessed via http or a tool that runs commands from the command line.
NPM - this is the package management system that comes with NodeJS for installing both tools to run via Node, and also it is used for install Javascript packages that will run on the client - more about that in a moment!
Webpack - a tool for taking Javascript dependencies that you need and bundling them with your Javascript so that you can be sure that everything will load correctly.

It is hard to cover all of this in depth in just one article, so what I'll do is give you a feel for how it all works, then you can decide whether to persue it further for your projects.

NodeJS

Before we move on to webpack, we need to look at NodeJS

Node can be installed on Windows, Mac or Linux and you can download from https://nodejs.org/

Once downloaded and installed you can open a command prompt or terminal and type in

node

and you will see a welcome like this:

Welcome to Node.js v12.8.0.
Type ".help" for more information.
>

This is in interactive mode. You can now type in Javascript, for example x = 3 and it will evaluate that and return the result.

You can type .exit to quit.

Node can also run a Javascript file in your computer, for example you can create a file index.js and enter the line:

console.log('hello')

Then from the command line run

node index.js

and see what you'd expect:

hello

The file index.js can contain a complicated program, and do many cool things. Javascript in Node, unlike the browser has access to the local file system and many other resources. That means it is possible to make complete development tools in Node. And indeed Webpack is one of those tools.

But how would you get the Webpack file to run locally?

Installing things into a NodeJS project

NodeJS has an installation and package managment tool called NPM. This is simlar to package managers in other programming languages, like pip in Python or even apt-get in Debian and Ubuntu.

With node installed you can use npm right away. Typing npm at the command line shows something like this:

Usage: npm <command>

where <command> is one of:
    access, adduser, audit, bin, bugs, c, cache, ci, cit,
    clean-install, clean-install-test, completion, config,
    create, ddp, dedupe, deprecate, dist-tag, docs, doctor,
    edit, explore, get, help, help-search, hook, i, init,
    install, install-ci-test, install-test, it, link, list, ln,
    login, logout, ls, org, outdated, owner, pack, ping, prefix,
    profile, prune, publish, rb, rebuild, repo, restart, root,
    run, run-script, s, se, search, set, shrinkwrap, star,
    stars, start, stop, t, team, test, token, tst, un,
    uninstall, unpublish, unstar, up, update, v, version, view,
    whoami

npm <command> -h  quick help on <command>
npm -l            display full usage info
npm help <term>   search for help on <term>
npm help npm      involved overview

You can use this to install node packages. You can either install them globally for your machine, or locally for a project. I recommend for now always installing things locally, so that you 'start fresh' with each new project and know exactly what is installed. Installing locally is the default.

To get started with npm, you first create a project. Just create a folder and run a command to initalise it as follows:

mkdir test
cd test
npm init --yes

The --yes means it will say yes to all of the questions, which gives good enough defaults about the name of the project etc.

Now if you look in the folder there is a package.json which is a file that describes your project, and it keeps track of what other projects you depend on.

Now let's install something. How about webpack!

npm install -save-dev webpack

The -save-dev means it will consider the thing you are installing a 'developer' dependencie, i.e. a tool that you will use to help you do development, but isn't part of your main program.

What happens when you install

The install will take a few seconds (depending on you connection to the internet). Once this is done, some things happen in your project folder:

A node_modules folder is generated which contains the source files for what you installed.
In package.json a devDependencies section has been added which contains information about the package you installed.
A package-lock.json file is created which details the actual versions of dependencies installed.

Take a look at all of these now, but don't worry if you don't understand them for now.

Now webpack is installed, how do you run it? This is a bit non-obvious, but not too hard.

Webpack has been installed into the node_modules folder. You could go and hunt for it's script in there and run it with the node command. That would totally work. However this is the hard way because every node module has it's files arranged in different ways.

The easier way is to edit package.json and add an entry to specify a script that you can run that in turn will run webpack. This seems convulted, and it is, but it means NPM takes care of finding the entry point of the package for you. If you edit the package.json file and add a value under scripts as follows for webpack:

...
"scripts": {
    "test": "echo \"Error: no test specified\" && exit 1",
    "webpack": "webpack"
  },
  ...

Running Webpack

You can now run webpack from npm itself:

npm run webpack

And it figures out where to find it.

Run that command now and let's see what happens.

After a moment of consideration, it will ask you if you want to install webpack-cli which is needed to use webpack from the command line as opposed from a node program. Say yes. It will download and install that for you, as if you typed npm install --save-dev webpack-cli.

You will get an error because there is nothing for webpack to process. Yes errors suck, sorry!

It will look for a source Javascipt file src/index.js (we can configure it to look elsewhere if we wish). So let's create a folder called src within the project folder test, and a file called index.js inside that, and add the following Javascipt code:

console.log('hello')

Now type npm run webpack again, and it will be succesful.

Once run, it will create a folder called dist (for distribution or distributable) and inside there is an main.js. What do you think will be in there? Have a guess then open it up.

Spoiler alert;

It's a long line of weird funny Javascript, with your console.log("hello") somewhere near the end. If you didn't guess that, it's fine, neither did I the first time I used it.

That funny JS is code that deals with module loading, something necessary for piecing together many different packages into one bundle. It helps with modularisation, something that Javascipt on it's own traditionally couldn't achieve.

This code happens to be code that works both in the browser and on NodeJS. That's not usually the case, but with a simple console.log it does work in both places. Let's run it:

node dist/main.js

And you get the expected 'hello'!

You can also put run it in a web page, by creating a file index.html with the following content:

<html>
    <head>
        <script src="dist/main.js"></script>
    </head>
</html>

When you open this file in a browser, and open the dev tools by pressing F12, and click the "console" tab you will see that it outputs hello there too.

Using a depedency

Webpack supports the require statment for using dependencies in your project. There are two main types:

Other files in your project
Other projects available via the NPM ecosystem.

Let's try using both, and see what webpack does with these.

Local files

To reference another javascript file from your index.js you just use:

var imported = require('./otherfile.js')

To give you an idea, index.js could contain:

var imported = require('./otherfile.js')

console.log(imported.hello)

and the otherfile.js contains:

exports.hello = 'hello'

and when you run webpack, and run the generated main.js it will do as it did before and print hello.

This works because in otherfile.js the object exports becomes the return value of the require call in the other file. It's how you teleport an object across! More accurately, the strange code we saw earlier in main.js modularises each file, then runs the module for the otherfile.js first, and get's its return value and provides it to the module for index.js.

Although we have used a simple string here, of course you can export anything - a function, an object or a class or value, and you can export multiple things by using different properties of the export object, allowing you to export a whole suite of utilities if you need to.

NPM packages

Earlier we used npm to install webpack, a tool that we are using to package our javascript files into a single file. We can also use npm to install dependencies that we need for our source javascript to run. For example we might want to use the popular jQuery library to manipulate the browser DOM and add or remove elements from the page.

Let's do that now.

Remember earlier we added webpack as a developer dependency. We will now add jQuery but as a regular dependency. jQuery is hosted on npm under the package named 'jquery'. To install it we run:

npm install jquery

If we now open our package.json we can see the dependency now inside the "dependencies" section rather than "devDependencies".

  "dependencies": {
    "jquery": "^3.4.1"
  },
  "devDependencies": {
    "webpack": "^4.41.5",
    "webpack-cli": "^3.3.10"
  }
}

With this installed we can now use require in our Javascript to use JQuery. This is how we would do it inde index.js:

var $ = require('jquery')

$(document).ready(function() {
  $("<p>Hello</p>").appendTo("body");
});

In this example we use require to get the jQuery reference and assign it to a variable called $ as is customary with jQuery. We then go ahead and use $ in 2 jQuery calls, one to wait for the document to be ready, and once it is, we use it to append "Hello" to the document.

If you are following along you can now reload the index.html and see the word Hello appear in the document. Previously it appeared in the console log.

I am going to leave it here for now talking about Webpack. What we have seen is it's default behaviour. It is very configurable though, and you can change where it gets files from, where it writes to, whether it 'minifies' the Javascript (make's it smaller) or not, and even use plugins and loaders to extend it to deal with other features like converting css files or compiling files written in other programming languages, like Typescript. It's an amazing powerful tool.

I hope to write a second article going into those options soon.

This article was originally posted at https://superjavascript.com/t/javascript-webpack/

Get More People To Your Site With These 8 Simple SEO Tips

Martin Capodici — Fri, 12 Jul 2019 10:57:24 +0000

If you want to get people to visit your website, then SEO (Search Engine Optimization) is a cost effective and sustainable way to help achieve that goal. The benefits of SEO are:

It's much cheaper than paid search or social media advertising.
The kind of visitors you get can be better "quality" i.e. more interested or more motivated having specifically searched for what you are offering.
Investment in SEO compounds over time.

The main drawbacks are:

It takes patience. You cannot get traffic on the same day from SEO, you may have to wait days or weeks for some traffic, and months for decent results.
You are dependent on Google and if things go wrong, through no fault of your own your traffic source could come to a screeching halt.

If you want to get started with SEO, here are some quick tips that won't take much time but could make a lot of difference compared with doing nothing:

1. Use the title and meta description tags.

This is an obvious one but people still miss it. It fulfills two purposes. One is to tell the search engine what your site is about, and secondly to configure how you want your site to look in the search results.

As such there are a few things to consider:

What is the target keyword(s) of this page, and how can I insert the exact keyword or something that is similar into the title and/or description?
When someone is searching for those keywords, what is their state of mind and what are they looking for?
When they see my title and description, will they be interested and click?
Once they click, will they be happy with what they find on the page, or feel mislead or disappointed or just "meh" and go back?

That is a lot to think about but get this right and you will get a lot more visitors than otherwise.

Here is an example of the tags, which need to go inside the head tag:

<title>This is my title</title>
<meta name="description" content="This is my description">

2. Use heading tags for your keyword.

Another tip that's as old as the hills!

Your page should have a main target keyword. It is usually a phrase such as "Dog Training" or "Dog Training Boston". You will use this in the title and meta tags and you should also use it (but not overuse it!) in heading tags in your page.

If it doesn't perhaps use a variant that means the same thing. Google can tell that "Dog Training" and "Training Dogs" are about the same thing, so if the second one is more natural then use it.

My rule of thumb for keywords is get them in, but if it looks odd to the reader that you are saying the same phrase too often then you are probably over doing it. Write for the human reader, not the robot search engine.

3. Write great original content.

You don't want to copy and paste content from elsewhere. Google will look at that page and consider it inferior to the original. You also don't want to generate or get cheap writers, you want solid content. Less but better content that people want to link to is better than lots of bad content.

4. Make sure your site is fast.

My tips for making your site fast:

Ideally use static hosting and a CDN. Netlify is a great option here and is free. I use it now for all my sites, including this one.
If not static hosting then make sure you are caching aggressively using an appropriate tool (e.g. for Wordpress look at https://wordpress.org/support/article/optimization-caching/) and have plenty of power in your hosting servers. It might be time to upgrade to a better hosting plan if not.
Check out how much assets the browser needs to download by viewing page(s) of your site in Chrome and pressing F12 and clicking the Network tab. Get this as low as possible without sacrificing what you are trying to do on the site. Less than 250kb total is a good thing to aim for. Removing unnecessary fonts, JavaScript and reducing image quality and/or size will help here.
Check your site speed with Google at https://developers.google.com/speed/pagespeed/insights/

Making your site fast has a twofold effect. One is you will be ranked better by Google. Second real visitors your site won't get frustrated and click away.

5. Build backlinks in an ethical way

You still need backlinks in 2019 and the best way to get them is by writing great content and letting people know about it. You can also write guest posts, and use your link in forum / social media signatures and profiles (where allowed) and then go on those forums and social media and help people out.

Links with the rel="nofollow" attribute will have little to no value for SEO. Also links that are dynamically created using JS are of no value. So if you are limited on time try to prioritize link building where you get a link that appears when you do "view source" on the page, and doesn't have the nofollow attribute.

Avoid using backlink building services, as many of them spam sites and those links at best will turn to dust quickly as the moderators of those sites delete the spam. Sites that are not moderated are likely to be not maintained and will eventually die, leaving you with nothing. Also Google is well aware of the difference between spammy links and good links from authority websites.

6. Google Search Console

This is a must, and it's so easy and it's free. Sign up for Google Search Console and verify your site. Now just wait a few days and log in and you will start seeing insights including:

Which keywords had you appear in Google.
How many clicks you got.
Where people landed on your site.
Typical search result positions.
Warnings about things you need to do to your site to improve SEO.

Using this information you could then decide what keywords to target in future articles, or which pages could do with some backlinks to help push them up from page 2 to page 1 of the Google search results.

7. Make a Sitemap

XML Sitemaps are a way to tell Google what pages exist on your site and will help in discovery and faster indexing. How to create on will depend on your site's technology. If you are using Wordpress then you can use sitemap plugins.

Once you have a sitemap, submit it to Google by logging in to the Google Search Console (useful isn't it!) and submitting.

8. See what the competition is doing

You can spy on what sites like yours are doing to get backlinks and what keywords people are using to find their site. Services such as ahrefs and moz will do this for a monthly fee that isn't cheap, but at the same time is affordable to a lot of businesses.

If you can afford it I would definitely give them a go and you can use their services for more than one site. Alternatively these sites also have free plans that will get you started somewhat.

These services will also help you research keywords and estimate the traffic you will get from them.

I recommend this book (I am not an affiliate or connected with the author): https://www.fullstackoptimization.com for learning all the technical details about SEO.

It will take about 4-16 hours to read depending on your technical level but it is definitely helpful and reasonably priced. The only catch is, at least when I purchased it, is that it needs to be shipped from Germany. I guess that'll get you in the patient mindset you need to do SEO!

That's it

Thanks for reading and I hope this get's you started on SEO and get some visitors to your blog, store or site.

JavaScript Dates

Martin Capodici — Sun, 23 Jun 2019 21:54:46 +0000

If you do any amount of programming, you will likely cross the thorny subject of dates and times. It's a tricky subject because of all of the world's time zones, changing clocks, leap years and other oddities. That said, it is simple to get started with dates in JavaScript, and this tutorial will explain most of what you can do with Dates in Javascript, as well as touch on a library you can use if you need more.

What is a JavaScript Date?

A Date in JavaScript is an object that represents a date and time. JS Dates can go back or forward about 280,000 years and are accurate to the nearest millisecond (thousandth of a second). Inside the Date object there is just one value which is the number of milliseconds before or after 01 Jan 1970 in UTC time. UTC time is a standard time used to help coordinate time across the world. It is similar to the Greenwich Mean Time (GMT) time zone.

When dealing with JavaScript dates, you are really just dealing with a number, such as 819131040000. However dealing with numbers like that is inconvenient so there are helper functions in JavaScript to make dates, query them and manipulate them. We'll look at those soon, but first lets make a date.

Creating a JavaScript Date

Create a JS Date from today's date and the current time

Here is an example of creating a date that sets it to today's date and time:

var today = new Date();

You can now view what is in that date by using one of the methods on Date, such as toDateString:

console.log(today.toDateString()); 
// Returns a string like: "Fri Mar 01 2019"

console.log(today.toTimeString()); 
// Returns a string like: "05:00:00 GMT+1100 (Australian Eastern Daylight Time)"

The format used in the toDateString and toTimeString methods isn't specified in Javascript. This means different web browsers could return different formats, and it might depend on your computer's locations settings too.

Aghhh! - time zones

Judging from the second result, which contains "GMT+1100" and "Australian Eastern Daylight Time", time zones are a factor here, so let's discuss them quickly.

The today variable from the previous example is set to today's date and time, and this is stored in UTC time. The code was run in Australia and the UTC date and time, at that time, was 6pm on February 28th 2019 UTC. The today variable knows that it is '6pm on February 28th 2019 UTC'.

When toDateString and toTimeString are called they use the current time zone of the browser, and convert the internal date and time to that time zone. In this case it happens to be in Australia again, so the results are produced according to that, i.e. '5am on March 1st 2019 AEDT'.

We will see this UTC conversion process a few more times through the tutorial, so don't worry if it seems a bit confusing now.

What if you want to create a date with a specific date and time?

There are 3 ways to do this:

Create a JS Date by parts

If you need a specific date and time, rather than the current time, the recommend way to create a date is to specify the year, month, day, hours, minutes and seconds. This can be done as follows:

var date = new Date(2019, 02, 03, 00, 00, 00); // 03 March 2019 in local time zone.

The arguments are as follows:

Year
Month (zero based, so 0 = January, 1 = February etc.)
Days
Hours
Minutes
Seconds

The date and time is converted from local time to a UTC-based number inside the Date object. This means you can call methods on Date to find it's UTC value. You can use other methods to see what that the time is locally too.

To see this in action, look at this example, when run in Australia, using getDate and getUTCDate to pull out the day of the month from a Date we have just created:

new Date(2019, 2, 3, 00, 00, 00).getUTCDate(); // 2
new Date(2019, 2, 3, 00, 00, 00).getDate(); // 3

Whenever you are sending dates to another system (e.g. a web server) it is often wise to send them as UTC, because the place where they end up might be in a different time zone, and using UTC makes it easier to coordinate because you only need one conversion to get to or to get from UTC.

Create a JS Date by String

This is not recommended as it is not standardized and different browsers (or other JavaScript environments such as Node, embedded devices, etc.) may give different results.

var date = new Date('2019-03-03T00:00:00');

This will also treat the time you give as a local time and convert it under the hood to UTC.

The only good time to use this is if you have an ISO formatted string. I'll tell you how to create one of those later in the tutorial.

Create a JS Date from a number

You can create a date from the "unix timestamp", which is the number of milliseconds since 01 Jan 1970. This number is how JavaScript Date stores the date internally anyway. It is a useful method if you are given such a timestamp from a server, which sometimes happens because some databases store dates that way.

var date = new Date(1551531600000);

Querying JavaScript Dates

Locally

There are a bunch of methods for getting the local date and time information from a date object. They are summarized here. All of them return the result for the local time zone:

Date

getFullYear() - Returns the year, with 4 digits if it is a 4 digit year.
getMonth() - Returns the month, with 0 being January, 1 is February etc.
getDate() - Returns the day of the month.
getDay() - Returns the day of the week, 0 being Monday, 1 being Tuesday etc.

Time

getHours() - Returns the hour.
getMinutes() - Returns the minutes.
getSeconds() - Returns the seconds.
getMilliseconds() - Returns the milliseconds.

Here is an example:

var date = new Date(2019, 2, 3, 0, 0, 5);
// Note that Date object stores the date and time as UTC internally, but it is converted
// back to local time when these methods are used:
date.getFullYear(); // 2019
date.getMonth(); // 2, i.e. March
date.getSeconds(); // 5

In UTC

You can also get the UTC date and time information, with these equivalent methods:

getUTCFullYear()
getUTCMonth()
getUTCDate()
getUTCDay()
getUTCHours()
getUTCMinutes()
getUTCSeconds()
getUTCMilliseconds()

Here is an example that, using toUTCString as well to show you the full date that is stored inside.

var date = new Date(1551531600000);
date.toUTCString(); // Returns "Sat, 02 Mar 2019 13:00:00 GMT"
date.getUTCDate(); // 2
date.getDate(); // 1,2 or 3 depending on your local time zone.

Changing JavaScript Dates

All of the get methods above also have a set equivalent, so that you can change a date:

setFullYear()
setMonth()
setDate()
setDay()
setHours()
setMinutes()
setSeconds()
setMilliseconds()
setUTCFullYear()
setUTCMonth()
setUTCDate()
setUTCDay()
setUTCHours()
setUTCMinutes()
setUTCSeconds()
setUTCMilliseconds()

Each of these methods takes in a number in the same format as returned by the equivalent get......() method. Here are some examples:

var date = new Date(2019, 2, 3, 0, 0, 5);
date.getFullYear(); // 2019
date.setFullYear(2020);
date.getFullYear(); // 2020

var date = new Date(1551531600000);
date.toUTCString(); // Returns "Sat, 02 Mar 2019 13:00:00 GMT"
date.setUTCHours(10); 
date.toUTCString(); // Returns "Sat, 02 Mar 2019 10:00:00 GMT"

Conversion

There are some methods for converting the date to different formats. Here are the most useful ones:

Date Method	Time Zone	Returns
toDateString()	Local	Date portion of the date, in a format to be read by a human. The format might vary from browser to browser.
toTimeString()	Local	Time portion of the date, in a format to be read by a human. The format might vary from browser to browser.
toUTCString()	UTC	Date and time portion of the date, in a format to be read by a human. The format might vary from browser to browser.
toISOString()	UTC	Date and time as UTC time in `YYYY-MM-DDTHH:mm:ss.sssZ` format ( ISO 8601 Extended Format).
toLocaleDateString()	Local	Date portion of the date, e.g. "3/2/2019", formatted based you on your computers's locale settings. The order of the month and day will depend on those settings, for example.
toLocaleTimeString()	Local	Time portion of the date, e.g. "9:00:00 PM", formatted based you on your computers's locale settings. The type of clock 12 hour or 24 hour will depend on those settings, for example.

There are some similar methods that are less useful, or are not a fully accepted standard so may not work in your browser. To see them click here

Examples, with results from my time zone. Your results might be different depending on your time zone, browser and computer's locale settings.

var date = new Date(2019, 2, 3, 0, 0, 5);
date.toDateString(); // "Sun Mar 03 2019"
date.toTimeString(); // "00:00:05 GMT+1100 (Australian Eastern Daylight Time)"
date.toUTCString(); // "Sat, 02 Mar 2019 13:00:05 GMT"
date.toISOString(); // "2019-03-02T13:00:05.000Z"
date.toLocaleDateString(); // "3/3/2019"
date.toLocaleTimeString(); // "12:00:05 AM"

None of those results above are guaranteed to be the same on your computer because when we create the date, the date is converted from the local time zone to UTC. So your date might be different to mine in the first place. And even then, most of those methods return formats that are dependent on your computer settings.

Be careful ...

...with methods that depend on time zone and computer settings

Most of the methods I just listed depend on time zone, your browser and your computer settings. They should be used with care. They are fine for showing a date to the user, but they are not recommended if that same date is going to be read again by your program (or someone else's). This is because you can't predict the format the string will be in. The day of month and month might be in a different order to what you expect, then the code using that date will get it wrong sometimes.

I recommend:

If you need to pass dates around your program, keep them as Date objects if possible.
When sending dates to a server or service, use the toISOString to format the date because it is a standard format and uses UTC. Alternatively use the getTime to get the number of milliseconds since 01/01/1970, and this is also in UTC.
When receiving dates from a server or service, if you have any control or influence, try to make sure they are sent as an ISO string or milliseconds since 01/01/1970, and then you can pass them as the argument to new Date().
Only use the toDateString, toTimeString, toUTCString, etc. methods for strings that will be shown to a user of your web page.
Be careful when using libraries with calendar controls, and read the documentation about what date format's they expect to work with.

Moment.js

Do you need to do something with dates that the Javascript Date object cannot support? For example formatting dates using a format string? Or more advanced time zone conversions?

Moment.js is a library that gives you more power to work with Dates in JS, and is recommended if you are struggling to achieve what you need with the standard JavaScript Date object. I will write a separate tutorial for moment.js in the future.

The first example from the moment.js website shows you that you can format dates to your liking with this library:

moment().format('MMMM Do YYYY, h:mm:ss a'); // March 2nd 2019, 9:06:12 am
moment().format('dddd');                    // Saturday
moment().format("MMM Do YY");               // Mar 2nd 19
moment().format('YYYY [escaped] YYYY');     // 2019 escaped 2019
moment().format();                          // 2019-03-02T09:06:12+11:00

Footnote on UTC

Early I said that UTC is similar to Greenwich Mean Time (the UK's winter time zone). However it is not the same! There can be up to 1 second difference between the two.

This is because there aren't exactly 24 hours in a day. Our planet is a big lump of rock, not a Rolex. And it's slowing down. GMT tracks the rotation of the earth to keep the kind of time we are used to. UTC uses scientifically accurate seconds, but needs adjustments along the way to keep it in sync with the Earth's rotation.

Did I say before dates and times are tricky? In any case, for almost all cases you won't have to worry about that 1 second but it is useful to know, and it shows that we must treat dates and times with a lot of care.

Summary

This tutorial covered the Date object in Javascript, and it is pretty good at giving you basic functionality to deal with dates. Sometimes this will be too limited for you and you will need to use a library like moment.js, or at least some 'utility functions' that you keep in a certain part of your program. It is good to be aware of time zones, and that a JavaScript Date is a UTC based date under the hood, even if you create it and query it in the local time zone.

This article was originally posted at Super JavaScript: JavaScript Dates

Using and Abusing Netlify's Free Tier

Martin Capodici — Sat, 22 Jun 2019 11:42:03 +0000

If you are hosting a website you absolutely have to be concerned about it's speed. How fast is it serving up your page? How long does it take from the user hitting enter to them seeing the complete page rendered?

There are many aspects to creating a fast site, because there are many touchpoints where things can be slow. The first point is the hosting company, plan and methodology that you use.

Many sites are served up dynamically using technologies such as PHP or Ruby on Rails. Sites that need to do this will usually be slower than sites that use static html files, because there is CPU work for the server to do. It needs to compute the contents of the web pages, rather than just serve them up from the hard disk.

If you can do it, moving the static file hosting will make a big difference to speed. (Or alternatively, using caching will make a big difference too).

Static file hosting gives you another speed boost that you many not have thought of. If your site is just a bunch of dumb files, those files can be served from many locations around the world. You could have a server in New York, Brazil and Hong Kong. Servers in different cities, with the closest server to the browser serving up the content.

This will help you overcome an unavoidable constraint in getting content to your visitors fast: the speed of light. If your content is coming from the other side of the world, and not 'as the crow flies' but via many hops through other countries - it will slow things down.

If you have servers near your audience, it will be much faster. This sounds hard to do, and that is why you use a CDN (content delivery network) to do it for you. Netlify is an example of a hosting company providing static file hosting on a CDN. Stick your site there and it will be fast wherever your audience is.

And this matters both for your visitors (so you don't lose them by being too slow!) and SEO - because Google cares about how fast your site is.

Netlify has a generous free tier, and on the site https://dealscombined.com.au we are using it (and abusing it - see below) to host thousands of deals from around Australia. Please take a look and hopefully you agree it is a fast loading site.

I say we are 'abusing' Netlify because we are using it in a much different and more aggressive way than most users. We have 8000 deals pages, and we push them from a server rather than let Netlify get them from Github. The reason for this is that the pages can change each day and so it is no good keeping the source data in a git repo as it will get way too big to be practical.

Make fewer JS mistakes with TypeScript

Martin Capodici — Fri, 21 Jun 2019 21:58:48 +0000

(This article original posted on https://superjavascript.com/t/typescript/)

Javascript (JS) was originally intended to make small alterations to web pages, such as spinning animations when you scroll over an email @ symbol, or to validate that you have typed an email address correctly. In the years since, JS has been used for more complex things. For example it is used for so called "Single Page Applications", which are like traditional desktop applications delivered on the web browser. The first famous application like this was Google's Gmail.

In 2011 NodeJS was released, letting developers write JS code to run on servers to provide back-ends for websites. These back-ends are required to create dynamic websites that allow people to log in, store information and communicate. Traditionally these would be written in languages like Java, C#, Ruby or Python, but NodeJS became popular, in part because you can use the same programming language of JS for the web page as you do on the back end.

In 2013 GitHub released Electron, a tool-kit for creating traditional desktop applications using ... you guessed it ... Javascript. Electron is based on web technologies, so the HTML, CSS and Javascript you write for the web can be reused in Electron. However Electron gives developers access to resources on the user's computer, such as local files, camera and sound.

With Javascript usage evolving from simple website tweaks to a language that is being used everywhere, it has shown signs of strain keeping up. Different versions of Javascript have been released such as ES5 in 2009 and ES6 in 2015 to modernise the language. However all versions of Javascript suffer from a problem that makes it hard to manage larger programs. This is a problem that is solved by TypeScript.

Javascript's Problem

The problem with Javascript is that is uses "dynamic typing".

What does that mean? It means that when you write the code, you have the flexibility to use objects, arrays etc. in any way you see fit, and there will be no errors at the time you write the code or compile it.

That sounds like a great thing right?

Well the catch is you will still get errors when you run the program. By then it is 'too late'. To make things worse some of those errors will only be uncovered in rare situations. Situations that users of your program might find themselves in, but that you didn't test for.

To illustrate the problem, imagine you have a variable passed in to a function called honestly_im_a_string. Then that variable could be a string (and probably should be!), but it could be anything at all, including:

undefined
null
a number that is a number like 4
a number that isn't a number like NaN or Infinity
an array
an object
an array where the first value is an object and the second is NaN
and so on...

If you just assume the value is a string and start doing things like find/replace with it, then the program will error if another piece of code passes an array into it.

Is this a likely mistake. Yes! Perhaps someone else is writing the code that calls your function and they didn't know. It could be the same programmer revisiting code from months ago making this mistake. With lots of functions and variables to keep track of, this sort of mistake could even happen if the same programmer writes both the function and the code that uses it on the same day!

As programs get larger and there are more functions and more objects, it can be easy to lose track of what these functions expect. For example if you pass in an object with coordinates, is it x and y or X and Y that are expected as properties? Is it expecting an array of these or just one. It is easy to make a mistake somewhere, and often only find out when it is too late and the program is being used by users who call you up with a problem.

The solution

The solution is "static typing".

This is where you define the type of variables when you write the program. Before you run your program a compiler will check that you haven't misused any types.

In the honestly_im_a_string example, this means you would be told before you run your program that there is an error, because someone is trying to pass an array into a function that wants a string.

In the coordinate example, you'd define that your object has an x and if you try to get it's X you will get an error from the compiler, so you can fix it before you even run the program for the first time.

TypeScript provides the ability to define the type of variables in your program. This works by writing your program in the TypeScript language, which is like Javascript but it allows you to "annotate" your variables with types.

TypeScript programs are compiled into Javascript by a tool called "tsc" or "TypeScript Compiler". During this process, it will check that you haven't made any mistakes.

As an example, less look at a TypeScript function that takes a string and adds a full stop (or period) to the end:

function addPeriod(sentence: string) {
  return sentence + ".";
}

The difference between this and the JavaScript equivalent is the type annotation of : string in the arguments to the function. This tells the TypeScript compiler that a string is expected. This has two effects.

One is that it knows that sentence is a string, and it will check whatever you do in the function is valid for a string. In this case we are appending another string to it, which is fine.

Secondly, it checks that any code calling this function is passing in a string and not something else. If this check fails it will fail to compile. With Javascript you wouldn't know until you run the program that there is a problem.

Another interesting thing TypeScript will do with the code above is it will figure out that you will always return a string. It knows this because it knows that sentence is a string, and you are appending another string to that and returning it.

For any code that calls addPeriod and puts the result in a variable, TypeScript will know that variable is a string.

For example the following code will give an error when trying to compile:

let result = addPeriod("This is a sentence");
result = result * 3;

Here is what the error message looks like when viewed in the online TypeScript compiler:

The escape hatch

The nice thing about TypeScript is you can sometimes cheat and use it like good ol' Javascript.

Javascript is excellent for dealing with responses from a server, which may have differently formatted data, missing data, etc. It is good because it treats the response like any other JavaScript object and you can call any properties you like without having to define them in advance. If you ask for something that is not defined, it's value will be the undefined value that is well known in JavaScript.

For example if you get an object for the result of a soccer game you can call say .teamAgoals and .teamBgoals to see the score. If later they change the response to also include other information, such as the time of the game, you can easily change the program to process that.

In TypeScript you can declare a variable as having type any and this will allow you to treat it like classic Javascript, and have it be an unknown type that you can probe to see what properties it has. Here is an example:

let serverResponse : any = { color: 'red', numbers: [1,2,3] };

// This will say 'red':
console.log(serverResponse.color);

// This will say undefined:
console.log(serverResponse.shapes);

This example will compile, even though we are referring to shapes which the TypeScript compiler could otherwise tell is not part of the serverResponse object.

Summary

Here we have scratched the surface of TypeScript, a programming language that allows you to write JavaScript programs with additional safety checks so that you will make fewer mistakes in your JavaScript programs.

(This article original posted on https://superjavascript.com/t/typescript/)

I have a free book on making a canvas game in JS. See https://superjavascript.com/book/

Build Yourself a Snake Game in JS

Martin Capodici — Fri, 31 May 2019 09:14:19 +0000

I have created a new book, for free, to create a game in JS: https://superjavascript.com/book/