DEV Community: mich

Shorter Way to Upload to Google Cloud Storage (Just Send the Buffer)

mich — Tue, 21 Jul 2020 23:43:06 +0000

Part of my current project is to take a disk image that's uploaded by the user and place it in google cloud storage. This is something that seems like it should be straightforward and it started close to that then moved further in that direction.

I started with this excellent thorough writeup and picked out the stuff that helped flesh out the mechanics. I use express-fileupload instead of multer just because I like it better for no other objective reason I could really name other than the free md5 sum which I use to know which files are duplicates based on their content.

Many of the examples I was seeing used .createWriteStream() which is likely a really good "learn the long way first" route. However there's a much simplified version (which is the job of a library) called .save(). Your files don't have a content type, but that can be added with a google cloud function (especially if you're already going to do other stuff with it) or a second call to the API after the upload. Thanks to this thread for pointing out this shortcut over three years ago.

Below is code that's roughly what I use.

import Cloud from "@google-cloud/storage";  // currently @5.1.2
const { Storage } = Cloud;

// same storage object for all uploads
const storage = new Storage({
  "keyFilename": "./PATH/MY_GCP_KEY.json"
});

// same bucket for all uploads
const bucket = storage.bucket(MY_BUCKET);

const uploadDiskImage = async file => {
  const { filename, buffer } = file;
  const fileHandle = bucket.file(filename);
  const [ fileExists ] = await fileHandle.exists();
  if (fileExists === false) {
    return fileHandle.save(buffer);
  }
  return new Promise((resolve, reject) => resolve(filename));
};

export default uploadDiskImage;

I believe most of it is pretty simple to read which was largely the point of this post. It could probably get simplified quite a bit more (the .exists() feels like an area of opportunity), but for now it's not too shabby.

Doing the Port Myself

mich — Sat, 18 Jul 2020 06:52:57 +0000

Part of the catalog software that I mentioned in my previous post will take a disk image, validate it, and get a listing of the files on the disk as well as a few other details (disk name, blocks free).

As I also mentioned previously, there's already software out there that does this called c1541 that's part of the VICE emulator. It was done and seemed that it would be easy to use (it's easy to use on a local machine), but this meant that I would need to use google run. Not that I mind that, but it meant building docker a container to have google, well, run.

Starting to get my docker VM running, it ALSO reported that it was out of space (it would appear that 10GB allocations of fixed disks isn't a very good idea). So instead of going through the whole process like I did last time, I went a different direction that I had planned to do later regardless: port c1541 to JavaScript myself. I looked to see if there was anything that already existed and nothing really hit the mark that I needed.

Loading a file is pretty easy in node.js. I just used fs.readFileSync since I really didn't need async and there wasn't much point in wrapping everything else in a promise. From there it was just jumping around the buffer.

I found two different sources that had almost identical information about the bytes of a .d64 file that appeared to both copy from roughtly the same place. It was fun to see that tripod is still around. I remember them from many years ago.

Using these sources, I was able to navigate a bit around the file buffer to get the parts that I needed. It was so simple since you can seek a node.js buffer just like it's an array. So disk[0x16500] gives me the byte at 0x16500. Pretty spiffy!

I'll probably write more about the actual code soon, but in the meantime you can take a look at the github project and/or play with the npm package.

Starting in the Middle

mich — Mon, 13 Jul 2020 06:11:05 +0000

I'm in the middle of building a fun little c64 disk catalog app for myself. The part which currently has my attention is handling the disk uploads. I have a react front end that's uploading the images to my google cloud storage bucket. I want to validate the disk image and get the listing of files for the disk. That's the start of today's adventure.

There's already software called c1541 that's bundled with the vice emulator AND it's already available for ubuntu. Yay! However, this means that I'm doing a google run instead of a google function. Functions can trigger themselves on a file upload, but run needs to pipe through pubsub. Bummer! Well, kind of. I get to play with more toys so that's fun. It's probably faster to do the port later and get this done now.

In my ubuntu 20 VM (host still running 18 until I get the backup and other prep done), I run through the install steps and have somehow popped over my 10GB fixed disk size! No clue if the apt install of the SDK worked, but I'm assuming that it didn't.

I found a quick tutorial for solving a disk that's too small. One thing that should be more clear is that the VM needs to be turned off completely before the storage can be changed. It can't just be in a saved state. This makes sense when you look at the whole process, but if you're just stepping through then you might run into this problem. The other thing that's obvious after a second, but can cause a stumble is to make sure you resize the extended partition that holds your partition so that you can resize your partition larger.

After fixing the partition and finishing the gcloud SDK install (on this newish ub20 install, I had to sudo apt install curl), I appear to be up and running with gsutil. Let's get this bucket wired.