DEV Community: Michael Rowlinson

🤯 Private S3 HTTP Server

Michael Rowlinson — Tue, 09 Nov 2021 13:12:04 +0000

What are we solving here?

In my AWS adventures, I've come across use cases where it'd be awesome to have easy access internally to some files in AWS without making something public....or creating an S3 bucket behind a Cloudfront behind a WAF with a whitelist rule.....

Anyway, http-server is an awesome package that does this exact thing for file shares. So I figured I'd fork the code base and replace all the file server bits with S3 calls.

Opening the hood of http-server, I realized for my specific use case, most of the code has features I didn't need. So, I opted to create s3-http-server from scratch with http-server as inspiration.

What follows is an explanation of the interesting bits.

A look at the 🥩🥔 code

Firstly, the stack used for this solution:

nodejs - javascript runtime
express - http server
pug - template engine

The most important features are listing, downloading, and uploading objects.

Listing Objects

The code snippet for this is straight forward, but ensuring you are only returning objects AND prefixes at a given level is a little obscure. Below is the excerpt of code from the express route's async function handler -

const AWS = require("aws-sdk");
const s3 = new AWS.S3();
const Bucket = 'MyBucket'
const Prefix = ''

...
    const s3Res = await s3
      .listObjectsV2({
        Bucket,
        Prefix,
        Delimiter: "/",
        MaxKeys: 1000,
      })
      .promise();
    const data = [
      ...s3Res.CommonPrefixes.map(({ Prefix }) => ({
        Key: Prefix,
        isDir: true,
      })),
      ...s3Res.Contents.filter((c) => c.Key !== Prefix).map(({ Key }) => ({
        Key,
        isDir: false,
      })),
    ];
...

The first part returns a list of s3 objects at a given prefix. Note that in a large bucket, you would want to handle pagination of the objects as well.

The shenanigans creating the data variable is the good part. If, for example, you call listObjectsV2 with a Prefix of "" (the root of the bucket), you only get objects in the return object's Content property. To get the prefixes at the root (or anywhere else) you'll need to look at the CommonPrefixes property.

Downloading Objects

The downloading of an object in S3 is a fun implementation as well. Here is an abstracted excerpt of that code.

...
      const readStream = new stream.PassThrough();
      const fileName = ...
      res.set("Content-disposition", "attachment; filename=" + fileName);
      s3.getObject({
        Bucket: bucket,
        Key: decodeURI(req.originalUrl.substring(1)),
      })
        .on("error", (err) => {
          console.log(err);
        })
        .on("httpData", (chunk) => {
          readStream.push(chunk);
        })
        .on("httpDone", () => {
          readStream.end();
        })
        .send();
      readStream.pipe(res);
...

This works by creating a pass through stream. We then call getObject and configure a listener for httpData. Each time the listener function fires the stream gets the current chunk pushed into it. Finally we pipe the stream to the express response stream.

Uploading Objects

The client website also allows uploading objects into the current S3 prefix.

...
    const form = new formidable.IncomingForm();
    form.parse(req, async function (err, fields, files) {
      const { originalFilename, filepath } = files.filetoupload;
      const rawData = fs.readFileSync(filepath);
      await s3
        .upload({
          Bucket: bucket,
          Key: req.originalUrl.substring(1) + originalFilename,
          Body: rawData,
        })
        .promise();
...

We leverage the formidable package to simplify the file upload. Simply call the IncomingForm function and follow that up with the parse function on the return object. The callback passed to the parse function is where the magic happens. We get a local file path of the file that has been uploaded and the file name. We then read the uploaded file into memory and create a key using the current prefix and supplied file name which is all passed to the s3.upload function.

Using the npm package

Here's the s3-http-server repo if you'd like to look at the code base in full.

Install the package with npm

npm install s3-http-server --global

Make sure you have AWS keys available in the environment

Run the following to fire up the server

s3-http-server my-bucket-name

Navigate to http://localhost:8080 and start browsing your bucket.

Wrap up

Thanks for reading. There are a few libraries in the wild that serve a similar purpose. This was my take on it. I'll add features like deletion of objects and website serving in the future. Let me know if you have any suggestions.

Peace ✌️

CryptoTracker in 70 lines of code!

Michael Rowlinson — Tue, 02 Nov 2021 12:14:52 +0000

🏗 Setup Project

Let's get this started by setting up the react project and installing the required package.

npx create-react-app cryptotracker
cd cryptotracker
npm i ez-react-table

🤑 Create Tracker

Open up src/App.js in your favorite editor and delete everything in it. Let's start from scratch.

Call CoinGecko API

import React, { useState, useEffect } from "react";
import EzReactTable from 'ez-react-table'

export default function App() {
  const [coins, setCoins] = useState([]);
  const fetchCoins = async () => {
    try {
      const res = await fetch(
        "https://api.coingecko.com/api/v3/coins/markets?vs_currency=usd&order=market_cap_desc&per_page=100&page=1&sparkline=false"
      );
      const data = await res.json();
      setCoins(data);
    } catch (e) {
      alert("Api error");
    }
  };
  useEffect(() => {
    fetchCoins();
  }, []);
...

What we are doing is setting up a coins state that defaults to an empty array. Once the App component mounts we run the fetchCoins function. The fetchCoins function calls the Coin Gecko API and sets coins to the array of crypto coins.

Configure Columns

Next up, let's configure the columns variable that will be passed to ez-react-table and tell it how to render the coins state to a table.

...
  const columns = [
    {
      title: "Name",
      key: "name",
      width: 200,
      render: (value, object) => {
        return (
          <div style={{ display: "flex", alignItems: "center" }}>
            <img height="20px" width="20px" src={object?.image} alt="coin" />
            <span style={{ marginLeft: "10px" }}>{object?.name}</span>
          </div>
        );
      },
    },
    {
      title: "Symbol",
      center: true,
      key: "symbol",
      width: 120,
      render: (value, object) => <div>{value.toUpperCase()}</div>,
    },
    {
      title: "Price",
      key: "current_price",
      width: 100,
      render: (value, object) => <div>{`$${value}`}</div>,
    },
    {
      title: "Change",
      key: "price_change_percentage_24h",
      width: 100,
      render: (value, object) => {
        return (
          <div style={{ color: /-/i.test(value) ? "#ff0374" : "#06a847" }}>
            {value}%
          </div>
        );
      },
    },
  ];
...

Couple of points of interest with the above code. The columns variable is an array of column objects. Each object defines the title for the column, the width of the column, the key name corresponds to the object property we are mapping to a column, and the render method takes a component which receives 2 arguments, the value of the current property and the object for the row. The component returns whatever should be rendered to the row cell. Take note that you could forego the render property altogether and ez-react-table would render the value to the cell.

Putting It All Together

...
return (
    <EzReactTable
      cols={columns}
      data={coins}
      darkMode
      title="Crypto Tracker"
      defaultSort="name"
      accentColor="#ffbc03"
      tableHeight={300}
    />
  );
};

Finally! Return the EzReactTable component. The cols property gets the columns variable we defined. The data property takes the coins state to be rendered. The defaultSort property is a key defined in the columns variable we'd like to default to sorting the table by. There are a few self explanatory properties as well.

👟 Run it!

Okay the code is done! Let's test it out by running npm start in the project directory and navigating to localhost:3000.

Bask in the glory! Virtualized rows, a global search input, column sorting, and style with almost no effort!

Here is the git repo if you'd like to look at the code.

🔮 What's Next?

The secret sauce for this article was ez-react-table. The package is built to be straight forward to use and powerful.

Try ez-react-table on your next project!

Thanks for coding along!

Easy React Table

Michael Rowlinson — Sat, 23 Oct 2021 00:56:31 +0000

What?

A simple batteries included table component for React projects.

Say hello to ez-react-table!

⚡️ Try the Demo! ⚡️

Why?

I build a lot of admin related dashboards and wanted a table that just works. A table with almost no setup and a nice user experience.

How?

Out of the box you get a search bar for filtering, sorting, virtualized rows, infinite scrolling ability, effective styling (including dark mode), and sweet cell overflow handling with tooltips. Additionally, it is arbitrary to add custom components to row cells.

Setup

npm i ez-react-table

Example

import * as React from "react";
import EzReactTable from "ez-react-table";

// fake data
const data = Array.from(Array(20))
  .map((i) => [
    { first: "Michael", last: "Myers" },
    { first: "Laurie", last: "Strode" },
    { first: "Samuel", last: "Loomis" },
  ])
  .reduce((a, c) => [...a, ...c], []);

// define columns
const cols = [
  {
    title: "First",
    width: 200,
    key: "first",
  },
  {
    title: "Last",
    width: 200,
    key: "last",
  },
  {
    title: "Actions",
    width: 100,
    key: "action",
    center: true,
    render: (value, object) => (
      <button onClick={() => alert(JSON.stringify(object))}>View</button>
    ),
  },
];

function App() {
  return (
    <div className="App">
      <EzReactTable cols={cols} data={data} />
    </div>
  );
}

End

Thanks for reading. In active development as I use across my projects. Stay tuned!

Peace.

Diagram AWS Org Accounts

Michael Rowlinson — Sat, 14 Aug 2021 02:32:08 +0000

Repo for code

Many enterprises in AWS are leveraging AWS Organizations and AWS SSO for central management of accounts and access.

In larger organizations with many accounts, it can require a lot of clicking to figure out what the applied SCPs and permission sets are for a given account.

Enter aws-org-mapper.

This simple python script collects information from both the Organization and SSO services and produces an HTML document that contains a diagram for each account. Each account diagram shows the associated OUs, SCPs, and permission sets.

Diagraming courtesy of Mermaid JS

To get started,

clone down the repo
ensure you have python 3.8+ and boto3 installed
ensure aws credentials are available
execute the script python aws-org-mapper.py
open aws-org-mapper.html file generated in directory

Thanks for reading. Peace!