DEV Community: Rick Delpo

Plain Javascript Refresher for those feeling left behind or not knowing where to start w/ Functions, Arrays, Loops, JSON & NoSQL

Rick Delpo — Sun, 27 Oct 2024 15:13:45 +0000

Here we will learn Plain Vanilla JS without using NodeJS or external libraries. We will start right out of the box with a use case by converting some Google sheet data into JSON and storing it in an AWS S3 bucket, then fetching it using plain js. Most lessons start with a hello world program and not much else but here we actually have something to program right away so we can practice our arrays and loops which are key pillars in any programming language. Plus you can have an introduction into the world of Data Science here and perhaps make a career out of it like I did.

When we first study data we think of SQL but there are many apps where SQL is overkill and not needed. In a dashboard with a few metrics we can get away with a simple JSON flat file as our data source with no relationships to other data. Dashboards can use this NoSQL format and are a popular choice for a Marketing groups reporting needs.

In order to set up our environment we only really need Google Chrome and the json chrome extension for converting spreadsheet data to json. Then we need free AWS and an S3 bucket for our generic website. Only a windows notepad will do as our IDE. We will also need a local network to fetch our data because fetching data from the C drive will not work as the javascript Fetch api uses http protocol so a web server is needed for this. Before going public on free AWS we will set up a local web server first to test our data. For this we use simple Python.

set up the environment
steps to set up a local Python server and an index file

Before creating an AWS remote server we need to set up a local web server for testing..here is how to do this

First find out if Python is installed... open a command prompt which will default to your home folder c:\Users\yourname and type in python. If version info shows then it is installed and u can go to step 6 below (but be sure that an index file is saved in your home folder first, see below)
If you dont have Python installed follow these instructions:

Python Server in windows

go to search area and type cmd then hit command prompt, a black screen will open with a path to your home folder (usually C:\Users\yourName)
type in python, if installed it will show a version number
if not installed then the get button shows up, press this and download will install over a few minutes (or just download python using chrome)
once fully installed reopen the cmd prompt and type in python again
version info will show....here is where we start if python is already installed
type in python -m http.server and this starts server (keep this cmd window open) 7/ be sure you saved an index file in home folder (in file explorer click c: then Users then yourName to open home folder) 7a. keep cmd open while u go to localhost in step 8...closing cmd requires having to reopen and start over
go to chrome and type in localhost:8000 and your default index page will appear....see below for creating an index file

Python server on a Mac
on a mac open a terminal and start with step 2 above ....except we might need to try 3 different options below depending on which version of python is pre installed. Our home folder should be the folder Python is installed in and the same as the terminal folder where we start server.

try this first

type in python -m http.server or
type python3 -m http.server if above does not work Hit return and Python 3 will instantly start a simple HTTP server from the directory in which the command was executed..this dir should also have an index file or option 3 if others dont work
type in python -m SimpleHTTPServer 8000 for older versions

How to create an index (home) file in our path..save it to the same folder where the web server resides. Copy this code and save it as index.html

     <!DOCTYPE html>
<html lang="en-US">
<header>
</header>
<body>
   <p> hi there, this is our first html page </p>
</body>
</html>

prepare some data

copy this data and paste it into a blank google sheet this is our fictitious company with an orders database

order_no,order_date,product1,product2,product3
12340,01-03-22,400,0,0
12341,01-02-22,0,50,0
12342,1-16-22,0,0,50
12343,1-17-22,100,0,0
12344,1-15-22,0,50,0
12345,2-5-22,100,0,0
12346,2-6-22,0,0,20
12347,2-7-22,100,0,0
12348,3-23-22,0,200,0
12349,3-5-22,0,0,20
123410,3-29-22,100,0,0
123411,3-25-22,100,0,0
123412,4-23-22,500,0,0
123413,4-24-22,0,100,0
123414,5-10-22,0,0,50
123415,5-15-22,500,0,0
123416,5-25-22,0,50,0

VERY IMPORTANT - after pasting data and while it is still highlighted, in google sheets, press data then split text to columns

get the json chrome extension
enable chrome to save as json before creating the sheet.
I found this easy shortcut that adds a JSON icon to the google sheet toolbar...this is a chrome extension
first go to this link https://chromewebstore.google.com/detail/sheets-to-json/enmkalgdnmcaljdfkojckdbhkjmffmoa
then press add to chrome, over to the far right of page
then open a blank google sheet and u will see the JSON icon as the last item in the toolbar near top of page
transform your data into json
paste above data into the sheet, then text to columns, then press the json icon and go to downloads to get ur json file
save this json file in the same folder where python and your index file reside...I saved it as orders.json

execute our program

fetch data from your index file...testing our server and files config

change your index.html file to contain the following code which is different from the code we will use below to access the data from a public server

<!DOCTYPE html>
<html>
<body>
    <div id="myData"></div> <!--data result displays in this html div-->

    <script type="text/javascript">
convert();  //run the convert function..this converts json to html and displays to the front end in the myData div
                 // async is needed to run await which is the newest way to return a promise..await must be used inside a function
    //When you then put await in front of a function you're instructing the program to wait until that operation is complete before moving on.
async function convert() {
                //let response = await fetch('https://rickd.s3.us-east-2.amazonaws.com/orders4.json'); //get data from rickd s3 bucket in aws
                      //replace rickd with your bucket name
   let response = await fetch('orders.json'); //get data locally instead of from s3..orders.json is in same folder where python is installed
                let data = await response.json(); //getting data array in json format...waiting for all the data to come in

                    //then iterate over javascript array for as many times as there are js objects inside the array
            for (var i = 0; i < data.length; i++) {
                data[i].product1 = parseInt(data[i].product1); //this converts from string to int just the val for product1
              } //end for loop
         //we are rendering the array and not data in table format
myData.innerHTML = JSON.stringify(data, null, "\t"); //first stringify json object then render result inside html div tag
console.log(JSON.stringify(data, null, "\t")) //this is more readable, this is the pretty print..press ctrl-shift-J to view
  //console.log(JSON.stringify(data)) //prints all as one line / a string, this is the record format

} //end function
    </script>
</body>

<!-- next up..we need to return this data in table format-->
</html>

call up localhost:8000 and view the data
after playing around with json on a local server we can then create a public S3 bucket in AWS

Configuring an AWS S3 Bucket as our Public Server

The simplest way to store a JSON file remotely is in AWS S3. By not creating a schema in a traditional data server we become serverless. We are out in the AWS cloud with S3 where we can connect to our bucket link from anywhere. As said previously the noSQL S3 approach has some limitations. But it also has big benefits. When working with data, human nature tends to want to only use one table similar to the ole days where we relied heavily on one excel table. This one flat file format can handle a few metrics so we don’t bombard our audience with complexity. An ideal use case for a flat file is a simple dashboard.

First we create an AWS s3 bucket then we upload the JSON file. Here is how we do it:

Sign up for free tier AWS, go to S3 from the AWS console and create a unique bucket name
make it public, by going to bucket permissions tab to turn off block public access - go to block public access section, edit, uncheck and save
stay in the permissions tab then edit bucket policy, erase what is there and replace with following, then save changes (remember to replace your bucket name below with the actual name)

{

"Version": "2012-10-17",

"Statement": [

{

"Sid": "PublicReadGetObject",

"Effect": "Allow",

"Principal": "*",

"Action": "s3:GetObject",

"Resource": "arn:aws:s3:::your bucket name/*"

}

]

}

then, while still in the permissions tab go to cross origin (cors), edit and replace with below and save

[

{

"AllowedHeaders": [

"*"

],

"AllowedMethods": [

"GET",

"HEAD"

],

"AllowedOrigins": [

"*"

],

"ExposeHeaders": []

}

]

Go to the properties tab and press static web hosting, the last item. Then click edit then enable then save. This transforms your bucket link into a public website.
Go back to the objects tab and press Upload, then go to where u saved ur newly created JSON file and complete this operation. Your json file will now be in s3.
Once the JSON file is uploaded successfully it will appear in the objects section of ur s3 bucket. Double click the JSON file and u will be in the properties tab where u see Object overview. Click on the object url provided and raw JSON data will appear. Copy and save this url link somewhere because u will need it to fetch data from Javascript. This url link provides access to the newly created generic website, a publicly accessible S3 bucket that serves as a temporary website. You can save files to this bucket and share them with others at any time.
accessing our json file from AWS requires different coding than used in a local server. First we need to provide the user with a link to our web page. My public link to our fictitious orders data is
https://rickd.s3.us-east-2.amazonaws.com/orders4.json Then our code should look like this which is different from our original code. For this code instead of displaying our array we add table elements so display is in table format

<html> <!-- Function to convert JSON data to HTML table using javascript fetch api
here we are adding table elements to the for loop
-->
<body>
   <div id="container"></div>  <!--the result appears in this div-->
   <script>
      convert(); //run convert function
      async function convert() {
         const response = await fetch('https://rickd.s3.us-east-2.amazonaws.com/orders4.json'); //fetch orders from aws S3 bucket..can just double click
         //const response = await fetch('orders.json'); //fetching from local server..need to run localhost:8000/file name.html for this
         const data = await response.json();

 let table = document.createElement("table"); // Create the table object
                //instead of traditional for loop we are using forEach loop here..a function is required and it is in arrow notation
         data.forEach((item) => {
            let tr = document.createElement("tr"); //the row object is created for each curly braced item in array
            let vals = Object.values(item); //obj.vals method in js returns array of vals, // Get the values of the current object
            vals.forEach((element) => {
            let td = document.createElement("td"); //create col element
td.innerText = element; // Set the value as the text of the table cell
               tr.appendChild(td); // while in each row Append each table cell val to the table row
                          });   //end of inner loop
            table.appendChild(tr); // Append the table row to the table
                      });     //end of original for each loop

         document.getElementById("container").appendChild(table) //append table to container
                               } //end of async function
   </script>
</body>
<!-- next up we add some styling and headers to our table-->
</html>

next, here is the same code where we add headers and styling

<html>
   <!--here we add borders to our table cells using style tags and we also add a header row. Instead of a for loop we are using a forEach loop-->
<head>
   <style>
      table, th, td {
         border: 1px solid black;
         border-collapse: collapse; 
      }
      td, th {
         padding: 10px;
      }
   </style>
</head>
<body>
 <!--  <div id="container"></div> -->
   <script>
           convert();  //run convert function .. a Function to convert JSON data to HTML table
      async function convert() {
                const response = await fetch('https://rickd.s3.us-east-2.amazonaws.com/orders4.json');
  const data = await response.json(); //getting data array in json format

         // Create the table element
         let table = document.createElement("table");

    // Get the keys (column names) of the first object in the JSON data
         let cols = Object.keys(data[0]);

         // Create the header element
         let thead = document.createElement("thead");
         let tr = document.createElement("tr");

         // Loop through the column names and create header cells
         cols.forEach((item) => {
         let th = document.createElement("th");
         th.innerText = item; // Set the column name as the text of the header cell
         tr.appendChild(th); // Append the header cell to the header row
         });
         thead.appendChild(tr); // Append the header row to the header
         table.append(tr) // Append the header to the table

     //below populates the table , above only populates header

         // Loop through the JSON data and create table rows
         data.forEach((item) => {
            let tr = document.createElement("tr");

            // Get the values of the current object in the JSON data
            let vals = Object.values(item);

            // Loop through the values and create table cells
            vals.forEach((elem) => {
               let td = document.createElement("td");
      td.innerText = elem; // Set the value as the text of the table cell
      tr.appendChild(td); // Append the table cell to the table row
            });
            table.appendChild(tr); // Append the table row to the table
         });

         document.body.append(table);
//container.appendChild(table) // or Append the table to the container element
      }
   </script>
</body>
</html>

Addendum added 7-23-25
I realized that introducing the ForEach method above without much explanation is going too fast for a beginner tutorial. We will ease our way into forEach using function with dot arrow notation later. Meanwhile here is another way of calling up json from s3 using the very popular insertrow and insertcell methods of the html table object

<html><body> 
<p id="output"></p> <!--output will display here -->
   <script>
           convert();  //run convert function in next line .. a Function to convert JSON data to HTML table
      async function convert() {
  let response = await fetch('https://rickd.s3.us-east-2.amazonaws.com/orders4.json'); //fetch data in s3 bucket
  let data = await response.json(); //getting data array in json format..assigning result to variable called data

       // EXTRACT VALUEs FOR HTML HEADER and push vals into array called col
  let firstRow = data[0]; //get only the data at index 0 of the data array = the first object = first row of data
  let col = [] //create empty array then use for in loop to grab keys
for (let key in firstRow) { 
        col.push(key) //for each key value found at index 0 populate the col array with only the key names
  }

// CREATE DYNAMIC TABLE.
        let table = document.createElement("table");

        // CREATE HTML TABLE HEADER ROW USING THE EXTRACTED HEADERS ABOVE.
        let tr = table.insertRow(); 

        for (let i = 0; i < col.length; i++) {
            let th = document.createElement("th");      // TABLE HEADER.
            th.textContent = col[i];  //this gets all the col names for the header from col array created above
            tr.appendChild(th);
        }

        // ADD JSON DATA TO THE TABLE AS ROWS.
        for (let i = 0; i < data.length; i++) {
             tr = table.insertRow();    // this creates a tr
        //then using inner loop while in row grab cell vals
            for (let j = 0; j < col.length; j++) {
                let tabCell = tr.insertCell(); //insertcell method creates a td
                tabCell.textContent = data[i][col[j]]; //while in row i get cells j
            }
        }

 // FINALLY ADD THE NEWLY CREATED TABLE WITH JSON DATA TO A CONTAINER.
        var divContainer = document.getElementById("output"); //send result to output tag in html
        divContainer.appendChild(table);

                                  } //end of async function
   </script>
</body></html>

Now we ease our way into using forEach with arrow notation

<html><body> 
<div id="table-container"></div>
   <script>
  function createTableWithDOM(data) { //data is the placeholder for array vals that the function receives when called below
    let container = document.getElementById("table-container");
    let table = document.createElement("table");
    let thead = table.createTHead();
    let tbody = table.createTBody();
    let headerRow = thead.insertRow();
    for (let key in data[0]) { //use for in loop to get keys at index 0 which is the first row                             
      let th = document.createElement("th");
      th.innerText = key; //grab only key vals from first row
      headerRow.appendChild(th);//passing data to headerRow created above
    }
   //now easing our way into the forEach method
       //first a dot is placed after the array indicating that the forEach method will be used
    //data.forEach(function(rowData) { //this is how to do it without using arrow
        //replace function with an arrow
    data.forEach((rowData) => { //in this new arrow function we create a parameter called rowData, a new placeholder
                                //also the forEach method calls a function which is referred to as a callback function
      let row = tbody.insertRow(); //we use the insertRow method instead of appendChild
        for (let columnValue in rowData) { // for in loop passes thru now called rowData..this gets confusing
        let cell = row.insertCell();
        cell.innerText = rowData[columnValue]; //i think we can also use textContent here but should not use innerHTML
      }
    });
    container.appendChild(table);
  }

  let myData = [
    { Name: "Alice", Age: 30, City: "New York" },
    { Name: "Bob", Age: 25, City: "London" },
    { Name: "Charlie", Age: 35, City: "Paris" },
  ];

  createTableWithDOM(myData); //tell program to run function above using myData array

  </script>
</body></html>

Part 2

Arrays, Loops and Functions, some building blocks of coding, the bedrock essentials of programming

Array - the most important data structure
In javascript our data is structured as key-value pairs surrounded by curly braces and delimited by commas, an array contains multiple javascript objects with each object representing a row of data. This is called JSON format or Javascript Object Notation.

Looping / Iteration - the most important loop is the traditional for loop but u should study up on other types of for loops. A for loop iterates over an array of data and performs an operation for each pass. In our use case it just displays the data to the frontend via a fetch request. It is highly recommended to learn the forEach method sooner rather than later because most tutorials will teach this way

a function
A function is a block of code, a set of instructions, that executes in response to some kind of event like pressing a button. In some cases functions are not necessary but in other cases a function is required for some code to work properly. Functions either have a name or are annonymous. They can be called or can be self executing. The dreaded arrow notation is intimidating at first but it is used widely so get used to it.

conditional logic last but not least we also have the if / else statement. if(condition) {then do this} else {do that}

in conclusion it is necessary to study up on all these topics to gain more knowledge as I am providing just a small synopsis here in this lesson. I am providing only a roadmap, a primer of sorts, and u as the programmer must learn the rest.

Happy coding !!

addendum

our plain js, nosql use case is a bar chart where u can find the code here....at https://dev.to/rickdelpo1/stacked-bar-chart-using-a-json-data-source-plain-vanilla-javascript-plain-css-and-no-chart-libraries-2j29

about the author Rick Delpo.. https://javasqlweb.org/about.html

Thank you for taking the time to subscribe to and learn this valuable lesson. Along with the above I offer unlimited support so just email me at rickdelpo@gmail.com with any questions.

Also as part of our package I share several code samples where you can feel free to copy and paste my code which can help you build a portfolio of projects

Interactive stock market S&P 500 line chart using Bokeh, Python, JS, Pyscript and a movable angle finder for Trend Line Analysis

Rick Delpo — Sat, 12 Oct 2024 19:47:12 +0000

Is the stock market overvalued in 2024? Thanks to a Python library called Bokeh we can chart the S&P 500 over time and superimpose an angle tool right over the graph for interactivity. As a bonus we can render all this in HTML using Pyscript. The angle finder is draggable and will show the degrees of our trend line (play video above for a demo). By moving our angle finder over recent data we are apparently trending way over the mean indicating that the market is indeed overvalued.

below, I present 3 code examples
1) a 30 year representation of the S&P 500 as a png screenshot (see caveats below for this version, can’t always find squared off data at Yahoo Finance)
2) a 15 year version of the same with a local json data source (more reliable than a png because we have control over squaring off the data)
3) a Pyscript version of example 2 importing JSON data from an AWS S3 bucket

problem statement

Measuring angles in stock market technical analysis has always been problematic because chart grid lines are usually elongated horizontally or vertically thus the angle of a trend line is distorted by several degrees. If u must use angles please ensure that the length of x axis ticks are equal to y axis ticks. This will square off our data. We can still have x represent time and y represent price but the units of time need to be the same as the units of price, so time is not a date here, it is just a unit. This seems paradoxical but if the graph is not equalized then we get the distortion. We can measure angles of ascent but in and of itself this measurement may not be very useful. Where we can make it useful is to compare this angle to a longer term mean trend line angle to see if we are above or below trend. When the angle of ascent exceeds the angle of the mean we are above trend and the stock price drops thus reverting to the mean.

Evolution of Technical Analysis - Trend line analysis

The use of angles in stock market analysis is long supported by Gann Theory developed by WD Gann in the early 1900s. But it is a tedious measurement because the graph must be equalized for an accurate result as described above. More importantly than the angle measurement itself is whether a stock is trending above the mean or below it. Note that in 2022 the s&p reverted to the mean. Reversion to the mean was theorized by Sir Francis Galton in the late 1800s. Charles Dow, founder of Dow Jones 30 Industrial Index, coined the term technical analysis in the late 1800s originally calling his approach the "Dow Theory". Many others jumped on this bandwagon figuring that charting stocks was more important than their fundamentals, hence line charts became their go to tool thus paving the way to the many charting tools available today.

taking our angle tool for a test run - play video at beginning of this article for a demo

Note we can draw a mean for 4 years of data between 2020 and 2024 (about 45 degrees). In late 21 we were trending way above the mean at 66 degrees but in 22 we reverted to the mean. With other charting tools we are unable to interact with our graph but here Bokeh allows it. First establish a mean trend of 3 lows over the long run then measure more recent data (angle of ascent) to see if trending higher than the mean

some basic definitions for our analysis

1 one unit of time must equal one unit of price to make a 1:1 ratio also known as the aspect ratio. On a chart 1:1 translates into a 45 degree angle, rise over run = 1:1
2 note, mean trend line should touch 3 low points on graph over a longer time frame
3 trendline and mean comparison of the underlying data = trend line analysis
4 don't think of my x axis as dates, it is strictly units of time
5 mean = average value of a set of data points
6 we must use a square in bokeh to equalize the x and y data, if we elongate our x axis then the angle flattens, so our png needs to be cropped to become a 650 x 650 square

Here is the code - Best used on a desktop..also note that Pyscript is very slow and even slower on mobile

First a quick note on why I am using Bokeh vs Matplotlib for this exercise. Bokeh is much easier for a beginner trying to combine Python with Javascript. Recently I have been migrating all my dev efforts over to Plain Javascript. But I needed some Python too for a line chart and I needed it to be interactive. Matplotlib is more complicated to learn and provides mainly static output whereas Bokeh is interactive. Also I wanted Pyscript for my presentation layer as it serves nicely as a simple IDE (Notepad only) for making changes to my code and plays nicely with Bokeh.

Example 1

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>stock chart with angle finder</title>
    <link rel="stylesheet" href="https://pyscript.net/releases/2024.10.1/core.css"> <!--most recent pyscript lib-->
    <script type="module" src="https://pyscript.net/releases/2024.10.1/core.js"></script> <!--most recent pyscript lib-->
    <script type="text/javascript" src="https://cdn.bokeh.org/bokeh/release/bokeh-3.4.1.min.js"></script> <!--recent bokeh lib-->
    <script type="text/javascript" src="https://cdn.bokeh.org/bokeh/release/bokeh-widgets-3.4.1.min.js"></script> <!--need for div to work-->
       <py-config>packages = ["bokeh"]</py-config>
<style>
.div {
  border: 2px solid;
  padding: 20px; 
  width: 740px; /* was 450*/
  height: 650px;
/*
  resize: both;  /* allow for resizing*/
  overflow: auto;
*/
}
</style>
</head>
<body>                 
   <b>wait 8 seconds to display Bokeh Line Chart due to Pyscript being slow, but well worth the wait</b>
    <div id="chart" class="div"></div>

        <script type="py" async>
        from bokeh.plotting import column, figure
        from bokeh.models import PointDrawTool, ColumnDataSource, Div, CustomJS
        from math import atan2, degrees
        from bokeh.layouts import column
        from bokeh.embed import json_item
        from pyscript import window, fetch, ffi 
        Bokeh = window.Bokeh #pass Bokeh into the main threads window object
        #import math
        from bokeh.models import Label
        async def get_data():
            p = figure(x_range=(0,7),y_range=(0,7), width = 740, height = 650) 
            p.image_url(url=['https://rickd.s3.us-east-2.amazonaws.com/s&p9.png'], x=0, y=0, anchor="bottom_left") #add s&p image to p
            #p.image_url(url=['https://rickd.s3.us-east-2.amazonaws.com/s&p15yrs4.png'], x=0, y=0, anchor="bottom_left")
            p.toolbar.autohide = True #only works in v3.0 #need to elim entirely later

                  # add label annotation   
            label = Label(x=.75, y=4, text="tap once then drag line endpoints, degrees show here", text_font_size="16pt")
            p.add_layout(label)

                  # add angle finder
            source = ColumnDataSource( data = {'x': [.25,2], 'y': [4,6]} )
            r_c = p.scatter(x = 'x', y = 'y', size = 15, fill_color = 'red', line_color = 'black', source=source ) #using scatter instead
            r_l = p.line(x = 'x', y = 'y', line_color = 'blue', line_width = 3, legend_label="S&P 500 Index with Angle Finder", source = source )
            tool = PointDrawTool(renderers=[r_c], add = False)
            p.add_tools(tool)
            p.toolbar.active_tap = tool

            p.legend.location = "top"

                  # add callback
            callback = CustomJS(
                  # pass above python vars as args here...we are declaring vars in our custom js
                args = {'src': source, 'p':p, 'label':label},
                code = '''
                    const dx = src.data.x[1] - src.data.x[0];
                    const dy = src.data.y[1] - src.data.y[0];
                    const angle = (180/Math.PI)*(Math.atan2(dy, dx)); //need to do the original way
                    label.text =  ` angle = `+`${angle.toFixed(2)}`+` degrees`;  
                    // this is a comment inside the callback not #....because callback is js and above is python where # denotes comment        
                '''
            )
                 # when angle is moved call the callback function
            source.js_on_change('data', callback)

            await Bokeh.embed.embed_item(ffi.to_js(json_item(p, "chart"))) #ffi converts a Python object into its JavaScript counterpart
        # lastly, run the get_data method above
        await get_data()
    </script>

</body>

<!--
instructions for getting png
go to yahoo finance and find a squared off chart, then take a screenshot and use this png file as an image in bokeh. With the same plot create our movable trend line over a plot with equalized x and y.
   resize 650 x 650 to fit on chart
  at yahoo must find squared chart if doing png but if doing actual data we can square off ourselves..this is very important
    the challenge is getting screen shot of squared chart
  at yahoo finance advanced chart 1 year and interval of 1 day is as square as i can get it..press blue arrow on right for full screen
  a 2 years, 1 week gets us close enough with both blue arrows making graph smaller

go to advanced chart then press indicaters and get rid of volume under..this way we only see the chart without the daily volume
press arrow on right side so chart fills screen
settings get rid of cross hair
chart at max with one month intervals should be squared
press window key and print screen (PrtSc) at the same time
go to pictures then screen shots then rename file and save where needed
then double click file press edit icon and make bigger (zoom to 100% then adjust top and side resize handles, crop to 650x650 ..and do final save)
-->
</html>

Example 2

from bokeh.plotting import column, figure, show
from bokeh.models import PointDrawTool, ColumnDataSource, CustomJS, HoverTool, CustomJSHover
import json
from bokeh.models import Label
import requests #this will not work in pyscript ******* it does work when using IDLE

#there are 2 options below for our data source...plain array and json array in s3

  #next 3 lines are for implementing s3 remote data source
  # for this to work from s3 save below json array to a .json file in AWS s3 and uncomment import requests above, uncomment next 2 lines, then comment out just the data array below
#response = requests.get('https://rickd.s3.us-east-2.amazonaws.com/data.json')  #this is my live working link 
#data = response.json()

#about the data, note time (x) is denoted in equal intervals matching y range so both axis are squared and so we can display the angle trend line
     #if alternatively we were to show time intervals as 1-10 for example then angle trend line would not be visable
     #values are annual closing prices on may 1 of each year from yahoo historical data pushed to google sheet, transformed to json and saved in s3 as data.json

data =[
    {
        "price": "919",
        "time": "919"
    },
    {
        "price": "1089",
        "time": "1217"
    },
    {
        "price": "1345",
        "time": "1515"
    },
    {
        "price": "1310",
        "time": "1813"
    },
    {
        "price": "1631",
        "time": "2111"
    },
    {
        "price": "1924",
        "time": "2409"
    },
    {
        "price": "2107",
        "time": "2707"
    },
    {
        "price": "2097",
        "time": "3005"
    },
    {
        "price": "2412",
        "time": "3303"
    },
    {
        "price": "2705",
        "time": "3601"
    },
    {
        "price": "2752",
        "time": "3899"
    },
    {
        "price": "3044",
        "time": "4197"
    },
    {
        "price": "4204",
        "time": "4495"
    },
    {
        "price": "4132",
        "time": "4793"
    },
    {
        "price": "4180",
        "time": "5091"
    },
    {
        "price": "5278",
        "time": "5389"
    },
    {
        "price": "5696",
        "time": "5696"
    }
]

 # if json in txt format like above, need to convert price and time to number format, either int or float, vals cannot have commas, reformatted while in google sheet
for format in data:
    for key in format:
        format["price"] = float(format["price"]) # will not convert numbers with commas in them
        format["time"] = float(format["time"])

x=[i['time'] for i in data] #grab s3 data into array using for in loop
y=[i['price'] for i in data] #grab s3 data into array using loop
# comment out above 6 lines and all of json array when using plain arrays below

    # comment out below 2 lines when json is in use from above data
    # these 2 lines x and y are the plain arrays that can be accessed when not using json 
#x = [919, 1217, 1515, 1813, 2111, 2409, 2707, 3005,3303,3601,3899,4197,4495,4793,5091,5389,5695] #time intervals increments of 281 to achieve squaring
           #these x increments are merely units of time, can also be 1 thru 17 but then angle trend line will not show up
#y = [919,1089,1345,1310,1630,1923,2107,2096,2411,2705,2752,3044,4204,4132,4179,5277,5695] # actual prices
p = figure(width=700, height=600, title='Bokeh line graph with S&P data and Angle Finder (hover over line for vals)')
p.xaxis.axis_label = 'time by year'
p.yaxis.axis_label = 'S&P price over 16 years'

    #add custom labels to x ticks because x ticks are only units of time so we override these with actual dates
p.xaxis.ticker = x
    #then overide existing x labels with custom labels 
p.xaxis.major_label_overrides = {919:'2009', 1217:'2010', 1515:'2011', 1813:'2012', 2111:'2013',2409:'2014',2707:'2015',3005:'2016',3303:'2017',3601:'2018',3899:'2019',4197:'2020',4495:'2021',4793:'2022',5091:'2023',5389:'2024',5696:'2025'}
new_labels = p.xaxis.major_label_overrides #assign above to new var to use in callback

p.line(x, y, line_width=2)
p2=p.line(x, y, line_width=2) #assign p.line to var p2 to use below in renderers

    #add some interactivity...a draggable trend line with angle finder
source = ColumnDataSource( data = {'x': [2000,4000], 'y': [2000,4000]} ) #needs to be within same coordinates as our stock data in order to be visable 
r_c = p.scatter(x = 'x', y = 'y', size = 15, fill_color = 'red', line_color = 'black', source=source )
r_l = p.line(x = 'x', y = 'y', line_color = 'blue', line_width = 3, source = source )
tool = PointDrawTool(renderers=[r_c], add = False)
p.add_tools(tool)
p.toolbar.active_tap = tool

  #add custom callback for hover , adding some more interactivity
x_custom = CustomJSHover(
    args=dict(special2=new_labels),  #pass in new_labels var to callback
    code="""
    return special2.get(value)
    """)

    #add hover tool plus tooltips to plot
p.add_tools(HoverTool(
    tooltips=[
        ( 'price = ','@y{0.}' ),
        ( 'year = ','@x{custom}' )
    ],
    formatters={'@x': x_custom}, # call customJSHover callback
    renderers=[p2]  #make only p2 show tooltips
))

    # add a label to carry an annotation..the result in degrees
label = Label(x=2000, y=5000, text="tap and drag red endpoints and angle will show here", text_font_size="12pt")
p.add_layout(label)

    #add callback for trend line
callback = CustomJS(
       #pass above python vars as args here...we are declaring vars in our custom js
    args = {'src': source, 'label':label },
    code = '''
        const dx = src.data.x[1] - src.data.x[0];
        const dy = src.data.y[1] - src.data.y[0];
        const angle = (180/Math.PI)*(Math.atan2(dy, dx)); //need to do the original way
        label.text =  ` angle = `+`${angle.toFixed(2)}`+` degrees`; //show this result in label object
        '''
)
source.js_on_change('data', callback) # when trend line endpoint is moved run callback, more interactivity included

# I am deliberatly hiding grid lines...data is squared but gridlines are not
         #note minor grids or factors do not resolve the visual
p.xgrid.grid_line_color = None
p.ygrid.grid_line_color = None

show(p)

Example 3

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>get json from AWS S3</title>
    <link rel="stylesheet" href="https://pyscript.net/releases/2024.10.1/core.css"> <!--most recent pyscript lib-->
    <script type="module" src="https://pyscript.net/releases/2024.10.1/core.js"></script> <!--most recent pyscript lib-->
    <script type="text/javascript" src="https://cdn.bokeh.org/bokeh/release/bokeh-3.4.1.min.js"></script> <!--recent bokeh lib-->
    <py-config>packages = ["bokeh"]</py-config>
</head>
<body>
    <h3>wait 8 seconds to display Bokeh Line Chart because Pyscript is very slow, but worth waiting for</h3>
    <div id="chart"></div> <!--inside the script tags below is my Python code-->

    <script type="py" async>
        from bokeh.plotting import figure
        from bokeh.models import PointDrawTool, ColumnDataSource, CustomJS, HoverTool, CustomJSHover
        import json
        from bokeh.models import Label
            #below 3 lines only needed when using Pyscript
        from bokeh.embed import json_item
        from pyscript import window, fetch, ffi 
        Bokeh = window.Bokeh #pass Bokeh into the main threads window object

        async def get_data():
               #getting json data from aws s3 ..in Pyscript we are using fetch but in plain python we import requests
            response = await fetch('https://rickd.s3.us-east-2.amazonaws.com/data.json')
            data = await response.json()
            for format in data: #need to convert txt in json to number format, either int or float
                for key in format:
                    format["price"] = float(format["price"]) # will not convert numbers with commas in them
                    format["time"] = float(format["time"])        
            x=[i['time'] for i in data] #grab s3 data into array using for in loop
            y=[i['price'] for i in data] #grab s3 data into array using loop

            p = figure(width=700, height=600, title='Bokeh line graph with S&P data and Angle Finder (hover over line for vals)')
            p.xaxis.axis_label = 'time by year'
            p.yaxis.axis_label = 'S&P price over 16 years'
            p.xgrid.grid_line_color = None #hides grid lines using IDLE but not in pyscript
            p.ygrid.grid_line_color = None    #trying to deliberately hide grid lines

                #add custom labels to ticks
            p.xaxis.ticker = x
                #then overide existing x labels with custom labels 
            p.xaxis.major_label_overrides = {919:'2009', 1217:'2010', 1515:'2011', 1813:'2012', 2111:'2013',2409:'2014',2707:'2015',3005:'2016',3303:'2017',3601:'2018',3899:'2019',4197:'2020',4495:'2021',4793:'2022',5091:'2023',5389:'2024',5696:'2024.5'}
            new_labels = p.xaxis.major_label_overrides #assign above to new var to use in callback

            p.line(x, y, line_width=2)
            p2=p.line(x, y, line_width=2) #assign p.line to var p2 to use below in renderers

                #add draggable trendline with angle finder
            source = ColumnDataSource( data = {'x': [2000,4000], 'y': [2000,4000]} ) #needs to be within same coordinates as our stock data in order to be visable 
            r_c = p.scatter(x = 'x', y = 'y', size = 15, fill_color = 'red', line_color = 'black', source=source )
            r_l = p.line(x = 'x', y = 'y', line_color = 'blue', line_width = 3, source = source )
            tool = PointDrawTool(renderers=[r_c], add = False)
            p.add_tools(tool)
            p.toolbar.active_tap = tool

                #add custom callback for hover 
            x_custom = CustomJSHover(
                args=dict(special2=new_labels),  #pass in new_labels var to callback
                code="""
                return special2.get(value)
                """)

                #add hover tool plus tooltips to plot
            p.add_tools(HoverTool(
                tooltips=[
                    ( 'price = ','@y{0.}' ),
                    ( 'year = ','@x{custom}' )
                ],
                formatters={'@x': x_custom}, # call customJSHover callback
                renderers=[p2]  #make only p2 show tooltips
            ))

                # add a label to carry an annotation..the result in degrees
            label = Label(x=2000, y=5000, text="tap and drag red endpoints and angle will show here", text_font_size="12pt")
            p.add_layout(label)

                #add callback for trendline
            callback = CustomJS(
                    #pass above python vars as args here...we are declaring vars in our custom js
                args = {'src': source, 'label':label },
                code = '''
                const dx = src.data.x[1] - src.data.x[0];
                const dy = src.data.y[1] - src.data.y[0];
                const angle = (180/Math.PI)*(Math.atan2(dy, dx)); //need to do the original way
                label.text =  ` angle = `+`${angle.toFixed(2)}`+` degrees`; //show this result in label object
                ''')
            source.js_on_change('data', callback) # when trendline endpoint is moved run callback
            await Bokeh.embed.embed_item(ffi.to_js(json_item(p, "chart"))) #ffi converts a Python object into its JavaScript counterpart
        await get_data()
    </script>
</body>
</html>

caveats

I have provided 3 code examples, all squared off but with slightly differing results because more or less data is fit into the same square. Our png model shows 30 years whereas our non png shows only 15 years of data. The trend line angles derived from our 15 year model result in a flattening of the line and less degrees. When we scrunch 30 years worth of data into the same square the slope steepens. This is why we cannot fully depend on angles in our analysis. But we can depend on trending over or below the mean.

Caveat #2
Our second example offers plain array data, json data locally and json data from s3
note x and y are squared off but does not show that way in grid lines because units of price are rounded off to the nearest 1000...this will require a question to the Bokeh help desk to rectify. For now I have hidden the grid lines.

Note in our example 3 that I can not hide the grid lines when using Pyscript..another help desk issue

Caveat #3
It is difficult to get squared data from yahoo finance for a screenshot png, plus the image needs cropping to 650px x 650px. So get as squared off as we can. We have more control over squaring if we do not use a png as in example 2 and 3

Happy Coding!

Embedding Bokeh into HTML with Pyscript, a CSS resize handle & Custom JS callback passing results back to a div on our html page

Rick Delpo — Fri, 28 Jun 2024 12:21:47 +0000

Here we explore a deeper dive into Pyscript where all my code below is circa 2024 (the newest version of Pyscript).

Pyscript allows us to embed a Bokeh Chart into HTML. Then we make our html interactive by adding some CSS with a resize handle. We also make our bokeh interactive using the stretch_both method which allows us to entirely fill the html div we are embedding into. When we drag the resize handle either up, down or sideways, our bokeh chart responds accordingly and also renders the plot width and height coordinates using a JS callback with custom JS.

In the old days, prior to 2022 when Pyscript was invented, in order to embed bokeh into html we needed bokeh on a server along with lots of python and javascript to make this solution work. Now Pyscript shortens the process with the embed.embed_item method and ffi (foreign function interface, circa pyscript 2024) which converts a Python object into its JavaScript counterpart, thus enabling web interactivity.

Embedding vs showing
In Pyscript we are embedding and in Python we are showing. Both involve an html file but it is the Pyscript html file we want to write our code in and share with others.

More specifically when working with Bokeh directly in Python there is the show() method and in Pyscript we have the embed.embed_item() method and these are separate ways to render Bokeh.

Our show() method creates an output html file and opens bokeh in a browser. This bare html template is not what we want because it is not usable code, it is mainly a combo of json and javascript just to render a page. The link to this html page is not the link we want to provide others either. Show() actually creates an html file after running from Python but we want to be in the html before running so we can manage our code. When writing Pyscript we start with html and script tags in a notepad and then save as html on our local drive before we click the link to run it. We will use this link to share with others.

Interactivity Everywhere
In this example the idea of interactivity encompasses more than just bokeh interactivity, it also combines it with html/css interactivity and javascript interactivity (ffi), plus optional custom js if needed. HTML, CSS, Bokeh and Javascript all play nicely together which is why I chose Bokeh in the first place. Bokeh offers a stretch_both method which allows the bokeh chart to completely fill the html object which is a div in this case. Then we apply pure css to resize the div and as div is wider so is bokeh. Our custom js works here too, so lots of interaction happening in our example below.

Pyscript link, not the show() link
In this use case we can save the Pyscript link to a remote html web server so as to provide a public link to others. When the user clicks the link he can also press ctrl u to view its source.

Here is my public link
https://rickd.s3.us-east-2.amazonaws.com/Pyscript+Template+with+div+resize+handle_2.html

With this pyscript link we can copy and edit the code in a notepad and then resave to your local link or upload to a public domain.

Custom JS in a Callback
Custom Javascript is optional and adds more interactivity to our app. This custom JS component allows us to pass Python args into the javascript and then out to an html div. We can trigger an event to call the callback which executes the custom js. Python allows us to build bokeh in Pyscript and Javascript allows us to build interactivity in our callback where we respond to the python event. Here in our example, the event is where we drag the resizing handle which triggers some custom js to display a result in an html div. Pyscript provides a healthy environment where Python, Javascript and HTML can play and interact together in harmony.

My code - Pyscript, Python, CSS all on one sharable HTML page

The purpose of this example is to show how an elongated grid distorts trend lines and angles. The chart grid lines need to be equalized or squared in order to give us an aspect ratio of 1:1 between height and width of the grid lines for the trends on a line chart to be accurate. For example, a stock chart has normally elongated grid lines thus flattening the angle of any slopes. This is why angles should never be applied to stock prices unless the height and width of grid lines are equal.

Note: in a previous version of bokeh the resizing method was abandoned so we need to do resizing on the html side now which is why I provide this example.

So here it is

<!DOCTYPE html>
<!-- here we embed bokeh into html and use pure css resize syntax along with stretch_both-->
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Using Pyscript, embed Bokeh into html, with resize handle and interactive results</title>
    <link rel="stylesheet" href="https://pyscript.net/releases/2024.6.1/core.css"> <!--most recent pyscript lib-->
    <script type="module" src="https://pyscript.net/releases/2024.6.1/core.js"></script> <!--most recent pyscript lib-->
    <script type="text/javascript" src="https://cdn.bokeh.org/bokeh/release/bokeh-3.4.1.min.js"></script> <!--recent bokeh lib-->
    <script type="text/javascript" src="https://cdn.bokeh.org/bokeh/release/bokeh-widgets-3.4.1.min.js"></script> <!--need for div to work-->
       <py-config>packages = ["bokeh"]</py-config>
<style>
.div {
  border: 2px solid;  padding: 20px; 
  width: 450px; /* here is where we size our html div then stretch_both makes plot below fill the div entirely*/
  height: 450px;  resize: both;  overflow: auto;
}
</style></head>
<body>
    <h3>wait 8 seconds to display Bokeh Line Chart, then drag resize handle in lower right corner and changes will display</h3>
    <p>as we widen the plot, the angle of the slope becomes flatter, squaring the plot provides accurate trend line, elongating the plot distorts the trendline</p>
        <script type="py" async>
        from bokeh.plotting import column, figure
        from bokeh.models import ColumnDataSource, Div, CustomJS
        from math import atan2, degrees
        from bokeh.embed import json_item
        from pyscript import window, fetch, ffi 
        Bokeh = window.Bokeh   #pass Bokeh into the main threads window object
        async def get_data():
            p = figure(x_range=(0,7),y_range=(0,7), sizing_mode="stretch_both")
            p.xgrid.grid_line_color = 'black'
            p.ygrid.grid_line_color = 'black'
            p.toolbar.autohide = True
            div = Div(text = ' ')  #initialize/define blank div and div2 here
            div2= Div(text = ' ')   
                #add angle line
            source = ColumnDataSource( data = {'x': [2,4], 'y': [3,5]} )
            r_c = p.scatter(x = 'x', y = 'y', size = 10, fill_color = 'red', line_color = 'black', source=source )
            r_l = p.line(x = 'x', y = 'y', line_color = 'blue', line_width = 3, source = source )     
                #add callback
            callback = CustomJS(
                     #pass above python vars as args here...we are declaring vars in our custom js
                args = {'p':p, 'div2': div2, 'div': div},
                code = '''
                    const dx = `${p.outer_width.toFixed(2)}`
                    const dy = `${p.outer_height.toFixed(2)}`
                    const angle = (180/Math.PI)*(Math.atan2(dy, dx));
                    div.text = `width = `+`${p.outer_width.toFixed(2)} `+ ` ......height = `+` ${p.outer_height.toFixed(2)}`;
                    div2.text = ` angle = `+`${angle.toFixed(2)}`+` degrees`;
                    const result2 = div.text;  //this is width / height...passing txt into result2
                    const result3 = div2.text;             //passing div2 text into result3
                          //prepare to send div to output tags outside pyscript in main body
                    let element = ' ';  //declare empty element
                    element = document.createElement('h3'); //h3 is the font size for output div below
                    element.innerText = result2;
                    document.getElementById("output").innerHTML = ""; //clears what is in output div so we dont keep appending
                    document.getElementById("output").append(element); //send element to output in html area outside pyscript tag
                    let element2 = ' ';  //declare empty element
                    element2 = document.createElement('h3');
                    element2.innerText = result3;
                    document.getElementById("output2").innerHTML = "";
                    document.getElementById("output2").append(element2);
                '''
            )
            p.js_on_change("outer_width", callback) #when outer width changes (the event) then execute custom js inside the callback above
            p.js_on_change("outer_height", callback)
            await Bokeh.embed.embed_item(ffi.to_js(json_item(p, "chart"))) #ffi converts a Python object into its JavaScript counterpart
        await get_data()
    </script>
    <div id="chart" class="div"></div> <!--bokeh chart renders in this div-->
    <div id="output"></div> <!--width and length render here-->
    <div id="output2"></div> <!-- degree of angle renders here-->
</body>
</html>

Happy Coding !!

Python in the Browser, Fetching JSON from an AWS S3 bucket into a Bokeh Line Chart, a Serverless solution

Rick Delpo — Sat, 15 Jun 2024 12:42:32 +0000

I needed some quick Python to replace broken code over in AWS Lambda. While looking at IDEs I stumbled upon a way to write and run Python directly in the Browser just with a Notepad. So I opened Notepad and wrote a boiler plate html file with a Python Library called pyscript...core.js. There was no IDE or server involved, just my frontend file. All I had to do was double click the HTML file and my Python, Javascript, JSON and HTML played very nicely together.

Also I had recently ditched React in favor of Plain JS and started fetching JSON from S3 versus the traditional SQL approach. So Fetch in Python? Was this possible? Fetch is JS and Python is not, so I thought. But thanks to WebAssembly now we can combine both Python and Javascript on an HTML page.

Since I never really used much Python and I was mostly Plain JS without Node I was accustomed to using CDN Libraries in my HTML. I was drawing some Python line charts one day and got to thinking about hosting them in JS too. Bokeh was the answer. Bokeh is both a Python library and a JS Library so it fit my new configuration. Now my solution was totally serverless.

So lets look at some Pyscript now. It has only been around since 2022. In 31 lines of code I can do all the above. The only drawback is that my chart takes 8 seconds to load and from Python IDLE it takes 3 seconds. But the data is worth waiting for because of the value it adds to our Marketing effort. Recently Pyscript had a complete rewrite. This code used to be 60 lines just 2 months ago and prior to that it was 100+ lines using popular competitor Matplotlib.

So now below is the no brainer code written in pyscript newest release, June 2024
Note: this is a first for this newest way of writing pyscript as of 6-15-24
Beware that versioning is critical with Pyscript and Bokeh and is not backward compatible.

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>get json from AWS S3</title>
    <link rel="stylesheet" href="https://pyscript.net/releases/2024.6.1/core.css"> <!--most recent pyscript lib-->
    <script type="module" src="https://pyscript.net/releases/2024.6.1/core.js"></script> <!--most recent pyscript lib-->
    <script type="text/javascript" src="https://cdn.bokeh.org/bokeh/release/bokeh-3.4.1.min.js"></script> <!--recent bokeh lib-->
    <py-config>packages = ["bokeh"]</py-config>
</head>
<body>
    <h1>wait 8 seconds to display Bokeh Line Chart</p>
    <div id="chart"></div> <!--inside the script tags below is my Python code-->
    <script type="py" async>
        from bokeh.plotting import figure
        from bokeh.embed import json_item
        from pyscript import window, fetch, ffi 
        Bokeh = window.Bokeh #pass Bokeh into the main threads window object
        async def get_data():
            response = await fetch('https://rickd.s3.us-east-2.amazonaws.com/04.json')
            data = await response.json()
            xAxis=[i['time'] for i in data] #grab s3 data using loop
            yAxis=[i['Quantity'] for i in data] #grab s3 data using loop
            p = figure(width = 700, height = 600) #beware that much code u see still uses plot_width /height which does not work anymore
            p.line(xAxis, yAxis, legend_label="data from AWS S3", line_width=2)
            await Bokeh.embed.embed_item(ffi.to_js(json_item(p, "chart"))) #ffi converts a Python object into its JavaScript counterpart
        await get_data()
    </script>
</body>
</html>

Warning, if u previously used urllib or requests packages in Python, note, these packages do not work in Pyscript but Fetch does work. You will need to do some recoding if transitioning from urllib or requests.

Happy coding folks!

13 reasons why Coding Tutorials can be Misleading resulting in Tutorial Hell

Rick Delpo — Wed, 08 Nov 2023 14:06:44 +0000

Ever get the feeling that searching for the appropriate tutorial can be quite challenging? Recently I found out the hard way. It took me 8 searches to find what I was looking for and I am an experienced programmer. What about the beginner? How do they fare when doing Google research?

First thing, to my dismay, was that the code did not work and this is what starts Research Hell or the continuous searching process. Luckily I can improvise some of the time but the beginner programmer won't know how to fill in the blanks for non working code.

I am writing this article now because I recently revisited a tutorial I wrote only 2 years ago and it is now outdated code. It is an AWS Lambda Tutorial based on Node version 12 or 14, I think, and we are now up to Node 18 which does some things differently.

While the intention of the writer may be good, their material becomes outdated or the writer is inexperienced and leaves out important steps. Many people get discouraged and give up because it becomes too difficult to self teach, as they go thru this research hell.

The most important thing is to first check the date on the tutorial. The problem with most research is that it becomes ancient history quickly and writers never take anything down or edit their writings so a ton of mis-information is awaiting the beginner out on Google.

Here are 13 things to watch for when Googling (my platform is windows and BTW it makes a difference when researching how-to topics)

1 things change a lot so stuff becomes deprecated and outdated sometimes within 1 year
check the date of the tutorial and get coroborating material

2 there can be multiple ways of doing something, it can get confusing when comparing tutorials
Example, many ways to do a basic for loop

3 the writer omits one thing because the tutorial is technically correct but incomplete
Frequently AWS tutorials leave out the IAM permissions part, then the code does not work right away without tweeks

4 the tutorial is sometimes downright wrong.
recently I relied on an AWS Cloudfront tutorial that gave me bad advice and I struggled for days to get it right.

5 the article paints with a broad brush thus is not detailed enough
Articles with concepts only and no code example, skip these

6 people sometimes write about dinosaur topics not knowing anything different
Writing as if it is current technology because they are not up to date with their skills.
Example, recently I read that someone was promoting jquery in 2023, not that I am against jquery. It still works and many people still use it but it is now old technology because plain javascript is so up to date now with their native code. In the past we needed jquery to fill in the gaps when plain javascript was not mature enough

7 the writer is obviously inexperienced but the beginner may not know this

8 the tutorial is old school coding..here is where the coroborating material comes in
here is where the code may not work anymore or the platform changed and does not accomodate the old way anymore

9 the tutorial is only for linux and not windows, the inexperienced dev might not know this

10 the reader might not understand the difference between sql and nosql
if it is in json this means it is nosql

11 know the difference between Javascript, Java and Python
example, the boto3 library is only for Python
the Fetch API is for javascript, it replaced jquery ajax and axios back in 2016

12 there is front end coding and back end coding
example php is backend, mysql is backend, Node.js can be both, plain javascript is front end

13 these days we can do without libraries in many cases with plain vanilla javascript
native javascript has come a very long way in recent years. Check out some examples at my website below where I dumped Java in favor of Javascript.

examples of my pet peeves

1 the traditional for loop has morphed into maybe five different ways to do one in 2023. Old ways go out of style. We have foreach, forin, forof, find method, map method. I still like a traditional for loop because it is the easiest to comprehend as a beginner

2 the arrow notation of a javascript function, i hate it because it does not seem intuitive to me and it really drives me crazy

some stuff is time tested so you will be safe learning below

1 plain vanilla javascript using the fetch api
2 plain css for only the basics
3 good ole csv data never goes out of style
4 json is a web standard for nosql data
5 coding from a windows perspective, not Linux if u are a beginner

conclusion
plain vanilla js and json nosql is a good bet for beginner programmers. Then graduate to harder stuff from there.
PS my opinion is that Node.js is too much overhead for the beginner, learn plain vanilla first but still graduate to Node later on because u will need it if u want to use AWS Lambda.

check my website for some tutorials at https://javasqlweb.org

Happy Coding !!

updated to include the term Tutorial Hell. Most of the above challenges create frustration for the beginner programmer. It is out of habit that we Google a topic in order to know something more. Tutorial Hell will never go away entirely, we just need to learn to do our research more carefully.

Forgot where I saved my AWS S3 files so I wrote this File Watcher Program

Rick Delpo — Fri, 15 Sep 2023 12:48:14 +0000

It is a File Watcher that watches my local S3 repository for changes. When any file is changed my AWS S3 Bucket file is updated magically with the below code. A simple NodeJS script, just save as myfile.js, open CMD prompt and enter node myfile.js to run. Read my story below.

Note: I have not yet found a tutorial that includes it all correctly. Hint, its all about pre setting content-type.

Most may use Github as their website repository but for aws S3 any old local folder will do. An inconvenience of hosting on s3 is u can't edit your html files right in the bucket. So after a few months of no edits, suddenly, where did I save my files and most importantly is this folder my most current version? more below...

const AWS = require('aws-sdk'); //add these 2 dependencies
const fs = require('fs');

var file_name;
  //next line polls the origin folder while program is open
 const WATCH_TARGET = 'C:/Users/rickd/Downloads/backup 6-4-23/howtolearnjava.com/other/';
    fs.watch(WATCH_TARGET, (eventType, filename) => {
console.log('File "' + filename + '" was changed: ' + eventType);
  file_name = filename;

if (eventType == 'change') {
    // a file change occurred
    //get aws user credentials
const s3 = new AWS.S3({
  accessKeyId: 'xxxxxxxxxxxxxxxxxxxxxxxxx',
  secretAccessKey: 'yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy'
});

const bucketName = 'howtolearnjava.com';//need to define this here
const fileName2 = file_name;  //must be defined here
   // this is the origin file that we are changing
const origin_file = 'C:/Users/rickd/Downloads/backup 6-4-23/howtolearnjava.com/other/'+fileName2;
//needs full path otherwise no such file or dir shows as error                                                                                                                                                                                                         //no fancy concat anymore...dont put syntax in the path
//******very important..this program will not overwrite s3 file
 //if content type is absent (in cf and perhaps always)
    //here is where we read our file
const fileData = fs.readFileSync(origin_file);

s3.upload({
  Bucket: bucketName,
  Key: file_name, //this is only the name of the file
 //ContentType: 'valid content type', //this is very major,
 //most tutorials leave this part out
//when hosting on cloudfront default content type is
 //octetstream which causes file to download
//to overcome we MUST specify content type on creation of file
  ContentType: 'text/html',
 //set content type here in top level params not in above const
  Body: fileData 
}, (err, data) => {
  if (err) {
    console.error(err);
  } else {
    console.log(`File uploaded successfully. ${data.Location}`);
  }
});

  } //end if changed condition

});

Caveats
1 cannot do this in aws lambda because lambda will not access the local windows file system
2 beware that content type MUST be stated upon creation of an html file or the default octetstream format will cause a file download and will not overwrite existing content (a cloudfront issue)
3 must use user credentials for s3 access, instructions are here.
4 when program is started a message appears saying in 2023 we must use aws-sdk (v3) vs v2. I am using v2 at the moment so this code may not function in a few months
5 best not to have s3 open when running this program as it causes it to crash. No damage so far but concerned. Check your files by httping to them instead.

Continuing Saga

Getting scared I figured I could go to my bucket and download the files. All well and good if u want to waste time downloading one by one since when trying to checkmark each file my download option goes away. We'll anyway, I did find the files but never want this to happen again.

Fs.watch to the rescue. A folder watcher that auto saves to s3 each time a file change is detected. Oh BTW u also need to copy updated file and paste to s3 bucket each time u make a change, forgot to tell u about this. I can live with all this but once I made my program s3 became a sinch to manage.

The program is a simple node file with only a few lines of code. When executed a listener opens waiting for updates. Keep the program running while making changes to html files. When the file change is detected the program passes the file name to an aws upload function which then uploads to s3 and overwrites the existing file. All we need to remember is the name of the node.js file. In cmd enter node my_program.js, go make some html updates and the rest is magic.

More about the Author, Rick Delpo at https://javasqlweb.org/about.html

Enabling AWS S3 to behave more like a Web Server

Rick Delpo — Sun, 03 Sep 2023 18:53:53 +0000

Cloudfront Functions (as of Nov 2021) can be used to mimic Apache htaccess behavior with Cloudfront in front of S3 thus providing more server like behavior for our AWS configuration. I have 4 things in mind. 1. We want to deal with Pretty Urls or files without extensions using redirects 2. We want to enable default index.html behavior for our sub folders 3. We want to fiddle around with our response security headers and 4. We want to do some url rewriting. In this article we will cover 1 thru 3.

Remember back in the day before https (circa 2018) when hosting in S3 was as simple as enabling web hosting right in the bucket? Well AWS s3 is not really meant to be a webserver so now adding cloudfront in front of s3 attempts to get us closer but more is still needed. And with the more I get really close to mimicking apache httpd like behavior. Beware by adding Cloudfront we do experience some challenges as some traditional S3 behavior ceases to work. Cloudfront overrides traditional programming thus getting in the way.

Part 1 - Pretty Urls and the need for Redirects

One vexing issue is how to deal with html pages without extensions. My previous Apache setup had some files with html extensions and some without. A couple lines in htaccess solved the extensionless issue. But S3 forces us to keep the html extension for all the files. My first inclination was to upload html files and then rename them without the extension. Then I tried to redirect my extensionless file in the metadata section of the S3 record. I even tried to use AWS CLI to upload my files. After struggling for a while it dawned on me that I do not even need the no extension file. It turns out that after all this I found out that Cloudfront does not honor any of the traditional S3 manipulation rules. Hence the Cloudfront function to remedy this. Wait what? does not honor rules, really?? no one tells us about this one up front. This means u can study aws up and down and nothing works because with cloudfront in the way there is an exception. This is a real gotcha but can be overcome with my new strategy. The gotcha part is that so much out there is misleading that we waste our time in Google research hell. So now that I know about not honoring, here is how we deal with it.

After much over thinking the solution is simply to include a 301 redirect instruction, in a cloudfront function, to redirect to a newly created html file. Using this approach also protects our SEO link equity, also known as link juice. We are simply asking if the URI request is the old file without an extension. No need to create this file, just ask in our request header. If the condition is true then we redirect the user to the html version of the file. We should thus create the html version of the file and also don’t forget to change our sitemap and canonicals to agree. Any previous links to the no extension syntax will continue to work. This is key.

PS. clarification, be sure that all pages agree with sitemap because some have the extension and some don't. Also don't forget to change ur canonicals to include the .html after ur migration.

So here we go, this is what it looks like
VERY IMPORTANT..since the writing of this article part 1 of this code is no longer working. Go to the end of this article for the new code **

function handler(event) {
    var request = event.request;
    var uri = request.uri;
    var requestURI = event.request.uri

    var newurl5 = `https://howtolearnjava.com/JDBC.html`;
    var newurl6 = `https://howtolearnjava.com/Java-Servlet.html`;


   //the uri is everything after the .com slash in our path including the slash and any subfolders


    if (requestURI === "/JDBC") {
            var response = {
                statusCode: 301,
                statusDescription: 'Found',
                headers:
                    { "location": { "value": newurl5 } }
                }
            return response;
        }

    if (requestURI === "/Java-Servlet") {
            var response = {
                statusCode: 301,
                statusDescription: 'Found',
                headers:
                    { "location": { "value": newurl6 } }
                }
            return response;
        }        



//Part 2 - below code will enable sub folders to host index.html as default
    // Check whether the URI is missing a file name. If so then it is a subfolder and we then concat and index.html file so when the user types in the subfolder, by default he gets the index.


    if (uri.endsWith('/')) {
        request.uri += 'index.html';
    } 
        // Check whether the URI is missing a file extension.
    else if (!uri.includes('.')) {
        request.uri += '/index.html';
    }
    return request;
}

Open AWS Cloudfront, click functions in left menu, name and build the function, test it, publish it then associate it to a cloudfront distribution.

Part 2 sub folders

Since S3 file storage is just simple storage S3 has no knowledge of a subfolder or its index file. Luckily when we first encounter web enabling an S3 bucket, this part gives our index file default like behavior. But generally nothing is ever discussed about subfolders. In Apache we make use of an htaccess file in each subfolder to deal with default indexes. But S3 and Cloudfront does not have an htaccess file so we hereby use a Cloudfront Function to mimic htaccess. In our above example the Cloudfront Function is like a listener listening for http requests (similar to apache httpd). Notice that when the user request comes in at the cdn edge our program asks if it is a subfolder and if it is then concat an index file to the uri. Voila we have the behavior we want.

Part 3 Response Security Headers

In our above 2 examples we are dealing with a request header but now we need to write a separate function to handle a response header. What we are doing here is providing our Cloudfront config with some additional headers for security purposes. Normally this is done once again in our Apache htaccess file but here we do it like this below. A separate function is needed because when we test it we want to check the box that says response and also want to do the same when we associate our published file. Likewise with the request function we check the box that says request.

Here we go

function handler(event) {
     var response = event.response;
     var headers = response.headers;


   // config security headers

headers['strict-transport-security'] = { value: 'max-age=63072000; includeSubdomains; preload'}; 


headers['content-security-policy'] = { value: "script-src-elem 'self' 'unsafe-eval' https://javasqlweb.org/js_new.js 'unsafe-inline' https://ipinfo.io/ 'unsafe-inline' https://www.googletagmanager.com/ 'unsafe-inline'; style-src 'self' 'unsafe-inline"};


headers['x-content-type-options'] = { value: 'nosniff'}; 


headers['x-xss-protection'] = {value: '1; mode=block'};


headers['referrer-policy'] = {value: 'no-referrer-when-downgrade'};


headers['x-frame-options'] = {value: 'DENY'};


return response;
}

The content security policy header is the challenging one. In my case I have a javascript file called js_new.js to enable and I also have 2 other js connections one to ipinfo and the other one to google tag manager. My last instruction for styles enables inline styles. It is best to google all these headers to gain a full understanding of their importance.

In order to pre and post test all of the above I use Screaming Frog, which absent the security headers will complain that we don’t include them or some of them. Frog is by all means a powerful tool to use frequently to test ur website for a number of issues. It is best to study these headers a bit more to know what they mean. You can also go into the network tab of chrome dev tools (ctrl-shift-j), click on ur page and view what the response header is returning there.

Conclusion

With all these little tweaks we slowly get s3 to behave more like a real webserver even though it is just simple storage. Like enhancing apache using htaccess we enhance s3 with a Cloudfront distribution and a Cloudfront function for the tweaks. Now I feel much better about using S3 to host my website.

2024 update on 301 redirects. from above **
copy the following code into a new cloudfront function

function handler(event) {
    var request = event.request;
    var headers = request.headers;
    var uri = request.uri;
    var requestURI = event.request.uri
    var response = event.response;
    var newurl5 = `https://howtolearnjava.com/JDBC.html`;
    var newurl6 = `https://howtolearnjava.com/Java-Servlet.html`;
    var newurl7 = `https://javasqlweb.org/about.html`;
    var newurl8 = `https://javasqlweb.org/home.html`;

    if (requestURI === "/JDBC") {
        return {
            statusCode: 301,
            statusDescription: "Moved Permanently",
            headers: {
                location: { "value": newurl5 },
            },
        };
    }

    if (requestURI === "/Java-Servlet") {
        return {
            statusCode: 301,
            statusDescription: "Moved Permanently",
            headers: {
                location: { "value": newurl6 },
            },
        };
    }


 if (requestURI === "/about.html") {
        return {
            statusCode: 301,
            statusDescription: "Moved Permanently",
            headers: {
                location: { "value": newurl7 },
            },
        };
    }

/*
if (requestURI === "/") {
        return {
            statusCode: 301,
            statusDescription: "Moved Permanently",
            headers: {
                location: { "value": newurl8 },
            },
        };
    }
*/

    return request;  //only needed once at end of all if statements
}

Happy Coding!!

AWS Gotchas Part 2 - corrections from part 1 plus new gotchas, A MUST Read (part of a series)

Rick Delpo — Fri, 23 Jun 2023 13:59:53 +0000

Last week I decided to host my website using S3 and Cloudfront and because I am not an expert I ran into lots of difficulties. While Googling out there I relied on some posts that turned out to be bad advice. Below advice only applies if u do not change ur website name and are just migrating over from a different web host. Also it seems like much of my advice below is SEO related. While many folks abhore SEO my opinion is that it is a necessary evil and best practice only if u want to get noticed out there (being on serp page 6 is better than not being noticed at all). But if ur web page is for internal corporate use only then no need to even care about SEO.

Firstly let me reiterate that gotcha #1 from my first post is the most major issue I faced. It took days for me to figure this one out. Your Cloudfront Distribution will not even resolve if u do not follow gotcha #1.

So while my migration was not working for days I created some more problems for myself that I had to correct after deployment.

Correction #1 - scratch what I tell u in gotcha #4 because it is all wrong. Yes ur files are now in a new path but it is ONLY a file path from Cloudfront to S3. There is ABSOLUTELY NO NEED to redo all ur links to this file path or ur canonicals. KEEP THEM AS IS using the legal path that u set up in Route53 and the S3 Bucket name. This is the legal path. Also it is imperative for ur sitemap to not have any references to the S3 file path and same for robots.txt. Bottom Line, make NO references to the cloudfront S3 path anywhere.

Correction #2 - more gotchas #1 is incorrect, you do not need to re seo everything. Your legal site name is still the same so don't create a new property in Google Search Console with the new S3 path name and start to request indexing as this will only create duplicate and confusing content out there on the web.

Correction #3 - be consistent about ur page names, if they did not have an html extension last time be sure this is true this time otherwise users will get a 404 error. My confusion was that by default all pages in S3 have an html extension. If u were on an old apache server where ur htaccess file stripped the html extension then this advice is for you. It is easy to just check the box in s3 bucket next to ur file, click actions, then rename and just remove the html extension and don't forget to re save.

Correction #4 - refers to more gotchas #4 that states that the CDN returns a 304 code vs a 301. While this is true and in some cases Google Search Console will not re index, here is what is really happening. When Google reindexes a page u can find that the status code of that page is 200 only for the indexing. This is because your robots.txt and sitemap are actually instructing the Google Bot to go to the source and not to the cached page over at the CDN. When this happens ur page will reindex. Here is the rub, much advice out there tells u that for small sites u do not need robots or a sitemap but I say ignore this and do both anyway only if u want web crawlere out there to notice u. Don't forget to do the same over at Bing Webmaster Tools. My suspicion is that if u don't have robots or sitemap then Google will then go out to ur 304 version of the page at the CDN and thus will not index the page. This is because Google Bot has no sitemap or robots to tell them what to do so Google has to guess. I think I am correct with this presumption but have not fully tested and professional feedback is a bit hard to come by these days.

This is all for my corrections. During my travels I found some new Gotchas that no one tells u about.

DO NOT create a sitemap using the cloudfront s3 file path name.
DO NOT create a new property at Google Search Console.
Even though all ur pre migration files are now deleted it seems like there is no place for a sitemap. Each Google property has a designated location to submit a sitemap.
DO NOT change any of ur old canonicals as AWS does its magic for you. If u try to change any canonicals (i use Screaming Frog to test crawl) Google will pick the legal canonical with the path name that matches ur legal name that u registered at Route53 which is also ur bucket name. It will not choose the user selected canonical if it is wrong and thus will not index ur page.
This one is kind of insideous. The default time for the Cloudfront CDN to update cache is 24 hours. Be advised that sitemaps and javascript files will work on a different schedule and update at different times. Don't be too hasty when checking all ur edits. Be patient. Cloudfront does warn us on Route53 that TTL is 48 hours. In my case some of my javascript was not working but 12 hours later everything was fine.

Thats all for now folks and THANKS for reading!

Find more about my NoSQL adventures at my home page

AWS Cloudfront DNS Migration Gotchas that no one tells you about

Rick Delpo — Sun, 18 Jun 2023 13:48:52 +0000

My overall observations, it takes a lot of digging to find these gotchas and many times AWS docs leave u hanging.

I migrated from an Apache Server using htaccess all the time, over to Cloudfront...started on June 7 and it is now June 18 and am still have issues but is working for the most part. This article assumes we already have some basic DNS and Networking knowledge but is still good for the beginner too.

I suspect that many developers go the Netlify route vs AWS Cloudfront due to an assumption of AWS complexity. I am not here to bash AWS, I think it is very worthwhile to do and HIGHLY recommend migrating to a better network neighborhood such as AWS S3 and Cloudfront.

first, the 4 major Gotchas

All names (comma separated), not just Alternate names must be in Alternate Domain Name Setting in the General tab on Cloudfront which is marked as 'optional'. The gotcha is that it says optional but wont work till u fill it in. Doesn't 'optional' mean optional?
Cors issue is in 2 places, S3 and Cloudfront, dont forget to enable cors also in Cloudfront. Gotcha is enabling only in s3 while forgetting to do in Cloudfront. Otherwise pre migration links used on the outside will show Cors issue in Dev Tools view. Go to Behavior tab in Cloudfront and under Origin Request policy put in CORS-CustomOrigin and under Response headers policy add Cors with preflight and security.....

Note that both these options are marked as optional (the gotcha here) but should not be skipped or CORS will only be enabled in S3 and not Cloudfront. Don't forget to press Save changes.
Using public DNS other than route53 does not migrate cleanly due to url forwarding issues. Route53 is only 50 cents per month to host DNS on AWS Route53 vs GoDaddy or other dns provider, and IT IS WELL WORTH IT. I was thrilled when I first found out u dont need route53 but ended up using it for a clean migration. The main issue was that my old links stopped working, they did not forward to my new page until I went back to using Route53. The thing about Route53 vs GoDaddy is that Route53 allows an A record to point to an alias and GoDaddy only allows an actual IP address. I think this is 301ing the right way and getting masking without really doing illegal masking.
This is not exactly a gotcha but could be. This is more of an FYI. Upon migration, pages now instantly live in S3 bucket behind a Cloudfront distribution so be sure to change ur canonicals right away on each page with the cloudfront path and submit a new sitemap to Google Search Console if u are on it. Also any links on your homepage need the cloudfront path changed right away. This applies to both buttons and href links. Even though many people say a sitemap is not necessary for a small site I say submit a new sitemap for a migration because if you are registered with Google search console this becomes a must do. Also be sure robots.txt is used and points to ur sitemap so web crawlers can find you.
Also another FYI is that Cloudfront seemlessly does a proper migration which includes your old name on the address bar. But if you do not use Route53 then your new cloudfront address will appear and not your web name. In my case my old links on facebook and twitter stopped working because of this, they timed out. When I changed back to Route53 this fixed all my old link issues. To me I could not have a situation where all my old links to my new home page now fail.

Note, since the original writing I found that this #4 gotcha about the canonicals and file paths is not true and I was given bad advice, please see my gotchas part2.

more Gotchas below - but many are just FYIs

Need to re-seo everything because files now live in a new place, GSC is still favoring my now defunct files and sitemap. When i request indexing a duplicate issue arises where Google chooses old canonical over user declared canonical....... still pending...will have an update on this soon.
My new finding is that we do not have to re seo, please see my updates in gotchas part2.
Beware that s3 requires the html extension until u tell it otherwise.....still pending
My old pages had no file extension as directed by htaccess file in Apache server. Now if user clicks on old page link from twitter or fb they get my 404 page which i finally revamped to accomodate this. Eventually i need to try out the now extension s3 option, but for now it is not a priority.
ps my index page old link works thank God
update.. just rename the page in S3 and remove html extension and all old links will work now

Since the original writing of this article I found that S3 files without extensions need special attention in Cloudfront. Removing the extension like I described above is not working out the way I expected. See my Part 3 of this series (coming soon) to find out why.

304 forwarding not 301, index pg only....impact to seo..but recently requesting indexing on home page returned a 200 response and not the expected 304. This is an SEO issue because if Google sees the 304 response they assume there are no updates to the page and thus will not index the page.

update - found out this is not true, see my gotchas part2 about this.

Best to use an AWS SSL Certificate and not ur own cert. I think that an AWS ssl cert does a proper TLS handshake with Cloudfront and I must say that I do not have direct experience with this but am using my common sense. I always used a free one from lets encrypt so with special certs I am not experienced but I suspect it can be a can of worms. So just get a new cert and dont try to move ur old one over. Not tested, just using my common sense here. PS. You will need a new SSL Certificate anyway because now pages reside at new address, the Cloudfront Endpoint.
Remember to have a separate S3 bucket for a www version. All the web crawlers take into consideration your
a. https://www
b. http://www
c. https://naked domain
d. http://naked domain
But I presume everyone already knows this part. Do this part only if u care about SEO. Perhaps it is best practice to do this otherwise site health gets dinged by Semrush, Ahrefs and GSC. All 4 DNS versions need to resolve to one https version (best practice).
Remember to have both A record and AAAA record pointing to cloudfront distribution in route53 and not the IP address of the new server. We use Alias for this, first we have our name and then for value press alias, then press cloudfront distribution then hopefully ur endpoint pops up and if it does not manually enter the new endpoint. Mine was dkrs578h9fllv.cloudfront.net which was assigned to me.
PS, dont forget to enable Brotli compression in cloudfront. Go to Behaviors tab in Cloudfront and be sure that compress objects automatically is set to yes, then go down and choose Cache Policy and Origin Request. Do not choose Legacy Cache Settings unless u need to fix ur TTL. Under Cache Policy choose CachingOptimized recommended for S3 and this will enable Brotli too. Your site will be blazing fast when deployed. Just fyi u can go to dev tools and click the network tab to see if response headers say Brotli is enabled. In addition to clicking network tab u also need to double click the actual html page after u refresh ur site with dev tools open. This is how to see the response header. Also u can find the age of the cached page which I think is very cool. 86400 seconds is the default so when this 24 hour period passes your page will recache and the afe will be 50 seconds approx and counting. For each refresh u can see the new age. Note 304 response code happens here because it is returning a cached page and no changes occur for 24 hours.
Cloudfront serves outdated content for 24 hours using a cached version of the page. This is a pain in the beginning because for a few days we are constantly updating our files, but the solution is not that bad. First of all, I could not get Invalidations to work properly so had to disable my cache only for a couple of days during the dev process. Also note that S3 files cannot be edited. You first need to delete them and then upload a new version of the page or just overwrite the existing file with a new one. Then I reset my cache to 24 hours when I was done. Note that I originally reset the default cache TTL to 3600 seconds but still needed to wait one hour to see my changes.
Note on URL masking, dont do this yourself from GoDaddy, (u can do url forwarding with masking) Dont do this!! Cloudfront will do it the correct way when u are on Route53. My lesson learned here was to just use route53 for my DNS and not my DNS provider which is Name.com
Using WAF on Cloudfront distribution incurs charges. Will work fine without WAF even though message comes up saying that WAF is recommended.
Remember that u need 2 cloudfront distributions one for main index page and other for your www page originally set up as a separate s3 bucket. Beware of enabling WAF, we easily live without it. Check your AWS cost explorer to find out current charges. I am not sure if u are still in the 12 month free tier if these charges apply.
It can take up to 48 hours for Cloudfront to fully propagate so be patient. Also the new SSL cert needs time to be cached too.

Conclusion - not sure if Netlify has any of these gotchas. Would be interesting to get some feedback on this. ALSO, fyi, I am not an expert at any of this just a casual ordinary Joe making my observations. But certainly would welcome feedback.

ps. more edits coming on this article, just wanted to get it out there for now

also find out more about Rick Delpo at my website
JavaSQLWeb.org Home Page
and don't forget to read more of my Dev.to articles. Thanks for reading and Happy Coding !!

A Brief History of the Internet and HTTP Protocol

Rick Delpo — Mon, 22 May 2023 14:29:00 +0000

Javascript wins out in my opinion, find out how below

Now and again we need reminders about our rich beginnings
Topics…. software architecting - request and response objects - the client server model - http
Origins of request and response objects - Major Milestones

1965 Donald Davies developed the concept of packet switching
http://www.nethistory.info/History%20of%20the%20Internet/beginnings.html#:~:text=In%201965%2C%20before%20Arpanet%20came,the%20NPL%20Data%20Communications%20Network.
1969 ARPANET the first public packet-switched computer network
1970s NCP preceded TCP - network control protocol used in arpanet
1975 Bill Gates starts Microsoft
1979 Compuserve dial up service launched - original online portal for the general public with news, weather, chats and email
1983 TCP/IP replaced NCP
1991 world wide web coined
1993 first version of HTML
1993 Mosaic - first web browser, then became netscape
1995 java released - also rise of javascript for netscape2, (the first ECMA standard)
1995 MySQL first appeared - open source vs proprietary Oracle db
1996 first mobile phone with internet connectivity - The Nokia 9000 Communicator
1998 java servlet api - also jsp - doGet doPost became servlet service method
This is why java was so prominent at the time - popularized server side computing
JSP was for the frontend
1998 soap api
2000 rest api - rise of the http network -> crud awareness - get, post, put
https://en.wikipedia.org/wiki/Representational_state_transfer re roy fielding
2006 jquery ajax - refreshed only part of page - div structure -turned networking upside down
Probably the most important milestone
<div id=”result”> only the content in this div refreshes/reloads <div>
2015 fetch api - genericized the ajax process - plain javascript now mainstream (ECMAScript 6)
2021 why is jquery still dominant - 58% of websites still use it
https://thenewstack.io/why-outdated-jquery-is-still-the-dominant-javascript-library/
2023 everything can be done in plain javascript these days, react just complicates matters by adding extra overhead, node.js does the same whereas plain javascript simplifies both, also no need for java or servers anymore, just do it all on the front end which now executes the http request
2023 age of serverless computing, cdn and edge network

Afterthoughts

1995 was a pivotal year - when both frontend and backend collided
But looks like frontend prevailed and won by 2023

Thoughts on Ditching JQuery in favor of Plain Vanilla Javascript
1 it requires a slow loading external library = unnecessary overhead
2 it has security issues
3 why use a whole library when u are only using a couple of its components
Also ditch Node, no need for all those dependencies when ECMA has solved them all
4 Fetch API replaced jquery ajax back in 2015

Inspecting the code of a stacked CSS Bar Chart data visualization (with no chart libraries)

Rick Delpo — Sun, 21 May 2023 18:51:56 +0000

In data analytics we use bar charts all the time. Here we construct a product sales stacked bar chart in plain javascript and css with no chart libraries (no chart.js, lodash or moment)

html bars are a for loop, 1 bar done 5 times, using css flexbox layout. This is how we build the chart.
grid lines using css after selector and css z index
section colors using css nth child
data source is json file saved in aws s3
data populated using array.indexOf() and array.reduce() . This is the plain js part where we do all our calcs
we use 2 methods in our JS, 1 to transform the data and 2 to build and populate the chart using array.push on an empty array to populate the final array

Our overall html div structure starts with the barchart-wrapper class inside where we construct the y axis and then the barchart class for grid lines. Then we drop the chart class inside a chart container which is inside the barchart class. It is the chart class that uses flexbox layout and contains our javascript to build everything.

<div class="barchart-Wrapper">
<div class="barChart-Container"> 
<div class="barchart">
<div class="chart" id="navContainer" style='width: 500px; height: 500px;'>

flexbox plus script goes here, this is where we drop in the chart class which uses css flexbox layout to give life to a bar 

</div> <!-- closing tag of chart class-->
     </div> <!--end barchart div-->
   </div> <!--end bar chart container-->
    </div> <!--end bar chart wrapper-->

Here is a more detailed view

 <div class="barchart-Wrapper">

     <div class="barchart-YCol"> <!-- this is y axis bar, we are populating y axis-->

      <div class="barchart-Y">
        <span class="barchart-YText">$1000</span>
      </div>

      <div class="barchart-Y">
        <span class="barchart-YText">$ 750</span>
      </div>

      <div class="barchart-Y">
        <span class="barchart-YText">$ 500</span>
      </div>

      <div class="barchart-Y">
        <span class="barchart-YText">$ 250</span>
      </div>

     </div> <!--end YCol-->

     <div class="barChart-Container">  <!--need container so width is applied to y axis-->
<div class="barchart"> <!--this is the class with grid lines across-->
           <!-- below dropped chart class inside barchart class which is in separate container-->
     <div class="chart" id="navContainer" style='width: 500px; height: 500px;'> <!--this is the chart container-->

the chart and script go here

</div> <!-- closing tag of chart class-->
     </div> <!--end barchart div-->
   </div> <!--end bar chart container-->
    </div> <!--end bar chart wrapper-->

Once inside the chart class div we use javascript to create a for loop which then repeats one bar five times. In order to make our for loop work we need to create a new div on the fly. This new div contains multiple html tags used in the for loop. Here we are getting the inner html from the new div to construct one bar.

var div = document.getElementById('navContainer'); //a wrapper div to be appended by new div
   // create a new div (with id and classes 'new div'): must be done to work in script below using i variable

var new_div = document.createElement('div'); //pass in div var so our new div behaves like old div
                   new_div.style.width = "500px";
                   new_div.style.height = "500px"; //needs height attribute
                   new_div.setAttribute("id", "navContainer");  //adding these id and class atts here displays bars from bottom and not upside down or backwards
                   new_div.setAttribute("class", "chart");       //adding, below fixed here plus above line, also done so bars display vertically and not horizontal

let dates = ["jan", "feb", "mar", "apr", "may"];  //x axis labels
for(var i = 0; i < 5; i++) {            
                         //we are constructing 1 bar here to be used 5 times using above constructed new_div
 new_div.innerHTML +="<div class='bar' style='--bar-height: "+percent_of_bar_height[i]+"%"+";'>"+ //setting bar height as percent of tot dollars on y axis
            //next 3 lines represent 1 bar with 3 sections, each bar has 3 data vals, 3 section heights and 1 bar height 
"<div class='section' style='--section-value: "+percent_of_section_height_3[i]+";' data-value='"+product_3_total[i]+"'></div>"+
"<div class='section' style='--section-value: "+percent_of_section_height_2[i]+";'data-value='"+product_2_total[i]+"'></div>"+
"<div class='section' style='--section-value: "+percent_of_section_height_1[i]+";'data-value='"+product_1_total[i]+"'></div>"+            //setting section percentage of product1 total dollars

"<div class='label'>"+dates[i]+"</div></div>";  //add month names to x axis

      // now append the element (as a whole) to wrapper
  div.appendChild(new_div); //appends newly created div to original div.. needed for script to work
}  //end for loop

Now that we have the chart built we need to run the transform_original_array method in our javascript. This method traverses our JSON dataset objects and transforms the result into a new array which we then make some calculations on, group the result by month and reduce (sum the data) for each month. These calcs are pushed into interim arrays which are then used to populate the final bar chart.

For Grouping and Summing click this link https://dev.to/rickdelpo1/group-by-month-and-sum-values-for-each-month-explained-in-plain-english-and-done-in-plain-javascript-with-example-4o9k

Find the Full Code here at my codepen https://codepen.io/rickdelpo/pen/KKGQrQv which is fully commented for easier understanding.

Array indexOf() method made simple for Data Analytics in charts and graphs..featuring Grouping, Summing and Rendering

Rick Delpo — Mon, 15 May 2023 16:55:56 +0000

Did you know that grouping is still unfinished in the Javascript API specification as of 2023? It's not even officially standardized yet. No wonder why it is so difficult to pull charts off in JS and why third party libraries are widely used. After researching this topic I discovered how much is out there and how confusing it is, to the point where we developers just want to give up. Read on and I will show u an easy way to build a quick bar chart dashboard.

We iterate over arrays in Data Analytics thus creating useful insights which add meaning to our data. Charts and graphs enable this process.

Since arrays are iterative by default I like using the indexOf() method to do my grouping because it shows the raw iterations, ie the status of our object for each iteration. Here I am trying to explain indexOf() in plain English.

IndexOf() returns the index position of an array for each iteration. Index is the location of an element in an array starting at position 0. We first query the array to find an element. var x = array.indexOf(element); On the second pass our index is incremented by 1 and so on.

Iteration begins with line 1 of our array for an object and if our search criteria is not found then the index position becomes -1. We then include our object and while this object is then found in the next iterations we sum values when index position is not -1 (index is zero at this iteration). When the object changes we have -1 again because on first pass the object is not found until we add it, and we start our summing all over again. This behavior is grouping. It is a while loop.

use case - group months for product sales and sum multiple product sales values for each month. Eg. while month is January sum 3 product's sales for records with month as January. Start over again for February.

var original_array =
[{"order_date":"01-03-22","product_line":"prod1","dollar_amt":100,"product1":200,"product2":0,"product3":0},
{"order_date":"01-02-22","product_line":"prod2","dollar_amt":50,"product1":0,"product2":50,"product3":0},
{"order_date":"1-16-22","product_line":"prod3","dollar_amt":50,"product1":0,"product2":200,"product3":50},
{"order_date":"1-17-22","product_line":"prod1","dollar_amt":100,"product1":100,"product2":0,"product3":0},
{"order_date":"1-15-22","product_line":"prod2","dollar_amt":50,"product1":0,"product2":50,"product3":0},
{"order_date":"2-5-22","product_line":"prod1","dollar_amt":100,"product1":100,"product2":0,"product3":0},
{"order_date":"2-6-22","product_line":"prod3","dollar_amt":20,"product1":0,"product2":30,"product3":20},
{"order_date":"2-7-22","product_line":"prod1","dollar_amt":100,"product1":100,"product2":0,"product3":0},
{"order_date":"3-23-22","product_line":"prod2","dollar_amt":200,"product1":0,"product2":200,"product3":0},
{"order_date":"3-5-22","product_line":"prod3","dollar_amt":20,"product1":0,"product2":100,"product3":20},
{"order_date":"3-29-22","product_line":"prod1","dollar_amt":100,"product1":100,"product2":0,"product3":0},
{"order_date":"3-25-22","product_line":"prod1","dollar_amt":100,"product1":100,"product2":0,"product3":0},
{"order_date":"4-23-22","product_line":"prod1","dollar_amt":500,"product1":500,"product2":50,"product3":50},
{"order_date":"4-24-22","product_line":"prod2","dollar_amt":100,"product1":0,"product2":100,"product3":50},
{"order_date":"5-10-22","product_line":"prod3","dollar_amt":50,"product1":0,"product2":0,"product3":50},
{"order_date":"5-15-22","product_line":"prod1","dollar_amt":500,"product1":100,"product2":0,"product3":0},
{"order_date":"5-25-22","product_line":"prod2","dollar_amt":50,"product1":0,"product2":50,"product3":150}]

; //end of array

       //first transform original array into product sums for each month array, then used to populate bar chart

  // Because JS doesn't have a nice way to name months
var monthNames = ['Jan', 'Feb', 'Mar', 'Apr', 'May', 'Jun', 'Jul', 'Aug', 'Sep', 'Oct', 'Nov', 'Dec'];
var result = [];   //transformed array after below operations are performed
var dateArr = [];   //starts empty, to keep track of month names
let initialValue = 0; //need to pass in an initial value or reduce omits first element of array

var dataByMonth= original_array.reduce(function(prev, currValue){ 
date = new Date(currValue['order_date']); //for js date to work must be inside a function, needs valid data format yyyy etc
      //next 3 lines converts non conforming date on the fly so conforming/valid date can be used in date object
var dateString = new Date(date.getTime() - (date.getTimezoneOffset() * 60000 ))
                    .toISOString()
                    .split("T")[0];
console.log(dateString);

  date = monthNames[date.getMonth()]; //gets month names to be grouped below     //we are iterating over month names array at each pass     //if cur mo is not found in dateArr then start summing //keep summing while cur mo is found in each subsequent pass
console.log("date.getMonth()= "+date);
               //next, indexOf is acting like a find function
   var index = dateArr.indexOf(date); //equaling -1 on first pass of ea mo until that mo is added to the array  //because there is no match for date in dateArr on first occurrance of mo
//first check to see if dateArr has mo name, on new mo will not have so index is -1
        console.log("index = "+index);

   if (index == -1) {  //is equal to -1 only for first pass, above forces -1 so we can initialize first pass
       dateArr.push(date);  //no match for month name until it is pushed..dateArr starts as empty until mo val is pushed
          var obj = {month: date, product1: currValue.product1, product2: currValue.product2, product3: currValue.product3};
      result.push(obj); //push in default vals for new mo
   } 
   else {
//index = 0 because dateArr has val now...then for next mo index=1
               //on subsequent passes index will not be -1 so just increment cur vals until new mo is encountered
           result[index].product1 += currValue.product1; //increment sum each pass starting index =0
           result[index].product2 += currValue.product2;   //no need to push, result val updates each time
           result[index].product3 += currValue.product3;
   }
console.log("dateArr "+dateArr);
return prev +currValue; //need to return both prev and curr, for first element to register
} //end of reduce

, initialValue);  //initial value here, this is the 2nd arg of array.reduce, first arg is callback function above

console.log(JSON.stringify(result, null, 1));

ps: there are other ways to group data such as array.groupBy() or lodash but 1 is confusing and 2 adds a library which I am trying to avoid.

note: above code is only a segment of the total solution to be found here: (make this stacked bar chart your next JS project)
https://dev.to/rickdelpo1/how-to-populate-a-stacked-bar-chart-in-plain-javascript-12p9

problem statement:
Grouping is critical in data analytics when we are using charts yet a Javascript standard is not in place yet forcing us to cobble together whatever we can and often just giving up.

Most charts have a time series on the x axis and for marketing and sales data it is usually a by month time line in our graph. It is 2023 and Javascript Ecma International is still trying to get this right. It is no wonder why many third party libraries popped up like underscore, lodash and moment. But these libs add complexity when we can solve the issue using only a few lines of native js code. Also external libraries slow down our web page load time.

Easy to understand solution:
The advantage of using indexOf() is that it demonstrates the iterative process on an array which is, bottom line, the whole essence of data structure Array. In my opinion it is the least confusing way to group data and leads us to an easy solution and a finished project.

caveat..Many examples out there show a grouping method but do not combine it with summing which is a must have in data analytics when building charts. We want the sum of products for each month not just the records grouped by month. These are 2 different things. It is a bit insidious at first where grouping alone does not solve the problem.

Thanks for reading folks and don't forget to click above for my stacked bar chart using this easy grouping logic. Once on my project page u can click my codepen for the full code.