DEV Community: sstores

Self Driving Cars... the Future is Now

sstores — Thu, 20 May 2021 19:06:51 +0000

Are We In The Future?

Self driving cars have been a staple of sci-fi and visions of the future for a long time. In recent years Artificial Intelligence technology has started to catch up with imagination. According to TechCrunch article, in 2021 there are over 1400 self-driving cars on the road, however, “nearly 43% of people in the US don’t feel safe in a driverless car”, according to PolicyAdvice.com. The German car company Audi has stated that they will invest $16 Billion into self-driving vehicles by 2023. That is quite a chunk of money, but the automated-vehicle global market is already a hefty $54 Billion, which will no doubt continue to expand over the next several years, with some projections estimating “to increase tenfold in the next 5–7 years”, according to Nissan.

Are All Automated Vehicles Made Equal?

Different types of Autonomous Vehicles (AVs) include a range from Level 1 including driver assistance functions, to Level 5 including fully-automated function with no need for human interaction. Level 5 AVs are the newest and least studied, largely because they are not fully developed and are not widely available yet.

How Comfortable Are We?

Results vary depending on age, gender, and whether working in a tech field or not, but according to US News of 1000 people polled in a survey, “only 57% of people familiar with self-driving cars would be willing to ride inside them”. There are a variety of statistics on this topic but generally they show that most people are not fully comfortable with the idea of self-driving cars yet. But as with any major change in paradigm, as something becomes more familiar and accessible, minds will change.

So, How Safe Are Self-Driving Cars?

Since the introduction of Autonomous Vehicles (AVs) on the road there have been deaths in accidents caused by a miscalculation or mistake by the AV. However, compared with the much larger number of people who are injured or killed in non-automated car crashes every year, the potential for increasing safety on the roads is significant. As the tech gets safer and more reliable, there will undoubtedly be more and more automated vehicles on the roads. Comfort with the changing environments on the road may be the largest hurdle that societies will have to overcome.

Big Change is Coming...

The question of autonomy is an interesting one in the context of AI and in this case, autonomous vehicles. Considerations should be taken for how certain vehicle-intensive industries like trucking and logistics will change with the influx of automated vehicles. Trucks that travel long distances on largely straight roads are a prime first candidate for automation. Full automation in shipping is still probably a ways off but we are much closer than people might think.

Sources:

https://policyadvice.net/insurance/insights/self-driving-car-statistics/
https://techcrunch.com/2019/06/11/over-1400-self-driving-vehicles-are-now-in-testing-by-80-companies-across-the-u-s/
https://www.vox.com/2018/5/17/17362308/elon-musk-tesla-self-driving-autopilot-fatalities
https://emerj.com/ai-adoption-timelines/self-drivin

Ember. js

sstores — Fri, 14 May 2021 00:26:44 +0000

One special thing about tech industry is that popularity of frameworks or libraries is largely based on usability and efficiency. How well the thing works for the purposes of the programmer. That means that libraries and framework fluctuate over the years. In the last few years React has been the newest hottest thing. But around 2012-2015 that place was arguably held by Ember. Ember js is a javascript framework like React, Angular, or Vue. Most of the current “Best frameworks in 2021” lists I have found list Ember among these top used. We will explore some of the characteristics particular to Ember and comparing it to the most popular framework/library today, React.

What is Ember js?

As we said before, Ember is a framework. On the Ember docs, it specifically says that Ember is “a framework for ambitious developers”. What makes this framework particularly appealing to the ambitious folks among developers? Well it is well suited for complex, industrial-scale apps. The code base itself is robust and has lots of great functionality and features build in. React in contract is specifically intended to be lightweight and non opinionated. So overall out of the box, react will be lighter and will be a bit simpler to get started.

Some other specific specialities on the Ember docs are the following:

“Offers HTML and CSS development model at the core.”
“Provides instance initializers.”
“Ember.js offers routes that are the core features used for managing the URL.”
“Ember.js uses templates that help to automatically update the model if the content of applications gets changed.”

There are of course some similarities between Ember and other libraries and frameworks like React. We will highlight some of the differences. React is a self-described library and Ember is a Framework. Though these are often lumped together in the “front-end” there are differences. React is backed and updated by facebook. Ember is community backed meaning it has a much smaller dedicated team keeping the code and documentation up to date. React was created in 2013 and Ember was created in 2011. Ember had a spike in popularity from around 2012-2015 where it was one of the most popular frameworks, widely taught in coding bootcamps.

When compared to React, Ember is larger and adept at handling complex programs. Therefore the learning curve is steeper and the barrier to entry is a bit higher than React. Because React is non-opinionated and the syntax is very similar to javascript, React is more beginner friendly as far as frameworks go. There are many companies that either use or have used Ember. The list includes Square, Bustle, TED, Netflix, Heroku, Microsoft, LinkedIn, Blue Apron, Travis CI and many more. Some of these, notably Square and Netflix have switched operations to React in recent years but that being said, Ember is still a useful thing to know in case you run into Ember legacy code.

Another helpful measure when looking at whether a technology is relevant and viable as far as usage and engagement by developers is Stack Overflow and GitHub stats. In August of 2020 there were over 236,000 questions about react on Stack Overflow. That number makes the 21,000 questions about Ember seem measly. But we can see trends in engagement through these numbers. React currently has more engagement and more folks exploring its functionality. The high number may also be due to the fact that React is much more DIY as far as adding plugins, additional libraries, and add ons, whereas Ember is more of a complete and robust system out of the box. That combined with the fact that Ember’s fan base tends to skew towards a more experienced developer crowd because of the steep learning curve, may account for the differential.

In short, Ember may not be the hottest thing any more but it is still good to know and very helpful for specific purposes. Only time will tell wha the next big thing will be!

Sources:
https://hackr.io/blog/best-javascript-frameworks
https://guides.emberjs.com/release/getting-started/anatomy-of-an-ember-app/
https://www.simform.com/react-vs-ember/#Table

Technical Animation… Where does the coding happen?

sstores — Thu, 06 May 2021 20:21:35 +0000

Animation has a long history in the film era. But even before the advent of film, the victorian’s created optical illusions that created the appearance of people dancing. Largely though, most animation as we think of it today stems from the film industry. We will go through a brief overview of animation and technical animation as disciplines.

What is Animation?

According to New York Film Academy, animation is “the act of creating the illusion of movement through still images”. Although there are of course many purveyors of animation arts, Disney is credited with many outstanding moments in the history of animation. The Illusion of Life by Frank Thomas and Ollie Johnston is often referred to as the animator’s bible. Thomas and Johnston were animators at Disney and wrote the book about Disney philosophy and process for animation.

The subject of “firsts” of animation in the film era, is fairly subjective because there were animated snippets in films earlier on that full-length feature films totally drawn by hand. In 1918 the Rotoscope was invented to paint over live-action film with animations. This could be considered some of the first instances of motion capture animation, taking footage of actors and creating an animated image. However, El Apóstol (The Apostle), was an Argentine animated political satire using cutout animation, released in 1917, and is sited as the first full length-animated film. It was a 70-minute long movie, and is often credited as the first commercially profitable animated movie ever made. Unfortunately it was lost in fire shortly after release, and was never seen again.

Moving forward, Snow White and the Seven Dwarfs originally released in 1937 is often sited as the first not-lost, feature-length animated fully hand-drawn film. Pixar’s release of Toy Story in 1995 was another huge leap forward in the animation history with the first use of Computer-Generated Imagery (CGI) animations. This was the start of the combination of animation and tech and in following Moore’s Law effect, every year the rate of technological increase is growing exponentially. Modern Animation can sometimes be terrifyingly lifelike.

Where does coding come into play?

With the advent of 3D Animation, many more technical professionals are required to manipulate computer generated images and animations with code. Technical animators are individuals who ideally know coding and animation because they will create animation tools and create functionality, motion, and world building. In short, a tech animators job is to make animators lives easier. It’s the animators job to Take tools build by the technical animators and give life to the character, story, and environments of the movie or game.

Motion Capture

Motions capture technology, also known as “mo-cap” is used to translate human movement to coded models. Actors, wearing suits covered in sensors, will perform the scene and provide the animation team a model to base their animations. An animator goes through mo-cap film and cleans up the movement and translates it to the character.

Movies vs Video Games

The creative process in animated movies and animated games is basically the same. The design team will start with sketching out character and landscape design. Interactive models are created through mo-cap or other animation techniques. With games however, there is much more variability and the programming side of things must be more robust. In movies there is a set storyline and a predictable final product to work towards. Where as in a video game, creators must account for movement and spatial interaction with user-generated variability. Characters need to be able to interact with the environment in a realistic way.

What languages/tools are used in TA?

There is a wide variety of tech that can be used in technical animation. However, similarly to javascript engines, Game engines are usually written in C++. For example, Chrome's Javascript engine, V8, is written in C++, and so are game engines Unreel and Unity.
Most code for motion and animation itself is written in Python. However, gaming companies may use proprietary coding languages for game animation. Maya is the proprietary language for Electronic Arts as an example. Key frames are like still images of the characters etc. The game engine is responsible for interpolating the data in each keyframe and smoothing out the animation.

In conclusion, animation and technical animation is a vast industry with a robust recent history. The more advanced the tech becomes, the more life-like animation becomes and the more opportunity can become possible for artists and technical folks alike.

Equity in Facial Recognition... How are we doing?

sstores — Mon, 26 Apr 2021 19:21:54 +0000

What is facial recognition?

At one point it may have been science fiction, but now facial recognition is as ubiquitous as having an iPhone. Facial recognition is a form of identity authentication, using mathematical mapping of the face, to identify a subject. The first computer to recognize facial features was created in 1964-66 by researcher Woodrow “Woody” Bledsoe, along with researchers Helen Chan, and Charles Bisson at Panoramic Research Inc. in Palo Alto, California. Woody Bledsoe is widely regarded as the father of facial recognition technology(FRT) and the controversy with this technology has a long history. Early in his career, Blesdoe worked on classified research relating to the atomic bomb, another controversial subject. Later on he worked on machine recognition of handwritten and printed letters and in the mid sixties he and his team took on the project for facial recognition technology, creating the first version of biometric comparison programs to identify individuals from multiple photos. This was the beginning of what has turned into everything from instagram filters, and unlocking your iPhone with just a scan of your face, to airport security.

Since its invention in the 1960’s there have been many improvements to the technology and a much more evolved version of FRT is now used in a variety of contexts for security and identity verification purposes. There have been some distinct biases and challenges presented by this technology, and although overall accuracy of facial recognition tech continues to improve year after year, there are still some important ethical and civil rights issues to consider with implementation of this tech.

How does it work?

Let’s look at some of tech itself. There are several ways that facial recognition algorithms can function. Two classes by which to distinguish these algorithms are, “geometric” vs “photometric”. A geometric algorithm relies on “biometrics” to create a two-dimensional image based on measurements from a subject’s face. These measurements create a batch of numbers also known as “feature vectors” which can be used to create a mapped image and compare it to other images mapped in the same way. This process can be challenging since accounting for tilt, or rotation of the head can be problematic.

Photometric algorithms will “normalize” and “compress” two-dimensional images and will compare with other similarly processed images. As an example, to normalize an image would be to turn it face-forward.There is also three-dimensional facial recognition where either 3D sensors, or three 2D cameras at different angles can be used to create an image of a subject. These are generally much more accurate than two-dimensional recognition.

Today, there are several big players in the facial recognition technologies game including Amazon’s Rekognition, Microsoft Azure’s Face API, and Facebook’s DeepFace, just to name a few. Although overall accuracy for these programs are above the 90th percentile overall, and have many helpful use cases that increase security, all have proved to be flawed, though improving, when accounting for differences in race and gender specifically less accurate for darker skinned individuals, and less accurate for women than men. We will explore this a bit more.

How is it being applied?

Today, facial recognition tech is as ubiquitous as having an iPhone, and using social media, but it is also being used in a wide variety of security and identity verification contexts. Some uses include European border controls, airlines passenger identity verification, airports security Paris-Charles de Gaulle and Paris-Orly airports and, mega event surveillance in Japan, and EU Summit security in Europe, even self selected “problem gamblers” can opt-in to having themselves barred from casinos using the tech. Facebook uses its ‘DeepFace’ tech to tag millions of user photos every day and claims an accuracy rate of 97.25%. Google’s similar product FaceNet, claims 99.63% accuracy when matching 13,000 pictures of faces from across the web.

There are ongoing and continually updated studies to verify and test the accuracy of facial recognition technology. Beginning in February of 2017, the National Institute of Standards and Technology(NIST) began an ongoing comprehensive Facial Recognition Vendor Test(FRVT). Anyone may submit an algorithm for evaluation and reports are published on an ongoing basis as results are available. The study compares accuracy of algorithms based on demographic categories male, female, black, white, Asian or American Indian.

The FRVT report published in 2019 reported performance of 116 facial recognition algorithms. The study found that the highest performant algorithms reach nearly 100% accuracy with ideal conditions. However there is a relatively higher margin of error when identifying darker-skinned people and women. The report points out that these margins are generally less than 0.1% depending on the algorithm, and overall the accuracy of these technologies are very high. There are additionally ongoing tests to evaluate the The most recent and updated reports from NIST can be found here: https://github.com/usnistgov/frvt/tree/nist-pages/reports/11

What is being done?

Since the beginning of facial recognition technology, it has been used by the CIA and law-enforcement agencies. Dating back to Blesdoe’s original research for the CIA, mugshots were the primary resource for facial recognition. Now of course, with the advent of social media and images of faces are much more accessible, however there are discussions and ethical dilemmas around how and where this technology should and can be used. Efforts to manage these questions and dilemmas are in many cases underpinned by discussions of protection of privacy rights and civil liberties.
In 2019, San Francisco blocked police and other law enforcement agencies from using face recognition software, and was the first major American city to do so. In June 2020, Microsoft, who had previously marketed its face recognition software to law-enforcement agencies announced that it will not sell and has not sold to the police. IBM, which was initially in the forefront of the facial recognition game, discontinued its Watson Visual Recognition API in June of 2020. And also in 2020 Amazon banned police from using Rekognition for a year, providing a window of delay for federal laws and regulation to be established and protect human rights and civil liberties in this arena. Continuing discussions and regulation is undoubtedly warranted and will continue to evolve, though perhaps not as fast as the technology in question.

Conclusion

This is by-no-means a comprehensive view of this topic and there are many questions related to the ethics and equity of utilizing this tech. It is clear that facial recognition technology has many important use cases that can benefit many people and make life better. However, we know that technology is not innate neutral. It reflects the biases and trends of its creators and developers must be conscious of this moving into the future. This sums up one of the major fundamental issues with how facial recognition technology is being used today, and discussions that should be continued as technology evolves.

Sources:
https://sitn.hms.harvard.edu/flash/2020/racial-discrimination-in-face-recognition-technology/
http://gendershades.org/overview.html
https://www.infoworld.com/article/3573069/what-is-face-recognition-ai-for-big-brother.html#:~:text=The%20early%20facial%20recognition%20algorithms,template
https://www.aaai.org/ojs/index.php/aimagazine/article/download/1207/1108
https://observer.com/2020/01/facial-recognition-development-history-woody-bledsoe-cia/
https://www.thalesgroup.com/en/markets/digital-identity-and-security/government/inspired/where-facial-recognition-used
NIST Report:
https://www.securityindustry.org/report/what-nist-data-shows-about-facial-recognition-and-demographics/
https://github.com/usnistgov/frvt/tree/nist-pages/reports/11

Advanced Axios

sstores — Mon, 19 Apr 2021 13:39:31 +0000

What is Axios?

Axios is a promise-based HTTP client which wraps the native javascript HTTP request processes for making requests to external servers from the browser. In other words, when the browser needs information to run code for the client, it may make requests to an outside server and Axios makes this process mor simplified and usable. As is the case with many elements of tech, improvements are constantly made over time, and network requests have had their fair share of makeover throughout the years. As an example, Asynchronous Javascript And XML (AJAX), is a combination of the browser built-in XMLHttpRequest object (to request data from a web server, and JavaScript and HTML DOM (to display or use the data). XMLHttpRequest (XHR) is the native JavaScript API to create AJAX requests. It provides build in methods which give the ability to send network requests between the browser and a server. Axios, is a library that abstracts some of the complexity away from these processes and provides more straightforward methods to handle these requests between browser and server.

Most front end programmers are probably familiar the basic Axios requests, Axios.get, Axios.post, etc. However, the Axios provides additional functionality and more extensive ways to handle requests and processing data. We will go through a non-exhaustive overview of some of these features in this blog.

Making Multiple Requests at Once

Axios.all is a handy function that allows you to make multiple Axios requests at once. Axios.all takes an array of Axios requests and will return an array of the responses of those requests. Keep in mind that if one of the requests fail, by default they will all fail if that possibility is not accounted for. You may choose to return null in the catch block of each individual Axios request. Therefore if one of the requests fail, the promise will still be able to resolve itself.

In order to access and process the responses, which are returned in an array, you may choose to map over the array using the Array.map function. However an alternative could be to use Axios.spread. Axios.Spread will give access to the responses in an array which can then be separated out as necessary. An example of Axios.all and Axios.spread is shown below from storyblok.com:

Although Axios.all and Axios.spread are technically deprecated according to the Axios docs, they are still good to know. You can use Promise.all to a similar effect since axios requests are dealing with promises anyways.

Creating an instance

The Axios.create function can be used to create a new Axios instance. You might want to create a new instance if you need to set custom defaults for your application. Axios comes out of the box with a default instance. It is not necessary to create new instances but can be helpful for specific configurations. For example, masteringjs.io has the following example. You might want to set a timeout on all the Axios requests in a given application. You might do that with the following:

Request Config

When making an Axios request, you have several options of how to build that request. The url is the only required is the only required config option while making a request. If the method such as GET, PUT, POST etc are not specified, it will default to GET. All of these config options are listed on the Axios docs but we will list some helpful ones here.

baseURL will be added to the beginning of url unless url is absolute. It can be helpful to set baseURL for an instance of axios to pass relative URLs to methods of that instance.
transformRequest is an array of functions that allows you to make changes to the request data before it is sent to the server. This is only applicable for request methods which send data, ie 'PUT', 'POST', and 'PATCH'. The last function in the array must return a string or an instance of Buffer, ArrayBuffer, FormData or Stream.
params are the URL parameters to be sent with the request. Params must be a plain object or a URLSearchParams object. data is the data to be sent as the request body. This options is only applicable for request methods PUT, POST, DELETE , and PATCH.
timeout specifies the time in milliseconds before the request times out. If the request does take longer than the timeout specifies`, the request will be aborted.

There are also several Node.js only options which can be specified as well.

maxContentLength defines the max size in bytes of the http response content allowed in node.js.
decompress indicates whether or not the response body should be decompressed automatically. If set to true will also remove the 'content-encoding' header from the responses objects of all decompressed responses. This is a Node only option, as XHR cannot turn off decompression. There are other options that may be utilized with request config listed on the Axios docs. A link is included in the Sources section below.

Response Schema

The response of an Axios request is an object which contains information sent from the server. It includes the following properties:

DATA is the response object that was provided by the server. This will include whatever data was requested.
STATUS is the HTTP status code from the server response, (ie 200, 404, 500).
STATUS TEXT is the HTTP status message from the server response.
HEADERS is the object of HTTP headers that the server responded with. All header names are lowercase and are accessible using the bracket notation.
CONFIG is the config that was provided to axios for the request
REQUEST is the request object that generated this response. It is the last ClientRequest instance in node.js (in redirects) and an XMLHttpRequest instance in the browser

Interceptors

Axios interceptors are functions that Axios calls for every request. You can intercept requests or responses and modify them in some way before they are handled by then or catch. According to masteringjs.io, “You can think of interceptors as Axios' equivalent to middleware in Express or Mongoose.” An example of an interceptor from the Axios docs can be seen below:

In conclusion, Axios is a very useful way to handle requests to a server. For more extensive look at the details above and many more, visit the Axios Docs. Also linked here https://kapeli.com/cheat_sheets/Axios.docset/Contents/Resources/Documents/index
Is a useful Axios“cheat sheet”. Happy coding!

Sources:
https://github.com/axios/axios#request-method-aliases
https://www.pluralsight.com/guides/all-need-to-know-about-axios
https://kapeli.com/cheat_sheets/Axios.docset/Contents/Resources/Documents/index
https://www.storyblok.com/tp/how-to-send-multiple-requests-using-axios
https://masteringjs.io/tutorials/axios/interceptors

Webpack

sstores — Mon, 15 Mar 2021 13:19:46 +0000

What is Webpack?

Webpack is a javascript module bundler which allows a developer to consolidate many files required to run an app into more manageable output files. This includes not only the JS files but also all required modules and dependencies. Essentially Webpack tells the builder how the app should be loaded via the configuration. It will take all the files and dependencies necessary to run the app and will “transform” them to create bundles which make the app more browser readable. Different bundlers or compilers have slightly varied ways of doing this but we will explore a brief overview of the core concepts of Webpack in this article.

What does it do?

“At its core, Webpack is a static module bundler for modern JavaScript applications.” Webpack processes and transforms the application, recursively building a “dependency graph” starting from the entry points designated by the config file. This creates a map of every module needed by the project and creates bundles (consolidated files) which are then loaded by the browser. This allows the browser to read files more efficiently and generally reduces the number of times the app has to wait for the browser to create a new request and return information. The Webpack docs lay out several core concepts integral to understanding and configuring Webpack’s operations. Entry, Output, Loaders, Plugins, Mode and Browser Compatibility.

Entry:

The entry point determines where the process of creating an “internal dependency graph” will begin. This will be established in the Webpack config.js file but if none are designated, the default will be ‘./src/index.js’. Another thing that Webpack does for us is to figure out what dependencies and other modules are needed by each file in order for the app to run successfully. It will figure this out for us during compilation.

Output:

Webpack creates bundles of files and then will output them into whatever output file is designated in the config file. The default is to output files to the ‘./dist folder’ with the main file having its own ‘./dist/main.js’. You must set the path and filename properties in the output object. The Path Node.js module should be imported at the top of the config file.

Loaders:

Loaders are additional customizations that allow Webpack to read and “transform” more than the default javascript and JSON files. There are two configured properties of loaders designated in the config file: Test and use. The test property indicates which files should be dealt with. The use property says which loader should be used to deal with that work. These properties will be designated in a rules array in a module object as seen below. The following example indicates that when the compiler comes across a txt file, it should use the ‘raw-loader’ to “transform it” before bundling it.

Plugins:

Plugins can be used to execute tasks outside of the loaders scope such as “bundle optimization, asset management and injection of environment variables.” There are built in plugins that some with Webpack out of the box but a developer may also customize plugins but requiring them in the file and adding them to the plugins array. Plugins can be used multiple times for different purposes, so in that case the new keyword should be used in the file such as the example below.

Mode:

The mode parameter can be set to ‘development’, ‘production’ or ‘none’ which will allow certain built-in customizations to run for each different environment. Production is the default mode, however there are more details about exact specifications for each mode in the docs.

Browser Compatibility:

Webpack includes browser support for all ES5 browsers.

What are the benefits and are there alternatives to using Webpack?

One benefit to using Webpack or a compiler in general is that it abstracts the complexity with consolidating and unifying necessary files to make the application run more efficiently in the browser. There are many options for bundlers and compilers and each will have slightly varied strengths and characteristics. The Webpack docs has a detailed comparison of six different bundlers, compared by feature.

In conclusion, Webpack is a very useful tool which runs on Node.js and creates a better experience for both the developer and ultimately the user experience of an application. With the advent of Node.js and the boom in use of node modules, Webpack is one solution for the problem of wrangling dependencies and supporting basic assets such as images, fonts and stylesheets, which otherwise might need to be specifically and individually managed. Webpack abstracts some of that process away from the developer and ultimately supports creating cleaner, easier to manage applications.

Sources:
https://webpack.js.org/
https://webpack.js.org/concepts/
https://webpack.js.org/comparison/
https://webpack.js.org/concepts/why-webpack/
https://medium.com/the-self-taught-programmer/what-is-webpack-and-why-should-i-care-part-1-introduction-ca4da7d0d8dc

Node Js File System

sstores — Sun, 07 Mar 2021 00:08:01 +0000

Javascript as a coding language was originally created to run in a browser. Node.js can allow javascript to run outside the browser, including on a web server. Node.js is an important tool for javascript and has many modules that hold additional functionality. Before we get into the Node.js File System module, let’s clarify some basics.

What is Node.js?

According to the Node js docs, “Node.js is an open-source, cross-platform, JavaScript runtime environment.” That means that Node.js allows us to run javascript outside of a browser. We can also think of it as a javascript interpreter. “As an asynchronous event-driven JavaScript runtime, Node.js is designed to build scalable network applications.” Node allows developers to do more than they would be able to do with just vanilla javascript and additionally allows for more add-on functionality including working on both the front-end and back-end of web applications.

What is a Node.js module?

Modules are analogous to javascript libraries, according to W3 School, ” a set of functions you want to include in your application.” The fs module is responsible for all the asynchronous or synchronous file operations. Explained well on the Turing School website, “While working in Node.js, we will encounter 3 different kinds of modules: internal modules that ship with Node, modules that we as developers create, and remote modules that we can download and use from the internet.” Node is a relatively small library because it is easy to customize and to add remote modules.

What is Node.js FileSystem?

FileSystem(fs) is an included internal module of Node.js. According to W3 School, “The Node.js file system module allows you to work with the file system on your computer.” It allows for asynchronous or synchronous access to file structures and is often also used in conjunction with the http Node module. To use fs you must require it in each file it will be used with the following line:

There are many methods in the fs module but some common ones are those to read, write, delete, append information to files in memory. We will go over some very common useful fs methods now:

fs.readFile()

The fs.readFile method allows you to asynchronously ready the entire contents of the given file. The first parameter is the path of the file to be read. The second parameter is an optional options object. The third parameter is a callback function which takes an error and data which is the contents of the file.

fs.writeFile()

The fs.writeFile method takes in four parameters, one of which is optional. The first is the file path of the file to be written, replacing the file if it already exists. The second parameter is the data to be written in the form of a string or a buffer (which we will discuss in just a moment). The third parameter is an optional options object. The last parameter is a callback which takes in an error.

fs.unlink()

The fs.unlink method is used to asynchronously remove a file or “symbolic link”. It takes in a file path and a callback similarly to those we’ve discussed before. This method will work on files but will not work on directories. To delete or unlink a directory you should use

fs.appendFile()

The fs.appendFile method asynchronously appends data to a file, creating the file if it does not already exist. The first parameter is the file path. The second parameter is the data to be appended in the form of a string or a buffer. The third parameter is an optional options object and the last is our callback which takes in an error.

Now, what is a buffer? A Buffer is a class in Node.js which is designed to handle and translate raw binary data. Buffers are usually used in the context of binary data from a stream. A stream is a way of handling data, potentially large amounts of data, and is used to read or write input into output sequentially. In other words, this allows data to be read in small chunks so the entire file can be processed continually with no interruption for the user, and does not need to be stored in memory. We can think of an example in Netflix and other streaming services. These process a huge amount of data and do not require a user to download all the data at once in order to utilize the service. This is just one of the benefits and uses of streams.

There are many more methods in the Node.js file system module than those listed above. The Node.js docs has categories for dealing with asynchronous, synchronous, callbacks, promises and more. FileSystem is a highly useful Node.js module and is worth some exploration.

Sources:

https://www.w3schools.com/nodejs/nodejs_filesystem.asp
https://www.tutorialsteacher.com/nodejs/nodejs-file-system
https://www.w3schools.com/nodejs/nodejs_modules.asp
https://nodejs.org/api/fs.html
https://nodesource.com/blog/understanding-streams-in-nodejs/
https://nodejs.org/en/knowledge/advanced/buffers/how-to-use-buffers/
https://teamtreehouse.com/library/what-is-nodejs
https://frontend.turing.io/lessons/

Cross Site Scripting: a contentious history

sstores — Mon, 01 Mar 2021 15:04:07 +0000

What is XSS?

“Cross-Site Scripting (XSS) attacks are a type of injection, in which malicious scripts are injected into otherwise benign and trusted websites.” -OWASP. This can happen when there are inputs from users that are unchecked. An attacker can inject a XSS attack by submitting valid code through user input forms, therefore altering the code of the program. There are different types of XSS attacks and strategies for how to protect against them. The categorization and description of different XSS has been a somewhat contentious point of discussion over the years and there are varied conventions and preferences for how to accurately refer to different types of XSS. As Ryan Oberfelder describes in his blog Describing XSS: The story hidden in time, “descriptions of XSS are living concepts, editable and expandable to uphold present understandings.” We will explore several of these distinctions in this blog.

Cross site scripting was first given this name in 1999 when a group of Microsoft security engineers recognized this type of attack, and later that year published an article naming it as Cross Site Scripting(XSS). The distinctions in cross site scripting at that point were Reflected and Stored. In 2005, DOM Based attacks were identified as a potential third sort of XSS. Later still the distinctions were a bit fuzzy and nowadays XSS can also be distinguished by Client vs Server side attacks.

REFLECTED XSS (Non-Persisted):

Reflected XSS is when malicious data from a user is sent to the server via a request and then is immediately returned as generated html with the malicious data included. For example, this could be in a hidden url including valid (but malicious) javascript code inside the url. When the malicious data gets back to the user via request and is loaded on the browser, it “weaponizes the injected code”. That code as an example could make another request containing user info to a site run by the malicious user to steal data about users. The following are examples from Oberfelder’s blog:
If this site,

utilizes a personal greeting with a link formatted like this,
a potential attack could look like this:

STORED XSS (Persisted):

Stored (or Persisted) XSS attacks are when the malicious data is stored on a server and then passed to every user who interacts with the site via HTTP response. In other words, “The payload [is] stored and injected over and over, affecting anyone who requested certain pages”, Oberfelder. If we again go back to our example links, what if that same site
has a Create New User functionality using this link .

This site would need to have input form for the user data. If an attackers for example made a username looking like this,

that again when stored to the server could pull the user data via cookie and send it directly to the attacker. Depending on where in the site this attack is targeted and if there is access to a list of usernames for example, it could affect many users. “The key difference between reflected and stored XSS is that a stored XSS vulnerability enables attacks that are self-contained within the application itself.” -PortSwigger. Stored XSS attacks can generally affect many more people than reflected attacks.

DOM BASED XSS:

Dom based XSS attacks occur when information from one part of the DOM is written to another part of the DOM. we’ll return to Oberfelder’s example. Perhaps our example site
has a footer element which is customized for the user and “appended dynamically”. If this link,
was sent to a user, it could again compromise the user’s info. This from the attackers perspective seems very much like a reflected attack but they are considered distinct by many because of the difference in strategy to prevent these attacks.

Server XSS vs Client XSS:

In 2012 the “research community” reorganized thoughts around how to describe the different cross site scripting attacks and came up with these distinctions. Server XSS are considered attacks where the problematic data is included in a HTTP response from the server. This could be reflected or could be stored data and therefor does not fit squarely into one of the original categories specified. “There are many different varieties of stored cross-site scripting. The location of the stored data within the application's response determines what type of payload is required to exploit it and might also affect the impact of the vulnerability.” -PortSwigger. Server vs Client distinctions deal more with where the origin of “payload” of the attack is stored.

How to Protect Against XSS Attacks:

The Open Web Application Security Project (OWASP) is a non-profit foundation, founded in 2001, dedicated to security improvements in software development.OWASP has detailed resources and “cheatsheets” detailing specific recommendations for protecting from specific types of attacks. However, they specify that generally a “White Listing” approach is generally a better more effective strategy than “Black Listing”. White listing is in essence, disallowing everything that is not specifically allowed. The top two rules that OWASP recommends developers to follow are the following:

Never Insert Untrusted Data Except in Allowed Locations
HTML Encode Before Inserting Untrusted Data into HTML Element Content

There are many more rules listed by OWASP that developers can explore to determine their security needs. Ultimately they recommend using a “security-focused encoding library” to ensure that the security rules are followed accurately. The technicality and implementation of the recommended rules will vary from situation to situation, but in short, a developer should limit the number of entry points that attackers can use, and protect those entry points using security libraries. Escaping alone is not enough to provide strong and stable security, however it can be a useful tool.

In conclusion, security can be a tricky business but especially when dealing with lots of user inputted data, it is a necessary concern.

Sources:
OWASP: https://owasp.org/www-community/attacks/xss/
https://owasp.org/www-community/Types_of_Cross-Site_Scripting
https://cheatsheetseries.owasp.org/cheatsheets/Cross_Site_Scripting_Prevention_Cheat_Sheet.html#rule-0-never-insert-untrusted-data-except-in-allowed-locations
PortSwigger: https://portswigger.net/web-security/cross-site-scripting/stored
Ryan Oberfelder, Describing XSS: https://medium.com/@ryoberfelder/describing-xss-the-story-hidden-in-time-80c3600ffe81

Hashing and why it is useful

sstores — Mon, 22 Feb 2021 03:10:24 +0000

What is Hashing?

Hashing is the process of creating a key that represents a piece of data in a given context. In other words, “Hashing is using some function or algorithm to map object data to some representative integer value” (FreeCodeCamp.com). Hashing can be used in many different contexts, some of which we will explore in this blog.

How does a hashing function work?

According to Bart Preneel in his article Mash Hash Functions (Modular Arithmetic Secure Hash), “A hash function is a cryptographic algorithm that takes input strings of arbitrary (or very large) length, and maps these to short fixed length output strings.” Essentially, hashing functions are one way to turn identifiable data into an anonymous key, or to create an index for storing a value in a data structure. There are many ways to implement a hashing function. It is important that the hashing function returns the same index for the same input every time it runs, or else the stored data would not be consistently accessible. When creating an index to store data, the produced hash must stay within the storage limit of the data structure.

There are a variety of hashing functions which have advantages for different uses. Three of the most common types of hashing functions are Arithmetic Modular, Truncation, and Folding, according to Jerry Ejonavi’s article Data Structures 101: Implement Hash Tables in JavaScript. These types of functions differ in the way by which they treat their input keys and specifically how they encode them. Below is a brief explanations of these three types from Ejonavi's blog:

Where can we use hashing?

Example 1: Hash Tables

Hash tables are part of the implementation of objects literals in javascript. Hash Tables use a hashing function to take in a key for an item, create an index for that item, which can then be used to store a value associated with the key. Given that the hashing function produces a consistent and reliable result for a given unique key, this can be a helpful tool for data storage.

Example 2: User Passwords Security

When a user is visiting a website, they may need to sign in to access certain usability of the site. In that case each user will have a username and password to provide security. When a user inputs their password that information will be passed to a server to provide access. However, this process introduces security concerns if passwords and user info were moved as plaintext. Hashing is a way to move that information between the client and server with much less risk to the user.

What is the difference between encryption and hashing?

According to William Jackson’s article Why Salted Hash Is As Good For Passwords As For Breakfast, “Encryption is a two-way function” which can be unlocked with the correct key. “Hashing, is a one-way function” that produces a unique index or id which if done well, cannot be decoded. For Additional security, a salt, or a random string of characters, can be added to the key before hashing to introduce an additional element of randomness, therefore producing a stronger hash. Hashing with salt is a common security strategy and helps to protect against malicious hackers using rainbow tables.

In conclusion, hashing has many applications and is an important concept and tool in program development. Hashing can provide accurate and secure data storage, and protect user data from attacks.

Sources:

https://www.freecodecamp.org/news/what-is-hashing/
https://www.educative.io/blog/data-strucutres-hash-table-javascript#function
https://docs.google.com/presentation/d/1RboEl1LSD9WutGQ_GiNKuecKLBFuOywEoZX5jo1QbGE/embed?slide=id.g4c07df579d_0_2414
https://link.springer.com/referenceworkentry/10.1007%2F0-387-23483-7_243#:~:text=MASH%2D1%20and%20MASH%2D2,short%20fixed%20length%20output%20strings.
https://gcn.com/articles/2013/12/02/hashing-vs-encryption.aspx
https://sectigostore.com/blog/hashing-vs-encryption-the-big-players-of-the-cyber-security-world/

Web Socket

sstores — Mon, 11 Jan 2021 06:40:36 +0000

What is a WebSocket?

“A WebSocket is a persistent connection between a client and server”, according to Kevin Sookocheff ‘s blog How Do Websockets Work. Usually, a web client requests a resource from a server with an http request, and the server responds with the requested information if it is available. This is a one-way connection also known as a “unidirectional protocol”. The unidirectional nature of http is a limitation that developers have found work arounds for, such as “long-polling”.

With HTTP long polling, the client polls the server requesting new information and the server holds the request open until new data is available. However, Long polling is more intensive on the server and can cause some issues with ordering of messages since multiple HTTP requests from the same client can be in process at the same time.

WebSockets are a less cumbersome fix for the limitations of http requests and builds on already existing protocols. A Web Socket uses an HTTP request as the initial “transport mechanism”, then uses a web socket “handshake” to keep the connection open. This allows for a two-way or “persisted” connection between client and server.

According to Kieran Kilbride-Singh’s blog Web Sockets Vs Long-Polling,
“WebSockets are a thin transport layer built on top of a device’s TCP/IP stack.” The TCP/IP stack is the Transmission Control Protocol which provides transport streams around the internet. WebSockets build on this existing network process. “The intent is to provide what is essentially an as-close-to-raw-as-possible TCP communication layer to web application developers while adding a few abstractions to eliminate certain friction that would otherwise exist concerning the way the web works.”

WebSockets came about in 2008 when developers Michael Carter and Ian Hickson became frustrated with existing capabilities in transporting robust programs and wanted to some up with a better solution. They launched WebSockets and Google Chrome 4 was the first browser to ship full support for WebSockets, in 2010.

So why do we love Web Sockets? Web Sockets allow for, “full-duplex asynchronous messaging. In other words, both the client and the server can stream messages to each other independently.” Additionally, WebSockets pass through most firewalls without any reconfiguration and do not use XMLHttpRequest, so headers are not sent every-time we need to get more information from the server. This reduces the “expensive data loads” being sent to the server.

How do we create a Web Socket?
First we make a new socket using 'ws' or 'wss' in the url:

Then when the socket is created we need to listen for events. There are four possible events, Open, Message, Error, and Close. The code might look like this:

When a new socket is created its starts connecting immediately. Browser headers should include Origin of client page, Connection: Upgrade, Upgrade: websocket, Sec-Websocket-Key (for security), and the current WebSocket version. The request headers might look like this:

If the server "agrees" to switch to WebSockets it will send a 101 response like this:

Once the connection is established, the client and server exchange data using the Web Sockets protocol. There are extensions and subprotocols that can be utilized with WebSockets as well which we won't get into here. But Web Sockets are a handy tool for more efficient messaging and asynchronous data transfer. Enjoy!

Sources:
https://sookocheff.com/post/networking/how-do-websockets-work/
https://www.pubnub.com/blog/http-long-polling/
https://www.ably.io/blog/websockets-vs-long-polling
https://en.wikipedia.org/wiki/Transmission_Control_Protocol
https://javascript.info/websocket

TDD and Why It's Important

sstores — Mon, 04 Jan 2021 14:13:39 +0000

What is TDD?

TDD stands for Test Driven Development. Typically when coding a program, a developer might think about the purpose of the program they are writing, think about what the code should do, write the code, then test if it works the way they expected. This is a more "traditional workflow" for software development. Test Driven Development is the process of deciding what you want your code to do, writing failing tests to specify functionality, then creating code to pass those tests. "In a nutshell, TDD changes our regular workflow" according to Peter Olson. By reversing the workflow in this way, and specifying the rules of the program's functionality first, a programmer will save a lot of time on debugging and ultimately write better code.

Why is TDD important?

Writing tests first can allow a developer to save time on debugging as well as many other benefits. According to Eric Elliott in his blog, TDD Changed My Life, "The point is not how long it takes to type this code. The point is how long it takes to debug it if something goes wrong." Note: Eric's blog is a great read which I recommend. (linked in sources below) He brings up several other good points and benefits to utilizing TDD rather than traditional development workflow. Utilizing TDD can allow easy separation of concerns, and testing sections or lines of an application separately, ultimately yielding better test coverage. Test Coverage refers to the amount of code covered by test cases. Having well-written effective tests before you code, can also help mitigate "fear of the merge button" or fear of change. If the code you've written essentially has the capability of testing itself before merging into production, there is less need to manually check and debug each piece before merging new code. If testing is well implemented it can be mostly automated and test coverage will be near 100%.

Types of Testing

Unit testing, refers to a self contained test or set of tests which focus on specific concepts or functionality. By writing unit tests for individual pieces of the program, debugging broken code becomes much easier to do. If a developer was only testing the application as a whole, utilizing End-to-End testing as an example, finding which pieces of code may or may not be broken would be very challenging. It is smart to utilize different testing methods to build an effective program. And allowing Tests to drive development can empower programmers to try new things, code with confidence, and build in checks and balances for merging new code.

Two types of testing to consider when talking about TDD would be Manual versus Automated testing. Manual Testing, according to Brandon Wozniewicz with Free Code Camp, is "the process of checking your application or code from the user's perspective." An example of this could be if you were building a YouTube look-alike, you might go to your rendered page in the browser and try using it like you would the actual YouTube site. If you are able to use the site as you expected then this manual test would be successful. Alternately, Automated Testing again according to Wozniewicz is "writing code that checks to see if other code works." In other words rather than relying on a user or yourself to test all the functionality of the YouTube look-alike, the criteria for the tests remain the same and allows for more and faster testing.

How To Write Tests

There are many tools and testing frameworks for utilizing TDD and other models like Behavior Driven Development. One non-tech analogy for TDD is the following example:

Using TDD does necessitate a deeper understanding of how the code should work and possibilities in development. In that way Elliott mentions, that there is a "learning curve" associated with using TDD well. Chai, Mocha, Jest, Jasmine and Karma are all common testing frameworks that allow a developer to more easily implement testing into their workflow. Each of theses has varying benefits and uses but ultimately, it is the responsibility of the developer to understand how to to implement tests into their program.

In conclusion, TDD requires some adjustment upfront from developers who are already used to coding from scratch. But by putting a bulk of the thinking work up front and preparing yourself as you go, ultimately it will yield better and tighter code.

Sources:
https://medium.com/javascript-scene/tdd-changed-my-life-5af0ce099f80
https://www.pluralsight.com/guides/introduction-to-test-driven-development-in-javascript
https://www.freecodecamp.org/news/an-introduction-to-test-driven-development-c4de6dce5c/
https://crossbrowsertesting.com/blog/test-automation/top-automation-frameworks-testers/
https://www.browserstack.com/guide/top-javascript-testing-frameworks

Using Snippets in DevTools

sstores — Mon, 21 Dec 2020 04:09:29 +0000

What is the Snippets Tool?

Snippets is part of the Google Chrome DevTools included on the Sources tab. It allows a developer to code, save, debug, and store pieces of code for repeated use. You can write your own snippets or find useful code snippets published online. Let’s look at how to access our snippets tool.

To access Snippets:

Open a Google Chrome tab and open the Chrome DevTools. You can do this by right clicking and selecting Inspect in the drop down menu, or use shortcut Command + Option + J on Mac or Ctrl + Shift + J on Windows.

Navigate to the Sources tab.

You can expand the navigation menu to the left by clicking here:

At the top of the Sources navigator, click the sideways arrows for more options and select Snippets.

Open a new Snippet or utilize an existing one.

Type your code and hit the play at the bottom to run your code.

Console.logs are helpful to see your code results in the console.

Use Cases for Snippets:

Great, we have this tool to use! So when might we actually utilize snippets? One use for snippets is when you want to save one or more bits of code that you might run in the console repeatedly during development. Rather than typing right into the console you can use snippets to store code snippets and run them multiple times.

Another advantage to using snippets, as Christian Nwamba says in his blog post 6 Snippets to Keep in Your Chrome DevTools, “when you run a snippet, it executes from the context of the currently open page”. (https://www.telerik.com/blogs/6-snippets-to-keep-in-your-chrome-devtools) So you can run a specific snippet that will give you information back about the current web page you are visiting or working on. Brian Grimstead is a developer that maintains a list of some helpful snippets which can be found here (https://bgrins.github.io/devtools-snippets/). Try some that may be useful for you and add them to your own library of snippets! One example of this is the allcolors.js. It returns all of the colors used on the page in a nice organized list in the console. Below you can see the list for twitter!

Other Characteristics of Snippets:

Gleb Bahmutov has a great blog post listing some features of Code Snippets which ive included here (https://glebbahmutov.com/blog/chrome-dev-tools-code-snippets/). To summarize a few, snippets can also be used to debug code from a larger application or program. You can use debugging tools like breakpoints on snippets just like you can with other Javascript code. Libraries available to the current page are also available in snippets. He lists some limitations as well including that you “cannot pass any arguments to a code snippet”, so the code may need to be modified to be useful. However, Snippets are an upgrade from tools like bookmarklets which allow developers to “quickly perform common web page tasks” according to the support.mozilla.org. Snippets have a similar functionality and are easier and more user friendly.

In conclusion, the snippets feature in Chrome DevTools is a great way to reuse bits of code to make your development life easier. Explore the possibilities and use your tools!

Sources:
devTools Documentation:
https://developers.google.com/web/tools/chrome-devtools/javascript/snippets
Other Sources:
https://briangrinstead.com/blog/devtools-snippets/
https://bgrins.github.io/devtools-snippets/
https://www.telerik.com/blogs/6-snippets-to-keep-in-your-chrome-devtools
https://glebbahmutov.com/blog/chrome-dev-tools-code-snippets/