DEV Community: ErrorGamer2000

Binary File Formats Explained

ErrorGamer2000 — Thu, 11 Aug 2022 17:38:00 +0000

When I first began researching binary file formats, I was met with a complete absence of human-friendly explanations anywhere online. All of the resources that I came across were full of unfamiliar technical terms that made me feel like I was reading a textbook written for someone with twice my IQ. Hence, this article. This is a summary of my research, explained in English so that you can understand it without the trouble that I had. Now, without further delays, let's get into the learning!

All of Those Confusing Technical Terms

As I said, there were a number of technical terms that I encountered in my research that I had never heard of before. these made little sense to me at the time, and took even more painful research. Here is a list of those terms, explained in friendly ways:

Binary - binary is a number system that has a base of 2. That means that the only digits it uses are 0 and 1.
Bit - a bit is the smallest unit of data in a computer, consisting of nothing more that a single binary digit (a 0 or 1).
Byte - a byte is the next unit of data in a computer. A byte consists of 8 individual bits.
Signed Integer - a signed integer is an integer (whole number) that is associated with a sign that declares whether the number is positive or negative. Basically, it is an integer with a + or - attached to it.
Unsigned integer - an unsigned integer is an integer that is not associated with a sign(+ or -), and is always considered to be a positive value.

What exactly is a file, anyway?

A file is a collection of bits that is stored within a computer's memory. Files are generally separated into bytes, and measured by the number of bytes that they contain (Kilo*bytes*, Mega*bytes*, Giga*bytes*, etc.).

Data Types

Before we can truly begin working with binary file encoding, we need to understand the types of data that can be stored within a file. there are two main types of data in a file: integers and strings.

Integers

Integers are separated into two sub-types, signed and unsigned, as I explained at the beginning of the article.

Unsigned Integers, thanks to the lack of a sign, can store a number with roughly twice the maximum value than that of a signed integer.

Note: unsigned integer is abbreviated as Uint in almost every programming situation.

Signed Integers(Ints) are pretty much exactly the same, aside from their reduced maximum value.

Both Uints and Ints are found in different sizes in binary files, and are generally named based on how many bits they use to store their value. For example, Uint8 and Int8 both use 8 bits in the file, Uint16 and Int16 use 16 bits, and so on. These numbers have different value limits based on the number of bits that they use:

Type	Minimum Value	Maximum Value
`Uint8`	0	255
`Int8`	-128	127
`Uint16`	0	65,535
`Int16`	-32,768	32,767
`Uint32`	0	4,294,967,295
`Int32`	-2,147,483,648	2,147,483,647
`Uint64`	0	18,446,744,073,709,551,615
`Int64`	-9,223,372,036,854,775,808	9,223,372,036,854,775,807

These are not the only sizes that can be used, however, and you may sometimes encounter some oddball sizes. Here are a couple that I ran into:

Type	Minimum Value	Maximum Value
`Uint4`	0	15
`Uint24`	0	8,388,608

Strings

Strings, as you may already know, are ordered sets of text characters. When a string is stored in a binary file, it is converted into a set of bytes, with each byte storing one utf-8 character in each byte as a Uint8 character ID.

Endianness

Endianness is one of the more difficult concepts to grasp, so I'm going to take an example-based approach here. Let's say I have a variable that is a Uint16 with the value of 255. In order to store this variable in a binary file, if must first be converted into a set of bytes. Because our variable is a Uint16, taking up 16 bits of space, it requires two bytes in the file to store. But ... Which of those two bytes comes first in the file? This is where endianness comes into play.

High and Low Bytes

Let's convert our Uint16 variable into binary, making sure to keep 16 digits: 0000000011111111. Notice how the number is split evenly between 0s and 1s. Each of those groups is a single byte. The byte on the left (the 0s) is the high byte, meaning it comes first in the normal order of the value. The byte on the right (the 1s) is called the low byte. As you go across the bytes from left to right, the bytes go from higher to lower "order". So, in this number, the left byte is of a higher order than the right byte.

Now, there are two types of "endianness" that a number can be encoded in: little endian and big endian format. A number that is encoded in little endian format will have the bytes ordered from highest to lowest, meaning that the lowest("littlest") byte comes at the end. Numbers encoded in big endian format are ordered in the reverse of little endian numbers, meaning that the highest("bigest") byte comes at the end instead.

Returning to our Uint16 variable, here is what the number will look like when in the file:

Endianness	Byte 0	Byte 1
little	00000000	11111111
big	11111111	00000000

Generic File Format

From here on, we can only discuss the general patter of binary file formats, so keep in mind any file format you see does not need to adhere to what is discussed here.

In most binary files, the data is generally split into two main types of sections: the file header and binary data blocks.

File Headers

A header in a binary file is generally a collection of bytes that have fixed positions at the beginning of the file. A file header typically contains data about the version of the file format that the file was encoded with along with some other data about the contents of the file. Headers do not, however, have to be the same length in every file of the same type. Sometimes, the header needs to contain data that has a variable length, such as a string that is provided by the program. Strings are not always the same length, so in order to accomodate for this, headers will typically define the total length of the header and a constant position for the string to start. Any program that parses the file will then start reading the string at the constant index and continue untill it has read the full length of the header.

Data Blocks

The rest of a file, after the header, is generally devoted to data blocks. Each data block can be either a fixed or variable size, and will commonly have it's own header that tells the program that is parsing the file how to use the data inside the block, as well as the lenght of the block if it is a variable size.

Hope that I have successfully explained how binary file formatting works. If you have any questions or see an issue with a part of this article, feel free to drop a comment below and I will fix it as soon as I can. Happy Hacking!

Feeling Generous?

I am a 17 year old, self-taught, web developer trying to make a living while stuck with oppressive parents, and trying to find a way to pay for college while not being allowed to have a job. I would appreciate any donations.

Not Going to Donate?

Even if you don't donate, simply liking this post or sharing it with anyone that might find it useful is a huge help.

Adding Recent Blog Posts to Your GitHub Readme

ErrorGamer2000 — Mon, 04 Apr 2022 00:13:53 +0000

This is my first post here. I am still learning, so If you see any issues with this post please let me know so that I can fix them.

GitHub is one of the most popular git repository websites, hosting a countless number of open source projects. Over the years, GitHub has grown into a diverse community, full of great code created by both solo web development hobbyists like myself and professional teams working for big companies like Facebook.

Hi there. 👋

I'm ErrorGamer2000 (I may reveal my real name in the future, but for now I'll remain hidden behind this pseudonym). I am a solo web developer working my way through my junior year in high school, and I spent the majority of my "virtual school" year during the COVID-19 pandemic pretty much ignoring my classes, using the time to instead teach myself web development. For my first blog post, I want to share my first REAL JavaScript (well, really, it's TypeScript) project with you.

Okay, we get it, back to the post already!

If you have used GitHub before, which I'm sure you have, you have most undoubtedly come across a profile readme before. While GitHub allows some level of profile personalization out-of-the-box, the profile readme feature allows you to add any content you want to your profile. Take my profile for example. As you can see, I have many different elements that are not a part of the normal GitHub profile.

Since these readme files provide your potential followers, customers, or even potential employers with a quick overview of both your activity and your ability, you want to put as much relevant content as you can on your readme. Just don't make it too long, or nobody will take the time to read it 😉.

Now, since you want your customers or employers to see the full extent of your skills, there are many tidbits of information that would be perfect to put on your profile, including any blog posts that you have made. If you are a blogger as well as a developer, then this is the perfect article for you (Even if you aren't feel free to keep reading, I don't mind).

As I planned to start blogging, I stumbled across this idea as I myself was looking to start a blogging presence of my own. As I looked around, I was able to only find one easy-to-use GitHub action that would allow me to add a list of my posts to my readme. Sadly, I was disappointed to discover that this action had little formatting, only setting up the posts as a bulleted list of links, which was less than appealing to the eye.

Alright, I'm lost. What is this post really about?

Okay, okay. I get it. A little much background, right? So, to the point. Unable to find a suitable solution, I began making one myself.

The Action itself

I figure I've kept you long enough by now. This action is extremely simple to use, all you need to do is give it a simple URL, and it does the rest.

If you want to go ahead and see what it looks like, go ahead and look at it on the GitHub Actions Marketplace.

Setup

As I said, this action is very easy to use, and there is minimal setup. Let's go ahead and start with a basic workflow file:

name: Generate Blog List

on:
  push:
  schedule:
    - cron: "0 */1 * * *"
jobs:
  blog:
    runs-on: ubuntu-latest
    name: Fetch and Generate Blog Posts
    steps:

This workflow, as you may be able to see, runs every time you push new code to the repository (so, if you update the readme it will automatically replace the old blog posts in case you changed the configuration of the action) as well as on a set schedule of every hour.

Now, in order for this action to be able to work, the action needs to have the ability to read your readme, so let's go ahead and add a checkout step to start with. Add a new step that consists simply of a uses property:

    steps:
      - uses: actions/checkout@v3

This will ensure that the repository is fully loaded for the action to run in. Now that we are about ready to actually add the step to load our posts, there is one last thing that we need: our RSS feed. Now the URL to your RSS feed will depend on the site that you are blogging on. For dev.to, it is https://dev.to/feed/{username}, where you replace {username} with your username. For example, my RSS feed URL here would be https://dev.to/feed/errorgamer2000.

Configuring the Action

Now we are finally ready to set up the action. Let's add another step to our workflow, but we will give this one a name.

    steps:
      - uses: actions/checkout@v3
      - name: GitHub Readme Blog Post Action
        uses: ErrorGamer2000/github-readme-blog-post-action@v1
        with:
          feed_urls: "{rss_feed}"

Now, just replace the {rss_feed} with the URL to your RSS feed. You may have noticed that the option we are using is called feed_urls rather than feed_url. Plural, not singular. That's right: you can give multiple URLs for different feeds, each of them separated by a comma.

Final configuration

Almost done, hang in there!

Now that we have added this action to our workflow, the action will automatically create all of the images it needs and save them to the folder blog-post-list-output. Go ahead. Save the file, commit it, and (if you are running a locally cloned repo) push your changes. Head over to the GitHub website for your repository and open the actions tab. You should see your brand new Generate Blog List action showing up there, with a little orange circle next to it. Wait until the action has finished (the circle will become a green checkmark) and open up the readme.md file.

Uhhh, nothing happened. There isn't even the blog-post-list-output folder...

Well, of course nothing happened. The action did make the changes to the repo, but we have not set up anything to actually save those changes. To do that, we need one final step: a git commit step. If you are handy with the git CLI, go ahead and do this yourself. I, however, prefer to use stefanzweifel/git-auto-commit-action. So, let's add another step to the workflow:

    steps:
      - uses: actions/checkout@v3
      - name: GitHub Readme Blog Post Action
        uses: ErrorGamer2000/github-readme-blog-post-action@v1
        with:
          feed_urls: "{rss_feed}"
      - name: Commit changed files
        uses: stefanzweifel/git-auto-commit-action@v4
        with:
          commit_message: Save Generated Blog Posts
          skip_checkout: true

This will save the changed files. Go ahead and try this out again, committing the changed workflow file and head over to the Actions tab again. Once your workflow has finished running, head back over to the readme.

Ugh, not again. Still no posts on the readme. But we are making progress: the new folder is there and everything.

Well, yes, there is nothing in the readme yet. But our workflow is complete and running smoothly, we simply have not told it where to put the list of posts.

Telling the Action Where to Put the Posts

Now that our workflow is *work*ing 😏, all that is left is to actually make it put stuff into the readme. to do that, just slap a couple of comments in it somewhere, like so:

<!-- blog-post-list:start -->
<!-- blog-post-list:end -->

Commit the changes again, wait for the workflow to finish, and then check the readme.

OMG IT WORKED!!!!!!!

Yes, congrats, you did it! you now have a working, automatically updated, list of posts on your readme! Have fun!🤗

Customizing the Action

There are a number of different options that you can use with the action:

Note: This table will most likely become out of date as I update the action. Head over to the GitHub Repository to see an up-to-date table.

Option Name	Type	Default Value	Description
`feed_urls`	`string`	`""`	A (comma-seperated) list of RSS feed URLs to load posts from. This is the only required input.
`max_posts_per_url`	`number`	`5`	The maximum number of posts to show for each feed. If the number of posts is less than this, then all of the posts will be shown.
`position_indicator`	`string`	`blog-post-list`	The text of the comments that the action uses to inject the images into the README file. Everything between the two comments in the form `<!-- position_indicator:start -->` and `<!-- position_indicator:end -->` is replaced. Changing this can allow you to use multiple configurations for different feeds by using different markers for each.
`show_feed_data`	`boolean`	`true`	Whether or not to show the generated markdown describing the feed, which includes the title of the feed, the description of the feed, the `Read More` link, the last updated date, and the post count. Each of these can also be individually toggled with the following options. This will override any of the specific options, so it is better to disable/enable them specifically if you want to remove some elements.
`show_feed_title`	`boolean`	`true`	Whether or not to show the header containing the title of the feed. This will be formatted as an `h2` header. An option to customize this header will be in a future update.
`show_feed_description`	`boolean`	`true`	Whether or not to show the title of the feed that is provided by the RSS feed.
`show_read_more`	`boolean`	`true`	Whether or not to show the `Read More` link under the feed description.
`show_last_updated_date`	`boolean`	`true`	Whether or not to show the date and time of the last update of the list.
`show_post_count`	`boolean`	`true`	Whether or not to show the number of posts shown and the total number of posts provided by the RSS feed.
`show_post_date`	`boolean`	`true`	Whether or not to show the date of each post on the card.
`locale`	`string`	`"en"`	The locale of the project. This is used purely for formatting the dates of the cards and last update.
`time_zone`	`string`	`"UTC"`	A valid time zone to use for the last updated date.
`output_dir`	`string`	`"blog-post-list-output"`	The directory to store the post card images in. Must be in the root directory (i.e. no path separators `/` or `\`) and cannot include the characters `/\?%*:\|"<>`.

How It Works

This action does a little more than just parsing a simple RSS feed. It does start with this, however, and it uses this data as a backup for missing information. What it does do is use the link that it receives from the feed to fetch the HTML for the webpage of the blog, and then uses The Open Graph protocol to find the preview image for the post as well as the title and description. Any of this missing information is taken from the RSS feed, if it is there. If an image is not found for the post, then it is removed from the card.

Happy coding! Please let me know how you feel about this post, and anything else you think I should add in the future!