DEV Community: Ian Jacobs

Automatic Function Redirecting: Navigating Dynamic Code Optimization in GCC

Ian Jacobs — Wed, 29 Nov 2023 17:44:09 +0000

Intro

In this blog post, I will be explaining the usage of indirect functions (or ifunc) and how they work to optimize code for the AARCH64 platform. This is a project for my software portability and optimization class where I am going to present a design implementation for an ifunc option when compiling with GCC.

PART-1: ifunc and its Intermediate Representation

To test ifunc usage in a program I used an "autoifunc" script that creates a function redirect scheme for ARM64 scalar vectorization instructions (SVE) and regular ARM Advanced SIMD instructions (ASIMD) before being compiled by GCC.

This script checks if a c-source file is vectorizable while using either SVE or SVE2. It then rewrites the functions for SVE and ASMID and creates a resolver function for processing at run time.

For testing this script I've used a program that adjusts the RGB values for a provided image which demonstrates this effect with one of the separated functions (in its own C file, referred to as function.c).

/*

adjust_channels :: adjust red/green/blue colour channels in an image

The function returns an adjusted image in the original location.

Copyright (C)2022 Seneca College of Applied Arts and Technology
Written by Chris Tyler
Distributed under the terms of the GNU GPL v2

*/

#include <stdio.h>
#include <stdlib.h>
#include <stdint.h>

//----Naive implementation in C

#include <sys/param.h>

void adjust_channels(unsigned char *image, int x_size, int y_size,
    float red_factor, float green_factor, float blue_factor) {

    for (int i = 0; i < x_size * y_size * 3; i += 3) {
        image[i]   = MIN((float)image[i]   * red_factor,   255);
        image[i+1] = MIN((float)image[i+1] * blue_factor,  255);
        image[i+2] = MIN((float)image[i+2] * green_factor, 255);
    }
}

After running the program through the script it produces a descriptive/verbose explanation of its function redirecting process and is similar to what was described earlier.

scripts/autoifunc function.c

* Auto-ifunc Tool v 0.001
> Input file: 'function.c'
> Input file does appear to contain C source.
> Vectorization using SVE/SVE2 is being applied.
> SVE2 optimizations are basically the same as the SVE optimizations, skipping.
> Compiling input file to assembly to get function names.
> Function #1: adjust_channels
> Writing output file.
> Output file function_ifunc.c has been created. Use this in place of function.c in your buld.

Here is the function.c (now function_ifunc.c) file after being run through the autoifunc script:

#include <sys/auxv.h>
#include <stdio.h>

void adjust_channels(unsigned char *image,int x_size,int y_size,
float red_factor,float green_factor,float blue_factor) _    
_attribute__(( ifunc("adjust_channels__resolver") ));

#pragma GCC target "arch=armv8-a+sve"
/*

adjust_channels__sve :: adjust red/green/blue colour 
channels in an image

The function returns an adjusted image in the original location.

Copyright (C)2022 Seneca College of Applied Arts and Technology
Written by Chris Tyler
Distributed under the terms of the GNU GPL v2

*/

#include <stdio.h>
#include <stdlib.h>
#include <stdint.h>

// -------------------------------------Naive implementation in C

#include <sys/param.h>

void adjust_channels__sve(unsigned char *image, int x_size, int y_size,
    float red_factor, float green_factor, float blue_factor) {

    for (int i = 0; i < x_size * y_size * 3; i += 3) {
            image[i]   = MIN((float)image[i]   * red_factor,   255);
            image[i+1] = MIN((float)image[i+1] * blue_factor,  255);
            image[i+2] = MIN((float)image[i+2] * green_factor, 255);
    }
}


#pragma GCC target "arch=armv8-a"
/*

adjust_channels__asimd :: adjust red/green/blue colour 
channels in an image

The function returns an adjusted image in the original location.

Copyright (C)2022 Seneca College of Applied Arts and Technology
Written by Chris Tyler
Distributed under the terms of the GNU GPL v2

*/

#include <stdio.h>
#include <stdlib.h>
#include <stdint.h>

// ----------------------------------Naive implementation in C

#include <sys/param.h>

void adjust_channels__asimd(unsigned char *image, int x_size, int y_size,
    float red_factor, float green_factor, float blue_factor) {

    for (int i = 0; i < x_size * y_size * 3; i += 3) {
            image[i]   = MIN((float)image[i]   * red_factor,   255);
            image[i+1] = MIN((float)image[i+1] * blue_factor,  255);
            image[i+2] = MIN((float)image[i+2] * green_factor, 255);
    }
}

static void (*adjust_channels__resolver(void)) {
        long hwcaps  = getauxval(AT_HWCAP);
        long hwcaps2 = getauxval(AT_HWCAP2);


 if (hwcaps & HWCAP_SVE) {
                return adjust_channels__sve;
        } else {
                return adjust_channels__asimd;
        }
};

As can be seen, the function has been broken down into 3 different components including a redirect function with an attribute value for a resolver function below:

void adjust_channels(unsigned char *image,int x_size,int y_size,
 float red_factor,float green_factor,float blue_factor) _    
 _attribute__(( ifunc("adjust_channels__resolver") ));

Next are the 2 remade functions for each variant's optimization where each function's name has been updated to indicate which method is being used for that implementation and to provide the resolver function with a way to call the specific remade function.

void adjust_channels__sve(unsigned char *image, int x_size, int y_size,
    float red_factor, float green_factor, float blue_factor) {

and

void adjust_channels__asimd(unsigned char *image, int x_size, int y_size,
    float red_factor, float green_factor, float blue_factor) {

In addition, there is the resolver function handling at the bottom which seems to get the current optimization strategy and choose the correct version of the function to work with.

static void (*adjust_channels__resolver(void)) {
        long hwcaps  = getauxval(AT_HWCAP);
        long hwcaps2 = getauxval(AT_HWCAP2);


 if (hwcaps & HWCAP_SVE) {
                return adjust_channels__sve;
        } else {
                return adjust_channels__asimd;
        }
};

This process is essentially what redirect functioning (ifunc) is doing.

To analyze the compilation process for this I then compiled the new function_ifunc.c into an intermediate representation using GIMPLE which dumps the compiler's intermediate interpretation of the code in a readable format that the compiler will then do optimizations on.

To do so I used the compiler option -fdump-tree-gimple with GCC to get the intermediate code.

The differences between the two different indirect functions are minimal and come down to styling and variable names.

A snippet of the resolver function

_attribute__((target ("arch=armv8-a+sve", "arch=armv8-a")))
void * adjust_channels__resolver ()
{
  void * D.5656;
  long int hwcaps;
  long int hwcaps2;

  # DEBUG BEGIN_STMT
  _1 = getauxval (16);
  hwcaps = (long int) _1;
  # DEBUG BEGIN_STMT
  _2 = getauxval (26);
  hwcaps2 = (long int) _2;
  # DEBUG BEGIN_STMT
  _3 = hwcaps & 4194304;
  if (_3 != 0) goto <D.5654>; else goto <D.5655>;
  <D.5654>:
  ...

As can be seen, the resolver function has morphed into this new intermediary function that handles each runtime compilation with the appropriate function just as it was described in the previous explanation.

Snippet from __sve function

__attribute__((target ("arch=armv8-a+sve")))
void adjust_channels__sve (unsigned char * image, int x_size, int y_size, float red_factor, float green_factor, float blue_factor)
{
  unsigned char iftmp.0;
  unsigned char iftmp.1;
  unsigned char iftmp.2;

  # DEBUG BEGIN_STMT
  {
    int i;

    # DEBUG BEGIN_STMT
    i = 0;
    goto <D.5634>;
    <D.5633>:
    # DEBUG BEGIN_STMT
    _1 = (sizetype) i;
    _2 = image + _1;
    _3 = *_2;
    _4 = (float) _3;
    _5 = red_factor * _4;
    if (_5 < 2.55e+2) goto <D.5659>; else goto <D.5660>;
    <D.5659>:
    _6 = (sizetype) i;
    _7 = image + _6;
    _8 = *_7;
    _9 = (float) _8;
    _10 = red_factor * _9;
    iftmp.0 = (unsigned char) _10;
    goto <D.5661>;
    <D.5660>:
    iftmp.0 = 255;
    <D.5661>:
    _11 = (sizetype) i;
    _12 = image + _11;
    *_12 = iftmp.0;
    # DEBUG BEGIN_STMT
    _13 = (sizetype) i;
    _14 = _13 + 1;
    _15 = image + _14;
    _16 = *_15;
    _17 = (float) _16;
    _18 = blue_factor * _17;
    if (_18 < 2.55e+2) goto <D.5663>; else goto <D.5664>;
    <D.5663>:
    _19 = (sizetype) i;
    _20 = _19 + 1;
    _21 = image + _20;
    _22 = *_21;
    _23 = (float) _22;
    _24 = blue_factor * _23;
    iftmp.1 = (unsigned char) _24;
    goto <D.5665>;
    <D.5664>:
    iftmp.1 = 255;
    <D.5665>:
    _25 = (sizetype) i;
    _26 = _25 + 1;
    _27 = image + _26;
    *_27 = iftmp.1;
    ...

Above is the __sve function that includes the _atttribute(target("arch=armv8-a+sve")) at the function declaration that was the #pragma GCC target "arch=armv8-a+sve" from the original. The code now is meant to be a readable explanation before optimizations are made.

Snippet from __asmid function:

__attribute__((target ("arch=armv8-a+sve")))
void adjust_channels__sve (unsigned char * image, int x_size, int y_size, float red_factor, float green_factor, float blue_factor)
{
  unsigned char iftmp.0;
  unsigned char iftmp.1;
  unsigned char iftmp.2;

  # DEBUG BEGIN_STMT
  {
    int i;

    # DEBUG BEGIN_STMT
    i = 0;
    goto <D.5634>;
    <D.5633>:
    # DEBUG BEGIN_STMT
    _1 = (sizetype) i;
    _2 = image + _1;
    _3 = *_2;
    _4 = (float) _3;
    _5 = red_factor * _4;
    if (_5 < 2.55e+2) goto <D.5659>; else goto <D.5660>;
    <D.5659>:
    _6 = (sizetype) i;
    _7 = image + _6;
    _8 = *_7;
    _9 = (float) _8;
    _10 = red_factor * _9;
    iftmp.0 = (unsigned char) _10;
    goto <D.5661>;
    <D.5660>:
    iftmp.0 = 255;
    <D.5661>:
    _11 = (sizetype) i;
    _12 = image + _11;
    *_12 = iftmp.0;
    # DEBUG BEGIN_STMT
    _13 = (sizetype) i;
    _14 = _13 + 1;
    _15 = image + _14;
    _16 = *_15;
    _17 = (float) _16;
    _18 = blue_factor * _17;
    if (_18 < 2.55e+2) goto <D.5663>; else goto <D.5664>;
    <D.5663>:
    _19 = (sizetype) i;
    _20 = _19 + 1;
    _21 = image + _20;
    _22 = *_21;
    _23 = (float) _22;
    _24 = blue_factor * _23;
    iftmp.1 = (unsigned char) _24;
    goto <D.5665>;
    <D.5664>:
    iftmp.1 = 255;
    <D.5665>:
    _25 = (sizetype) i;
    _26 = _25 + 1;
    _27 = image + _26;
    *_27 = iftmp.1;
    ...

Just like the previous function, it includes the target attribute and looks very similar to the other GIMPLE example except for the different function/location calls.

It seems that the GIMPLE code at this point hasn't gone through any vectorization or any optimizations as the code still contains loops (in the form of goto) and scalar operations, as well as math operations which vectorization would attempt to optimize at the highest optimization level meaning that this current implementation in GIMPLE doesn't apply vectorization yet.

If we were to get the GIMPLE representation of function.c it would look like just one of the two different functions and thus it can be seen that the compiler at this stage is performing Link Time Optimizations

So with that, the process of converting function.c into function_ifunc.c is the following:

Choose a specific architecture variant and extension to use optimizations from.
Decide which parts of a program are vectorizable and determine which functions should be remade into an ifunc variant.
Create named versions of each of the chose functions for different architecture variant optimizations.
Create a resolver function that uses the correct function during runtime.
Create a prototype "driver" function that the resolver function can load the correct auto-vectorization function into (the original function as a prototype in function_ifunc.c).

These steps are how indirect functions are created and are important to note for the next part of this blog where I talk about a theoretical design for an ifunc feature in GCC.

PART 2: Designing ifunc for GCC

For the next part of this blog, I want to discuss how an auto-ifunc feature could be added to GCC to be usable by any user for compilation.

Command Line Argument

Like any of the many commands that can be used in GCC enabling it would come in the form of a command line argument. For example, the option could take the name -fauto-ifunc. The 'f' in the function is used for other arguments in GCC to denote that whatever follows it is a flag (more info here)

Using this flag would then enable the automatic ifunc capability during compilation and looks like a good name for the feature to be called from GCC.

Specifying Architecture

To facilitate this the target architecture could be specified, in a -march=[base]+[ext1]+[ext2]... where m at the start, in this case, is the mode flag used in other GCC flags that deal with mode (again more info about -m in GCC can be found here).

For example, the argument -march=armv8-a+sve would be used to specify aarch64 armv8-a type architecture variant with the SVE extension.

The base architecture in this case would need to be mandatory but the additional extensions can be chosen to be added.

With this implementation, there will need to be a lot of constraints that need to be added for the architecture variants for the resulting program to be runnable.

These constraints would include:

1. Compatibility:

Making sure that these architecture variants are compatible with each other so that GCC doesn't combine incompatible features or instructions into a single executable.
Handling mutually exclusive variants (32-bit vs 64-bit)
If specifying specific instruction set extensions (e.g., SIMD), ensure that the selected extensions are compatible with each other.
Making sure that the compiler version supports the specified architecture variant

2. Consistency:

Ensure that the selected variants share common features so that there is no unpredictable behavior.
Library Consistency in the program by checking if the libraries that are used in the program are compatible with the specified architecture variants.
If the compilation is for a different architecture than the host, ensure that cross-compilation is set up correctly, and the target architecture is supported.

These are some of the main constraints that would need to be applied for the architecture variants. There are more that would need to be handled including things like runtime detection, but with proper testing and documentation in the source code, I think that this is the best solution if the constraints can all be accounted for.

Specifying Functions

The auto-ifunc script demonstrated at the beginning of the blog shows that the script works on a per-function basis. With this in mind when adding this to GCC the names of the functions in which automatic ifunc capability should be applied.

I believe that a user should have a choice as to whether they should specify specific functions or have the auto-ifunc automatically detect functions.

User-specified Functions:

This could come in the form of another extended initial argument -fauto-ifunc-functions=[function1],[function2],....

For example from part 1: -fauto-ifunc-functions=adjust_channels

In the source code, these specific functions to be targeted will be added as #pragma statements.

This flag would be optional and if not provided the -fauto-ifunc will need to auto-select functions based on specified selection criteria provided by the user.

Automatic Function Selection & Specifying Selection Criteria

This option would allow a user to specify which criteria should be used when automatically selecting functions. For example, using the criteria terms that could be specified would be: "vectorizable", "complexity", or "execution-frequency". would have the script check each function for that compatibility.

To add specific criteria you could specify another default argument -fauto-ifunc-criteria=[criteria].

An example from part 1 would be -fauto-ifunc-criteria=vectorizable.

This argument would be optional and have a default value that would be applied if not used. This default value should be a balance between the main criteria; targeting functions that benefit from multiple implementations based on the different architectures provided (either by default or through the -march=[arch] mode flag).

Allowing for both user-specified and automatically selected functions would allow for more flexibility and accessibility to both advanced and novice users which in my opinion is the most effective when adding this option to GCC.

Diagnostics

When running the -fauto-ifunc option in GCC the user should be fully informed of the selected architecture and its variants, the name of each function that will be processed, and any potential issues that arise in attempting to run this command.

An option should be included to display a more verbose diagnostic message (something like -fauto-ifunc-verbose) and forgoing them completely (with something like -fauto-ifunc-quiet)

Conclusion

I believe that the process I have presented here if implemented as described would be a good way to add automatic indirect function creation to GCC as it goes through the constraints and user-level control over how the option could be used seamlessly with GCC.

There exist many more considerations that need to be reviewed including its many interactions with existing GCC options and its effect on the source code when used but I believe the current theoretical implementation works well for this feature.

Building GCC on Different Architectures: A Study of Cross-Compilation

Ian Jacobs — Mon, 30 Oct 2023 01:56:09 +0000

In this blog post, for my software optimization and portability course, I will go over the process of installing GCC on two distinct architectures x86_64 and AArch64. I will go over my process of installation for each and show them working as intended.

I worked on two Linux environments with x86 and AArch64 architecture specifically.

Building GCC for x86_64

Obtaining the Source Code

First, I cloned the GCC git repo from https://gcc.gnu.org/git/gcc.git.

$ git clone https://gcc.gnu.org/git/gcc.git

This step took a couple of minutes as there were a lot of individual files to download

Creating a Build Directory:

I then created a new directory outside the source directory where I would have GCC installed and gave it an appropriate name.

$ mkdir x86_64_gcc_install
$ cd x86_64_gcc_install

Configuring GCC:

Before I could make the GCC installation in this new directory I needed to configure GCC using the configure script in the git repo which would also configure GCC for the current architecture.

$ cd x86_64_gcc_install 
$ $PWD/../gcc/configure --prefix=$HOME/spo600/x86_64_gcc_install/

I set the configure file from within the install directory and also included the prefix argument to tell the configure script where I want GCC to be installed

Building GCC:

The makefile was used with the -j 4 argument, which set the amount of CPU cores during the installation to 4.

$ make -j 4

The make process for GCC on x86_64 architecture took almost 2 hours with four parallel jobs.

After this, the makefile and configuration options were added to the install directory. To finish the installation and create the working program I installed the binaries to the x86_64_gcc_install/bin directory:

$ make install

Building GCC for AArch64

I began building GCC on the other environment with AArch64 architecture instead of x86_64

Initial setup

The process for obtaining the source code and creating the install directory was the same

$ git clone https://gcc.gnu.org/git/gcc.git

Again, this step took a couple of minutes as there were a lot of individual files to download

$ mkdir aarch64_gcc_install
$ cd aarch64_gcc_install

Configuring GCC:

Once again, configuring GCC using the configure script in the git repo which would also configure GCC for the current architecture with the prefix argument for configure location.

$ cd x86_64_gcc_install 
$ $PWD/../gcc/configure --prefix=$HOME/spo600/x86_64_gcc_install/

After this, the makefile and configuration options were added to the install directory

Building GCC:

I then used 5 CPU cores with the -j 5 option to see if there was any difference in speed.

$ make -j 5

With this, it took just under 2 hours and was marginally faster as I had added more multiple cores.

I then as with x86, installed binaries to aarch64_gcc_install/bin:

$ make install

Confirming installation

For each installation, the runnable programs were available in the install_dir/bin directory and to show that GCC was installed for the specific architecture I ran the architecture-specific program (usually including the x86/AArch64 name). For example here was the x86_64 bin folder:

To confirm that the newly built GCC compilers were fully functional, I created a Hello World C program in each environment which had the following structure:

#include <stdio.h>
int main(){

        printf("Hello x86_64!\n");

        return 0;
}

The x86_64 part is replaced with AArch64 in its version

Compiling and Running x86_64

To run a program using the installed GCC I used the relative path to the bin folder in my install directory. Usually when fresh installing GCC you would set the bin folder as a $PATH variable but in this case there was already a version of GCC in the environment.

To 100% confirm that it is built on x86_64 I used the file command on the object file:

Compiling and Running AArch64

Same as above I used the relative path name with the specific AArch64 compiler.

Confirming with the file command:

Conclusion

This experience serves as a testament to the power of open-source development and the versatility of GCC as a cross-compiler tool.

Code Refactoring and Git Rebase

Ian Jacobs — Sat, 14 Oct 2023 01:10:28 +0000

This week in my open source program I spent time refactoring my code in my textml program located on my GitHub here https://github.com/ijacobs-cpa/textml. In addition, I also experimented with git rebase to edit commits made on a refactoring branch.

Refactoring Process

Before beginning to refactor my code for this week I had to find where best to focus on refactoring as I had already refactored some code on my own time.

I mainly focused on refactoring my main file textml.py and improving my modules as the main function had barely been updated since the first version of the program and could be reduced. The file with the main function was also updated recently with a lot of command-line processing which could be concentrated elsewhere. I also focused here as I had refactored the convertUtils module on my own time, so I wanted to focus elsewhere.

To refactor this section of code I first wanted to separate code that dealt with processing the command line into its own module. I had already removed the converting functions into their own module so to organize this I decided to store all future modules in their own bin/ directory except for the main textml.py program which would remain at the root of the project.

With this, I moved any argparse usage and TOML file processing into its own module and then referenced the module from the main to get the command line arguments. This effectively split the file into two.

Next, I began looking at how I stored these command line args as variables in my main program and I decided to move them to a class object effectively extracting a class from the code. This helps when these variables need to be passed to a function as I can just pass the entire class allowing for easier maintainability and expansion in the future:

from bin.cli import args as ARGS

class ConvertProperties:   # Properties for converted file 
    def __init__(this, userInput, outDir, lang):
        this.userInput = userInput
        this.outDir = outDir
        this.lang = lang

props = ConvertProperties(  # Instanced class of properties
    ARGS.input, 
    ARGS.output, 
    ARGS.lang
)

With this implemented I then went to work on extracting a function out of the code in my main where it would run the convert functions. This was repeated for directories or files and could be simplified into a single function effectively extracting a function from reused code.

This was completed by providing the class object of converting properties with a file name into a function like below:

def process_file(file, fileProps):      # Function to process file in proper file type
    ext = os.path.splitext(file)[1]

    if ext == ".txt":                                       # Checking if each file's type is supported before converting
        utils.convertText(file, props.outDir, props.lang)   # Text Conversion
    elif ext == ".md":                               
        utils.convertMD(file, fileProps.outDir, fileProps.lang) # Markdown conversion
    elif not os.path.isdir(file):                                                     # Prints error if file is not supported
        raise Exception("Error!: Invalid file type for: " + file + " (Expected .txt, .md)")

Lastly, I updated some variable names in my program to make them more descriptive.

Git rebase

For this refactoring, I created a separate refactoring branch and committed each step separately. I wanted to group these commits into one single commit with descriptions of all the changes.

This was done using git rebase allowing me to squash all commits into one picked one like the demonstration below:

git rebase main -i

In a text editor:

pick 4710009 Commit message 1
squash 0c01069 Commit message 2
squash fd8e932 Commit message 3
squash f9d85bf Commit message 4
squash bfac3d3 Commit message 5

This then converted all my individual commits into one commit with all commit info. I then amended the commit message to indicate that this singular commit was all refactoring.

Its information and all code changes mentioned can be viewed Here

Code Contributions Using Git Remotes

Ian Jacobs — Sat, 07 Oct 2023 02:58:14 +0000

Intro

For my Open-Source class this week I collaborated with a classmate to add a new feature to their text-to-HTML converter adding support for TOML configuration files.

Before beginning contributing with this addition I familiarized myself with their project's code and structure while beginning to think about how I was going to implement it.

Contributing

To start, I created an Issue in my classmate's project detailing what I was going to add and how I would add it.

To add support for TOML files I first needed to find a proper .toml file parser that would be compatible with my classmate's project in Python. I ended up using the tomli on recommendation from the project owner for parsing program options from a .toml file. The tomli library was added to version 3.11 of Python.

After being approved to work on the project I began to implement the tomli TOML file parser in my classmates' repo. Their project was separated into many different modules which made adding any changes easier as I wouldn't have to update as many functions as they all referenced one module.

The module in question dealt with command line arguments passed to the program and used Python's argparse library to export an object of required program options that are used around the program. In this module is where I would do most of the work by adding a -c, and --config argument option using argparse that accepted .toml files, then adding a check to see if a config file was passed where if there was it would process and overwrite any program options with ones in the file.

I then sent the changes as a pull request and after updating the readme and some other things requested by the maintainer my addition was merged into the main project.

Reviewing a Pull Request with Git Remote

In my own text-to-HTML project, I also received a pull request adding support for TOML config files.

Once the pull request was made I then needed to review the change and test it. To do this I set a remote to the forked branch of my repo and then fetched the contents into my repo.

$ git remote add <name-of-student> <https://git-url-of-other-studnet-fork.git>
$ git fetch <name-of-student>

I then created a new branch with the remote additions added to my main repo so that I could test the new additions as if they were merged:

$ git checkout -b <branch-name> <name-of-student>/<branch-name>

This allowed me to easily make updates or pull any new changes the contributor makes.

After reviewing the pull request I saw it was good to go I used:

$ git merge <student-name>/issue-10

which automatically merged the pull request into my repo from my created remote mentioned above.

Final Thoughts

After completing these activities, I've learned a lot about remotes and how they can be set up to directly work with pull requests which is very useful for reviewing. I also was able to view another person's code and test my ability to add and try to emulate how the code was used which improved my overall skill.

The difficulty that I encountered when working on these was that I made the mistake of adding the remote branch of my repos forked pull request branch to the repo of another project. Thinking that I screwed up I ended up learning how to remove remotes by using git push origin -d to delete the remote branch from the incorrect repo.

An Example of Merging Parallel Branches in GitHub

Ian Jacobs — Fri, 29 Sep 2023 18:51:04 +0000

This week in my open-source course I am practicing creating and merging branches in my textml text-to-HTML converter located on my GitHub at https://github.com/ijacobs-cpa/textml. The program takes a .txt or .md file and converts it to a usable HTML file with proper tags.

To practice creating and merging branches for my program. I decided to add 2 new features to my program:

Adding support for an optional -l, --lang language argument that specifies the attribute in the root <html> element in the final converted HTML file.

For example, ./textml.py french.txt --lang fr would set <html lang="fr"> the language to french in the root <html> element.

Adding the conversion of --- in markdown files to the <hr> horizontal rule element in HTML.

For each of these new features, I created an issue on my projects repository:

Once this was done I created a new branch for each issue on my local repository:

$ git checkout -b issue-10
$ git checkout -b issue-11

I then started to implement the first new feature into my code on its respective branch making sure not to add any changes to the main branch.

After this was done I moved to the other branch to implement the second new feature:

$ git checkout issue-11

During this process, I found moments when I was testing the feature that I was working on and tried to run the program with the other already implemented feature from the other branch as an optional argument only to see it not work. This shows that if you are working on multiple unmerged branches make sure you understand the base limitations of your code.

Merging the Branches

After completing and testing the current issue branch's new feature I was ready to merge the parallel branches into the main branch and I started with the second one:

$ git checkout main
...
$ git merge issue-11
Updating 5f96795..23098af
Fast-forward
 convertUtils.py          |  4 +++-
 examples/mdTest/test3.md | 11 +++++++++++
 2 files changed, 14 insertions(+), 1 deletion(-)
 create mode 100644 examples/mdTest/test3.md

As can be seen above git uses a "Fast-forward" strategy merge as the unchanged main branch has no changes so no conflicts come up as the branches are merged.

I then went and merged the issue-10 branch with the main branch:

$ git merge issue-10
Auto-merging convertUtils.py
Merge made by the 'recursive' strategy.
 README.md       |  6 +++++-
 convertUtils.py | 12 ++++++------
 textml.py       | 13 +++++++++----
 3 files changed, 20 insertions(+), 11 deletions(-)

This time when merging it used the recursive strategy as we are trying to merge the issue-10 branch into the main branch which has been merged already with the issue-11 branch essentially making this a 3-way recursive merge.

Thankfully, there were no merge conflicts and the issue branches were merged into the main branch successfully. I then pushed the changes to GitHub to finish up.

Here are links to the merge commits on GitHub:

Final Thoughts

Working on branches seems like a very good way to go about adding changes to a project on GitHub as it provides a good way to modify and test your code without removing the original and makes a project compatible with the open-source community when wanting to work with others simultaneously.

6502 Programming

Ian Jacobs — Thu, 28 Sep 2023 04:37:41 +0000

For my portability and optimization class, I am taking a look at the 6502 assembly code in preparation for modern assembly code. I am using a 6502 emulator located here that includes a bitmap display and a text output so we can visually see the output for the assembly code including a memory monitor to see the added memory.

I will be running some 6502 assembly code to see the results while calculating the time it takes to run and performing some extra experiments in this blog.

The instructions for creating programs with this CPU are very minimal and a manual used for this lab and in the course can be found here with all instructions for the 6502 processer.

Calculating Performance

The following code fills the bitmap with a solid colour:

lda #$00    ; set a pointer in memory location $40 to point to $0200
    sta $40     ; ... low byte ($00) goes in address $40
    lda #$02    
    sta $41     ; ... high byte ($02) goes into address $41

    lda #$07    ; colour number

    ldy #$00    ; set index to 0

loop:   sta ($40),y ; set pixel colour at the address (pointer)+Y

    iny     ; increment index
    bne loop    ; continue until done the page (256 pixels)

    inc $41     ; increment the page
    ldx $41     ; get the current page number
    cpx #$06    ; compare with 6
    bne loop    ; continue until done all pages

Timing Results

Below are the timing results from each instruction used above including how many cycles it would take to complete which is then transformed into time.

A bit of clarification, the bitmap display is 32x32 pixels long and takes that amount of space in bits in the 6502's memory, and because we are looping and changing each pixel. The cycle count for those operations is 32x32 = 1024 split into 4 pages of 256 pixels.

This overall means that the program requires 10326 cycles to complete which assuming a 1 MHz CPU speed would take 0.01036 seconds to complete.

Memory

During the execution of the code, memory is used for each instruction used and for variables. The total memory used for this program was the following:

Just to note: The answer of 8KB is not the max used during the program but it is the total amount of bytes used during the entire program. The byte values for the instructions were found in the manual here.

Optimized code

To optimize this code for better time I removed the loading of the x register to the beginning of the program instead of in the page loop. I then compare x to the current high-bit (since it's incremented) which would tell me if it is within the page's boundary for the bitmap.

    lda #$00     ; a pointer in memory location $40 to point to $0200
    sta $40     ; low byte ($00) goes in address $40
    lda #$02    
    sta $41     ; high byte ($02) goes into address $41

    LDX #$06    ; Loading number of pages

    LDA #$06    ; Yellow color code
    LDY #$00    ; set Y register index to 0

loop:   
      STA ($40),y   ; Set pixel colour at the address (pointer)+Y
    INY        ; Increment Y register
    BNE loop    ; Continue until done the page (256 pixels)

    INC $41      ; increment the page
    CPX $41      ; Comparing the high bit to the number of pages
    BNE loop     ; continue until done all pages

Here are the new timings below

As can be seen, my new "optimized" implementation doesn't really shave off that much time only 0.000006 seconds.

I am not sure exactly how to cut more time from this program. I knew that it most likely had something to do with how the loops were constantly repeating instructions and that it could be optimized there somehow.

Modifying the Code

Colouring Each Page

Here is code that will add 1 to the accumulator used for getting the colour code for each pixel after every page so that its colour updates each page change.

  lda #$00  ; set a pointer in memory location $40 to point to $0200
    sta $40     ; low byte ($00) goes in address $40
    lda #$02    
    sta $41     ; high byte ($02) goes into address $41

    LDX #$06    ; Loading number of pages

    LDA #$06    ; Yellow colour code
    LDY #$00    ; set Y register index to 0

loop:   
        STA ($40),y     ; Set pixel colour at the address (pointer)+Y
    INY     ; Increment Y register
    BNE loop    ; Continue until done the page (256 pixels)

    ADC #$01    ; Adds 1 to the accumulator to update the color
                 ; for each page
    INC $41      ; increment the page
    CPX $41      ; Comparing the high bit to the number of pages in x
    BNE loop     ; continue until done all pages

Here is the result:

Experiments

TYA

Adding the TYA instruction at the beginning of the loop:

After adding TYA it causes the bit map to display vertical lines of different colours. It appears that it is displaying each available colour in a loop for each pixel as each horizontal bit on the display is a different colour on each line and there are both 16 different colours and 32 bits on the display so it shows each colour twice.

LSR

Including LSR after and with the TYA instruction:

When adding LSR after TYA it causes each different coloured vertical line to become thicker meaning that there is only 1 set of the 16 different colours showing.

Adding more LSR instructions continually causes the lines to become thicker as after 3 LSR instructions there are 8 lines but since there are 16 colours available they overlap each other on every other line.

This eventually causes the single-width lines to be displayed horizontally once 5 instructions of LSR are performed. Additionally, it only repeats the first 8 colors instead of the 16.

ASL

Restarting the process by adding TYA I added ASL instead which caused the following.

The display once again displayed vertical lines for different colours, however, this time compared to using LSR some of the colours were skipped altogether. When referring to the 6502 emulator colour code chart on this page it seems that the ASL operation ignores one colour every time it loops meaning that only 8 out of the 16 colours are repeated for the 32 bits.

When adding more ASL operations like before, it seems that the program skips more and more for each ASL opcode added with each ASL added effectively halving the number of colours available (starting from 16) as seen below.

This culminates in 4 ASL instructions as it only shows the first colour: black, because 4 ASL instructions == 2^4 = 16 | 16/16 =1 meaning that the first colour is repeated for each pixel.

When referencing the manual located here ASL is an "Arithmetic shift left" which means that ASL performs a bitshift to the left which ends up dividing the byte by 2 explaining what is happening here with each additional instruction.

INY

Another experiment to perform is to add more INY (increment y register) instructions for each loop. I would expect this to increment the counter in 5's meaning it makes skip colouring pixels.

However, what happens is the screen still fills with colour but when running at the lowest speed you can see the pixels being filled in with 5 spaces apart for each pixel repeating 256 times which means that it overflows like (255 -> 300 (overflow) x --> 004) and so on meaning that eventually, the pixels fill up each page as usual then the entire display.

Random Colours

lda #$00    ; set a pointer in memory location $40 to point to $0200
    sta $40     ; ... low byte ($00) goes in address $40
    lda #$02    
    sta $41     ; ... high byte ($02) goes into address $41

    lda $FE         ; Sets random colour (number) to accumulator
    ldx #$06    ; get the current page number
    ldy #$00    ; set index to 0

loop:   LDA $FE       ; Adds random colour for next pixel
    STA ($40), y
    iny
    BNE loop

    inc $41     ; increment the page
    cpx $41       ; compare with 6
    bne loop    ; continue until done all pages

This code loads the hexadecimal number 0xFE (254) which is defined as a pseudo-random number in the peripherals of the 6502 processor. This generates a random number assigned to the accumulator before each pixel is stored, allowing each pixel to be a random colour. Here are two examples with the above code on the 6502 Emulator:

Challenges

The code below sets the bit display to a single colour except for the middle 4 pixels.

    lda #$00    ; set a pointer in memory location $40 to point to $0200
    sta $40     ; ... low byte ($00) goes in address $40
    lda #$02    
    sta $41     ; ... high byte ($02) goes into address $41

    lda $FE        ; Sets random colour (number) to accumulator
    ldx #$06    ; get the current page number
    ldy #$00    ; set index to 0

loop:   

    STA ($40), y
    iny
    BNE loop

    inc $41     ; increment the page
    cpx $41         ; compare current page to x (0600)
    bne loop    ; continue until done all pages

    ADC #$01    ; Add one to the colour code

    STA $03EF   ; Storing the new coloured pixels at middle location
    STA $03F0
    STA $040F
    STA $0410

My solution to this challenge was to manually add each of the middle pixel locations and assign them +1 to the colour chosen (kept the random colour choice in this code so it is still random).

Here is the output:

Final Thoughts

After completing this lab, and doing the experiments and challenges. I believe that I have learned the basic instructions and addressing methods for this assembler including the little-endian design philosophy when it comes to byte storage.

When first viewing the assembler demos and lab I was sort of intimidated by the randomness of how the code operated but throughout this lab, I experimented and got a good beginner's understanding of how the 6502 processer operates and how assembly operates in general.

First Code Contributions

Ian Jacobs — Fri, 22 Sep 2023 03:50:18 +0000

This week in my open source development course I practiced creating and managing pull requests to another person's project and my for my project textml.

For my part, I was working on the Waypoint project located here at https://github.com/rabroldan/Waypoint which is a text-to-HTML converter that allows for custom CSS styling and allows live creation and editing of files submitted to the program.

Adding a Feature

My task was to add support for converting markdown/.md files to HTML and this started with creating a Issue. After receiving approval I then forked the repo and created an issue branch to add my changes located here: Forked Issue branch.

I then began making the changes. My main goal was to not overly change the design of the code that was already there so I made sure that any changes were similar to how the project owner's code was made. I ended up creating a new function that would process any .md files that were passed to the program or created by the program.

After creating the function I had to find examples of when the program was checking the file type and have it also check for a markdown file which made me edit multiple points around their code so that it would correctly convert .md and .txt files to HTML in there respective locations.

Making a Pull Request

Once this was done I tested the program under different scenarios (folder, single file, creating a new file, and a new folder) with markdown files and made sure that it would properly be converted and then pushed all the changes to my branch and submitted a pull request which can be seen here.

As can be seen, it was reviewed by the owner and merged into the project after we had a small chat about some of the features on Slack.

Handling a pull request

My project on my GitHub here which is a text to HTML converted also received a pull request to add markdown file/folder support.

Just like my previous process they:

Forked the repo
Created an issue branch
Added changes to the branch
Submitted the branch as a pull request

Pull requests can be found here

After receiving the pull request I went to work on testing if it would work and found that it had a lot of problems:

It had some oversights that caused the program to crash, it wasn't saving files to the correct location, and it would only convert a single .md file and had no directory support.

I opened a thread on the pull request explained some of the issues that were present with their implementation and then asked them to fix them before I could merge.

Later, they updated their commit branch with changes that fixed their previous issues. After a couple more fixes that I required from them, I merged the new code into my repo.

Overall I liked the whole process I just would have preferred if the contributor stuck to how my code was formatted and operated but their additions were solid and they kept most of the continuity of my program intact.

Final Thoughts

Overall this process was interesting. At first, I had a couple of problems understanding how best to implement the code for the project I was working on as I found the program to be written very uniquely and different from what I was used to.

After running and testing the different outcomes I understood where I needed to add something. I also wanted to make sure that I didn't interfere with the original design of the code too much

Being new to Python I learned a good amount from reading the code. I saw a very good use of the argparse Python library which has a much more efficient way of handling command line arguments and I plan to implement it into a future version of my program.

I also learned a good amount about creating and managing pull requests including linking issues that automatically close when pull requests are complete.

If I were to do something differently next time I would ask for more info on how the maintainer would like me to modify their program so I could know how much I could drift away from their design.

Textml

Ian Jacobs — Mon, 18 Sep 2023 19:54:18 +0000

I have created my first version of a truly open-source program called Textml. It is a Python program that converts text files to basic HTML files.

To use the program you provide either a text file or a folder containing text files to be converted to HTML. They will then be saved to a default directory textml/ for use.

Features

Provide custom output directory
- Creates new directory if output directory doesn't exist
Title parsing from text files (First line followed by 2 newlines only)

The project is located on my GitHub at https://github.com/ijacobs-cpa/textml

Instructions

To run this program you first need to clone the repository from GitHub and open the repository locally.

$ git clone https://github.com/ijacobs-cpa/textml.git
$ cd textml/

To run make sure that you have Python installed on your system:
$ python --version

Once installed you can then run the Python script:

$ python textml.py [file/directory] <commands> [command argument]
or
$ python textml.py <options>

Options

Command	Description
-v, --version	Displays version of the program
-h, --help	Displays a help message
-o, --output	Specifies the location to save converted files, creates a new folder if location doesn't exist, defaults to textml/

Examples

Converting a file:

$ python textml.py myfile.txt

Converting all files in a directory:

$ python textml.py mydirectory/

Running Commands:

$ python textml.py -v

Specifying Output Directory:

$ python textml.py myFile.txt -o newDir/

Results

Here is an example of a text file being converted:

command:

$ python textml.py examples/liveTest/index.txt -o examples/liveTest/

Today I learned


Documenting some information I've learned today

Here is some information:


6502 Assembly Notes (KEY_terms):

#{number} adds a nuber
${hex} adds a hexadecimal

LDA = Load the accumulator

LDY = Load the Y register
LDX = Load the X register

STA = Store the accumulator
STY = Store the Y register
STX = Store the X register

CPX = compare THE x register

<!doctype html>
<html lang="en">
<head>
  <meta charset="utf-8">
  <title>Today I learned</title>
  <meta name="viewport" content="width=device-width, initial-scale=1">
</head>
<body>
<h1>Today I learned</h1>
  <br />
  <p>Documenting some information I've learned today</p>
  <br />
  <p>Here is some information:</p>
  <br />
  <p>6502 Assembly Notes (KEY_terms):</p>
  <br />
  <p>#{number} adds a nuber</p>
  <p>${hex} adds a hexadecimal</p>
  <br />
  <p>LDA = Load the accumulator</p>
  <br />
  <p>LDY = Load the Y register</p>
  <p>LDX = Load the X register</p>
  <br />
  <p>STA = Store the accumulator</p>
  <p>STY = Store the Y register</p>
  <p>STX = Store the X register</p>
  <br />
  <p>CPX = compare THE x register</p>
  <br />
</body>
</html>

Static site: https://ijacobs-cpa.github.io/textml/examples/liveTest/index.html

Investigating Open Source Software Packages

Ian Jacobs — Sun, 17 Sep 2023 20:28:18 +0000

This is my first blog post for my Optimization and Portability class at Seneca College. I am researching some open-source software packages and explaining how they handle code changes in the open-source environment.

For this investigation, I've found 2 software packages with different licenses:

First, I'll be taking a look at OpenCV which uses a Apache license. They're an open-source computer vision software package that uses machine learning and its goal is the commercialization of machine perception for use in the mainstream including things like identifying faces and objects, tracking objects, and extracting 3D models of objects (Learn more at https://opencv.org/about/).

Their primary mode of accepting code changes is through their GitHub repository here where users can contribute to the project by submitting issues and making pull requests. An example I'll be looking at is this Pull Request for an update to the code.

The process is pretty typical for a GitHub project the contributor Creates a branch, Clones the branch repository locally, Creates the change/patch, and then Submits the patch as a pull/merge request to the main repo on GitHub.

The review process then begins and a maintainer is assigned to the issue who tags the request and adds it as a part of a future patch milestone. In this case, the maintainer asks another maintainer to review the code and small discussions begin about the methods the contributor uses where other contributors discuss some of the changes. The original contributor then makes some changes and then the patch is approved and merged into the main project. 4 people are involved in getting this patch accepted into the main project

Another project that I'll be looking at is the VLC media player and the VideoLAN software located at https://www.videolan.org/ which is a popular multimedia video player that uses a GPLv3 (GNU Public License). They have their own organization GitLab located here for contributions to their code and require people to have an account with their organization before they are allowed to make any changes.

I'll be describing another accepted patch through a merge request located at this Merge Request. Like GitHub, the contributor creates a branch including their addition for their merge request.

In this request, the contributor creates the patch and submits it for review. The maintainer then provides tags for the request and addresses some of the methods used for changing. A small discussion between the maintainer and the contributor happens which then gets some others involved (including the president of VideoLAN Jean-Baptiste Kempf) on the methods used which results the the original contributor updating their merge branch. After this, the branch is approved by the maintainer and given a tag saying it can be added to a future update, and a couple of days later it is grouped together with many other merge requests which all together are finally pushed to the main branch.

With this, you can see how the head maintainers organize the patches provided by merge requests by grouping approved ones and adding them all at once. This patch involved 5 people from the review stage to the approving and merging stage.

By looking at these different open-source methods for software packages we see how many people contributing and reviewing the project help maintain it as both a relevant project and a free and open-source one.

Reviewing and Being Reviewed: First Reviews in Open-Source

Ian Jacobs — Sat, 16 Sep 2023 00:56:57 +0000

In my Open-Source course this week we worked on completing a small project to convert text files to HTML files. While it wasn't due for a couple of days after this task; before then we needed to review another student's release and get ours reviewed.

To get someone to review my project I made use of Slack where I listed my GitHub repo asking for someone to provide feedback for it by creating issues in GitHub. I responded to another request of someone to have their project reviewed which allowed me to find someone who did not have any feedback already and was willing to examine my project.

When testing and reviewing my partner's code I found we both used the same programming language, Python which I am very unfamiliar with. Initially, when looking at the code I saw that they were more proficient and used a lot of different methods that I was unaware of. Through testing, I found that their code was solid and worked fine under a lot of different circumstances, and was learning from just looking at their code. For example, I was unaware you could run a Python script using the ./{python program} method if you gave the file execute privileges as I kept using the python <program name> method.

Getting my project reviewed I thought my partner who looked more proficient in Python would be more critical of my implementation. However, they focused more on the project details and future flexibility of the code which I would find helpful. Instead of going into detail on things that I might have overlooked when it came to efficiency.

The main issue that I found in my partner's project was that they were missing many README details including installation instructions making it slightly unclear to someone trying to install it. The other issue/improvement I found was when a text file was converted to HTML it didn't add any `
tags for when there was a double newline. This meant that if there were any extra spacings between paragraphs they wouldn't be in the HTML document. This however was something that was optional and something that each project creator could choose to include or not but mentioned as an improvement.

Issues with my partner's project:

Issue 1 | In this issue I discussed how my partner's README could be improved as theirs was lacking some of the required details and details that could help one understand how to use the program.

Issue 2 | In this issue I discussed how their version and help flags could use some improvements including more details on how to run the program in the --help tag.

Issue 3 | This issue dealt with the fact that any optional features that are present in the project are not documented or explained and suggested a feature that could be added for future optional requirements.

Issue 4 | This issue dealt with how their program dealt with spaces in HTML and how they could have included some <br /> tags to more accurately reflect the text document they were processing.

Issue 5 | This issue dealt with code formatting issues and for them to add a couple more comments in the program.

Issues for my project:

Issue 1 & Issue 4 | These issues dealt with reformatting of the README document to make it more user friendly and optimized. This involved moving some sections closer to the top and making sure to use constant styling.

Issue 2 | This issue dealt with displaying the differences between different versions and creating a changelog to document them in the readme.

Issue 3 | This issue was about an error in my help message that showed an incorrect use of the program when used.

Issue 5 | This was an issue that one of my initial functions was not dynamic enough that it could be expanded upon in the future so they suggested I make it able to fulfill future users requirements.

These issues made by my partner helped me document my project more efficiently and helped with making sure my program passed all the specifications required for submission while also providing improvements to the code for flexibility in the future.

To sum it up, I gained a lot of experience opening and closing issues in GitHub and testing another person's program, and found that you can learn a lot from reviewing other people's code. I feel like it is good to have someone with a fresh perspective view code that is not theirs as they can catch things that others can't.

Open-Source Introduction

Ian Jacobs — Fri, 08 Sep 2023 18:02:23 +0000

Hey, my name is Ian and I am a programming student at Seneca College in the 6th semester of the CPA program and am going to get my start in the world of open-source development through this course.

I took this course because I want a good introduction to the open-source development community as I've only worked on more closed projects for assignments and personal learning. It also will teach me to interact more with open source communities and work on related projects which will help me with a future career in software development.

My goals in this term are to learn the proper methods and etiquette for contributing to open-source projects while also testing the many programming languages and skills that I have acquired during my time in college with real-world projects.

I'm not sure what types of projects I would like to contribute to exactly and would like to research and experiment with many different types of projects. I do however have a focus on tools that help with back-end development and useful tools that perform a task simply and efficiently. This leads to the project that I've chosen to research for this post.

scrcpy is a program written mainly in C that allows you to view and control your mobile device on your computer. I first came across this project a year ago when trying to find a program to emulate my mobile device for a school assignment. When I first downloaded it was extremely easy to set up and start and worked seamlessly, and besides allowing you to view your mobile's screen it also allowed you to control your phone with your mouse and keyboard which I found great as many other screen capturing methods don't have this level of emulation.

The project is located at https://github.com/Genymobile/scrcpy and was created by Genymoblie which is a company specializing in Android emulation that has other similar projects that allow for the use of Android devices on other platforms include in the cloud.

I find companies like these that specialize in efficient software to be very interesting and insightful for an open-source project as one of the reasons it works so well may have come from help from the open-source community and it inspires me to participate in projects like these in the open-source community.