DEV Community: arinak1017

GCC: Automatic Function Multi-Versioning Wrap Up

arinak1017 — Thu, 12 Dec 2024 09:35:22 +0000

Before we start...

If you have just stumbled upon my SPO600 series of blog posts, it has been created to document and share my learnings as I progress through my Software Portability and Optimization course at Seneca Polytechnic.

As this term comes to an end and we officially wrap up our course project, implementing a new experimental feature (Automatic Function Multi-Versioning) in the GNU Compiler Collection, I want to reflect on the progress I’ve made and summarize my work. Throughout this project, I delved into GCC compiler passes, tackled the complexities of the GIMPLE intermediate representation, and developed initial logic for identifying and pruning redundant function clones.

If you are curious to learn more about the project, feel free to check out my previous blog posts:

How did I approach the experiments?

Over the course of the last few weeks, I spent a significant amount of time navigating the GCC documentation, and experimenting with the dummy passes provided to us by our Professor, Chris Tyler.

While working on my test pass, I approached the problem by isolating it from the complexity of the GCC compiler. Imagine designing a simple function that scans a piece of text or a file and flags redundant paragraphs. This analogy reflects what we’re trying to achieve but makes the concept easier to grasp. Instead of analyzing text, however, we’re examining the binaries of function clones and flagging them for pruning if they are redundant.

Let’s take a look at the draft logic

bool compare_gimple_functions(gimple_seq g1, gimple_seq g2) {
    gimple_stmt_iterator gsi1 = gsi_start(g1);
    gimple_stmt_iterator gsi2 = gsi_start(g2);

    while (!gsi_end_p(gsi1) && !gsi_end_p(gsi2)) {
        gimple *stmt1 = gsi_stmt(gsi1);
        gimple *stmt2 = gsi_stmt(gsi2);

        // Check if the statements are equivalent
        if (!stmt1 || !stmt2 || !simple_cst_equal(stmt1, stmt2)) {
            return false;
        }

        gsi_next(&gsi1);
        gsi_next(&gsi2);
    }

    return gsi_end_p(gsi1) == gsi_end_p(gsi2);
}

unsigned int
pass_testprune::execute (function *fun)
  {
     struct cgraph_node *node;

    FOR_EACH_FUNCTION(node) {
        if (!node->analyzed)
            continue;

        std::string function_name = IDENTIFIER_POINTER(DECL_NAME(node->decl));
        if (function_name.find("_1") != std::string::npos) {
            size_t pos = function_name.find_last_of("_");
            std::string base_func_name = function_name.substr(0, pos);

            struct cgraph_node *base_node = node->get_base_function();
            if (base_node && base_node != node) {
                if (compare_gimple_functions(base_node->get_body(), node->get_body())) {
                    fprintf(dump_file, "PRUNE: %s\n", base_func_name.c_str());
                } else {
                    fprintf(dump_file, "NOPRUNE: %s\n", base_func_name.c_str());
                }
            }
        }
    }
    return 0;

  }
} // anon namespace

Understanding the proposed logic:

compare_gimple_functions
The function compare_gimple_functions attempts to compare two GIMPLE statement sequences using a simple while loop. It iterates through the sequences with iterators and checks if the statements are equivalent using the simple_cst_equal function. This logic is designed to mimic comparing lines in two files.
execute function
The execute function is the core of this pass. It aims to process function clones within the binaries. Clones are typically identified by a suffix like _1 or _2 in their names. The pass compares each clone to its base function, flagging it for pruning if it matches.
Step-by-step execution
- Initial Safeguard: the if (!node->analyzed) block is intended to ensure that the compiler has analyzed the function before processing it. If not, the function is skipped. This check is meant to act as a safeguard.
- Iterating Through Functions: inside the FOR_EACH_FUNCTION loop, the pass attempts to identify clones by checking for function names ending with _1 or similar suffixes.
- Comparing Clones: the base function and its clone are compared using the compare_gimple_functions method. If they appear identical, the pass prints PRUNE: <function_name>. Otherwise, it prints NOPRUNE: <function_name>.

You can find the source code with my newly added pass (tree-testprune.cc) in my personal fork of gcc-mirror

To try running it:

Clone the repo
git clone https://github.com/arilloid/gcc.git
Switch to the afmv-function-clone-comparison branch
git checkout afmv-function-clone-comparison
Follow the standard build process to configure, build, and install the GCC compiler with the added pass.

Code refrences:

To see which files were modified to add a pass, refer to this commit: 5ece499

The proposed logic for the pass is added in this commit: 61428c7

The issues with the proposed solution

The primary issue stems from the incorrect use of GIMPLE functions and macros. To address this, I spent a substantial amount of time studying the GCC GIMPLE documentation. Through this effort, I gained a better understanding of how GIMPLE passes work. However, due to time constraints, I couldn’t get the pass to run successfully.

Advice for future students

My main advice to future students taking this course is to become very comfortable with configuring, building, and installing the GCC compiler. These foundational skills will save you a lot of time and frustration as you progress through the project.

Additionally, dedicate time to thoroughly reading the documentation, particularly on GIMPLE, as the project revolves around designing a GIMPLE pass. The more familiar you are with the documentation, the more confident you’ll feel working with the GCC compiler. If we had a clear understanding of the necessary macros and operands from the start, completing the pass would have been much easier. However, since the documentation is mostly incomplete or hard to navigate, patience and persistence are key.

Finally, get comfortable navigating the codebase and always use Git for version control. Even for small changes, commit your work regularly. Mistakes are inevitable, and being able to revert changes will save you countless hours. Trust me, this is an underrated yet crucial tip when working with a project as complex as GCC.

Afterthoughts

All in all, this journey has been a great experience. Although I wasn’t able to achieve a functional solution, I’m proud of the effort I put in, actively making changes, experimenting, and continuously rebuilding the GCC compiler. This process has given me confidence in my ability to work with large and complex codebases.

In the professional world, the reality is that most of the time we’ll be working on someone else’s code, relying on company-specific libraries, and navigating official documentation to understand and extend functionality. This project closely mirrored that experience, and I’m grateful for the valuable skills and knowledge I gained along the way.

6502 Length Converter

arinak1017 — Thu, 12 Dec 2024 07:21:20 +0000

Hello, Blog!

If you have just stumbled upon my SPO600 series of blog posts, it has been created to document and share my learnings as I progress through my Software Portability and Optimization college course.

The end of the term is around the corner, and I’m continuing to tackle my backlog of blog posts. Today, I’m sharing my experience completing Lab 3 for the Software Portability and Optimization course.

Check out my 1-st post about 6502 Assembly: 6502 Assembly Intro
Find the 6502 emulator I used here: 6502 Emulator
Nice and colourful 6502 reference: Ultimate 6502 Reference
Sample code provided by our professor, Chris Tyler: 6502js-code

An inches-and-feet to centimeter converter

In Lab 3, we were tasked to write a 6502 assembly program involving math operations and string manipulation. To be brutally honest, I chose to write a length converter due to time constraints, creating a full game in assembly felt too ambitious given the complexity.

That said, even writing a "simple" inches-and-feet-to-centimeters converter turned out to be incredibly challenging (just like writing anything in assembly).

To write this code I had to deal with:

Handling user input and strings: learning to work with ROM routines for basic I/O.
Branching and jumps: refreshing the fundamentals of conditional branching and program flow.
Challenging arithmetic operations: implementing multi-step calculations using decimal and binary-coded decimal (BCD) math.
A LOT of trial and error: iterating through countless debugging cycles to get everything working.

I borrowed the printing logic from our professor, Chris Tyler.
You can find the original implementation here

The Final Solution

This program is a basic unit converter for lengths. Users can choose to convert from inches or feet to centimeters. After selecting a unit (1 for inches or 2 for feet), they enter a one-digit number (0-9). The program then calculates the result using approximated conversion factors (3 cm per inch or 30 cm per foot) and displays it in centimeters. It uses binary-coded decimal (BCD) arithmetic for calculating conversions. Once the result is shown, the program starts over, letting users make another conversion.

The main loop illustrates the overall structure of the program:

MAINLOOP:
    JSR RESET             ; Reset variables
    JSR PRINT_WELCOME     ; Show welcome message
    JSR PROMPT_UNIT_INPUT ; Get unit selection
    JSR GET_NUMBER        ; Get number input
    JSR CONVERT           ; Convert to centimeters
    JSR SHOW_RESULT       ; Display the result
    JMP MAINLOOP          ; Restart

Here is the key code logic:

; Get unit selection
GET_UNIT_INPUT:
    JSR CHRIN          ; Get input    
    CMP #$31           ; Check if 1 pressed
    BEQ UNIT_INCHES
    CMP #$32           ; Check if 2 pressed
    BEQ UNIT_FEET

    JMP GET_UNIT_INPUT ; If neither, keep waiting

; Handle inch selection
UNIT_INCHES:
    JSR PRINT
    DCB "I","n","c","h","e","s",32,"s","e","l","e","c","t","e","d","!",$0d,00
    LDA #$01
    STA UNIT_TYPE
    JMP PROMPT_NUMBER

; Handle feet selection
UNIT_FEET:
    JSR PRINT
    DCB "F","e","e","t",32,"s","e","l","e","c","t","e","d","!",$0d,00
    LDA #$02
    STA UNIT_TYPE
    JMP PROMPT_NUMBER

PROMPT_NUMBER:
    JSR PRINT
    DCB $0d,"E","N","T","E","R",32,"N","U","M","B","E","R",32
    DCB "(","0","-","9",")",":"
    DCB 32,00
    RTS

; Get number from user
GET_NUMBER:
    LDA #$00
    STA INPUT_NUM      ; Clear input storage

READ_DIGIT:
    JSR CHRIN          ; Get character from keyboard
    CMP #$0d           ; Check for ENTER key
    BEQ PROCESS_NUMBER

    CMP #$30           ; Check if less than 0
    BCC READ_DIGIT
    CMP #$3a           ; Check if greater than 9
    BCS READ_DIGIT

    CPX #$01           ; Already have a digit?
    BCS READ_DIGIT     ; If yes, ignore

    PHA                ; Save digit temporarily
    JSR CHROUT         ; Display it
    PLA                ; Retrieve digit

    AND #$0f           ; Convert from ASCII to binary
    STA INPUT_NUM      ; Store directly
    INX
    JMP READ_DIGIT

; Process completed number input
PROCESS_NUMBER:
    LDA #$0d           ; Print newline
    JSR CHROUT
    LDA INPUT_NUM      ; Load final number
    CPX #$00           ; Check if no digits entered
    BEQ READ_DIGIT     ; If none, keep waiting
    RTS

; Convert number based on selected unit
CONVERT:
    SED                ; Set decimal mode for BCD math
    LDA INPUT_NUM
    STA RESULT_L       ; Store initial value
    LDA #$00
    STA RESULT_H

    LDA UNIT_TYPE      ; Check which conversion to do
    CMP #$01
    BEQ CONVERT_INCHES
    JMP CONVERT_FEET

; Convert inches to centimeters (×3)
CONVERT_INCHES:
    LDA INPUT_NUM
    CLC
    ADC INPUT_NUM      ; ×2
    CLC
    ADC INPUT_NUM      ; ×3
    STA RESULT_L
    LDA #$00           ; Clear high byte
    STA RESULT_H
    CLD
    RTS

; Convert feet to centimeters (×30)
CONVERT_FEET:
    SED                ; Set decimal mode (for BCD math)

    ; Multiply by 10
    LDA INPUT_NUM
    STA RESULT_L       ; Store in RESULT_L
    LDA #$00
    STA RESULT_H       ; Clear high byte (start at zero)

    LDX #$09           ; Loop 9 times to add the number to itself (×10)
multiply_by_10:
    CLC
    LDA RESULT_L
    ADC INPUT_NUM
    STA RESULT_L
    LDA RESULT_H
    ADC #$00
    STA RESULT_H
    DEX
    BNE multiply_by_10

    ; Store the result of ×10 in a temporary variable
    LDA RESULT_L
    STA TEMP_L         ; Store low byte
    LDA RESULT_H
    STA TEMP_H         ; Store high byte

    ; Multiply by 3 (add twice more)
    LDY #$02           ; Loop 2 times to add the current result to itself
multiply_by_3:
    CLC
    LDA RESULT_L
    ADC TEMP_L
    STA RESULT_L
    LDA RESULT_H
    ADC TEMP_H
    STA RESULT_H
    DEY
    BNE multiply_by_3

    CLD                ; Clear decimal mode
    RTS

If you’d like to see the complete implementation, feel free to check it out on GitHub.

Execution of my converter

Conversion from feet to cm (9 feet -> ~270 cm)
Conversion from inch to cm (8 inch -> ~24 cm)

Limitations

Single-digit input
No floating-point support
Approximated conversion factors (1 inch = 3 cm vs. 1 inch = 2.54 cm; 1 foot = 30 cm vs. 1 foot = 30.48 cm)
Lack of thorough error handling

Afterthoughts

This is the most complex assembly program I have written as of yet. Completing it required refreshing my understanding of how the 6502 processor handles data, performs arithmetic, and manages program flow. Along the way, I encountered some of the limitations of the 6502, such as its lack of native support for multiplication or floating-point arithmetic. Getting the multiplications right and implementing I/O operations were definitely the most challenging parts of this project. This lab really pushed me making the results more rewarding.

6502 Math

arinak1017 — Wed, 11 Dec 2024 23:05:14 +0000

Hello, Blog!

If you have just stumbled upon my SPO600 series of blog posts, it has been created to document and share my learnings as I progress through my Software Portability and Optimization college course.

The semester is wrapping up, and I'm working to catch up on my backlog of blog posts - bringing you a sudden throwback to 6502 assembly.

In this post, I'll cover the second lab of the course. The main goal of this lab was getting familiar with performing math operations in 6502 assembly, specifically to animate the graphics on the screen.

Check out my 1-st post about 6502 Assembly: 6502 Assembly Intro
Find the 6502 emulator I used here: 6502 Emulator
Nice and colourful 6502 reference: Ultimate 6502 Reference

Let's take a look at the starter code

;
; draw-image-subroutine.6502
;
; This is a routine that can place an arbitrary 
; rectangular image on to the screen at given
; coordinates.
;
; Chris Tyler 2024-09-17
; Licensed under GPLv2+
;

;
; The subroutine is below starting at the 
; label "DRAW:"
;

; Test code for our subroutine
; Moves an image diagonally across the screen

; Zero-page variables
define XPOS $20
define YPOS $21

; Set up the data structure
; The syntax #<LABEL returns the low byte of LABEL
; The syntax #>LABEL returns the high byte of LABEL
LDA #<G_X     ; POINTER TO GRAPHIC
STA $10
LDA #>G_X
STA $11
LDA #$05
STA $12       ; IMAGE WIDTH
STA $13       ; IMAGE HEIGHT

; Set initial position X=Y=0
LDA #$00
STA XPOS
STA YPOS

; Main loop for diagonal animation
MAINLOOP:

  ; Set pointer to the image
  ; Use G_O or G_X as desired
  LDA #<G_O
  STA $10
  LDA #>G_O
  STA $11

  ; Place the image on the screen
  LDA #$10  ; Address in zeropage of the data structure
  LDX XPOS  ; X position
  LDY YPOS  ; Y position
  JSR DRAW  ; Call the subroutine

  ; Delay to show the image
  LDY #$00
  LDX #$50
DELAY:
  DEY
  BNE DELAY
  DEX
  BNE DELAY

  ; Set pointer to the blank graphic
  LDA #<G_BLANK
  STA $10
  LDA #>G_BLANK
  STA $11

  ; Draw the blank graphic to clear the old image
  LDA #$10 ; LOCATION OF DATA STRUCTURE
  LDX XPOS
  LDY YPOS
  JSR DRAW

  ; Increment the position
  INC XPOS
  INC YPOS

  ; Continue for 29 frames of animation
  LDA #28
  CMP XPOS
  BNE MAINLOOP

  ; Repeat infinitely
  JMP $0600

; ==========================================
;
; DRAW :: Subroutine to draw an image on 
;         the bitmapped display
;
; Entry conditions:
;    A - location in zero page of: 
;        a pointer to the image (2 bytes)
;        followed by the image width (1 byte)
;        followed by the image height (1 byte)
;    X - horizontal location to put the image
;    Y - vertical location to put the image
;
; Exit conditions:
;    All registers are undefined
;
; Zero-page memory locations
define IMGPTR    $A0
define IMGPTRH   $A1
define IMGWIDTH  $A2
define IMGHEIGHT $A3
define SCRPTR    $A4
define SCRPTRH   $A5
define SCRX      $A6
define SCRY      $A7

DRAW:
  ; SAVE THE X AND Y REG VALUES
  STY SCRY
  STX SCRX

  ; GET THE DATA STRUCTURE
  TAY
  LDA $0000,Y
  STA IMGPTR
  LDA $0001,Y
  STA IMGPTRH
  LDA $0002,Y
  STA IMGWIDTH
  LDA $0003,Y
  STA IMGHEIGHT

  ; CALCULATE THE START OF THE IMAGE ON
  ; SCREEN AND PLACE IN SCRPTRH
  ;
  ; THIS IS $0200 (START OF SCREEN) +
  ; SCRX + SCRY * 32
  ; 
  ; WE'LL DO THE MULTIPLICATION FIRST
  ; START BY PLACING SCRY INTO SCRPTR
  LDA #$00
  STA SCRPTRH
  LDA SCRY
  STA SCRPTR
  ; NOW DO 5 LEFT SHIFTS TO MULTIPLY BY 32
  LDY #$05     ; NUMBER OF SHIFTS
MULT:
  ASL SCRPTR   ; PERFORM 16-BIT LEFT SHIFT
  ROL SCRPTRH
  DEY
  BNE MULT

  ; NOW ADD THE X VALUE
  LDA SCRX
  CLC
  ADC SCRPTR
  STA SCRPTR
  LDA #$00
  ADC SCRPTRH
  STA SCRPTRH

  ; NOW ADD THE SCREEN BASE ADDRESS OF $0200
  ; SINCE THE LOW BYTE IS $00 WE CAN IGNORE IT
  LDA #$02
  CLC
  ADC SCRPTRH
  STA SCRPTRH
  ; NOTE WE COULD HAVE DONE TWO: INC SCRPTRH

  ; NOW WE HAVE A POINTER TO THE IMAGE IN MEM
  ; COPY A ROW OF IMAGE DATA
COPYROW:
  LDY #$00
ROWLOOP:
  LDA (IMGPTR),Y
  STA (SCRPTR),Y
  INY
  CPY IMGWIDTH
  BNE ROWLOOP

  ; NOW WE NEED TO ADVANCE TO THE NEXT ROW
  ; ADD IMGWIDTH TO THE IMGPTR
  LDA IMGWIDTH
  CLC
  ADC IMGPTR
  STA IMGPTR
  LDA #$00
  ADC IMGPTRH
  STA IMGPTRH

  ; ADD 32 TO THE SCRPTR
  LDA #32
  CLC
  ADC SCRPTR
  STA SCRPTR
  LDA #$00
  ADC SCRPTRH
  STA SCRPTRH

  ; DECREMENT THE LINE COUNT AND SEE IF WE'RE
  ; DONE
  DEC IMGHEIGHT
  BNE COPYROW

  RTS

; ==========================================

; 5x5 pixel images

; Image of a blue "O" on black background
G_O:
DCB $00,$0e,$0e,$0e,$00
DCB $0e,$00,$00,$00,$0e
DCB $0e,$00,$00,$00,$0e
DCB $0e,$00,$00,$00,$0e
DCB $00,$0e,$0e,$0e,$00

; Image of a yellow "X" on a black background
G_X:
DCB $07,$00,$00,$00,$07
DCB $00,$07,$00,$07,$00
DCB $00,$00,$07,$00,$00
DCB $00,$07,$00,$07,$00
DCB $07,$00,$00,$00,$07

; Image of a black square
G_BLANK:
DCB $00,$00,$00,$00,$00
DCB $00,$00,$00,$00,$00
DCB $00,$00,$00,$00,$00
DCB $00,$00,$00,$00,$00
DCB $00,$00,$00,$00,$00

This initial code creates a simple animation that moves the graphic diagonally from the top-left corner to the bottom-right. It uses a subroutine (DRAW) to place the image at the correct screen address each frame, then erases the old position and updates XPOS and YPOS until it has moved 29 times.

Let's make it bounce

Our goal in this lab is to make the graphic the bitmapped display using 6502 math, instead of resetting after a set number of frames.

Steps taken:

1. Set the initial position for the graphic: I chose a non-corner starting point, so the graphic can properly bounce around the screen once I am finished with the logic

; Set initial position X=Y=0
LDA #$00
STA XPOS
LDA #$05
STA YPOS

2. Select an X increment that is -1 or +1, and a Y increment that is -1 or +1: I created two new zero-page variables, XINCREMENT and YINCREMENT, and set them to 1. This means the image initially moves down-right. Using #$01 and #$FF will allow us to move in different directions (#$01 - right/down; #$FF - left/up)

; Zero-page variables
define XINCREMENT $22
define YINCREMENT $23

; Select the increments
LDA #$01
STA XINCREMENT  ; start moving right (+1)
LDA #$01
STA YINCREMENT  ; start moving down (+1)

3. Successively move the graphic and make it bounce around the screen:

Figuring out the logic took quite some time, but the final code essentially works as follows:

Increment the X coordinate: on each iteration, the X position is adjusted by adding either +1 or -1, depending on the current horizontal increment value.
Check horizontal boundaries: if the updated X position hits the right or left edge of the screen, the increment is reversed, causing the graphic to change direction horizontally.
Increment the Y coordinate: next, the Y position is similarly updated based on its current vertical increment.
Check vertical boundaries: if the Y position reaches the top or bottom edge, the vertical increment is also flipped, making the graphic bounce vertically as well.

  ; setup code
  ; start of the main loop

  ; Increment the X position
  ; ---------------------------
  ; If XINCREMENT is #$01, we move right by adding 1
  ; If XINCREMENT is #$FF, we move left by adding -1

  LDA XPOS
  CLC             ; Clear carry before addition
  ADC XINCREMENT  ; Add XINCREMENT to XPOS
  STA XPOS

  ; Check horizontal boundaries
  ; ---------------------------

  ; Check if XPOS == XBOUNDARY (right edge)
  LDA XPOS
  CMP #XBOUNDARY
  BNE CHECK_X_LEFT ; If not equal, jump to check left boundary

  ; If we hit the right boundary, reverse direction
  LDA #$FF
  STA XINCREMENT

CHECK_X_LEFT:
  ; Check if XPOS == 0 (left edge)
  LDA XPOS
  CMP #0
  BNE UPDATE_Y ; If not equal, jump to update Y coordinate

  ; If we hit the left boundary and direction == left, reverse direction
  LDA #$01
  STA XINCREMENT

UPDATE_Y:
  ; Update YPOS using YINCREMENT
  ; ----------------------------
  ; If YINCREMENT is #$01, we move down by adding 1
  ; If YINCREMENT is #$FF, we move up by adding -1

  LDA YPOS
  CLC             ; Clear carry before addition
  ADC YINCREMENT  ; Add YINCREMENT to YPOS
  STA YPOS

  ; Check vertical boundaries
  ; ---------------------------

  ; Check if YPOS == YBOUNDARY (bottom edge)
  LDA YPOS
  CMP #YBOUNDARY
  BNE CHECK_Y_TOP  ; If not equal, jump to check the top edge

  ; If we hit bottom edge, reverse direction
  LDA #$FF
  STA YINCREMENT

CHECK_Y_TOP:
  ; Check if YPOS == 0 (top edge)
  LDA YPOS
  CMP #0  ; If no boundary hit, continue moving
  BNE MAINLOOP

  ; If we hit top edge, reverse direction
  LDA #$01
  STA YINCREMENT

  JMP MAINLOOP  ; Return to the main loop

; DRAW subroutine

Full Solution

If you’d like to see the complete implementation, feel free to check it out on GitHub.

Afterthoughts

Assembly programming is certainly very tedious, especially after stepping away from it for a while (the process of readjusting is very real!). However, the complete lack of abstraction also has its benefits: it forces you to gain an intimate understanding of every tiny detail of the code's logic.

The Open Source Finale: Pt. 3

arinak1017 — Wed, 11 Dec 2024 13:47:59 +0000

Hello, Blog!

If you have just stumbled upon my OSD600 series of blog posts, it has been created to document and share my learnings as I progress through my Open Source Development college course.

This post wraps up the final stretch of my Open Source course, Release 0.4. In it, I’ll share updates on my latest contributions and give an overview of what I’ve achieved in this assignment.

A bit about Release 0.4

For our final assignment, we were tasked to leverage the skills and experience we’ve built over the term to work on something meaningful to us in one way or another.

If you missed how I started working on 0.4, check out my previous posts:

The Open Source Finale: Pt.1

The Open Source Finale: Pt. 2

Last Issue follow up

In my last post, I shared how I created a draft Pull Request for the following issue:

feat: add ability to view the current game without having to log in #43

yurijmikhalevich posted on Sep 29, 2024

Display the current game at the login screen.

E.g. we should update the login screen to look exactly like the screen after login, but instead of the code editor, we should display the login form.

View on GitHub

I had implemented the base requirements and left a comment asking for feedback from the maintainer to clarify some doubts:

Comment for #92

arilloid commented on Dec 08, 2024

Hello @yurijmikhalevich!

I have refactored the login page!

I have a doubt about the login form. Would it be better to move into a sperate component, or is it okay just to leave it as is? (I tried to move the form into a separate component, but this seems to have disrupted how the E2E tests interact with the page elements)

Also, would you suggest displaying any kind of placeholder/animation while the game screen is loading?

View on GitHub

What happened next?

I quickly received feedback from the maintainer. First, they pointed out that copying the resizing logic from another page wasn’t ideal and questioned whether it was necessary on the login page. They also addressed my question about the login form. Initially, I had moved it into a separate component, but since this caused the e2e tests to fail, I kept it inside the login page for the time being.

The maintainer suggested that it would be better to move the login form into a separate component and encouraged me to adjust the e2e tests if necessary.

Addressing feedback

Most of the changes were straightforward. I simplified the /login page by removing the resizing logic, moved the login form into a separate component, and investigated the e2e tests.

Initially, I couldn’t run the e2e tests locally due to an issue where one of the testing commands failed because my absolute file path contained a space. Once I fixed that, I was able to run the tests successfully. However, the login tests were still failing due to the default 30-second timeout being insufficient for the tests to complete.

Since the project uses Playwright for testing, I consulted their documentation on timeouts to find a solution. I increased the timeout for both tests to 60 seconds to resolve the issue. With this adjustment, the tests ran smoothly. After confirming everything was working as expected, I pushed my changes and reached out to the maintainer for another review.

Final Touches

The maintainer left two minor comments, which I quickly addressed. Now, I’m waiting for my PR to be merged.

Wrap up

This assignment concludes my Open Source Development course. While I couldn’t get both of my Pull Requests merged by the deadline due to some life circumstances, I still learned a lot and accomplished most of my goals. Here’s what I achieved:

Contributed to high-priority issues: I addressed two high priority issues, making key features of the app publicly accessible for unauthenticated users.
Gained hands-on experience with new tools and frameworks: now I know how to build full-stack Vue.js apps using Nuxt, write Playwright tests. /+ I learned a bit more about Prisma ORM.
Improved communication with project maintainers: Unlike my previous contributions, where feedback was limited, I had a more engaging back-and-forth with the maintainer. So, I finally had the chance to polish features based on feedback and make necessary adjustments to meet project expectations.

Afterthoughts

The Open Source course at Seneca pushed me out of my comfort zone. I’m no longer as intimidated by collaboration or sharing my code with others. I’ve become much more proficient with Git and, overall, feel more confident in my programming skills despite the hiccups I faced throughout the course.

Update! My changes got merged 😄 - commit

The Open Source Finale: Pt. 2

arinak1017 — Tue, 10 Dec 2024 06:03:49 +0000

Hello, Blog

If you have just stumbled upon my OSD600 series of blog posts, it has been created to document and share my learnings as I progress through my Open Source Development college course.

In this blog post, I’ll share my progress on the course's final assignment, Release 0.4.

A bit about Release 0.4

For our final assignment, we were tasked to leverage the skills and experience we’ve built over the term to work on something meaningful to us in one way or another.

If you missed how I started working on 0.4, check out my previous post

Where am I at right now?

At this stage, I’ve already made significant progress. I’ve gotten a hang of the stack, picked up two issues, closed one, and created a draft Pull Request for the second.

The Stack

move-fast-and-break-things / aibyss

🚧 UNDER DEVELOPMENT 🚧 Aibyss: code your AI to compete in a survival game

Setup

install node.js
(on macOS only) install node-gyp: npm install -g node-gyp
install the dependencies: npm ci

Development server

Start the development server on http://localhost:3000:

npm run dev

Create new user

npm run create-user <username>

Running the tests

unit tests

npm run test

e2e tests

First, setup e2e tests by running npm run test:e2e:install, then run the tests with:

npm run test:e2e

Contributors guide

We follow conventional commits, name your PRs accordingly

Production

Build the application for production:

npm run build

Locally preview production build:

npm run preview

Check out the deployment documentation for more information.

View on GitHub

The project I’m contributing to, AIbyss, uses various tools and technologies, many of which were entirely new to me. Familiarizing myself with the stack was one of the goals I set for Release 0.4, so I took the time to explore the project structure and dive into the documentation for each tool and framework.

Now, I have a basic understanding of building Vue.js apps!

The key components of the stack:

Nuxt- a powerful open-source framework built on top of Vue.js, designed for creating dynamic full-stack web applications. It provides a great way to build full-stack Vue apps.

Just take a look at Nuxt DevTools!

Vue.js - a "progressive JavaScript framework" focused on building user interfaces. Vue.js piqued my interest for a while, and after finally using it in this project, I can see why it’s so popular in the developer community - it's both intuitive and flexible.
Prisma - a modern Object-Relational Mapping (ORM) tool for simplifying database interactions. Having heard much about it, I am excited to incorporate it into future projects.

My contributions (as of now)

If you’ve read my first post, you’ll know that my ultimate goal was to contribute to the project in a meaningful way. To achieve this, I focused on taking and resolving issues labelled as high priority. So far, I’ve successfully closed one issue and created a draft Pull Request for the second, marking steady progress toward my goal.

Issue 1 - Making the Rating Page publicly accessible

feat: let the users view the rating page without having to log in #42

yurijmikhalevich posted on Sep 29, 2024

This is a follow-up idea to the #37 and should be done after it is merged.

View on GitHub

When I first picked up this issue, I thought it would be straightforward since the repo maintainer provided clear guidance on where to start. However, it turned out to be more challenging than expected. The task involved a lot of debugging and required a deeper understanding of how routing and middleware were set up in the project.

Comment for #42

yurijmikhalevich commented on Dec 02, 2024

Hi @arilloid! 👋

Thank you for reaching out! Yes, of course, you can take this issue! It's a good one to start with. It should take a smaller change to the PUBLIC_PATHS in server/middleware/auth.ts, and let's move the "rating" link to the header so it's accessible from any page. Let me know if you have any questions!

View on GitHub

I began by moving the rating link from GameScreen.vue to GlobalHeader.vue. This required locating the necessary components, removing the link from the game screen, and placing it in the global header to ensure it was displayed for both authenticated and unauthenticated users.

After confirming the link was accessible from the header, I modified the public paths array inside the authentication middleware to include the /rating path. The middleware checks if a requested path is public and redirects unauthenticated users to the login page if it isn’t.

I assumed this would resolve the issue, but I was met with a rendering error when I clicked the rating link while not logged in. The error indicated that .toFixed(2) was being called on null values in RatingTable.vue. I suspected this was because the api/rating fetch wasn’t returning any data in the development environment. To test this, I updated the component to conditionally render the values with .toFixed(2) only if rating data was available.

This allowed me to access the rating page without logging in, but something was still wrong, as the table displayed blank entries.

To investigate the issue, I checked rating.get.ts to confirm that fetching rating data didn’t rely on user-specific information. Then, I added console.log() statements to the middleware logic in /server/middleware/auth.ts to understand how redirects were handled. I also logged the status of the api/rating fetch on the rating page.

I noticed that, for some reason, there was a redirect to /API/auth/users. Then, I thought: "It must be it! Somewhere in the code, there is an attempt to fetch user details while not being authenticated, resulting in the redirect to the /login being sent back to the rating page instead of the rating info". My assumption was partially confirmed by displaying the result of the /api/rating fetch.

The logs revealed a redirect to /api/auth/user. I thought that it was IT, that I found the underlying issue, as this suggested that an fetching user details was causing a redirect to the login page, which was being returned to the rating page instead of the actual ratings data. I confirmed an attempted redirect to /login by inspecting the result of the /api/rating fetch.

I discovered that the global header was making a fetch request to /api/auth/user, and I suspected this was causing the issue. To test this, I temporarily removed the logic that fetched user data from the header. However, the problem persisted, and the rating page was still rendered incorrectly.

At this point, I decided to revisit my changes and carefully re-examine the logs. It became clear that I also needed to include /api/rating in the public paths array to allow it to be accessed without authentication.

After adding /api/rating to the public paths, everything worked as expected! It surprised me that such a small oversight caused so much debugging, but I was glad to have resolved the issue and was ready to submit my PR.

feat(rating): make rating page publicly accessible #90

arilloid posted on Dec 06, 2024

How does this PR impact the user?

https://github.com/user-attachments/assets/41f5cb0d-b87a-4852-bbc3-a905fdebe2bb

Description

Made the rating page accessible to unauthenticated users by moving the link from game screen to the global header, and adding /rating and /api/rating to the list of public paths.

Checklist

[x] my PR is focused and contains one wholistic change
[x] I have added screenshots or screen recordings to show the changes

View on GitHub

I opened a draft PR and asked the maintainer for feedback on the link’s placement in the global header. I also suggested conditionally rendering the link so it wouldn’t appear on the rating page itself. However, the maintainer was satisfied with the changes and merged my PR.

Issue 2 - Make the Current Game publicly available

After my 1-st Pull Request, I quickly picked up a high priority issue.

feat: add ability to view the current game without having to log in #43

yurijmikhalevich posted on Sep 29, 2024

Display the current game at the login screen.

E.g. we should update the login screen to look exactly like the screen after login, but instead of the code editor, we should display the login form.

View on GitHub

Implementing the base requirements for this issue was relatively straightforward, as I was already familiar with the project and its routing setup. The maintainer requested that the login screen be updated to be exactly like the main game screen, with the only difference being that the left panel should instead display the login form.

To achieve this, I added /api/state route used to fetch game state information for the game screen to the public paths in the middleware. I then reused the layout template and separator logic from the index.vue (main game page) to build the login screen.

Code Reference: index.vue

<template>
  <div class="flex flex-col h-screen w-screen">
    <GlobalHeader />
    <div class="flex flex-grow p-1 h-[90%]">
      <div
        ref="resizeEditorElement"
        class="flex min-w-[300px] h-full w-1/2 mr-2"
      >
        <CodeEditor />
      </div>
      <div
        ref="separator"
        class="w-1 bg-gray-300 cursor-ew-resize"
      />
      <div
        ref="resizeGameElement"
        class="flex min-w-[300px] w-1/2 overflow-auto"
      >
        <GameScreen />
      </div>
    </div>
  </div>
</template>

<script lang="ts">
import { defineComponent, ref, onMounted, onUnmounted } from "vue";

export default defineComponent({
  setup() {
    const isResizing = ref(false);
    const initialX = ref(0);
    const initialWidthEditor = ref(0);
    const initialWidthGame = ref(0);
    const resizeEditorElement = ref<HTMLDivElement | null>(null);
    const resizeGameElement = ref<HTMLDivElement | null>(null);
    const separator = ref<HTMLDivElement | null>(null);

    const startResize = (e: MouseEvent) => {
      if (!resizeEditorElement.value) {
        throw new Error("unexpected: no resizeEditorElement");
      }
      if (!resizeGameElement.value) {
        throw new Error("unexpected: no resizeGameElement");
      }
      isResizing.value = true;
      initialX.value = e.clientX;
      initialWidthEditor.value = resizeEditorElement.value.offsetWidth;
      initialWidthGame.value = resizeGameElement.value.offsetWidth;
    };

    const handleResize = (e: MouseEvent) => {
      if (!resizeEditorElement.value) {
        throw new Error("unexpected: no resizeEditorElement");
      }
      if (!resizeGameElement.value) {
        throw new Error("unexpected: no resizeGameElement");
      }
      if (isResizing.value) {
        const dx = e.clientX - initialX.value;
        const newWidthEditor = initialWidthEditor.value + dx;
        const newWidthGame = initialWidthGame.value - dx;
        resizeEditorElement.value.style.width = `${newWidthEditor}px`;
        resizeGameElement.value.style.width = `${newWidthGame}px`;
      }
    };

    const stopResize = () => {
      isResizing.value = false;
    };

    onMounted(() => {
      if (!separator.value) {
        throw new Error("unexpected: no separator");
      }
      separator.value.addEventListener("mousedown", startResize);
      document.addEventListener("mousemove", handleResize);
      document.addEventListener("mouseup", stopResize);
    });

    onUnmounted(() => {
      if (!separator.value) {
        throw new Error("unexpected: no separator");
      }
      separator.value.removeEventListener("mousedown", startResize);
      document.removeEventListener("mousemove", handleResize);
      document.removeEventListener("mouseup", stopResize);
    });
    return {
      resizeEditorElement,
      resizeGameElement,
      separator,
    };
  },
});
</script>

My only issue with the code was that I initially moved the login form into a separate component, LoginForm.vue, for better modularity. However, I discovered that this change caused issues with existing e2e (end-to-end) tests. The tests appeared to break because the final DOM structure rendered differently when the login form was imported as a separate component. (But I noticed a cool thing: because of Nuxt, there was no need for explicit component export/import!).

I moved the login form back to the login page to resolve the issue, ensuring the DOM structure remained consistent. Once I confirmed that the login form worked as expected and the game screen displayed correctly on the login page, I submitted a draft pull request.

I also asked the maintainer for feedback on whether the login form should remain part of the page or be modularized into a separate component. Additionally, I mentioned how trying to move the form to a separate component tempered with the existing e2e tests.

feat(login): view current game without login #92

arilloid posted on Dec 08, 2024

How does this PR impact the user?

https://github.com/user-attachments/assets/7f2fea98-3c8b-4cc6-a313-8e19c729f4ab

Description

Added /api/state to public paths to make current game publicly accessible.
Updated the login screen to display the login form on the left and the current game on the right (reused the code from index.vue).

Limitations

The game screen takes a couple of seconds to load.

Checklist

[x] my PR is focused and contains one wholistic change
[x] I have added screenshots or screen recordings to show the changes

View on GitHub

Comment for #92

arilloid commented on Dec 08, 2024

Hello @yurijmikhalevich!

I have refactored the login page!

Also, would you suggest displaying any kind of placeholder/animation while the game screen is loading?

View on GitHub

At the time of writing this post, I am awaiting feedback from the maintainer. As soon as I receive their suggestions, I plan to go ahead and implement any necessary changes.

Afterthoughts...

I’ve already learned quite a lot about Vue.js and Nuxt and feel satisfied with my progress. + One thing I’ve noticed is how much easier it has become, especially toward the end of the term, to navigate the structure of unfamiliar projects, even those built with frameworks I’ve never worked with before!

The Open Source Finale: Pt.1

arinak1017 — Fri, 06 Dec 2024 15:55:33 +0000

Hello, Blog!

If you have just stumbled upon my OSD600 series of blog posts, it has been created to document and share my learnings as I progress through my Open Source Development college course.

Lately, I have been falling behind on blogging. This post, for instance, should have gone up last week, right after I wrapped up the planning phase for our final Open Source assignment. But as they say, better late than never! Today, I’ll take you back to where I was a week ago to share how I approached the planning process, the project I chose to work on, and the personal goals I set for myself.

A bit about Release 0.4

For our final assignment, we were tasked to leverage the skills and experience we’ve built over the term to work on something meaningful to us in one way or another.

Looking for the right project

Finding the right project to work on was the toughest part of the planning stage. I spent hours browsing popular open-source repositories and reading countless articles on "best open-source projects for beginners" (half of which list projects like Linux Kernel as a great starting point for beginners). As that approach wasn’t working, I decided to focus on something fun. Something that I wanted to try for a long time but didn't have a chance (or at least so I thought) to do.

The first idea that came to mind was exploring fun VSCode extensions. I’ve always loved quirky extensions made to bring a smile to a coder's face right in their IDE. However, after doing some research, I discovered that most of these extensions are more like personal side projects than active open-source initiatives. Many of the repositories hadn’t been updated in over a year.

For example, I tried contributing to the repo of a VSCode extension featuring one of my favourite memes, (Incredibly In Your Face). It’s a simple but genuinely funny project with over 100k downloads on the VSCode Marketplace. I picked an issue, submitted a comment, and waited for a response. Unfortunately, I haven't heard back since the project is no longer maintained.

Comment for #6

arilloid commented on Nov 16, 2024

Hello @virejdasani!

I know it's been a while since you have released the extension, but I was wondering if you are still open to accepting Pull Requests.

I’m a student just starting my Open Source journey and looking to play around with some VSCode extensions & I would love to contribute to yours as I find it genuinely funny!

View on GitHub

With time running out, I decided to pivot again and look into the repositories my classmates worked on during Release 0.3.

And then, finally, I found the perfect project!

The project I picked

The project I chose is a game in active development created by a group with ties to KubSu, one of my country's most renowned tech universities. It’s maintained by a close-knit community of former students, many of whom are now immigrants (essentially, people in the same boat as me). The atmosphere felt welcoming, and the project maintainer stood out as very supportive and helpful, thoroughly responding to my classmate's comments. It feels like a community I could genuinely see myself joining if I continue to contribute.

Even the name of the community is compelling:
Move Fast and Break Things - an ultimate developer's motto!

+ An online game, what can be more fun than that?

move-fast-and-break-things / aibyss

🚧 UNDER DEVELOPMENT 🚧 Aibyss: code your AI to compete in a survival game

Setup

install node.js
(on macOS only) install node-gyp: npm install -g node-gyp
install the dependencies: npm ci

Development server

Start the development server on http://localhost:3000:

npm run dev

Create new user

npm run create-user <username>

Running the tests

unit tests

npm run test

e2e tests

First, setup e2e tests by running npm run test:e2e:install, then run the tests with:

npm run test:e2e

Contributors guide

We follow conventional commits, name your PRs accordingly

Production

Build the application for production:

npm run build

Locally preview production build:

npm run preview

Check out the deployment documentation for more information.

View on GitHub

So, that’s how I decided to work on AIbyss, an online competition game where players program AI bots to compete against one another.

While the project is relatively small, it offers many learning opportunities. I’m unfamiliar with most of the stack (Nuxt, Vue.js), have never worked on online games, and also haven’t tackled any frontend-related issues in my open-source journey. All this combined makes the project a great fit for my Release 0.4.

The goals I set for Release 0.4

Contributing to the project in a meaningful way - to achieve this, I decided to start by picking up an issue with a high priority label

feat: let the users view the rating page without having to log in #42

yurijmikhalevich posted on Sep 29, 2024

This is a follow-up idea to the #37 and should be done after it is merged.

View on GitHub

Getting comfortable with the tools - as mentioned, the project utilizes a lot of tools that are new to me, so I was excited to see them in action.
Communicating with the maintainers more actively - in this release, I wanted to engage more actively with the maintainers. Instead of waiting until my work was complete, I planned to create a draft PR early on to gather feedback and collaborate more effectively throughout the process.

Stay tuned

I’ve already picked up an issue and started working on it, so stay tuned — I’ll share my progress in an upcoming post!

GCC: Automatic Function Multi-Versioning Pt.2

arinak1017 — Fri, 06 Dec 2024 06:56:16 +0000

Before we start...

If you have just stumbled upon my SPO600 series of blog posts, it has been created to document and share my learnings as I progress through my Software Portability and Optimization course at Seneca Polytechnic.

In this blog post, I will talk about the code experiments I conducted as part of Stage 2 of our course Project: implementing a new experimental feature in the GNU Compiler Collection (GCC) — Automatic Function Multi-Versioning.

If you are curious to learn more about the project, feel free to check out my previous blog posts:

Analyzing the Demo Pass

In my previous post, I shared how I integrated the demo pass provided to us by our professor.

After I made sure that I could get the new passes running, I decided to take a closer look at the functionality of the demo pass:

// Omitted the License and the include statements
// ============================================================= vvv
// Test pass
namespace {

const pass_data pass_data_ctyler =
{
  GIMPLE_PASS, /* type */
  "ctyler", /* name */
  OPTGROUP_NONE, /* optinfo_flags */
  TV_NONE, /* tv_id */
  PROP_cfg, /* properties_required */
  0, /* properties_provided */
  0, /* properties_destroyed */
  0, /* todo_flags_start */
  0, /* todo_flags_finish */
};

class pass_ctyler : public gimple_opt_pass
{ 
public:
  pass_ctyler (gcc::context *ctxt)
    : gimple_opt_pass (pass_data_ctyler, ctxt)
  {}

  /* opt_pass methods: */
  bool gate (function *)  final override { 
    return 1; // always execute pass
  }
  unsigned int execute (function *) final override;

}; // class pass_ctyler

unsigned int
pass_ctyler::execute (function *fun)
  {
    basic_block bb;

    int bb_cnt = 0, stmt_cnt = 0;

    FOR_EACH_BB_FN (bb, fun)
      {
        bb_cnt++;
        if (dump_file)
          {
            fprintf (dump_file, "===== Basic block count: %d =====\n", bb_cnt);
          }

        for (gimple_stmt_iterator gsi = gsi_start_bb (bb); !gsi_end_p (gsi); gsi_next (&gsi))
          {
            gimple *g = gsi_stmt (gsi);
            stmt_cnt++;
            if (dump_file)
              {
                fprintf (dump_file, "----- Statement count: %d -----\n", stmt_cnt);
                print_gimple_stmt (dump_file, g, 0, TDF_VOPS|TDF_MEMSYMS);
              }
         }
       }

    return 0;

    if (dump_file)
      {
        fprintf (dump_file, "\n\n##### End ctyler diagnostics, start regular dump of current gimple #####\n\n\n");
      }

  }


} // anon namespace

gimple_opt_pass *
make_pass_ctyler (gcc::context *ctxt)
{
  return new pass_ctyler (ctxt);
}

// ============================================================= ^^^

I first noticed that the if loop was placed below a return statement, making it ineffective. I fixed this by moving the if loop above the return statement.

Then, I decided to break the code down into sections to understand the logic of the demo pass better:

Metadata Declaration: in the first section, the metadata of the pass is defined (type, name, options, and properties)
Pass Object: a pass object of type Gimple Opt Pass is declared
Gate Method: this method determined whether the pass should run.
Execution Logic: the actual logic of the pass is implemented in the execute function. It uses the FOR_EACH_BB_FN macro to loop through basic blocks. Inside this loop, a nested loop iterates through statements, printing the statement number and content
The if loop to print the final message was initially placed below the return statement, making it unreachable
Pass Object Declaration: finally, the pass object of type pass_tyler is declared

Shortly after, the updated version of the pass was released:

/* Test pass
   Chris Tyler, Seneca Polytechnic College, 2024-11 
   Modelled on tree-nrv.cc

This file is part of GCC.

GCC is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation; either version 3, or (at your option)
any later version.

GCC is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with GCC; see the file COPYING3.  If not see
<http://www.gnu.org/licenses/>.  */

#define INCLUDE_MEMORY
#include "config.h"
#include "system.h"
#include "coretypes.h"
#include "backend.h"
#include "tree.h"
#include "gimple.h"
#include "tree-pass.h"
#include "ssa.h"
#include "gimple-iterator.h"
#include "gimple-walk.h"
#include "internal-fn.h"
#include "gimple-pretty-print.h"

// Added headers:
#include "gimple-ssa.h"
#include "cgraph.h"
#include "attribs.h"
#include "pretty-print.h"
#include "tree-inline.h"
#include "intl.h"

// Included for dump_printf:
//#include "tree-pretty-print.h"
//#include "diagnostic.h"
//#include "dumpfile.h"
//#include "builtins.h"
//#include <stdlib.h>

// =================================================== vvv
// Test pass



namespace {

const pass_data pass_data_ctyler =
{
  GIMPLE_PASS, /* type */
  "ctyler", /* name */
  OPTGROUP_NONE, /* optinfo_flags */
  TV_NONE, /* tv_id */
  PROP_cfg, /* properties_required */
  0, /* properties_provided */
  0, /* properties_destroyed */
  0, /* todo_flags_start */
  0, /* todo_flags_finish */
};

class pass_ctyler : public gimple_opt_pass
{ 
public:
  pass_ctyler (gcc::context *ctxt)
    : gimple_opt_pass (pass_data_ctyler, ctxt)
  {}

  /* opt_pass methods: */
  bool gate (function *)  final override { 
    return 1; // always execute pass
  }
  unsigned int execute (function *) final override;

}; // class pass_ctyler

unsigned int
pass_ctyler::execute (function *)
  {
    struct cgraph_node *node;
    int func_cnt = 0;

    FOR_EACH_FUNCTION (node)
      {
        if (dump_file)
          {
            fprintf (dump_file, "=== Function %d Name '%s' ===\n", ++func_cnt, node->name() );
          }
      }

    if (dump_file)
      {
        fprintf (dump_file, "\n\n#### End ctyler diagnostics, start regular dump of current gimple ####\n\n\n");
      }

    return 0;

  }


} // anon namespace

gimple_opt_pass *
make_pass_ctyler (gcc::context *ctxt)
{
  return new pass_ctyler (ctxt);
}

// =================================================== ^^^

The contents of the dump file:

Hello World Program, pass output

;; Function main (main, funcdef_no=0, decl_uid=3267, cgraph_uid=1, symbol_order=0)

=== Function 1 Name 'printf' ===
=== Function 2 Name 'main' ===


#### End ctyler diagnostics, start regular dump of current gimple ####


int main ()
{
  int D.3270;
  int _3;

  <bb 2> :
  printf ("Hello");
  _3 = 0;

  <bb 3> :
<L0>:
  return _3;

}

This pass uses a different accessor macro called FOR_EACH_FUNCTION, which cycles through the code and prints the function name; this version of the pass seems to have a considerably simpler logic.

To test the pass, I extracted the test case files provided by our professor in the /public/spo600-test-clone.tgz file.

In the Makefile, I set DUMP_ALL to 1 to make sure that all files would be dumped into the current directory during the build process.

The test files contain two implementations of the scale_samples function, which were effectively identical, and two implementations of the scale_sample function, which were significantly different. The goal of our pass = pruning the identical functions while leaving the significantly different ones untouched.

After building the files, our pass generated two dump files:

Each dump file listed the function names at the top, followed by the full content of the dump. Since the files are quite lengthy, I can’t display them in full here, but they looked something like this:

;; Function main (main, funcdef_no=24, decl_uid=3961, cgraph_uid=25, symbol_order=24) (executed once)

=== Function 1 Name '__builtin_cpu_supports' ===
=== Function 2 Name '__builtin_cpu_init' ===
=== Function 3 Name 'scale_samples.resolver' ===
=== Function 4 Name 'scale_samples' ===
=== Function 5 Name 'scale_samples.arch_x86_64_v3' ===
=== Function 6 Name 'printf' ===
=== Function 7 Name 'vol_createsample' ===
=== Function 8 Name 'calloc' ===
=== Function 9 Name 'main' ===
=== Function 10 Name 'scale_samples' ===
=== Function 11 Name 'sum_sample' ===


#### End ctyler diagnostics, start regular dump of current gimple ####


int main ()
{
  unsigned long ivtmp.123;
  vector(4) int vect_t_16.116;
  vector(4) int vect_t_22.115;
  vector(4) int vect__14.114;
  vector(8) short int vect__13.113;
  int t;
  int16_t * out;
  int16_t * in;
  void * _10;
  unsigned long _12;
  int _33;
  int _34;

  <bb 2> [local count: 10737416]:
  # DEBUG BEGIN_STMT
  # DEBUG BEGIN_STMT
  # DEBUG ttl => 0
  # DEBUG BEGIN_STMT
  # DEBUG BEGIN_STMT
  # DEBUG BEGIN_STMT
  in_3 = calloc (50000000, 2);
  # DEBUG in => in_3
  # DEBUG BEGIN_STMT
  out_5 = calloc (50000000, 2);
  # DEBUG out => out_5
  # DEBUG BEGIN_STMT
  vol_createsample (in_3, 50000000);
  # DEBUG BEGIN_STMT
  scale_samples (in_3, out_5, 50000000, 50);
  # DEBUG BEGIN_STMT
  # DEBUG buff => out_5
  # DEBUG samples => 50000000
  # DEBUG INLINE_ENTRY sum_sample
  # DEBUG BEGIN_STMT
  # DEBUG BEGIN_STMT
  # DEBUG BEGIN_STMT
  # DEBUG x => 0
  # DEBUG t => t_18(D)
  # DEBUG BEGIN_STMT
  ivtmp.123_21 = (unsigned long) out_5;
  _12 = ivtmp.123_21 + 100000000;

  <bb 3> [local count: 1063004408]:
  # vect_t_22.115_20 = PHI <vect_t_16.116_15(5), { 0, 0, 0, 0 }(2)>
  # ivtmp.123_31 = PHI <ivtmp.123_22(5), ivtmp.123_21(2)>
  # DEBUG x => NULL
  # DEBUG t => NULL
  # DEBUG BEGIN_STMT
  # DEBUG D#26 => D#27 * 2
  # DEBUG D#25 => out_5 + D#26
  _10 = (void *) ivtmp.123_31;
  vect__13.113_25 = MEM <vector(8) short int> [(int16_t *)_10];
  vect__14.114_24 = [vec_unpack_lo_expr] vect__13.113_25;
  vect__14.114_23 = [vec_unpack_hi_expr] vect__13.113_25;
  vect_t_16.116_19 = vect_t_22.115_20 + vect__14.114_24;
  vect_t_16.116_15 = vect_t_16.116_19 + vect__14.114_23;
  # DEBUG D#24 => *D#25
  # DEBUG D#23 => (int) D#24
  # DEBUG t => D#22
  # DEBUG BEGIN_STMT
  # DEBUG x => NULL
  # DEBUG t => D#22
  # DEBUG BEGIN_STMT
  ivtmp.123_22 = ivtmp.123_31 + 16;
  if (_12 != ivtmp.123_22)
    goto <bb 5>; [98.99%]
  else
    goto <bb 4>; [1.01%]

  <bb 5> [local count: 1052266995]:
  goto <bb 3>; [100.00%]

  <bb 4> [local count: 10737416]:
  _33 = .REDUC_PLUS (vect_t_16.116_15);
  _34 = t_18(D) + _33;
  # DEBUG BEGIN_STMT
  # DEBUG buff => NULL
  # DEBUG samples => NULL
  # DEBUG x => NULL
  # DEBUG t => NULL
  # DEBUG ttl => _34
  # DEBUG BEGIN_STMT
  printf ("Result: %d\n", _34);
  # DEBUG BEGIN_STMT
  return 0;

}

This is just one of the blocks from the dump file, specifically from the main function. My goal is to compare the binaries of the two cloned functions within the dump file to check if they are identical. However, as discussed in class (and as John pointed out), the binaries of the two cloned functions in the dump file are not identical, even when the functions themselves are effectively the same.

For a deeper dive into this issue, you can check out my classmate John’s blog, where he explores this problem in detail, providing really good examples.

My Approach

Looking at the problem from a bird’s eye view, I think we should first develop a working logic that compares the binaries of both of the cloned functions, even if it classifies them as no-prune due to some minor differences, it’s essential to establish a basic comparison mechanism. Once this is achieved, we can fine-tune the comparing logic to ignore some specific parts of the binaries. For example:

Credit: Borrowed from John’s blog
Identifying the Essence of a Function Clone

In this specific example, we could focus on checking the type instead of performing a word-by-word comparison. The advantage of this approach is that it at least provides some working foundation to build upon.

However, developing this logic presents challenges since it goes far beyond simple loops and logic operations typically used in day-to-day coding. I spent a significant amount of time exploring the GCC Gimple Documentationto better understand the required macros and functions. But, to be honest, navigating the documentation felt like diving into a rabbit hole... There is no efficient way to search for a specific macro or loop, so finding clues/helpful pointers usually requires looking through the whole docs. On top of that, some of the documentation sections are incomplete, unclear, or perhaps just too complex for me to understand.

While doing the experiments, I was able to come up with draft logic for the execute and compare_gimple_functions:

bool compare_gimple_functions(gimple_seq g1, gimple_seq g2) {
    gimple_stmt_iterator gsi1 = gsi_start(g1);
    gimple_stmt_iterator gsi2 = gsi_start(g2);

    while (!gsi_end_p(gsi1) && !gsi_end_p(gsi2)) {
        gimple *stmt1 = gsi_stmt(gsi1);
        gimple *stmt2 = gsi_stmt(gsi2);

        // Check if the statements are equivalent
        if (!stmt1 || !stmt2 || !simple_cst_equal(stmt1, stmt2)) {
            return false;
        }

        gsi_next(&gsi1);
        gsi_next(&gsi2);
    }

    return gsi_end_p(gsi1) == gsi_end_p(gsi2);
}

unsigned int pass_ctyler::execute(function *) {
    struct cgraph_node *node;

    FOR_EACH_FUNCTION(node) {
        if (!node->analyzed)
            continue;

        std::string function_name = IDENTIFIER_POINTER(DECL_NAME(node->decl));
        if (function_name.find("_1") != std::string::npos) {
            size_t pos = function_name.find_last_of("_");
            std::string base_func_name = function_name.substr(0, pos);

            struct cgraph_node *base_node = node->get_base_function();
            if (base_node && base_node != node) {
                if (compare_gimple_functions(base_node->get_body(), node->get_body())) {
                    fprintf(dump_file, "PRUNE: %s\n", base_func_name.c_str());
                } else {
                    fprintf(dump_file, "NOPRUNE: %s\n", base_func_name.c_str());
                }
            }
        }
    }
    return 0;
}

The logic I developed currently has three errors, mainly due to incorrect usage of GIMPLE functions and macros. I tried digging into cgraph.cc to better understand the issue, but this only added to my confusion. Unfortunately, I ended up corrupting the project and had to rebuild it from scratch. For now, my logic attempts to get the binaries for the cloned functions and compare them using the compare_gimple_functions; if they are the same, it is supposed to print PRUNE, and if they differ, it will print NOPRUNE.

Afterthoughts

This code is the result of a lot of trial and error, and although I don’t have any feasible results to show, I will continue learning more about the code base and try to fix my logic.

All in all, this journey has been rewarding despite constant setbacks due to the extreme complexity of the GCC codebase.

GCC: Automatic Function Multi-Versioning Pt.1

arinak1017 — Fri, 06 Dec 2024 03:22:53 +0000

Before we start...

If you have just stumbled upon my SPO600 series of blog posts, it has been created to document and share my learnings as I progress through my Software Portability and Optimization college course.

In this blog post, I’ll start sharing the progress I made after transitioning to Stage 2 of our class project: implementing a new experimental feature in the GNU Compiler Collection (GCC) — Automatic Function Multi-Versioning.

What is Automatic Function Multi-Versioning?

As promised in my 1st project-related blog post, before diving into the details of Project Stage 2, I’d like to give you a better idea of the project's specifics.

Automatic Function Multi-Versioning should enable the compiler to automatically identify functions that could benefit from being cloned and optimized for specific architectural variants without requiring developers to modify the source code. By using a compiler option to specify the target architectures, AFMV will clone, optimize, and prune redundant function versions, enhancing software performance without requiring any manual efforts from the devs.

However, this 2024 Fall term, our focus is determining how to detect when two function versions are "fundamentally the same," which is essential for implementing function pruning.

Project Stage 2 Objective

In Stage 2 of the Project, we are aiming to add a compilation pass to the GCC compiler that determines whether functions should be pruned. This pass should iterate through the program's code, identify cloned functions, and analyze their gimple representations to determine if their implementations are identical. If duplicates are detected, the pass will output a decision indicating that the redundant clones can be pruned (by returning true or false).

To limit the scope, we will be making the following assumptions:

There is only one cloned function in a program

There are only two versions of that function.

Initial Setup

For the initial setup, I created a folder called Stage-2 in my home directory, along with the following sub-folders:

/src - contains the source code of the GCC compiler

/build - used to configure and build the compiler

/install - serves as the destination directory for installing the GCC binaries

The server where I initially built the compiler was put down, so I rebuilt it on our class x86_64 server. The clean build took about 22 minutes.

You can read about building GCC from source in my previous blog post: Building GCC from Source

Adding a New Pass

Before conducting any code experiments, I made sure that I could successfully integrate new passes into GCC.

Our professor, Chris Tyler, gave us his demo pass as a starter code for Project Stage 2. The pass was located in the public folder (/public/spo600-gcc-pass-demo.tgz). I moved it to my home directory and extracted it using the following command:

tar -xvzf spo600-gcc-pass-demo.tgz

The extracted files included three modified files from the /gcc subdirectory of the source tree (Makefile.in, passes.def, tree-pass.h) and one new file, tree-ctyler.cc, containing the pass logic.

Let's Take a Look at the Files, What Changes Were Made to Add a Pass?

Makefile.in: on line 1712, tree-ctyler.o was added. tree-ctyler.o is a language-independent object file generated from the corresponding source file tree-ctyler.cc.
```
tree-ctyler.o \
```
passes.def: this file defines the sequence of passes that transform and optimize code during compilation. The demo pass, pass_ctyler, is added on line 444, ensuring it is executed late in the compilation process, after all significant optimizations have been completed.
```
NEXT_PASS (pass_ctyler)
```
tree-pass.h: this file contains the structures for defining optimization passes in GCC. It includes a new line that declares the entry point for our custom pass. The declaration introduces the external function make_pass_ctyler, which initializes the pass in the GCC compiler framework. This function takes a pointer to a gcc::context object as its argument and returns a pointer to a gimple_opt_pass object, representing the configuration of our custom pass.
```
//Line: 491
extern gimple_opt_pass *make_pass_ctyler (gcc::context *ctxt);
```

tree-ctyler.cc: this file contains the logic of the demo pass. The pass iterates over each basic block and statement within a function, printing diagnostic information about the number of basic blocks and statements and detailed GIMPLE statements if a dump file is enabled.

// Omitted the License and the include statements
// ============================================================= vvv
// Test pass
namespace {

const pass_data pass_data_ctyler =
{
  GIMPLE_PASS, /* type */
  "ctyler", /* name */
  OPTGROUP_NONE, /* optinfo_flags */
  TV_NONE, /* tv_id */
  PROP_cfg, /* properties_required */
  0, /* properties_provided */
  0, /* properties_destroyed */
  0, /* todo_flags_start */
  0, /* todo_flags_finish */
};

class pass_ctyler : public gimple_opt_pass
{ 
public:
  pass_ctyler (gcc::context *ctxt)
    : gimple_opt_pass (pass_data_ctyler, ctxt)
  {}

  /* opt_pass methods: */
  bool gate (function *)  final override { 
    return 1; // always execute pass
  }
  unsigned int execute (function *) final override;

}; // class pass_ctyler

unsigned int
pass_ctyler::execute (function *fun)
  {
    basic_block bb;

    int bb_cnt = 0, stmt_cnt = 0;

    FOR_EACH_BB_FN (bb, fun)
      {
        bb_cnt++;
        if (dump_file)
          {
            fprintf (dump_file, "===== Basic block count: %d =====\n", bb_cnt);
          }

        for (gimple_stmt_iterator gsi = gsi_start_bb (bb); !gsi_end_p 
    (gsi); gsi_next (&gsi))
          {
            gimple *g = gsi_stmt (gsi);
            stmt_cnt++;
            if (dump_file)
              {
                fprintf (dump_file, "----- Statement count: %d -----\n", stmt_cnt);
                print_gimple_stmt (dump_file, g, 0, 
TDF_VOPS|TDF_MEMSYMS);
              }
         }
       }

    return 0;

    if (dump_file)
      {
        fprintf (dump_file, "\n\n##### End ctyler diagnostics, start regular dump of current gimple #####\n\n\n");
      }

  }

} // anon namespace

gimple_opt_pass *
make_pass_ctyler (gcc::context *ctxt)
{
  return new pass_ctyler (ctxt);
}

// ============================================================= ^^^

While the GCC documentation is rather sparse, you can find some helpful info on adding a pass here:

Pass manager (GCC Internals)

Writing a new pass (GCC Wiki)

Rebuilding GCC with a New Pass

After wrapping my head around the changes required to add the sample pass, I integrated it into the source code and rebuilt the project.

I started by copying the necessary source code files using the following commands:

[akolodeznikova@x86-001 gcc]$ pwd
/home/akolodeznikova/Stage-2/src/gcc
cp ../../../test/gcc/Makefile.in .
cp ../../../test/gcc/passes.def .
cp ../../../test/gcc/tree-pass.h .
cp ../../../test/gcc/tree-ctyler.cc .

Then, I went into the build folder and ran the make command to rebuild the project with applied changes.

The build process took approximately five minutes. Once completed, I ran the make install command to install the newly built GCC compiler into the /install folder.

Note: the make utility efficiently rebuilds a codebase by comparing the timestamps of dependencies (inputs) and targets (outputs). It determines which source files have been modified and rebuilds only the affected targets (Chris Tyler’s Wiki).

With the build and installation complete, I tested the setup to verify that the sample pass was working.

To do this, I quickly created a sample file in the test folder:

hello.c

#include<stdio.h>

int main(){
    printf("Hello");
    return 0;
}

And then compiled it, triggering the newly added pass using the following command:

~/Stage-2/install/bin/gcc hello.c -fdump-tree-ctyler

After running the command, a file named a-hello.c.263t.ctyler was generated. The file contained the following output:

//a-hello.c.263t.ctyler
;; Function main (main, funcdef_no=0, decl_uid=3267, cgraph_uid=1, symbol_order=0)

===== Basic block count: 1 =====
----- Statement count: 1 -----
# .MEM_2 = VDEF <.MEM_1(D)>
printf ("Hello");
----- Statement count: 2 -----
_3 = 0;
===== Basic block count: 2 =====
----- Statement count: 3 -----
<L0>:
----- Statement count: 4 -----
# VUSE <.MEM_2>
return _3;
int main ()
{
  int D.3270;
  int _3;

  <bb 2> :
  printf ("Hello");
  _3 = 0;

  <bb 3> :
<L0>:
  return _3;

}

The output confirmed the added pass was working as expected. It successfully iterated through the code, producing diagnostic information about basic blocks and statements.

🎉 🎉 🎉

Stay tuned...

In my next post, I’ll share the code experiments I conducted to implement the proposed functionality. So, stay tuned for more updates on my progress!

Publishing ADDCOM!

arinak1017 — Sat, 23 Nov 2024 00:46:53 +0000

Hello, Blog!

If you have just stumbled upon my OSD600 series of blog posts, it has been created to document and share my learnings as I progress through my Open Source Development college course.

In this blog post, I want to go over the steps I took to publish my CLI tool - addcom to PyPi.

arilloid / addcom

a CLI tool for adding comments to your source code files

ADDCOM

addcom is a CLI source code documenter tool which provides coders with an easy way to add comments to their source code files Give it a relative/absolute path to your file and it will analyze its contents and add comments using a Large Language Model's chat completion.

See a demo of the functionality on YouTube: addcom-demo

Setup Instructions

Prerequisites

Make sure Python is installed on your system (you can download it here: https://www.python.org/downloads/).

1. Install the tool using `pip`

pip install addcom

2. Default: Create an account and generate the API key here: https://console.groq.com/

By default, addcom uses the Groq API endpoint for chat completion. However, you can specify a custom endpoint using the --base-url or -u flag option. (If you do this, make sure to obtain an appropriate API key and specify the model supported by the chosen provider using the --model/ -m option).

3. Set

…

View on GitHub

I welcome feedback/contributions, so don’t hesitate to take a look and get involved 😊

Preparing for Release

Before releasing my tool, I integrated Poetry for dependency management and packaging into my project, replacing the good old setuptools. This decision was inspired by recommendations from peers and various blog posts highlighting Poetry’s simplicity and modern features.

I removed the setup.py file, which handled the tools' packaging. Then, moved the metadata from __init__.py into a pyproject.toml file, which I created a couple of labs back to add configs for the source code formatter and linter.

Additionally, I did a bit of refactoring to make the project's structure a bit cleaner. To do so I moved all source code files into a single app package and updated the imports across the project.

For guidance, I referred to Poetry's official documentation and studied similar Python projects on GitHub.

Below are the final configurations I used for Poetry:

# Poetry - project metadata
[tool.poetry]
name = "addcom"
version = "1.0.2"
description = "a CLI tool for adding comments to your source code files"
authors = ["Arina Kolodeznikova <arinak1017@gmail.com>"]
license = "MIT"
readme = "README.md"
repository = "https://github.com/arilloid/addcom"
packages = [{include = "app"}]

# Dependencies required for running the project
[tool.poetry.dependencies]
python = "^3.12"
typer = "^0.12"
openai = "^1.46"
rich = "^13.8"
typing_extensions = "^4.12"

# Dev dependencies (not required for run-time)
[tool.poetry.group.dev.dependencies]
black = "^24.10"
ruff = "^0.7"
pytest = "^8.3"
pytest-mock = "^3.14"

# Project's CLI commands
[tool.poetry.scripts]
addcom = 'app.main:app'

# Build configurations
[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

Publishing addcom to PyPi

After setting up Poetry and verifying that everything was working as expected, I tweaked my CI workflow file, updated the tool’s documentation, and went on to publishing my tool to PyPi (The Python Package Index).

You can find the guide that I followed here: Publishing packages with Poetry

Steps to publish

1. Tagging the Release

I started by tagging the release using an annotated Git tag to mark the version:

git tag -a v1.0.0 -m "my version 1.0.0"

2. Configuring PyPI Credentials

Next, I configured Poetry to use my PyPI credentials (this step required obtaining a PyPI API token):

poetry config pypi-token.pypi <your-pypi-token>

To generate a PyPI token, I created an account on PyPI, navigated to Account Settings, and selected Add API Tokens.

3. Building and Publishing the Package

Finally, I built and published the package using Poetry (which I specified as the build backend in the pyproject.toml):

poetry publish --build

You can find my release here: addcom

Testing the Release

After testing the release myself, I invited my friend, roommate, and fellow developer, Tatiana, for a dedicated testing session. Although I was nervous to show her my tool, the testing process went pretty smoothly, thanks to the effort I put into creating detailed instructions in the projects README.md. However, there were still a few issues during the review. The instructions for setting up the API key were poorly structured, causing confusion. Tatiana also noted that the .toml configuration section assumed users were working from a cloned repository, which wouldn't be the case for most users installing the tool using pip. The most significant feedback, which echoed a remark made by my professor David, was about the tool’s handling of multiple files. When generating comments for several files, the tool saved everything to a single file, with no option to split the output. Additionally, at times the generated comments were inconsistent, even with the given "context" code used for reference.

To address these issues, I edited the API key setup instructions in the README and updated the .toml configuration section. As for the file output logic and comment consistency, I plan to look into updating them after the term comes to an end.

How to install ADDCOM

You can now install addcom using pip:

pip install addcom

By default, addcom uses the Groq API for chat completion. To get started quickly, create an account on the Groq Console, generate an API key, and run the tool with the following command:

addcom FILE_PATH(S) -a "your_api_key"

For detailed information on the tool's functionality, usage, and instructions, visit the release page on PyPI.

Contributing to ByteChef

arinak1017 — Thu, 21 Nov 2024 08:06:10 +0000

Before we start...

If you have just stumbled upon my OSD600 series of blog posts, it has been created to document and share my learnings as I progress through my Open Source Development college course.

After completing Hacktoberfest, my classmates and I were challenged to make more substantial Open Source contributions. It was time to move beyond typo fixes and minor code changes and focus on implementing full-fledged features.

If you read my Hacktoberfest Recap post, you might remember that my largest contribution was adding a settings variable to one project's backend. This was mainly because, in the final week of Hacktoberfest, I decided to unassign myself from a more difficult issue that involved implementing a feature in the Java Spring Boot backend of a project called ByteChef.

As a result, when the new course task was announced, I knew I wanted to revisit ByteChef and face my fear of Java!

In this blog post, I'll share how I learned from the mistakes of my previous contribution attempt and successfully contributed two new features to ByteChef.

A bit about ByteChef

ByteChef is an open-source, low-code platform for API integration and workflow automation (currently under active development). It aims to help users easily create and manage workflows across different applications and systems by adding components (each responsible for interacting with a specific service) and defining their relations.

Cool fact: the company is based in Croatia! 🌍

bytechefhq / bytechef

Open-source, low-code, extendable API integration & workflow automation platform. Integrate your organizations or your SaaS product with any third party API

API integration and workflow automation platform

Website - Documentation - Discord - Twitter

UPDATE: ByteChef is under active development. We are in the alpha stage, and some features might be missing or disabled.

What is ByteChef?

ByteChef is an open-source, low-code, extendable API integration and workflow automation platform. ByteChef can help you as:

An automation solution that allows you to integrate and build automation workflows across your SaaS apps, internal APIs, and databases.
An embedded solution targeted explicitly for SaaS products, allowing your customers to integrate applications they use with your product.

Key Features

Intuitive UI Workflow Editor: build and visualize workflows via the UI editor by dragging and dropping components and defining their relations.
Event-Driven & Scheduled Workflows: automate scheduled and real-time event-driven workflows via a simple trigger definition.
Multiple flow controls: use the range of various flow controls such as condition, switch, loop, each, parallel, etc.
Built-In Code…

View on GitHub

The 1-st Issue I worked on

I decided to give myself a head start by picking up an issue similar to the one I had previously dropped (Baserow - Get Row action #1643). So, I found an issue related to the same workflow component and reached out to the issue creator, asking for a second chance to contribute.

Comment for #1642

arilloid commented on Nov 05, 2024

Hello @monikakuster ,

I previously had to unassign myself from the related issue due to some personal circumstances, but I now have more time and resources to tackle adding new actions to the Baserow component. Could you please assign me to this issue? I’m ready to get started and see it through this time.

Thank you!

View on GitHub

To be honest, I felt a bit nervous returning to the project, as I was still ashamed for unassigning myself from the previous issue. However, the project maintainer was kind enough to give me another shot.

The Issue I ended up working on: The task was to implement the Delete Row action for the Baserow (Open Source database platform) component, enabling users to remove specific rows from a table in their Baserow database.

Baserow - Delete Row action #1642

monikakuster posted on Oct 22, 2024

Implement the Delete Row action for the Baserow component to enable users to remove specific rows from a table within their Baserow database.

Action properties:

table id - id of the table from which the row should be deleted
row id - id of the row that needs to be deleted

Ouput:

no output

Documentation Reference: https://baserow.io/api-docs

</div>
<div class="gh-btn-container"><a class="gh-btn" href="https://github.com/bytechefhq/bytechef/issues/1642">View on GitHub</a></div>

Contribution Process

Dev Environment Setup

During my first attempt, I struggled to get the project running, even though I closely followed the instructions in the CONTRIBUTING.md (or at least I thought I did), which left me unable to test my draft code. This time, I decided to focus on correctly setting up my development environment from the very beginning.

Here is an overview of the entire setup process, combining the problems I faced during both my initial and current attempts:

The first issue I encountered was an error when cloning the repository: "File name too long" / "Clone succeeded but checkout failed." I resolved this by configuring Git to accept long paths.

Next, I downloaded the compatible version of the Java SDK (21+) and the Java support extension for VS Code. I then started the necessary Docker containers for the dependent infrastructure (Redis, PostgreSQL, and Mailhog). After compiling the codebase, I attempted to start the server but encountered an error with a terrifyingly long stack trace. I thought the issue was related to unresolved dependencies, as, at first glance, the error message seemed to point in that direction. So, I checked the prerequisites and tried rebuilding the project, but the problem persisted. Upon further investigation (of the stack trace that was too long for the LLM to process), I discovered that the issue was related to the server's inability to connect to PostgreSQL. I checked the PostgreSQL Docker container's logs, but everything appeared to be running as expected. Confused, I decided to seek help on the project's Discord. One of the maintainers asked if I was trying to access an external database, and that’s when the realization struck me: something else was likely occupying the port where the PostgreSQL container was supposed to run. Sure enough, I found that my local PostgreSQL server server was already using that port. After terminating the process and restarting the Docker containers, the server started up as expected.

With the development environment finally set up, I was ready to move forward with implementing a feature.

Feature Implementation

As a Java noob, I was intimidated by the project, especially since I had no experience with Gradle or Java Spring Boot.

On my first attempt, I started by locating where the component actions needed to be implemented. Although I found the relevant functions, I quickly realized that this approach wasn't ideal. Without a clear understanding of the project setup, I struggled to make sense of imported custom classes, methods, and constants, and, inevitably, got overwhelmed.

On my second attempt, I took a step back to study the project structure and familiarize myself with the tools used. I also revisited the project's history to see how similar features had been implemented, which helped me understand the workflow. (By this point, more component actions had been implemented, so I had a chance to learn by looking at previous PRs + the docs on adding new components/component actions were updated, making the implementation process clear).

Lesson learned: It's much easier to tune into a project that is further along in its development, as there's more work to reference and learn from.

After studying the project, I began coding by following the instructions in the ByteChef Developer Guide and referencing existing actions for guidance.

I discovered that creating an action starts with the action() method from the ComponentDsl package. It initializes an instance of ModifiableActionDefinition which provides various methods, such as title(), description(), properties(), and perform(), to define the action’s properties and logic. Once all the properties are defined, the action is registered in the component handler, making it available for use.

For the action I was implementing, no output was required. Its sole purpose was to send an HTTP DELETE request to a specific API endpoint, using a table ID and a row ID as inputs. I constructed the URL for this request by referring to the Baserow documentation (you need to create an account to see the docs!). To execute the request, I utilized the http() method provided by the ActionContext class.

Overall, the project already provided all the necessary building blocks. My main task was to understand how these components worked together and supply the required parameters to define the action correctly. Once I understood the structure and flow, implementing the action became a straightforward and logical process.

BaserowDeleteRowAction.java:

package com.bytechef.component.baserow.action;

import static com.bytechef.component.baserow.constant.BaserowConstants.ROW_ID;
import static com.bytechef.component.baserow.constant.BaserowConstants.TABLE_ID;
import static com.bytechef.component.definition.ComponentDsl.action;
import static com.bytechef.component.definition.ComponentDsl.integer;

import com.bytechef.component.definition.ActionContext;
import com.bytechef.component.definition.ComponentDsl.ModifiableActionDefinition;
import com.bytechef.component.definition.Parameters;

/**
 * @author Arina Kolodeznikova
 */
public class BaserowDeleteRowAction {

    public static final ModifiableActionDefinition ACTION_DEFINITION = action("deleteRow")
        .title("Delete Row")
        .description("Deletes the specified row.")
        .properties(
            integer(TABLE_ID)
                .label("Table ID")
                .description("ID of the table containing the row to be deleted.")
                .required(true),
            integer(ROW_ID)
                .label("Row ID")
                .description("ID of the row to be deleted.")
                .required(true))
        .perform(BaserowDeleteRowAction::perform);

    private BaserowDeleteRowAction() {
    }

    protected static Object perform(
        Parameters inputParameters, Parameters connectionParameters, ActionContext actionContext) {

        actionContext
            .http(http -> http.delete(
                "/database/rows/table/" + inputParameters.getRequiredString(TABLE_ID) + "/" +
                    inputParameters.getRequiredString(ROW_ID) + "/"))
            .execute();

        return null;
    }
}

Testing

After implementing the action, I tested the row deletion by rebuilding the project, starting the server and client, creating a test table in my Baserow account, setting up the test workflow in the frontend, adding the delete row action for the Baserow component, connecting the component to the service using my Baserow credentials, inputting the table and row IDs, and executing the workflow — and it worked as expected! (For details on creating and testing components, refer to ByteChef's Quick Start Guide).

Creating the Delete Row action:

Workflow successfully running:

The row I specified (row #1) is no longer in the DB:

After confirming that my action worked as expected, I began working on the unit test. I reviewed tests for other Baserow actions and found that they were fairly simple, involving the use of JUnit and Mockito for mocking API interactions and input parameters to ensure the correct API calls were made and expected results returned.

Following this approach, I mocked the ActionContext, Http.Executor, and Http.Response to simulate the API. Using MockParametersFactory, I created mock parameters for the table and row IDs. I then configured the Http.Executor to return the mocked response and tested the perform() method, confirming it returned null as expected for a successful row deletion. (See my unit test here).

After several iterations of formatting the code, running tests, and fixing minor issues, all tests passed, and my changes were ready for a PR. I went ahead and submitted a Pull Request, and it was merged the following day!

1642 - baserow delete row action #1743

arilloid posted on Nov 15, 2024

Description

Implemented the Delete Row action for the Baserow component to enable users to remove specific rows from a table within their Baserow database.

Fixes #1642

Type of change

New feature (non-breaking change which adds functionality)

How Has This Been Tested?

Added a new unit test and executed it alongside the existing test in the module to ensure all tests pass successfully.

Manually tested the action by creating and executing a test workflow on the client side.

Created a test workflow that included a Baserow delete row action. Configured the workflow by adding the authentication token to connections and specifying the table ID and row ID for the testing Baserow table:

Executed the workflow:

Checked the specified Baserow database table to confirm that the specified row was successfully deleted.:

Checklist:

[x] My code follows the style guidelines of this project
[x] I have performed a self-review of my own code
[ ] I have commented my code, particularly in hard-to-understand areas
- Referenced the code for other Baserow component actions
[ ] I have made corresponding changes to the documentation
- Running gradlew generateDocumentation for the module did not produce any documentation updates
[x] My changes generate no new warnings
[x] I have added tests that prove my fix is effective or that my feature works
[x] New and existing unit tests pass locally with my changes

View on GitHub

The 2-nd issue I worked on

After my first PR was merged, I was curious to implement an action for another component to explore how the backend integrates interactions with other services. I checked the open issues and found one related to adding an action for the Google Drive component.

Google Drive - Get File action #1647

monikakuster posted on Oct 24, 2024

The Get File action for the Google Drive component is designed to retrieve a specified file from the user's Google Drive. This action allows users to access file metadata and content.

Action properties:

file ID - the ID of the file you wish to retrieve

Ouput:

file - a file object containing detailed information about the retrieved file.

Documentation Reference: https://developers.google.com/drive/api/reference/rest/v3/files/get

View on GitHub

I left a comment asking to be assigned the issue and also requesting feedback on possible areas for improvement in my previous PR.

Although I didn't receive any feedback, the issue was assigned to me a couple of days later. At first, I was a bit surprised by the delay in responses (mainly because the maintainers had been super quick to reply and assign issues before), but after a quick Google search, I learned that there was a long weekend in Croatia, where most of the project maintainers are based.

Note: The ability to put yourself in someone else's shoes is especially important when working with people from all around the world. As for me, I certainly wouldn't want to be bothered during the holidays.

Contribution Process

Implementing the feature

I was already familiar with the project, so writing the second feature was much easier. The main difference between writing the action for Baserow and Google Drive was that it used different custom classes. So, this time, I essentially needed to familiarize myself with a few new "building blocks", the key one being the Google API SDK library for Java.

I read the documentation reference linked by the maintainer and realized that I saw the same method in the Google Drive Copy File action. By extracting the necessary components from the existing logic, I was able to write an action that returns the metadata for the selected file in the linked Google Drive (See the code on GitHub).

Testing the feature

Manually testing the feature was fun, as I got to set up a Google Cloud account and OAuth2 credentials to access my Google Drive from the frontend application. (I followed the instructions for the Google Drive component in the docs to generate the necessary credentials).

Retrieving the metadata for one of my Google Drive documents:

After manually testing the feature in the frontend, I created a unit test in the same way as before, using JUnit and Mockito (GoogleDriveGetFileActionTest.java).

Then, I ran the code formatter, ensured that my code passed all the checks, and submitted a Pull Request, asking a few questions about the expected output in the comments. (However, as of writing this blog post, the maintainers haven’t yet responded).

1647 - Added Google Drive Get File Action #1753

arilloid posted on Nov 19, 2024

Description

Added Get File action to the Google Drive component.

Fixes #1647

Type of change

New feature (non-breaking change which adds functionality)
This change requires a documentation update

How Has This Been Tested?

Added a new unit test and executed it alongside the existing test in the module to ensure all tests pass successfully.
Manually tested the action by creating and executing a test workflow on the client side.

Checklist:

[x] My code follows the style guidelines of this project
[x] I have performed a self-review of my own code
[ ] I have commented my code, particularly in hard-to-understand areas
- Referenced the code for other Google Drive component actions
[ ] I have made corresponding changes to the documentation
- Running gradlew generateDocumentation for the module did not produce any documentation updates
[x] My changes generate no new warnings
[x] I have added tests that prove my fix is effective or that my feature works
[x] New and existing unit tests pass locally with my changes

View on GitHub

Afterthoughts

Contributing to ByteChef was a valuable learning experience. I got to author new features and unit tests, learned the importance of taking time to understand the project structure and leveraging existing code as a reference for implementing new features. + I gained hands-on experience with new tools and libraries, including Gradle, Baserow, the Google Drive API SDK, JUnit, and Mockito.

Adding CI Workflows with GitHub Actions

arinak1017 — Sat, 16 Nov 2024 23:53:15 +0000

Hello, Blog!

If you have just stumbled upon my OSD600 series of blog posts, it has been created to document and share my learnings as I progress through my Open Source Development college course.

Another week, another blog post. This week, my classmates and I focused on creating a Continuous Integration workflow for our projects using GitHub Actions. Our mission was to create a GitHub Actions workflow that ensures our projects never get broken by automatically building the code and running tests on every Push and Pull Request to the main branch.

If you are curious, feel free to check out my project on GitHub:

arilloid / addcom

a CLI tool for adding comments to your source code files

ADDCOM

See a demo of the functionality on YouTube: addcom-demo

Setup Instructions

Prerequisites

Make sure Python is installed on your system (you can download it here: https://www.python.org/downloads/).

1. After cloning the repo cd into the project folder and simply run:

pip install .

2. Default: Create an account and generate the API key here: https://console.groq.com/

View on GitHub

I welcome feedback/contributions, so don’t hesitate to take a look and get involved 😊

Creating a GitHub Actions Workflow

To set up my workflow, I used the GitHub Actions web interface and followed GitHub's Building and testing Python guide. I started with the Python Application workflow template. I tweaked it to build and test my project based on the guide's instructions.

The final workflow file with the build-and-test job: python-app.yml

name: Run Unit Tests

on:
  push:
    branches: [ "main" ]
  pull_request:
    branches: [ "main" ]

jobs:
  build-and-test:
    runs-on: ubuntu-latest

    steps:
    - name: Checkout code
      uses: actions/checkout@v4

    - name: Set up Python
      uses: actions/setup-python@v5
      with:
        python-version: "3.12"

    - name: Build the tool
      run: |
        pip install .

    - name: Test with pytest
      run: |
        pip install pytest pytest-mock
        pytest

On every push and pull request to main, this workflow sets up a Python 3.12 environment, installs the project as a package, and runs tests using pytest to maintain code reliability.

Adding Tests to Another Project

After setting up and testing my CI workflow, the next step was adding a new test case to one of my classmates' projects. This week, I worked on Jin's project:

gitdevjin / code-mage

Description

CodeMage is a tool that translates a source file written in one programming language into another language The translation will be done by Large Language Model AI(such as ChatGPT)

Release 0.1

Features

1. Supported Languages: Javascript, Python, C++, Java
2. Default target Language is Python
3. Supported LLM model: openrouter, groq
4. Default LLM model is openrouter(sao10k/l3-euryale-70b)

Getting Started

1. Install Python : https://www.python.org/downloads/

2. Install Poetry

curl -sSL https://install.python-poetry.org | python3 -

OR if you have pipx installed on your local machine, you can use the following commend

How to download pipx

pipx install poetry

3. Clone the repository

git clone https://github.com/gitdevjin/code-mage.git
cd code_mage

4. Install Poetry Package

poetry install

5. Create your API_KEY at openrouter Or Groq

It's free with sign-up. You can easily sign-up with your google account

6. Create `.env` file in the root directory and save the following:

OPENROUTER_API_KEY=your_open_router_api_key
GROQ_API_KEY=your_groq_api_key

Now you are ready…

View on GitHub

As I scanned Jin's repository, I noticed that our testing and workflow setups share many similarities, likely because we both have Python projects and use pytest for testing. However, there were a few notable differences: Jin uses Poetry for packaging and dependency management, causing the differences in the build and test setup; he organizes test cases by file rather than by function, as I do; and he relies on unittest.mock instead of pytest-mock for mocking.

Since I was already familiar with the stack, writing a new test case for Jin's project didn’t pose any challenges. The main hurdle was identifying an untested area, as Jin had already written comprehensive tests for most of the functionality. After discussing it with him, he suggested focusing on argument parsing.

Following his recommendation, I reviewed the existing argument parsing function and tests and decided to write a test case for handling invalid flag argument options:

# Test invalid flag argument option
def test_arg_parser_invalid_option():
    with patch("sys.argv", ["code_mage.py", "--invalid"]):
        with pytest.raises(SystemExit) as excinfo:
            arg_parser({})
        assert excinfo.type is SystemExit
        assert excinfo.value.code != 0

I added this test to the tests/test_argsParser.py, ran it locally to ensure that it works as intended, and submitted a PR. My code passed Jin's CI workflow and got merged to main almost right away.

Added a new test case in test_argsParser.py #17

arilloid posted on Nov 15, 2024

Hello @gitdevjin ,

I added a new test case in test_argsParser.py to make sure that the argument parser terminates the program with a non-zero code when invoked with an invalid flag.

# Test invalid flag argument option
def test_arg_parser_invalid_option():
    with patch("sys.argv", ["code_mage.py", "--invalid"]):
        with pytest.raises(SystemExit) as excinfo:
            arg_parser({})
        assert excinfo.type is SystemExit
        assert excinfo.value.code != 0

View on GitHub

Afterthoughts

Although I wasn’t too familiar with writing CI workflows at the start of this week, the entire process turned out to be straightforward (largely thanks to the simplicity of my CLI tool). + I’m glad I had the opportunity to dive into the GitHub Actions documentation, as I know that it will certainly come in handy in my future projects.

Having been on both sides of the process - as a contributor and a repository maintainer, I can confidently say that a solid CI workflow is very helpful. It streamlines the development and provides peace of mind, knowing that no breaking changes will sneak into my main branch.

Automated Testing with Pytest

arinak1017 — Sun, 10 Nov 2024 01:06:22 +0000

Hello, Blog!

If you have just stumbled upon my OSD600 series of blog posts, it has been created to document and share my learnings as I progress through my Open Source Development college course.

Aside from continuing to work on contributing to Open Source projects, this week's focus was on looking into "managing project complexity through the use of Automated Testing" & starting to write test cases for our CLI tools.

Testing is a fundamental aspect of software development. Tests allow us to refine the code's logic, ensuring each component performs as expected even when handling edge cases. Automated tests make it easier to introduce changes to code without breaking its core logic, which is especially important in the Open Source project (as they rely on collaboration).

Helpful links:

arilloid / addcom

a CLI tool for adding comments to your source code files

ADDCOM

See a demo of the functionality on YouTube: addcom-demo

Setup Instructions

Prerequisites

Make sure Python is installed on your system (you can download it here: https://www.python.org/downloads/).

1. After cloning the repo cd into the project folder and simply run:

pip install .

2. Default: Create an account and generate the API key here: https://console.groq.com/

View on GitHub

I welcome feedback/contributions, so don’t hesitate to take a look and get involved 😊

Starting to write tests in Python

Beforehand, I had only written tests using Jest in one of my Node.js projects, so I had a general understanding of how to approach testing. However, I had no experience with testing in Python.

So, to start, I went "shopping" for the right testing library. After reading a couple of articles and community discussions, I decided to begin writing tests using Pytest in combination with the pytest-mock plugin, which I used to mock the API interactions with the LLM.

Why Pytest?

Pytest is one of the most popular testing frameworks that combines simple syntax (e.g. less boilerplate code compared to Python's built-in unittest library) with powerful features and a vast ecosystem of plugins for nearly every testing scenario.

Prep & Setup

Before diving straight into the documentation, I like to go through a few simple beginner guides to get a quick overview of the tools and their functionality. I did the same with Pytest, reading two beginner guides before jumping into setup and coding (A Beginner's Guide to Unit Testing with Pytest, Mastering Unit Tests in Python with pytest: A Comprehensive Guide).

The setup was close to none, I simply installed the library and the mocking plugin using pip, created a new tests/ folder at the root of my project, and was good to go:

pip install pytest pytest-mock
mkdir tests/

Writing the first test

I began testing with the simplest function in my project: the version_callback() function, which prints the version of the CLI tool when the --version/-v flag is provided.

I covered three test cases:

Successful version output – Verifying that the version prints correctly to the terminal when the flag is used.
No flag behaviour – Ensuring that not providing the flag does not cause any unexpected behaviour.
Handling unset version – Confirming that the function prints an appropriate message if the version is not set.

Writing the tests was straightforward, I mostly followed examples from guides and documentation, with occasional clarifications from an AI chatbot.

I used pytest assertions for stdout comparisons and pytest.raises for exception checks and also got to try out the patching to replace the imported version value with None.

import pytest
from typer import Exit
from app.core.callbacks import version_callback
from app import __version__


def test_version_callback_without_version_defined(mocker, capfd):
    """
    Test version callback function when the version is not defined
    """
    mocker.patch("app.core.callbacks.__version__", None)
    with pytest.raises(Exit):
        version_callback(provided=True)

    # Capture the output
    captured = capfd.readouterr()

    # Check that it prints a fallback message instead of the version
    assert "version not defined" in captured.out.strip()

See the full code on GitHub

Testing the core functionality/Mocking LLM interactions

The core functionality of the tool revolves around interacting with the LLM, which is handled in the generate_comments() function. To test this, I needed to mock the LLM interaction.

To simulate the OpenAI client interaction, I used the pytest-mock plugin with its mocker fixture. This allowed me to mock the behaviour of the OpenAI client and control the responses returned by its methods during testing.

In pytest, a fixture is a function that prepares and provides the necessary setup for tests. Fixtures are defined with the @pytest.fixture decorator and can be automatically passed into test functions as arguments.

@pytest.fixture
def mock_openai_client(mocker):
    """
    Fixture to mock the OpenAI client and its responses
    """
    # Create a mock object with MagicMock()
    mock_client = mocker.MagicMock()
    # Replace the client with the mock object whenever 
    # OpenaAI() is called in the code
    mocker.patch("app.core.api.OpenAI", return_value=mock_client)
    return mock_client

####################################################################

def test_generate_comments_with_api_key(test_parameters, mocker, mock_openai_client):
    """
    Test that the function generates expected output when a valid file path and API key are provided
    """
    file_path, content = test_parameters

    commented_code = """
                    def generate_fibonacci(limit):
                        # Initialize the Fibonacci sequence with the first two numbers
                        fibonacci_sequence = [0, 1]
                        # Generate Fibonacci numbers until the next number exceeds the given limit
                        while True:
                            # Calculate the next number in the sequence
                            next_number = fibonacci_sequence[-1] + fibonacci_sequence[-2]
                            # Stop generating if the next number exceeds the limit
                            if next_number > limit:
                                break
                            # Append the next number to the sequence
                            fibonacci_sequence.append(next_number)
                        return fibonacci_sequence

                    """

    # Replicate the OpenAI client response structure using mock objects
    mock_response = mocker.MagicMock()
    mock_response.choices = [
        mocker.MagicMock(message=mocker.MagicMock(content=commented_code))
    ]
    # Set the result of the mock chat completion to the mock response,
    # so when the value is accessed in the code it gives the expected result
    mock_openai_client.chat.completions.create.return_value = mock_response

    result = generate_comments(
        file_path, content, None, "test_api_key", None, None, False
    )

    assert result == commented_code

See the full code on GitHub

Afterthoughts

From the start, I tried making my code clean and modular by breaking it down into smaller functions. This approach certainly made my project more testable. But, along the way, I realized that I hadn't fully accounted for all possible edge cases ("Wait, does the logic break when this value is None?"). From now on, I will try to look at my code more critically by asking more "what if?" questions.

Writing the first few tests helped me gain a solid understanding of pytest. I learned how to do simple assertions, test exceptions, capture data written to the terminal, use fixtures for setup, and how to replace values and mock different objects using mocker.

Although testing the generate_comments() and version_callback() functions has given me a good foundation, I still need to improve test coverage, explore testing argument parsing and figure out a way to test the main entry point function.