DEV Community: Qiwen Yu

OSD600 Release 0.4: Release and Final Thoughts

Qiwen Yu — Thu, 16 Dec 2021 04:09:02 +0000

As approaching to the end of this semester, I have leared a lot about the open source project, such as how to contribute to an open source project, what is the contribution process and how to find a proper issue to work on.

The issue[https://github.com/Seneca-CDOT/telescope/issues/2511] I worked on in the release 0.4 is about a bug fixing. To investigate the code to figure out why the referenceButtons is a null. It shows that the referenceButtons has an uncaught type error.

Through investigating the code and trying to debugging this issue, I found out it is kind of difficult for me to locate the specific spot to fix this issue. I spent time on understanding the whole project structure and logic.

Sometimes it took me more time on reading and understanding the code than actually changing the code.

OSD600 Release 0.4: Progress

Qiwen Yu — Thu, 16 Dec 2021 04:08:29 +0000

Updates on Issues

At this stage of the project, I have received updates on issues, and have some thoughts on how to write pull requests to solve most issues.

IBM Issue

This issue is about creation of a customized jinja tag to insert a specific markdown content header. I have found two useful tutorials, https://github.com/dldevinc/jinja2-simple-tags and https://michaelabrahamsen.com/posts/jinja2-custom-template-tags/. With customized tag, the Python library jinja can have new functionality unique to your own project.

This issue is pretty similar to what we did about static site generator for generating a complete HTML web site from raw data and files. Jinja doesn't have the functionality that IBM wants. So this issue is about to create a jinja annotation for developers to use.
I have left the comment below this issue and communicated with the project maintainer. Turned out that, even though this issue currently shows open on the board, it actually was being fixed in another issue.
In this case, I was not be able to get this issue assigned to me.

Microsoft Issue

This issue[https://github.com/microsoft/qlib/issues/691] is a bug fixing issue for qlib. The module_path is wrong. This is a pretty simple issue. After contacting the project maintainer, it turned out this issue has been resolved in another issue as well. So that I couldn't get a chance to work on it as well.

Telescope Issues

Telescope issue - 01:

This issue[https://github.com/Seneca-CDOT/telescope/issues/2521] is about to fix the backend end point url. Now the url has ":4000" inside, and it's expected to be changed into "/deploy" to the places where we are acting as a consumer of the deploy routes vs. defining them.

Telescope issue - 02:

This issue[https://github.com/Seneca-CDOT/telescope/issues/2511] is to fix a bug of an uncaught type error.

OSD600 Release 0.4: Plan

Qiwen Yu — Wed, 15 Dec 2021 23:03:36 +0000

Introduction

In this blog, I am going to give a brief review of what I have planed to do for this stage of the project. At this stage, we are encouraged to submit more pull requests in the GitHub open source development community, meanwhile, do some code reviews in

Plan

The plan including the issues I was trying to get assigned, including an issue from IBM, this was an issue about "Create a custom jinja tag to insert a specific markdown content header". I found this issue is interesting and doable. Another issue was from Microsoft China, this is a project about Quant Trading.
Moreover, since I was not sure if I could get those issues assigned, I reached out to a Seneca Open Source project Telescope. For this project, I took two issues, the first one is a small issue of update of the API url, and the second one is a larger issue. This issue is about debugging the value of referenceButton value of the Telescope build-log dashboard. This variable had an uncaught type error, and its details remain to be investigated through the Console log.

Code Review

The most interesting part of contributing to Open Source project is reading the Code Review and providing Code Reviews. With the Seneca Open Source projects, we are given the right to give Code Reviews. So I will provide at least one Code Review during this project.

SPO600-Lab6, Project Stage 3: SVE2

Qiwen Yu — Wed, 15 Dec 2021 21:20:43 +0000

Introduction

In the previous blog, I introduced the Python numeric calculation package NumPy and the SIMD optimization implemented in it. I have checked the code:

#ifdef _MSC_VER
     #include <Intrin.h>
 #endif
 #include <arm_neon.h>

 int main(void)
 {
     float32x4_t v1 = vdupq_n_f32(1.0f), v2 = vdupq_n_f32(2.0f);
     /* MAXMIN */
     int ret  = (int)vgetq_lane_f32(vmaxnmq_f32(v1, v2), 0);
         ret += (int)vgetq_lane_f32(vminnmq_f32(v1, v2), 0);
     /* ROUNDING */
     ret += (int)vgetq_lane_f32(vrndq_f32(v1), 0);
 #ifdef __aarch64__
     {
         float64x2_t vd1 = vdupq_n_f64(1.0), vd2 = vdupq_n_f64(2.0);
         /* MAXMIN */
         ret += (int)vgetq_lane_f64(vmaxnmq_f64(vd1, vd2), 0);
         ret += (int)vgetq_lane_f64(vminnmq_f64(vd1, vd2), 0);
         /* ROUNDING */
         ret += (int)vgetq_lane_f64(vrndq_f64(vd1), 0);
     }
 #endif
     return ret;
 }

As ARMv9 is approaching, I am thinking that NumPy should support the improved SIMD implementation called Scalable Vector Extensions v2 (SVE2) coming together with ARMv9.

SVE2

It is important to know that the existing implementations are for fixed-width SIMD, while SVE2 is variable-width, and that AArch64 code will need to detect either at compile-time or runtime whether advanced SIMD or SVE2 SIMD instructions should be used.

SVE2 is following the development of Neon architecture, which has a fixed 128-bit vector length for the instruction set. SVE is an extension to AArch64, and a superset of SVE and Neon, to allow for flexible vector length implementations. SVE improves the suitability of the architecture for High Performance Computing (HPC) applications, which require very large quantities of data processing. Particularly, SVE2 can be implemented from 128 bits up to 2048 bits with 128 bits increment.

Code Example and Recommendation

The code examples were provided in official documentation.
In this documentation, the Section B of Generic Vector and Matrix Operations examples are very useful for update of NumPy SIMD optimization.

They have example such as shown below for vectors dot-product with complex SP floating-point elements:

struct cplx_f32_t {
float re;
float im;
};
void vecdot(int64_t n, cplx_f32_t* a, cplx_f32_t* b, cplx_f32_t* c) {
cplx_f32_t acc;
acc.re = 0;
acc.im = 0;
for (int64_t i = 0; i < n; ++i) {
acc.re += a[i].re * b[i].re - a[i].im * b[i].im;
acc.im += a[i].re * b[i].im + a[i].im * b[i].re;
}
c->re = acc.re;
c->im = acc.im;
}

SPO600-Lab6, Project Stage 2: Analysis of an AArch64 SIMD Optimization

Qiwen Yu — Wed, 15 Dec 2021 21:08:42 +0000

Introduction

At this stage of project, I am going to focus on one of the open source projects which has applied SIMD (See Definition Below) Optimization, locate and analysis the SIMD code for AArch64 and other architectures including x86_64, SSE/SSE2, and AVX512.

Open Source Projects

It is worth mentioning that all of the following projects have SIMD optimization implemented:

ardour
eigen
ffmpeg
gnuradio
gromacs
lame
mjpegtools
nettle
numpy
pipewire
pulseaudio
vlc
xz

SIMD

SIMD optimization or SIMD operations both mean the same thing. Single instruction, multiple data (SIMD) is a type of parallel processing. SIMD describes computers with multiple processing elements that perform the same operation on multiple data points simultaneously.

numpy

The source code of NumPy can be found here. This project, NumPy is the fundamental package for scientific computing with Python. Originally Python was not designed for numeric computation. And the Numpy was created in 2005 to address this challenge. As the array size increase, Numpy gets around 30 times faster than Python List.

Several years ago, a StackOverflow answer clearly pointed out, there are SIMD vectorized operations in NumPy project. In this program, it was wrote as

"""
/* Currently contains sse2 functions that are built on amd64, x32 or non-generic builds (CFLAGS=-march=...) */
"""

However, this is only a simple example of SIMD optimization applied in this project.

Details of SIMD in NumPy

A detailed paper published in NumPy support documentation, CPU/SIMD Optimizations, clearly states that:

"""
NumPy comes with a flexible working mechanism that allows it to harness the SIMD features that CPUs own, in order to provide faster and more stable performance on all popular platforms. Currently, NumPy supports the X86, IBM/Power, ARM7 and ARM8 architectures.
"""

Where the AArch64, aka ARM64 was first introduced with the ARMv8-A architecture. Another review paper was published here.

The SIMD optimization support for AArch64 was summarized here:

The SIMD optimization support for X86 was summarized here:

However, the above StackOverflow question is outdated now, a very important Pull Request was merged in the last year, ENH: enable multi-platform SIMD compiler optimizations. This PR is very important to trace back all of the SIMD optimization applied in NumPy, it contains both a lot of discussions on how to implement SIMD in NumPy and corresponding changed programs.

Code examples, the code shown below is about the selection mechanisms for the SIMD code, including the compile-time mechanisms:

#ifndef NPY_DISABLE_OPTIMIZATION
     #if defined(__powerpc64__) && !defined(__cplusplus) && defined(bool)
         /**
          * "altivec.h" header contains the definitions(bool, vector, pixel),
          * usually in c++ we undefine them after including the header.
          * It's better anyway to take them off and use built-in types(__vector, __pixel, __bool) instead,
          * since c99 supports bool variables which may lead to ambiguous errors.
         */
         // backup 'bool' before including '_cpu_dispatch.h', since it may not defiend as a compiler token.
         #define NPY__DISPATCH_DEFBOOL
         typedef bool npy__dispatch_bkbool;
     #endif
     #include "_cpu_dispatch.h"
     #ifdef NPY_HAVE_VSX
         #undef bool
         #undef vector
         #undef pixel
         #ifdef NPY__DISPATCH_DEFBOOL
             #define bool npy__dispatch_bkbool
         #endif
     #endif
 #endif

Summary

In general, the SIMD code in NumPy is used for linear algebra. ARMv9 is on the horizon, and it includes an improved SIMD implementation called Scalable Vector Extensions v2 (SVE2). Now, NumPy does not support of ARMv9.

The following code:

#ifdef _MSC_VER
     #include <Intrin.h>
 #endif
 #include <arm_neon.h>

 int main(void)
 {
     float32x4_t v1 = vdupq_n_f32(1.0f), v2 = vdupq_n_f32(2.0f);
     /* MAXMIN */
     int ret  = (int)vgetq_lane_f32(vmaxnmq_f32(v1, v2), 0);
         ret += (int)vgetq_lane_f32(vminnmq_f32(v1, v2), 0);
     /* ROUNDING */
     ret += (int)vgetq_lane_f32(vrndq_f32(v1), 0);
 #ifdef __aarch64__
     {
         float64x2_t vd1 = vdupq_n_f64(1.0), vd2 = vdupq_n_f64(2.0);
         /* MAXMIN */
         ret += (int)vgetq_lane_f64(vmaxnmq_f64(vd1, vd2), 0);
         ret += (int)vgetq_lane_f64(vminnmq_f64(vd1, vd2), 0);
         /* ROUNDING */
         ret += (int)vgetq_lane_f64(vrndq_f64(vd1), 0);
     }
 #endif
     return ret;
 }

The code is for AArch64, might be updated for ARMv9.

SPO600-Lab6, Project Stage 1: Algorithm Selection Lab

Qiwen Yu — Wed, 15 Dec 2021 00:23:09 +0000

Introduction

This is the Algorithm Selection Lab.In this lab, the goal is to investigate the impact of different algorithms which produce the same effect. Based on benchmarking, three algorithms for adjusting the volume of PCM audio samples were tested. It is crucial to remember not to compare the relative performance across different machines, because various systems have different microarchitectures, memory configurations, peripheral implementations, and clock speeds, from mobile-class to server-class (e.g. Intel Atom vs. Xeon; AMD APU vs. Threadripper; ARM Cortex-A35 vs. Neoverse-V1).

However, do compare the relative performance of the various algorithms on the same machine.

Procedures

Digital sound is typically represented as two streams of signed 16-bit integer signal samples. To change the volume of sound, each sample can be scaled (multiplied) by a volume factor, in the range of 0.00 (silence) to 1.00 (full volume). On a mobile device, the amount of processing required to scale sound will affect battery life.

The given six programs are:

vol0.c is the basic or naive algorithm, which using casting between integer and floating points. Expensive.
vol1.c does the math using fixed-point calculations.
vol2.c pre-calculates all 65536 different results, and then looks up the answer for each input value.
vol3.c is a dummy program, negative control.
vol4.c uses Single Instruction, Multiple Data (SIMD) instructions accessed through inline assembley (assembly language code inserted into a C program). Only for Aarch64.
vol5.c uses SIMD instructions accessed through Complier Intrinsics. This program is also specific to AArch64.

Get example programs using tar -xvzf spo600-volume-examples.tgz -C ~/Lab6/
Change directory to where the Makefile stored. Using make to compile c-scripts.
Define SAMPLES in the header file vol.h and the scale of volume was set to 50% in this header file.
The memory used by these algorithm can be measured using free -m command in terminal.
Simple measure of time using time command.
Find a way to measure performance, using #include <sys/time.h>

Prediction

My prediction is that algorithms vol5 and vol4 are faster than others, vol2 is medium speed, while vol1 and vol0 are slowest, may vary depends on situations.

Test in Aarch64

Simple test

Simple test using time command:

Questions.

The prediction can be found at the above.

Q: Measure performance

struct timbal start, stop;
float elapsed;

gettimeofday(&start, 0);

for (x=0; x< SAMPLES; x++){
  data[x] = scale_sample(data[x], 0.5);
}

gettimeofday(&stop, 0);

elapsed = (stop.tv_sec - start.tv_sec)*1000.0f

The running time is roughly between start and stop.

SPO600-Lab5 64-bit Assembler Lab: Aarch64 VS. X84_64

Qiwen Yu — Tue, 14 Dec 2021 23:10:59 +0000

Instruction

This lab is to experiment with assembler on the x86_64 and aarch64 platforms.

Particularly, example scripts were provided in the server. First we need to extract the content to a different folder with command:

tar -xvzf spo600-assembler-lab-examples.tgz -C ~/

The code examples were shown as below:

Tasks:

Review assembly language programs using objdump -d objectfile, compared to source code, X86_64 VS. Aarch64.
Write loop with digit in assembly language X86_64 VS. Aarch64.

Aarch64

objdump

Overall, the objdump command is used to inspect compiled binaries. Thus, first using gcc to compile hello.c code to hello, then using objdump to inspect, taking the main section as the example:

loop

The next part was to write and run loops with assembly language. For example, if you have a assembly language file helloWorld.s, then you have to use command as -g -o helloWorld.o helloWorld.s to compile it. And then you have to take the helloWorld.o file and link it using this command ld -o helloWorld helloWorld.o and then you can actually run the program using this command ./helloWorld.

The following code can give us a sequence of 'Loop: ' in the terminal

.text
.globl _start

min = 0                          /* starting value for the loop index; note that this is a symbol (constant) */
max = 11                         /* loop exits when the index hits this number (loop condition is i<max) */
ten = 10
_start:

        mov     x19, min
        mov     x17, ten
        mov     x0, 0           /* status -> 0 */

loop:
        mov     x0, 1           /* file descriptor: 1 is stdout */
        adr     x1, msg         /* message location (memory address) */
        mov     x2, len         /* message length (bytes) */

        mov     x18, x19        /*mov x19 into x18 */
        udiv    x9, x18, x17
        add     x13, x9, 0x30   
        msub    x10, x9, x17, x18 /*get remainder*/
        add     x14, x10, 0x30  
        adr     x15, msg        
        strb    w13, [x15, 8]   

        strb    w14, [x15, 9]  
        mov     x8, 64          /* write is syscall #64 */
        svc     0               /* invoke syscall */

        add     x19, x19, 1
        cmp     x19, max
        b.ne    loop

.data
msg:    .ascii      "Loop:  \n"
len= .- msg

X86_64

objdump

Similarly, the main section:

As shown above, the compiled code in Aarch64 and X86_64 are very different.

loop

The similar program to produce a list of 'Loops: ' in X86_64 was shown as below:

.text
.globl  _start

_start:
        movq    $min, %r10      /*store the min vlaue into r10 as a loop index*/
        movq    $division, %r9   /*store the division value(10) into r9*/

loop:
        cmp     %r9, %r10      /*compare r10(loop index) with 10*/
        jl      digit_1          /*if r10 is less than 10, go to the subroutine digit_1*/
        jmp     digit_2          /*if r10 is greater or equal to 10, go to the subroutine digit_2*/


digit_1:
        movq    %r10, %r15      /*store the r10 value into r15*/
        add     $'0', %r15       /*add '0' to r15 so the value will be ascii number character value*/

        movq    $msg+15, %r11   /*the digit location within string*/
        movb    %r15b, (%r11)   /*store the digit at the location*/

        jmp     print           /*go to the subroutine print*/

digit_2:
        movq    $0, %rdx         /*initialize rdx to 0 for division*/
        movq    %r10, %rax     /*store the r10 value into rax for division*/
        div     %r9              /*divide rax value by 10(r9)*/

        movq    %rax, %r14    /*store the rax value(quotient) into r14*/
        movq    %rdx, %r15    /*store the rdx value(remainder) int r15*/

        add     $'0', %r14       /*add '0' to r14 so the value will be ascii number character value*/
        add     $'0', %r15       /*add '0' to r15 so the value will be ascii number character value*/

        movq    $msg+14, %r11   /*the digit location within string*/
        movb    %r14b, (%r11)   /*store the digit at the location*/

        movq    $msg+15, %r12   /*the digit loctaion within string*/
        movb    %r15b, (%r12)   /*store the digit at the location*/

        jmp     print   /*go to the subroutine print*/


print:
        movq    $len,%rdx                       /* message length */
        movq    $msg,%rsi                       /* message location */


        movq    $1,%rdi                         /* file descriptor stdout */
        movq    $1,%rax                         /* syscall sys_write */
        syscall
        inc             %r12b

        inc             %r10      /*increment loop index*/
        cmp             $max, %r10  /*compare the max with r10*/
        jne             loop      /*if the max value and the r10(loop index) value is not equal,*/
                                  /*   redo the loop subroutine*/

        movq    $0,%rdi                         /* exit status */
        movq    $60,%rax                        /* syscall sys_exit */
        syscall


.section .data
msg:    .ascii      "Loop:   \n"
        len = . - msg
min = 0
max = 10
division = 10

Conclusion

It is relatively straight-forward to get the loop to print a list of 'Loop: ', but it is very difficult to figure out how to make the loop to print 'Loop: 0', 'Loop: 1', ..., 'Loop: 9', 'Loop: 10'. The challenging parts include to show the digit and to show only the second digit but not to show as 'Loop: 00', 'Loop: 01' ... 'Loop: 09'.

SPO600 Lab4(Option 4): Screen Colour Selector Part B

Qiwen Yu — Tue, 14 Dec 2021 19:42:25 +0000

Summary

This dev blog is the second part of the previous one. In this part, I have figured out all the subroutines required and put together as the final scripts.

Final Implementation

; 

 lda #$00 
 sta COLOUR
 sta COLOUR_INDEX 

 jsr initialize_print ; jump to subroutine

; = get key input =
getKey:
 lda $ff
 sty $ff

 cmp #UP_KEY
 beq decrement_key

 cmp #DOWN_KEY 
 beq increment_key

 jmp getKey

decrement_key:
 lda COLOUR
 cmp #$01
 bpl decrement_colour

 jmp getKey

decrement_colour:
 dec COLOUR

 jsr initialize_print
 jsr initialize_paint
 jmp getKey

increment_key:
 lda COLOUR
 cmp #$0f
 bmi increment_colour

 jmp getKey

increment_colour:
 inc COLOUR
 jsr initialize_print
 jsr initialize_paint
 jmp getKey

; = print screen (Output) =
initialize_print:
 jsr SCINIT
        ldy #$00

title_write:
 lda title,y
        beq title_done
        jsr CHROUT
        iny
        bne title_write

title_done:
 lda #$00
 sta COLOUR_INDEX

write_colourStart:
 ora #$00
 ldy #$00

write_colourName:
 jsr select_colour
 beq write_done
 jsr toggle_highlight

 jsr CHROUT

 iny
 bne write_colourName

write_done:
 inc COLOUR_INDEX
 lda COLOUR_INDEX
 cmp #$10
 bne write_colourStart

select_colour:
 lda COLOUR_INDEX

 cmp #$00
 beq print_colour0

 cmp #$01
 beq print_colour1

 cmp #$02
 beq print_colour2

 cmp #$03
 beq print_colour3

 cmp #$04
 beq print_colour4

 cmp #$05
 beq print_colour5

 cmp #$06
 beq print_colour6

 cmp #$07
 beq print_colour7

 cmp #$08
 beq print_colour8

 cmp #$09
 beq print_colour9

 cmp #$0a
 beq print_colour10

 cmp #$0b
 beq print_colour11

 cmp #$0c
 beq print_colour12

 cmp #$0d
 beq print_colour13

 cmp #$0e
 beq print_colour14

 cmp #$0f
 beq print_colour15

 rts

print_colour0:
 lda colour0,y
 rts

print_colour1:
 lda colour1,y
 rts

print_colour2:
 lda colour2,y
 rts

print_colour3:
 lda colour3,y
 rts

print_colour4:
 lda colour4,y
 rts

print_colour5:
 lda colour5,y
 rts

print_colour6:
 lda colour6,y
 rts

print_colour7:
 lda colour7,y
 rts

print_colour8:
 lda colour8,y
 rts

print_colour9:
 lda colour9,y
 rts

print_colour10:
 lda colour10,y
 rts

print_colour11:
 lda colour11,y
 rts

print_colour12:
 lda colour12,y
 rts

print_colour13:
 lda colour13,y
 rts

print_colour14:
 lda colour14,y
 rts

print_colour15: 
 lda colour15,y
 rts

toggle_highlight:
 ldx COLOUR_INDEX
 cpx COLOUR
 beq highlight

 ora #$00
 rts

highlight:
 ora #$80
 rts

; = Screen Paint =
; = Initialize =
initialize_paint:
 lda #$00         ; set a pointer at $40 to point to $0200
        sta POINTER
        lda #$02
        sta POINTER_H

 ldy #$00

 lda COLOUR

draw_screen:
  sta ($40), y     ; set pixel

        iny              ; increment index
        bne draw_screen  ; continue until done the page

        inc $41          ; increment the page
        ldx $41          ; get the page
        cpx #$06         ; compare with 6
        bne draw_screen  ; continue until done all pages

 rts

; = for screen printing =
title:
dcb "C","o","l","o","u","r","s",13,00

colour0:
dcb "0",".",32,"B","l","a","c","k",13,00

colour1:
dcb "1",".",32,"W","h","i","t","e",13,00

colour2:
dcb "2",".",32,"R","e","d",13,00

colour3:
dcb "3",".",32,"C","y","a","n",13,00

colour4:
dcb "4",".",32,"P","u","r","p","l","e",13,00

colour5:
dcb "5",".",32","G","r","e","e","n",13,00

colour6:
dcb "6",".",32,"B","l","u","e",13,00

colour7:
dcb "7",".",32,"Y","e","l","l","o","w",13,00

colour8:
dcb "8",".",32,"O","r","a","n","g","e",13,00

colour9:
dcb "9",".",32,"B","r","o","w","n",13,00

colour10:
dcb "1","0",".",32,"L","i","g","h","t",32,"r","e","d",13,00

colour11:
dcb "1","1",".",32,"D","a","r","k",32,"g","r","e","y",13,00

colour12:
dcb "1","2",".",32,"G","r","e","y",13,00

colour13:
dcb "1","3",".",32,"L","i","g","h","t",32,"g","r","e","e","n",13,00

colour14:
dcb "1","4",".",32,"L","i","g","h","t",32,"b","l","u","e",13,00

colour15:
dcb "1","5",".",32,"L","i","g","h","t",32,"g","r","e","y",13,00

What I have Learned

From this lab, I have learned how to work with strings in assembly language. Moreover, one need to use ROM, input and output key, and implement highlight characters as well.

Release the TextHTMLPress package to PyPI

Qiwen Yu — Fri, 26 Nov 2021 23:18:44 +0000

In this blog, we are focusing software releases and getting our code into the hands of users from a GitHub repository. My static html generator, TextHTMLPress was used as the package to be published on PyPI.

Setup the Python Packge structure

Based on references on setup Python project, package structure, and a production-level Python package, I refactor the package as shown below:

TextHTMLPress/
|-- tests/
|   |-- inputs/
|   |-- test.py
|   |
|-- TextHTMLPress/
|   |-- __main__.py
|   |-- __init__.py
|   |-- generator.py
|   
|-- .gitignore
|-- config.yml
|-- requirements.txt
|-- LICENSE
|-- CONTRIBUTING.md
|-- setup.cfg
|-- pyproject.toml
|-- README.md

Write pyproject.toml and setup.cfg

To build a Python Package with setuptools and wheel, pyproject.toml and setup.cfg files are required. Follow the documentation, one thing worth noting is that now, static metadata setup.cfg should be preferred over setup.py, and now setup.py is not required.

Entry point

Adding a console script entry point in setup.cfg allows the package to define a user-friendly name for installers of the package to execute, seeing the reference.

Build and Upload

python -m build
python -m twine upload dist/*

The final package can be found at here.

Flytekit: Create a Conda package for flytekit

Qiwen Yu — Mon, 22 Nov 2021 07:56:33 +0000

Flyte and Flytekit

Flyte is a large open source community and project, which is mainly developed and maintained by Lyft. This a production-level project with hundreds of developers working on a significant sized code base spread across several repos. This type of work is often characterized by longer review cycles, stricter guidelines on what your contribution must be, and fixes that take a long time to make due to the need to read and understand big code.

Flytekit Python is the Python Library for easily authoring, testing, deploying, and interacting with Flyte tasks, workflows, and launch plans.

Issue

This issue was to create a Conda package for flytekit, now is pip install flytekit, the goal is to apply conda install flytekit.

Solution

The first step was to propose a proper plan to publish this package on conda. Initially, I wrote a comment addressing this issue with a draft plan. Finally, as the first PR submitted, the plan was settled.

Then, I recieved very useful comments from the project maintainers and submitted a follow up revision. This PR is still under review, but the picture is very clear right now. The meta.yaml file will eventually submit to conda-forge channel via the flyteorg fork of https://github.com/conda-forge/staged-recipes, following the steps in this guide: https://conda-forge.org/#add_recipe.

Code review

Qiwen Yu — Mon, 22 Nov 2021 07:55:13 +0000

Overview

This blog is to summarize what I have done in code review for internal open source projects of Seneca College.

Overall, this is my first GitHub code review experience, which is very exciting and educating. Code review is an important stage in the completion and merge of a pull request.

Code Review 1

This is the one I used as the template and example to learn from. In this code review, our instructor demonstrated the whole process from request a code review, add reviews to changed file (code), add comment to finish a review, and request changes as follow up.

Code Review 2

This is a simple code review I did for the Telescope project. This pull request (PR) was to update the version of a dependency in package.json file. The changes applied in the PR were successfully deployed. In addition, I checked the changed file (raw code) and it looks good to me. In summary, my opinion is that this PR is good to be merged.

Seneca Telescope: Update Traefik

Qiwen Yu — Mon, 22 Nov 2021 07:54:21 +0000

Telescope

Telescope is a tool for tracking blogs in orbit around Seneca's open source involvement. In more details, Telescope is an open source web server and client application for aggregating and presenting a timeline of Seneca's open source blogs. Telescope makes it easy to see what's happening with open source at Seneca right now. In this way, blogs posted on a variety of platforms are aggregated on one site.

Issue

The issue here is to update Traefik to newer version. Traefik is a cloud native edge router. And Traefik is a modern HTTP reverse proxy and load balancer that makes deploying microservices easy. The challenge to me is to understand the framework and structure of this project. Because Telescope uses microservices, Docker and Docker-compose. Those are frameworks I have never used.

Fixes

I followed the contributing documentation, and tried to setup this project both locally and on Gitpod. Gitpod is a convenient automated development tools supporting cloud development. The challenge of local setup is to install, setup, and run docker and docker-compose properly on Mac system.

Summary

Finally, I checked the changelog, and felt confident to migrate to v2.5 from v2.4 for Traefik. The final PR was simple, just changed the image in docker-compose.yml file. The newer version was tested good on Gitpod and locally by reviewers ang got merged successfully.