DEV Community: Emily

Flask-Assets

Emily — Tue, 21 Dec 2021 21:54:21 +0000

This week, I had a task to implement the SCSS with the Python Flask application. The project uses Jinja as a template engine and Flask for web Framework. This is the first time I have used SCSS, so I would love to take some introductory notes.

Why SCSS

We all know that CSS is Cascading style language using to style webpage. SCSS stands for Sassy CSS, which is CSS pre-processor that give developer ability to use variables, nested syntax, imports, etc. In the pre-processing stage, SCSS files will be converted to CSS to be ready for the browser.
The biggest advantages of SCSS over CSS is removing code redundant and utilizing code reuse. For example:
CSS Code

.container {
  background-color: white;
  padding: 20px;
}
.container h1{
  color: red;
  text-align: center;
  margin-top: 20px;
}
.container p{
  color: red;
  text-align: center;
  margin-top: 10px;
}

SCSS Code

$color-container: white;
$color-text: red;
.container {
  background-color: white;
  padding: 20px;
  h1 {     // Notice here we don't need to repeat .container class
  color: $color-text;
  text-align: center;
  margin-top: 20px; 
  }
  p {
  color: $color-text;
  text-align: center;
  margin-top: 10px;
  }
}

Install Flask-assets and pyscss

To compile SCSS to CSS, I need Flask-Assets to manage the assets/static files of my Flask application and Pyscss as CSS compiler to convert SCSS markup to real CSS.

pip install Flask-Assets
pip install pyscss

My application structure:

Folder    
├───App
│   ├───Config
│   │   └───__pycache__
│   ├───Constants
│   │   └───__pycache__
│   ├───flask_session
│   ├───Models
│   │   └───__pycache__
│   ├───static 
│   │   ├───Gen
│   │   ├───SCSS
│   ├───templates
│   │   ├───example.html
|   ├───__init__.py
├───Setup
│   ├───PIP
│   └───QRIOUS
├───application.py

According to the documentation, Environment object is used to hold a collection of bundles and configuration.
A bundle object contains source files, filters _ pyscss in my case, and the output file. It is important to note that all paths in the bundle are relative to my flask application's static folder. For example: output='Gen/example.css' , the output CSS file will be in static\Gen folder.
My code:

from flask import Flask
from flask_assets import Environment, Bundle
app = Flask(__name__)
assets = Environment(app) # create an Environment instance
bundles = {  # define nested Bundle
  'example_style': Bundle(
            'SCSS/example.scss',
            filters='pyscss',
            output='Gen/example.css',
  )
} 
assets.register(bundles) # register name for every bundle in bundles object

And lately, I can insert my bundle by using its name in the block assets of jinja template. Flask-Assets will help merge and compress my bundle source files the first time when the template is rendered. Moreover, it also watches the change of the source files and will update the compressed file when necessary.

<head>
    {% assets "example_style" %}
    <link rel="stylesheet" type="text/css" href="{{ ASSET_URL }}">
    {% endassets %}
</head>

I found that Flask-assets is easy to use, and it becomes convenient for pre-processing files such as SCSS files or JS files.

Thinking about Open source

Emily — Wed, 15 Dec 2021 02:56:11 +0000

Open-source

Before OSD600, I saw open-source as an interesting, yet difficult field to get in. I thought that people should have been familiar with open-source projects to be able to join as contributors. However, after working on Telescope and Medusa, I learned that even beginner can start with good first issue even if they don't have any knowledge on it. The amazing thing about open-sources community is that they can give you opportunity to work on their projects without questioning about your ability. In my eyes, open source is a source code that open for anyone to use, to share and to write. From contributor's point of view, Open source world is like a huge boot-camp where you can learn any topic you like and get practice, get feedback on your code and help resolve real-life challenges.

Advices

The advices I would love to give my-self and other new students from day 1 is that you should follow the documentation and ask the right questions. It will save you lots of time when you understand thoroughly the documentation and you know how to ask question.

Conclusion

I learn a lot from the OSD600 course and I am grateful for this opportunity. My point of view about blogging has also changed. Before, this is the scariest part of the course. I was terrified of writing something wrong or scared that maybe I am not good enough. However, as Prof David said from beginning:

A blog gives you a voice, which you can use when and how you want.

Now, I see blog as as a journal of my learning and working journey. It is a safe place that I can express my thoughts.

Cover image from MIT News

The final result

Emily — Sun, 12 Dec 2021 20:49:01 +0000

In the last week of OSD600, all my PRs for Telescope were merged.
The summary of them is as follows:

1st: Adding feeds/delayed route to Posts service: PR
2nd: Upgrade Elasticsearch version in docker-compose.yml file: PR

In my last two blog posts, I discussed the issues related to the first PR and how I resolved them. For this blog, I would love to write about my experience working on the Telescope project in general and what I learned from my second PR.

Elasticsearch version

Regarding this issue, I just need to change the version number of Elasticsearch in the docker-compose.yml file and it is just a 5-min fix:

// old code
image: docker.elastic.co/elasticsearch/elasticsearch:7.9.3
// my fix
image: docker.elastic.co/elasticsearch/elasticsearch:7.14.2

However, after I sent my PR, Prof. David filed an issue about a high severity vulnerability (CVE-2021-44228) which has impacted the Elasticsearch versions 5.0.0+. So the solution is to upgrade to Elasticsearch 7.16.1 once it is released. This experience made me think that we should be aware and keep updated on the third-party engines/tools used in our project.

Experience

To put it simply, this is the best open source project I have contributed to. Telescope's Slack channel is always active, and when there is a question, the maintainers and contributors reply instantly. What I love most about it is code review, as I learnt a ton of good practices from Prof. David's comments.
For example:

I think Telescope's maintainers did a fantastic job in fostering engagement. They are always there to help contributors. They provide excellent and constructive feedbacks, and they value every contribution, even small ones.

Conclusion

This semester, I did not have a chance to work on Front-end issues. However, I am looking forward to taking some React issues in the OSD700 course.

Progress Report

Emily — Sat, 11 Dec 2021 05:52:02 +0000

Progress

Until now, my first PR for adding route \feeds\invalid to Posts service merged, and I also sent another PR for the issue #2413. So everything is going well, and I have not had any significant challenges working on these issues.

Finding

The hardest part for implementing the route \feeds\delayed to Posts service is testing my code. When microservices finished downloading feeds and processing them, I checked the Redis server to find the delayed one. And I found nothing. After asking a question on Telescope's Slack channel, I got an answer from prof. David that this case is not usual. The reason is a feed is marked delayed only when the processor receives an HTTP 429 response ("the user has sent too many requests in a given amount of time ("rate limiting")"). This issue made it hard to test my new functions against the real-time database.
The workaround suggested by prof. David is implementing an integration test to create delayed feeds on the fly. Surprisingly, those feeds did not have the time to live when I used ttl command. I found that the setDelayedFeeds() function did not set the expire time for the key.

// old code
setDelayedFeed: (id, seconds) => redis.set(createDelayedFeedKey(id), seconds, 1),
// new code
setDelayedFeed: (id, seconds) => redis.set(createDelayedFeedKey(id), seconds, 'EX', seconds),

Take-away

The biggest take-away for me this time is learning about Redis commands and Readable stream in NodeJs.
Streams is one of the core concepts in NodeJs, and it is often used when we need to work with a large amount of data.
When calling the scanStream() method on a Redis client, I actually created an object of readable stream.

Scanstream : Convenient class to convert the process of scaning keys to a readable stream

My code:

const getFeedKeysUsingScanStream = (matchPattern) => {
  const keys = new Set(); 
  const stream = redis.scanStream({
    match: matchPattern,
  }); //create a readable stream object 
  return new Promise((resolve, reject) => {
    stream.on('data', (data = []) => {
      data.forEach((k) => keys.add(k));
    });
    stream.on('error', (err) => {
      logger.error({ err }, 'Error while scanning redis keys');
      reject(new Error('Error while scanning redis keys'));
    });
    stream.on('end', () => {
      resolve([...keys]);
    });

A stream object has many events: 'closed', 'data', 'end', 'error', etc. Because the stream processes data piece by piece, we need to wrap it inside a Promise and only resolve when there is no more data to consume _ the end event.

In the end, I was able to make the route work, and I am happy with the progress.

Second PR for Telescope project

Emily — Thu, 09 Dec 2021 04:02:54 +0000

Issue

In the last Release 04 of the OSD600 course, I decided to work on implementing the /feeds/delayed route for the Telescope project and finishing the /feeds/invalid route. In general, there are feeds that have been flagged as delayed in the Redis server and the project owners want to see them from new endpoint in the Posts service. So, the new endpoint will return a JSON array of objects containing feeds Id, URL and the time to live of the feed.

Planning

The first part is finding the keys with the delayed suffix on the Redis server. This issue is similar to issue 2412 that I have sent a PR. I realized that I could utilize the getFeedKeysUsingScanStream in the old PR to find the delayed keys.

The second part is retrieving the information of the founded keys. The issue also requires finding the time to leave the delayed feeds. Luckily, prof. David has given detailed instructions about the time to live of delayed feeds on the issue. I will research ttl command and finish this task soon.

Auditting Compiler.md for IPC 144 Project

Emily — Wed, 01 Dec 2021 05:09:40 +0000

In Release 03 of the Open source course, I had a chance to work on IPC 144 project that helps improve and maintain the IPC144 course's notes. I was working on auditing and editing the Compiler.md page.

Issues

Prof. David has listed issues that should be checked and edited to ensure the markdown page looks well. I simply followed the guide, and I improved the page by:

Fixing typos and Markdown syntax
Fixing Frontmatter metadata
Adding inter-site links to other pages that have related info
Adding alt text on images
Improving page's Accessibility

There were some requirements I have not fixed on this page, such as centring images horizontally or making sure images work in dark and light modes. I feel like these requirements/issues should be set up globally instead of locally so the project will have a consistent style throughout all pages.

Result

I also got some reviews from my classmate to improve my PR. Thank you, guys, for taking the time to review it. It could not complete without your recommendations.
Finally, my Pr is merged, and this task helps me learn more about Docusaurus, LightHouse tool, and Web accessibility.

Publishing shinny-ssg Nuget Package

Emily — Mon, 29 Nov 2021 18:45:58 +0000

In the last lab of OSD600, I have a chance to publish and release my static site generation application. I chose Nuget.org as my package registry. I have always loved Microsoft documentation as it is clear and easy to follow. This time, I simply followed their instructions and got my package available here.

Pack

To start, I needed to set the application's PropertyGroup in the project files

    <PropertyGroup>
        <OutputType>Exe</OutputType>
        <TargetFramework>netcoreapp3.1</TargetFramework>
        <RootNamespace>shinny_ssg</RootNamespace>
        <PackAsTool>true</PackAsTool>
        <PackageId>shinny_ssg</PackageId>
        <Version>1.0.1</Version>
        <Authors>Emily Phan</Authors>
        <PackageOutputPath>./bin</PackageOutputPath>
        <IsPackable>true</IsPackable>
        <RunPostBuildEvent>Always</RunPostBuildEvent>
    </PropertyGroup>

Then you can use the command line to pack the package:

dotnet pack

This command generated a .nugkp in the /bin folder.

Publish

The next part is publishing the package to nuget.org.
To prepare for that, I created a Nuget account and retrieved the API key. This key is used to identify your account to the Nuget Gallery.
After that, I ran this command to publish Shinny-ssg

dotnet nuget push shinny_ssg.1.0.1.nupkg --api-key <token> --source https://api.nuget.org/v3/index.json

And that was it, my tool was ready to use.

Testing

I asked one of my friends to test the tools. We have found a bug in my application. When she ran shinny-ssg -v , she got Multiple options with name "v" found. This is usually due to nested options. I realized that I did implemented the -v|--version twice and I did not specify it right in my Options class. I will fix this issue and update the version.

Publishing the package is a happy ending for the project I have worked on the last 2 months. After this, I understand more about a project's lifecycle: developing, testing and publishing.

Working on Telescope Project

Emily — Fri, 26 Nov 2021 22:03:56 +0000

Introduction

Telescope is an open-source project written by Seneca students to aggregate and present a timeline of Seneca's student blogs. Telescope is a fantastic project, and it has a solid impact on to open source community in Seneca. I got a chance to contribute to this project by implementing the route posts/feeds/invalid for the posts service.

Docker

Contributors to the Telescope project can use Docker to develop and run the application locally. They have provided great documentation to set up Docker containers for different operating systems. I always have problems with setting up the environment to work on any projects, but with Docker, this task is a little bit easier this time.
The first challenge I had is error response from Deamon when I first ran npm run service:start

Error response from daemon: Ports are not available: listen tcp 0.0.0.0:80: bind: An attempt was made to access a socket in a way forbidden by its access permissions.
Error starting services with docker-compose: undefined

I asked a question on the Slack channel, and Prof.David quickly pointed out that my port 80 is being used by an IIS application. I checked, and there was an old application that was running to port 80 all the time. I have closed this application and resolved the problem.

Redis

Redis is an in-memory database, a data-structure server and a message broker that stores key-value data. That is a lot of words. But what is in-memory database? It means all data are stored in the main memory of the computer instead of the disk storage, making Redis operations exceptional faster than on-disk databases.
Telescope uses Redis database and ioredis to interact with Redis.

Pr

I have added a new route to the Post service and added functions to query the invalid Feeds data to the project. I have sent a draft PR, and I got some comments from Prof. David to improve it.

Telescope uses many advanced technology/concepts such as microservices architecture, Redis database, Elasticsearch, Docker, etc. There is so much to learn from it, and I am glad that I can take part in the project.

GitHub Actions

Emily — Fri, 26 Nov 2021 16:56:28 +0000

Introduction

What is GitHub Action? According to Github, it is a series of commands that can be executed when a special event (defined by user) occurred. The series of commands is put in the context of workflow and every steps can be seen as a job.
This week, I had a chance to explore GitHub Actions for my Shinny-SSG project and it is a great improvement for my application.

Define Workflows

I created my first workflow by using Github's default workflow template for .Net project. Github created a YAML file in the .github/workflows directory inside my git repository. You can read more about it here.
My dotnet.yml file :

name: .NET

on:
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]

jobs:
  build:

    runs-on: ubuntu-latest

    steps:
    - uses: actions/checkout@v2
    - name: Setup .NET Core 3.1 
      uses: actions/setup-dotnet@v1 
      with: 
       dotnet-version: 3.1.x
    - name: Setup .NET Core 5.0
      uses: actions/setup-dotnet@v1
      with:
       dotnet-version: 5.0.x
    - name: dotnet-format
      run: dotnet tool install -g dotnet-format --version 5.1.225507
    - name: Restore dependencies
      run: dotnet restore shinny-ssg/shinny-ssg.sln
    - name: Build
      run: dotnet build shinny-ssg/shinny-ssg.sln --no-restore 
    - name: Test
      run: dotnet test shinny-ssgTests --no-build --verbosity normal

Testing

Later, I tested my Github Actions by making a pull request to my origin repository. It worked like a charm and it denied merging if my PR did not pass the test.

I also made a PR for my classmate's project. It also worked as expected.

Writing tests for CLI tool

Emily — Sun, 14 Nov 2021 18:27:00 +0000

This week, I was working on writing tests for my Shinny-SSG project. It was the most challenging lab in the OSD600 course since I had to modify both my code and my project's folder structure to implement the tests.

Set-up

The testing framework that I chose is XUnit. The first reason is that it is trendy compared to another test framework such as NUnits. I created test method stubs from the existing code by Create Unit Tests command. To use it with Xunit, I have to implement the XUnit.net.TestGenerator extension to my project.

Challenges

I want to test how my tools generated files and folders in the destination with different arguments passed to the program. However, in my old code, I put all the logic of working with arguments in the static int main( string[] args) function. I could not use Interface and Dependency injection to mock the CommandLineApplication because CommandLineUtils does not have an interface for this class. Luckily, I found this guidance from the owner of CommandLineUtils, and he advised that "Split the command-line argument parser and application execution into separate class structures" to test various options programmatically. It is a great suggestion, and I rewrote my program by adding class CommandLineOptions and adding logic to the constructor of class Generator. I can kill two birds with one stone by this change: code refactoring and writing better tests.

Another problem I had was my folder structure. Before, I put the project's sln file, .git file , and src files in the root of the folder. However, when I added a new test project for Shinny-SSG, I had it outside my git folder, and it would be impossible to commit the change and put it in my remote repository. To resolve that, I had to change my folder structure to this:

C:.
├───shinny-ssg
│   ├───bin
│   │   ├───Debug
│   │   │   └───netcoreapp3.1
│   │   │       ├───dist
│   │   │       └───publish
│   │   ├───Destination
│   │   └───Release
│   │       └───netcoreapp3.1
│   ├───obj
│   │   ├───Debug
│   │   │   └───netcoreapp3.1
│   │   └───Release
│   │       └───netcoreapp3.1
│   ├───Properties
│   └───src
└───shinny-ssgTests
    ├───bin
    │   └───Debug
    │       ├───.netcoreapp,version=v3.1  
    │       └───netcoreapp3.1
    ├───obj
    │   └───Debug
    │       ├───.netcoreapp,version=v3.1
    │       └───netcoreapp3.1
    └───src

Testing

I wrote 3 test for Generator class run() function that cover 3 different cases: config file option, input path option and invalid input path option. My tests help uncovering a huge bug in my application. Before, I thought that default keyword was used to specified the default value of a variable.

 cssUrl = cssOption.HasValue() ? cssOption.Value() : default;

However, the default literal is for producing the default value of a type that is null in this case ( CssUrl is the string type).
I also wrote a test that testes the core feature of my application: Given a text and checked if the generated HTML value matched the expected output.
Pull request

Throughout this experience, I learn a lot about software testing and why it is essential for software development. In the future, I will implement more tests for my project and explore other test frameworks.

Hackoberfest recap

Emily — Wed, 10 Nov 2021 03:33:20 +0000

The Hacktobestfest ended last month and, through this journey, I have learned many new things.

Challenges

The first challenge I had when working on Hacktoberfest's issues was setting up the project in the local machine. I was a little bit ambitious at the beginning and I chose only the big projects. Obviously, these projects had many packages, test projects, and databases setup processes. In the end, I gave up working on some issues due to the complexity of installation and local development.

Lessons

Even thought I have not successfully made four merge pull requests this time, this experience is still worth trying. I have discovered how to write integration tests and what .Net Bot is. Another thing I realize is the way to contribute to an open-source project and git commands.
In addition, I appreciate the project's owners for spending time reviewing my pr. They were patient, they suggested changes and waited for me to fix my code.
I contributed my part to the project, learned a lot in return, and that is the beauty of open source and good karma.

Prettier your C# code with DotNet Format

Emily — Sat, 06 Nov 2021 16:36:24 +0000

This week, I learned about Static Analysis Tool, and I practice applying it to my Shinny-SSG application. The two tools I used are Dotnet Format and StyleCopAnalyzers. I chose them because they are widely used and they are both analyzing source code based on the .editorconfig file.

Installation

EditorConfig

In the first step, I generated the .editorconfig file for my project. This Editorconfig file is where you define your coding style in Visual Studio, such as Indentation and Spacing or Wrapping Preference.

DotnetFormat

The second step is installing DotnetFormat tools in my local machine by this command:

dotnet tool install -g dotnet-format

This tool will format my code based on the style preferences on the .editorconfig file.

StyleCop.Analyzers

The third step is installing StyleCop Analyzers. As specified in the documentation, I followed the preferable way is to add the NuGet package StyleCop.Analyzers. I also found this blog is helping, and I followed step by step which the author recommended. In the Project File extension, I added this line to the project so it can trigger StyleCop at the build time:

<PackageReference Include="StyleCop.Analyzers" Version="1.1.118" PrivateAssets="all" Condition="$(MSBuildProjectExtension)=='.csproj'">
    <PrivateAssets>all</PrivateAssets>
    <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
</PackageReference>

The last step is to edit my .editorconfig file to disable the rule that I don't want to my project, such as: Using directive should appear within a namespace declaration.

dotnet_diagnostic.SA1200.severity = None

However, at the build time of my project, it still showed a warning for this rule, so maybe I should research more about it.

Build Event

I was struggling to find a way to fix the warning from StyleCop for my project automatically. I had 360 warnings, and it will be time-consuming to fix them manually. Luckily, I read the blog of my classmate, Minsu Kim, he mentioned that dotnet format can be used with third-party analyzers. It is a great find, and I can use this command to fix all the warnings from style-cop:

dotnet-format -a warn ./shinny-ssg.sln

By using this command, I did not need to modify my Project file extension to trigger StyleCop.Analyzers at the build time and I can leave all the works to dotnet-format.

Microsoft Visual Studio has build events configuration for projects, and I used it to trigger auto-format code action at the time of building project. I have set up my project as follows:

After I ran the dotnet build in my project, two dotnet-format commands have fixed six formatting errors, and 360 warnings.

Editor/IDE Integration

Visual Studio provides a great way to replicate the development environment through a vsssetings file. New contributors to my project can have the same development environment as me by importing the ShinnySSG-2021-11-06.vssettings, which is available in the Project repository.

Conclusion

Formatting code is the first way to improve code quality, and it makes easier for reviewers to spot bugs and give suggestions. I am happy that I have found great tools to make this task much more easier.