DEV Community:

Google Summer of Code 2024: A Retrospective

Mon, 26 Aug 2024 01:20:32 +0000

Organization: Chromium
Projects Overview:

Web Audio Test and Live Suite
Rainfly: Audio Worklets Playground

Projects

Web Audio Test and Live Suite

The Web Audio Test Suite is comprised of performance and benchmark tests for Web Audio. These tests can be run either through Playwright or the Live Suite. Playwright is a browser automation testing framework that runs via CLI or CI. This is useful for automated testing across different browsers or browser versions. Web Audio Test Live Suite is an interactive webpage for running tests locally in your browser and monitoring runtime.

Contributions

My contribution to the Web Audio Test and Live Suite centered around its creation, design, and development with @terryzfeng. After researching between Puppeteer and Playwright, we decided to use Playwright due to its testing capabilities in addition to its browser automation, which we could use for CI.

Documentation

The original draft PR for integrating the Web Audio Test Suite into `web-audio-samples`

[Working Draft] Introduce Playwright test framework #368

terryzfeng posted on Jun 11, 2024

Preparing Playwright Test Framework for AudioWorklet Testing

Tests written here src/tests/playwright (.spec.ts) Test pages are in src/tests/playwrights/pages

More documentation is written here https://docs.google.com/document/d/1e7G9jc4kuVZWvjylTtFE76DzQBsOG6lZSFmV19Kc5cY/edit#heading=h.wb3akxg1nco

View on GitHub

The first PR to introduce the Web Audio Test and Live Suite

Playwright Test Suite: realtime-sine (MVP) #372

terryzfeng posted on Jul 10, 2024

This PR Implements the "realtime-sine" test with basic infrastructure/framework.

Instruction for local testing

npm install
npx playwright install
Then npm run test or npm run test-live
- npm run test for Playwright tests.
- npm run test-live for live suite; after the comment, open up a browser and go to localhost:8080.

The test result looks like this for npm run test:

The test result looks like this for npm run test-live:

View on GitHub

Adding documentation to the Test and Live Suites, allowing for more crowdsourced tests

Web Audio Test Suite Documentation #381

Kizjkre posted on Jul 17, 2024

Includes:

How to write tests for Playwright and Live Suite running
Sample template.html for test writing
Notes for Playwright Configuration and Browser setup
API documentation of audit.js and recorder object.

View on GitHub

Overcoming cold start on the Test and Live Suites by porting performance tests from WPT

Playwright Port Performance Tests #384

terryzfeng posted on Jul 20, 2024

Porting Performance Tests from https://source.chromium.org/chromium/chromium/src/+/main:third_party/blink/perf_tests/webaudio/ to Web Audio Test Suite

[x] Gain Test
[x] Panner Test
[x] Timeline Insert Event
[x] Audio Buffer Source Node
[x] Audio Worklet Node
[x] Biquad FIlter Node
[x] Dynamics Compressor Node Knee
[x] Dynamics Compressor Node Post Knee
[x] Dynamics Compressor Node Pre Knee

View on GitHub

Fix an issue where the build process did not include all relevant files

Add style and realtime sine reference to live suite build #385

Kizjkre posted on Jul 20, 2024

Fix #382.

View on GitHub

Linking the Live Suite to the index page of `web-audio-samples`

Add live suite to tests page and tests page to homepage #386

Kizjkre posted on Jul 23, 2024

View on GitHub

Rainfly

Rainfly is an Audio Worklet playground, useful for inspecting, developing, and rapid prototyping of Audio Worklet code.

Contributions

Along with @terryzfeng, we created, designed, and developed an MVP of Rainfly using SvelteKit and TailwindCSS.

Documentation

Figma med-fi prototype

The original MVP hosted on a separate repository due to separate framework and development environment

Kizjkre / rainfly

Google Chrome Labs Web Audio Samples Audio Worklets Sandbox v1.0.0-beta

Rainfly

An AudioWorklet DSP Playground for Chromium Web Audio API Project (2024)

Developing

Install dependencies with npm install (or pnpm install or yarn), and then start a development server:

npm run dev

# or start the server and open the app in a new browser tab
npm run dev -- --open

Building

To create a production version of Rainfly:

npm run build

You can preview the production build with npm run preview.

View on GitHub

Introducing Rainfly into `web-audio-samples`

Rainfly - An AudioWorklet DSP Playground [MVP] #395

terryzfeng posted on Aug 25, 2024

This PR is to introduce Rainfly: An AudioWorklet DSP Playground for Web Audio. This online AudioWorklet IDE streamlines writing custom JavaScript AudioWorklet code, building Web Audio DSP graphs, and recording output for easy visualization, analysis and validation.

Rainfly MVP Features:

Monaco Code Editor for writing AudioWorkletProcessor and AudioContext graph
Realtime DSP execution via AudioContext, can pause and play
Recording and visualization of DSP output a la Canopy
Example AudioWorklet code and boilerplate
Error console for JS execution error feedback
Exporting JS code or recorded output to a .wav
Vim support
Built with Svelte

Authors:

@terryzfeng
@kizjkre

GSoC 2024

View on GitHub

Other Miscellaneous Work

Onboarding PR to familiarize myself with Google standards

Preparing resources documentation for Audio Worklets playground. #359

Kizjkre posted on May 10, 2024

Added a comment to landing_data in anticipation of the Audio Worklets playground.

View on GitHub

Fixing linter issues

Downgrade eslint to be compatible with eslintrc #373

Kizjkre posted on Jul 10, 2024

Downgrade eslint from 9.5.0 to 8.57.0 to be compatible with eslintrc.

View on GitHub

AbleMail: Furthering independence through email

Thu, 21 May 2020 19:58:05 +0000

My Final Project

What is AbleMail?

AbleMail is an email client designed for people with cognitive disabilities. This started as a project addressing the needs of my school community, but ended up growing much further and much bigger. I originally wanted to make this for Jon, an important member of our school community. He works in our school’s media department with recording assemblies as well as organizing all of the A/V equipment. He was introduced to our school via On-the-Move, a nonprofit organization dedicated to helping people with intellectual and developmental disabilities find employment opportunities. I had the fortune of working with him and his parents throughout the development of AbleMail. The hope of this project is to let Jon, and others like him, be able to use email independently.

Why?

Email is a crucial tool for workplace communication and collaboration; however, current email clients present accessibility barriers for people with cognitive disabilities. For example, Jon is unable to use email independently. Any job related communication, such as changes to his work schedule, must be emailed to his job coach or his mother who handle the logistics of his employment. AbleMail features a visually simple interface so Jon can (1) comprehend, (2) respond, and (3) compose emails independently. Since Jon struggles with a slow reading rate, text-to-speech capability converts the text of an email into audible words to improve email comprehension. AbleMail provides large and colorful icons that allow Jon to respond to emails with customized pre-written responses. Under AbleMail settings, icons and written responses can easily be changed and new ones added. Lastly, AbleMail allows Jon to compose emails through dictation and to attach an audio clip of his recorded response. AbleMail empowers Jon to communicate with his employer by email, thereby increasing his independence, ownership of his work, privacy, and self-esteem.

Demo Link

AbleMail

Link to Code

ablemail / ablemail

Furthering independence through email.

Furthering independence through email: AbleMail is a free email client web app that allows people with cognitive disabilities to independently use email.

Try the app here!

Created and maintained by Tristan Peng.

This project was made with Next.js, Material UI, Express, and React.

Setup

Clone this repository git clone https://github.com/TheAmazingness/ablemail.git <your-project-folder>
Run yarn on the folder to install all dependencies.
Run yarn run dev for the dev version.
Visit http://localhost:3000.

Signing In With a Gmail Account

Visit https://myaccount.google.com/.
Click on Security
Turn on Less secure app access
Go back to AbleMail
Sign in

Usage

It is recommended to use Google Chrome to use AbleMail because AbleMail uses experimential Web APIs such as SpeechRecognition and SpeechSynthesis. To see what browsers to support these experimental technologies, click on the words to learn more.

This branch is for AbleMail with Next.js, React hooks…

View on GitHub

ablemail / ablemail-backend

Backend for AbleMail. Written in Node.js

AbleMail Backend

This repository holds the server code for AbleMail. Check out the api.

API

GET

GET `/`

Redirects to the AbleMail client.

GET `/auth/signup`

Signs up a new non-Google user and generates a session.

Query

`key`	`email`	`pass`	`first`	`pass`
String	String	String	String	String
A key that authorizes the connection.	User's email. Make sure it does not end in `@gmail.com`.	User's password.	User's first name.	User's last name.

GET `/auth/google`

Authorizes a Google user

Query

`key`	`email`	`pass`
String	String	String
A key that authorizes the connection.	User's email. Make sure it does not end in `@gmail.com`.	User's password.

GET `/auth/google/redirect`

Redirects to Google auth page.

Query

None

GET `/auth/google/callback`

Redirects to the AbleMail client with the access token.

Query

`code`
String
Google access token.

GET `/get-mail/google`

Gets last 6 messages from Google.

Query

`key`	`token`
String	String
A key that authorizes the connection.

…

View on GitHub

How I built it

This is probably my largest project I've worked on, and I have learned a lot throughout my development process. For AbleMail, I used React, Redux and Express to build the application. Since I was also just starting to learn React when I built AbleMail, I faced a lot of challenges with React's steep learning curve, especially getting used to React's model of organizing components. Even though the syntax presented a lot of things for me to learn, I think the biggest challenge that I faced during this process is the mail protocols. At the inception of this project, I knew nothing about IMAP, SMTP, and POP, but this was integral to AbleMail, so I had to learn about how each protocol worked and its implementation in Node.js.

Another learning opportunity that I'm facing right now has to do with authentication and security. I use passport.js to manage all of the logging in and authentication, but I am facing bugs relating to sessions that are giving me lots of trouble at the moment. However, I am confident that, like the other bugs that have challenged me, that I will be able to fix it.

DEV Community: ​

Google Summer of Code 2024: A Retrospective

Projects

Web Audio Test and Live Suite

Contributions

Documentation

The original draft PR for integrating the Web Audio Test Suite into web-audio-samples

The first PR to introduce the Web Audio Test and Live Suite

Instruction for local testing

Adding documentation to the Test and Live Suites, allowing for more crowdsourced tests

Overcoming cold start on the Test and Live Suites by porting performance tests from WPT

Fix an issue where the build process did not include all relevant files

Linking the Live Suite to the index page of web-audio-samples

Rainfly

Contributions

Documentation

Figma med-fi prototype

The original MVP hosted on a separate repository due to separate framework and development environment

Kizjkre / rainfly

Google Chrome Labs Web Audio Samples Audio Worklets Sandbox v1.0.0-beta

Rainfly

Developing

Building

Introducing Rainfly into web-audio-samples

Other Miscellaneous Work

Onboarding PR to familiarize myself with Google standards

Fixing linter issues

AbleMail: Furthering independence through email

My Final Project

What is AbleMail?

Why?

Demo Link

Link to Code

ablemail / ablemail

Furthering independence through email.

Setup

Signing In With a Gmail Account

Usage

Next

ablemail / ablemail-backend

Backend for AbleMail. Written in Node.js

AbleMail Backend

API

GET

GET /

GET /auth/signup

Query

GET /auth/google

Query

GET /auth/google/redirect

Query

GET /auth/google/callback

Query

GET /get-mail/google

Query

How I built it

Additional Thoughts / Feelings / Stories

YouTube Video

DEV Community:

The original draft PR for integrating the Web Audio Test Suite into `web-audio-samples`

Linking the Live Suite to the index page of `web-audio-samples`

Introducing Rainfly into `web-audio-samples`

GET `/`

GET `/auth/signup`

GET `/auth/google`

GET `/auth/google/redirect`

GET `/auth/google/callback`

GET `/get-mail/google`