DEV Community: Emilie Ma

Towards Web Monetization

Emilie Ma — Mon, 16 Jan 2023 15:18:37 +0000

It's been a couple years since Dev.to hosted the Grant for the Web hackathon, a month-long sprint to develop innovative projects with Web Monetization. Web Monetization is a proposed JavaScript API that allows browsers to create payment streams directly to websites, allowing for micropayments and unlocking exclusive content on a pay-per-use basis. It's still being incubated at the Web Incubator Community Group, but it's an active project and an exciting technology.

✨ I've recently built a project that aims to bring Web Monetization to the Ethereum blockchain, and would love any feedback on it! Scroll down to the bottom to learn more. ✨

The Current State of Web Monetization

Currently, the dominant player in the Web Monetization space is Coil, a subscription service that provides a Chrome extension to users. It connects to your cryptocurrency wallet - right now, Web Monetization works primarily on the Ripple blockchain. Whenever the Chrome extension detects a payment pointer, or a wallet address, it starts streaming micropayments. The page to which payments are being streamed can detect this and unlock exclusive paywalls.

It's straightforward to get set up: create a payment pointer with one of the supported wallets, and add a <meta> tag to your site. Bingo - any users with a Web Monetization extension that navigated to your pages would start streaming you small fractions of a cent as they spent time on your site.

Web Monetization has been touted as the future of micropayments and monetization on the Web. With enough work to boost adoption and reframe mindsets around current ad-based monetization models, I think it will be. But as it stands, the project seems to have stagnated and become more centralized around the key players above.

We need more providers

One of the more glaring issues seems to be adoption, specifically with WM providers. There are only a couple wallet providers, a handful of browser integrations, and only one payment provider, Coil, which is a for-profit company. Even with the technology and open source models these providers build with, I see a lot of centralization in the claims of decentralization.

Part of this is the feedback loop where companies don't see the benefits of investing in creating a payment processor for a niche area of monetization when there isn't widespread adoption and low profits to be made. In turn, because there's low adoption among existing companies, users perhaps don't see the point of creating a whole new account and subscribing to yet another service. Developers see that few people have accounts, and without a wallet at one of the few providers, might decide that the possible revenue would be too low to bother spending valuable time on it. And without high-profile companies integrating WM into their products, competing companies wouldn't bother to use WM, and so the cycle continues.

Consumer mental models must shift

I think it'll take a lot of time and promotion before WM starts growing to something that even a decent fraction of an app or content creator's userbase will readily have.

With the ads-based models that seem to be the standard on the web, users don't have to explicitly fork over any payment. The money creators make is not directly from you, but is still from you - your data, your analytics, and your preferences. The important thing, though, to note, is that the consumer never sees any of this. It's taken for granted that things are 'free' on the web, and that's a model we need to shift before WM becomes more dominant.

Conclusion

So many amazing projects came out of the Grant for the Web hackathon, including my project Revshare, and its sister project Revshare for GitHub. These projects let users configure payment pointers to split revenue coming into their sites, allowing users to each get a share of the Web Monetization proceeds.

But there's so much more innovation that can be done in the space, and the issues of centralization and consumer mindsets I've detailed above need to shift first. While this might seem daunting, the Web Monetization community remains active through a handful of forums and discussion boards. I'm hopeful and optimistic we can see more growth in the WM space in the future!

Side plug!

I've recently soft-launched Cobweb, a way for teens to learn and earn on the blockchain. It's built on the Web Monetization protocol with the Ethereum blockchain.

I hope Cobweb is a step towards making Web Monetization truly decentralized: to my knowledge, it's the only other monetization provider besides Coil. It's truly peer-to-peer, fully open source, and never takes a cut of any of your transactions. While it's targeted at teen creators for now, it can be used by anyone on Cobweb-monetized content. I'm excited to be contributing some innovation in the space!

I'd love any feedback on the project - comment down below if you have any thoughts!

Webdev in Vim

Emilie Ma — Thu, 01 Oct 2020 19:32:33 +0000

This post is edited from my original blog post. Find it, and other posts by me on kewbi.sh/blog.

Introduction

As you may have noticed from my last post, I've been doing a lot of editing and writing in Vim. I mostly used VSCode before this, which came with a lot of nice housekeeping features, such as integrated git menus, and lots of autoformatting and intellisense options. I was definitely missing the autoformat on save setting, and the nice indenting that was automatically in place.

Vim definitely can be made to do all these things. The thing is, configuring Vim to do something that VSCode does by default usually takes time. I can see why this might be a reason that people'd prefer just going with an IDE and clicking for one theme change rather than trawl through the Vim Wiki for the solution to a problem that doesn't exist. However, I do also like the customizability of Vim - I can change every colour, snippet, expansion, and keybinding, among many, many other configuration options.

This post will serve as a sort of journal during my first couple weeks doing webdev in Vim, which isn't something I've seen a lot of resources and blog posts for (that being said, I didn't go much further than the first couple pages of Google). For context, these couple weeks I've mostly been finishing revshare-gh and redesigning my website - so I've mostly been working in Javascript, HTML / CSS, and then a bit of YAML.

Phase One

Phase One of my webdev in Vim experiment was mostly the first week or so. At this point, I didn't have any autocomplete, or much else other than a colourscheme, really.

Immediately, when I tried to edit HTML, I was sorely missing the automatic Emmet expansion and auto-end bracket pairing. I still haven't figured out how to add that bracket bit, but to fix the very obvious lack of Emmet, I just installed emmet-vim. This adds a <C-y>, expansion for Emmet abbreviations, which definitely made it less painful to write HTML - I didn't have to struggle around the angle bracket keys.

Within a couple hours of editing, I was also missing the select-surround functionality of VSCode. Essentially, if I selected a block of HTML, and started writing, it'd surround the current tag in whatever I typed. (Also, they had a command palette option to surround in Emmet, which I haven't quite figured out how to do in Vim yet). I added this via vim-surround. This excellent plugin lets me go into Visual Block to select a tag or so, and press S<tag> to surround with a tag. It makes HTML much, much easier.

I also wanted an autocomplete popup menu, and was a bit hesitant to look into Coc or one of the Vim language server protocol extensions, so I decided to just use this workaround from the Vim Wiki. Nothing fancy, and I didn't really get much in terms of 'autocomplete' - it'd only fill with tabs and words from the current open file. That was fine for what I was doing - basic HTML and JS where I wouldn't really need to do much documentation referencing. AutoComplPop added the IDE-like tab completion instead of <C-Space>.

It was also extremely annoying to have to type sequences like () => {} - I was used to the VSCode style of just typing the beginning bracket. All in all, it wasn't too annoying, but I still installed auto-pairs to bring this into Vim.

Formatting was basically done manually - I ran js-beautify (available here) manually in a terminal split before I committed. Very simple, but a little annoying to have to remember it. No big deal though.

It took me a little bit of research to understand Vim's splits and tabs feature, but once I learned how to navigate around properly, it was very useful. I usually didn't work with split windows in VSCode, but I really liked being able to view both HTML and CSS side by side - don't know why I didn't enable it in VSCode. When working in my Hugo projects, for example, I'd have a tab open with my HTML layouts and main CSS files, and another tab with a mostly-fullscreen terminal. This let me switch between hugo server and git commands.

You might notice that I didn't have one of those filetree plugins. I found :Explore more than enough for my needs. My projects are pretty small anyway, just a main file tree in a format that's familiar enough for me to know mostly where things where.

Phase Two

Phase Two is when I actually started investigating autocomplete. I had to properly look into what a language server even was, and understand what the client / server relationship was (i.e. where I was supposed to install things, and what helper extensions that Vim needed to work). I looked into vim-lsp and asyncomplete, both works of Prabir Shrestha. They're probably better for lightweight setups, but I, for some reason, could not for the life of me get the popup tabs to, well, popup. I tried with HTML and CSS, and I found, after a lot of trawling Reddit, that I was supposed to get another plugin for snippets, and a couple more for actually configuring said snippets.

This was a bit overkill - even just adding the HTML language server involved adding around 5 plugins in my .vimrc, and I honestly didn't understand them much. I decided it was probably not worth the time to try to debug this. Besides, I have a decently powerful laptop - I could handle the extra resources that Coc needed.

As to why I chose Coc, I found it very easy to set up in comparison to the various lsp plugins, which had a more 'diy' approach. I guess this is a bit of a counterpoint to my preference for having configurability, but anyhow, I thought it'd be a better use of time to just put one plugin into place instead of having lots of smaller plugins that I didn't have any idea as to how they went together. Also, I'd heard it was very fast and replicated a lot of VSCode Intellisense functionality (which, by this point, I was very much missing), so I gave it a go.

Maybe two or three :CocInstalls later, I had a nice setup running. I've spent a little too much time doing things in Vim during the last couple weeks, so I'm probably going to take a break from configuration. This setup works nicely for me at the moment, and I can write homework assignments / do math / code all in Vim, which is nice.

For those of you who do webdev in Vim as well - share your experiences below!

Making a Vim colorscheme

Emilie Ma — Wed, 23 Sep 2020 19:24:37 +0000

This post is edited from my original blog post. Find it, and other posts by me on kewbi.sh/blog.

Introduction

In the middle of getting back to school and trying to redesign my website, I've also been spending quite a bit of time in Vim. Since installing Manjaro a couple weeks ago, I've decided to experiment with only programming in Vim.

With spending 100% of my writing and editing time in Vim also came a slight obsession with customizing things. Looking up random settings to put into my .vimrc was quite fun - I've managed to do most of what I usually rely on in VSCode anyhow. The Vim wiki really does happen to have everything.

One of these tweaking points was the colourscheme. (Or colorscheme, according to my vimrc. I use both terms interchangeably out of habit from American / Canadian English, so please make sure you've spelled things correctly.) I could have just spent two minutes scrolling through default colourschemes, but I didn't like any of the inbuilt ones. Coming from VSCode, I thought it'd be nice to have the same colourscheme, so I popped vim-code-dark into my plugins and went about my merry day.

However, I wasn't quite happy with the colours - it was a bit too colourful for my taste, since the rest of my setup was relatively monochrome anyway. So - I spent some time researching how to change colourschemes. There are a bunch of internet tutorials for building a basic colourscheme, and even a generator for that™️, but I didn't find much about how to tweak an existing one.

Finding a colourscheme

After watching several people flip through the endless catelog of vimcolors, I'd probably suggest finding a colourscheme that you like ≥90% and tweaking that last 10% yourself. Or if you're planning on just monochroming something, a scheme that has your preferred range of saturation and colour. It'll save you time in customizing and also in finding schemes.

How colourschemes work

Note: most of this is from reverse engineering vim-code-dark, so probably take this with a grain of salt or two.

From my trawling of several Vim colourscheme repos, I've kind of figured out that Vim takes colorschemes from whatever colors/ folder your plugin or colourscheme is installed in. I'm not entirely sure how Vim-Plug works, but whatever I edited inside .vim/plugged/vim-code-dark/colors happened to show up in my colourscheme, so I assume this is controlled via Vim-Plug and whatever you reference inside your .vimrc.

Vim also has a bunch of terms regarding the colours that are used for highlighting and styling.

guifg and guibg: the foreground (text) and background (background of interface) colours for gVim (the GUI version of Vim).
gui: the text style, which obviously isn't a colour, but it fits in here.
ctermfg, ctermbg, cterm: the same as the gui equivalents, but for the terminal.
cterm256: which seems to be cterm but for 256 colour terminals.

Vim does highlighting and colours with highlighting groups, which are all in :help syntax.txt:, under naming conventions. Here, the syntax is something like:

highlight groupname key=value

However, most of this is usually already filled in with whatever colourscheme, so you don't have to worry too much about this.

Another interesting note is that these ctermfg and cterm256fg (and the others as well) can be aliased, and the entire process made a lot easier with function aliases and things. While going through vim-code-dark, I noticed they were using a custom function, which provides shorthand for assigning fore and background colours. If you're interested, it might look a little like this implementation.

Editing colours

Most colourschemes usually have a bunch of colour definitions, with a long list of let statements somewhere, so go look for those. In vim-code-dark, I had to edit the cterm256 definitions. The guifg parts get set with hexadecimal codes, it seems, but the rest of the colours seem to be from the 256 terminal colour chart. You can see the list on this cheat sheet.

What I did here was essentially look up each colour, copy-paste its hex code into Google's colour picker, then choose an appropriately saturated grey. (And yes, the whole colourscheme is grey. I might want to add some colour pastels but for now it works fine. Not on dotfiles yet - I need to set those up properly as well.)

The default color definition bit looked something like:

let s:cdLeftDark = {'gui': '#252526', 'cterm': s:cterm01, 'cterm256': '235'}

and I just edited the cterm256 mapping. (Don't use gVim, but if you're using gVim it's the same, only editing the hex code.)

Some other schemes I looked at while trying to figure things out just used a bunch of definitions without variables - here, you can probably just do a :%s/whatever/whatever2/g.

Right now, it looks something like:

My Vim at the moment.

Conclusion

I'm still going to tweak some of the colours later, but for now, I think that this grey theme's pretty nice. For markdown and writing there aren't a lot of random colours jostling for attention, and when editing proper code it's half decent too.

Anyhow, back to tweaking my .vimrc (and definitely not spending too much time doing that).

Switching to Linux

Emilie Ma — Wed, 16 Sep 2020 21:43:45 +0000

Introduction

As you may have guessed from the title of this blog post, I have switched to Linux. And no, not Arch Linux - though I guess Manjaro is close. Specifically, XFCE Manjaro (and I've also been experimenting with i3).

Before this, I had a very strange split where most of my development things (like Python packages, programs, scripts) would be on the Windows side, and then only the programs that I absolutely needed from Bash and WSL (like C compilers for CS50 and vim) on WSL. However, I've been recently having to lean more on the WSL side, and having dependencies on both sides was a little annoying to manage. For example, I wanted to try Vim and configuring autocomplete with Python there, but the problem was, I needed the packages to be on the WSL side.

As well, I find myself surrounded by a lot of people who use Linux, and their constant harping on about this super cool project and interesting configurations made me curious. I wanted to see what it was like, and I'd been spending a lot of time browsing shiny rices on Reddit. While it's possible to be a productive developer on Windows, I felt that it'd be easier (and more fun besides) to just try installing Linux.

Day -∞

Leading up to the install, which I kept putting off, I spent a decent amount of time researching how to dual boot, partition, configure Linux, and choosing a distro. Various distro quizzes (yes, somewhat cringe) and recommendations led me to Manjaro. It has the configurability of Arch, while having a GUI installer and being less of a pain to install (according to several blog posts, which were extremely reliable sources of information) than pure Arch.

I also was thinking about KDE versus just a tiling WM, but I decided to go with XFCE (also with the help of several probably biased articles). It'd provide graphical options, and would probably make it easier to transition from Windows and its infinitely explorable menus to the already unfamiliar Linux. I liked the idea of a tiling WM, but reading that it didn't provide the same menus to click through was a bit off-putting. After a while, I also installed i3, but more on that later.

And of course, I wanted to experience the ever-mentioned Vim in its full glory. I'd tried it in WSL so this wasn't too much of a leap.

Day 1

Day 1 was a fine Thursday - where I really wanted to just spend some time doing mindless work that'd work without headaches. In the morning, I backed up all my old data and while that was happening, I burned the ISO.

After backing things up and doing some last minute research, I finally booted to the USB and followed this very fine guide on the Manjaro forum. I only shrunk my Windows partition by maybe half (this is important for Day 2), but besides that I was pretty proud of myself. I was quite pleasantly surprised that I ended up not messing anything major up, and the entire installation was done in 15 minutes.

For the remainder of the day, I mostly messed around with terminal settings, and trying to explore XFCE settings. I managed to import my Chrome history and also set up a theme. I did a bit of terminal theming tweaking, and messing around with LightDM backgrounds. Didn't really do much to mess things up, just a lot of installing. I was also pleasantly surprised how fast things and packages download on Linux, and how relatively easy pacman was to use.

Day 2

In the morning, I did a bit of work, and in the background, tried to install more packages. In the meantime though, I also realized that I wouldn't really be on Windows much. So - I tried to resize partitions. (This is where the bit about partitions from last section comes into play). This is what my partitions approximately looked like:

||| windows things ||| linux things + + |

The Linux side had about half, and I wanted to make it more like:

||| win ||| lots of linux space         |

Essentially, more space. The problem was, I didn't know how to rearrange the space after shrinking Windows to move capacity to the home directory. I realize now that I could probably have just formatted the other directories and kept home, but I didn't know that then. In my defense, I hadn't realized that yet. So, I got the trusty Manjaro USB out again - and reinstalled. This unfortunately wiped a lot of my packages, so I had to reinstall things yet again. Unfortunate.

At least this time round, it was a lot easier to get things set up the way I'd like - I knew where each of the relevant settings were, and I even set up OneDrive properly this time. It was very tedious to reinstall everything yet again, but in the grand scheme of things, I spent more time configuring each and removing unneeded packages.

Day 3

Today, I was messing around with power settings and things, and I found that normal Chrome was using a bunch of resources relative to the other programs, at least according to powertop. I did some research, and apparently it was due to YouTube and GPU decoding videos. It wasn't really enough to be noticeable, but I was interested in trying to tweak it anyway. So, I (tried to) download Chromium from pamac. I thought it wouldn't do anything major - just another install. (Spoiler alert: I was wrong.)

It was a pretty big install, so I just did more work and let it run in the background. However, once I finished a problem or two and I was checking on the install progress, I realized it was stuck. What's more, when I tried to close pamac (bad idea #1) just by exiting, it didn't close. Huh. I then tried to force quit it through processes (bad idea #2). It did close, but before that, I noticed that it was stuck particularly on 'Updating grub menu'. In hindsight, that was probably a very bad idea. But I go about my day, and after finishing work, I shut down (kinda bad idea #3).

When I tried to boot my laptop again, I noticed that Manjaro's bootloader kept getting stuck on the HP logo screen, where the spinner would appear for a while, and it'd go dark as if it were properly booting, but then getting stuck in a loop again. I thought it was an issue with the partition things I'd done yesterday, so I re-enter the Manjaro boot live environment. Problem was, it was stuck waiting for a module forever.

I then tried booting into Windows, but for some reason, that was extremely slow and things weren't opening properly. I was pretty stuck on this, but then a friend donated a Manjaro wiki link. It detailed some grub menu fixes, which I attempted to do, but by then, the live USB had managed to fix itself (I guess turning things on and off again does somehow fix things), and I just ended up formatting every partition except my home, which I kept.

Cue a couple minutes of waiting, and then tedious reinstalls. After this though, I managed to not break my system (fingers crossed I don't manage to break it again), and I've been fiddling for the past few days, tweaking configuration files and the like.

Conclusion

In the future, I'm going to continue configuring i3 (which I also installed, just to see what a tiling manager was like) and try to set this up just as I like it. I'd like to fix the status bar to be not quite so colourful and information-full, and make the window tabs a different colour and design. In addition, I'd like to make all my colourschemes nice and greyscale, and maybe properly set up my dotfiles.

In case you wanted a screenshot, here's how my desktop looks at the moment (yes, this is a metascreenshot):

My desktop at the moment

I'm actually pretty pleased with what I've managed to learn and create in the past week or so. It's quite bare - mostly just the default i3 config. I'd like to 'rice' it some more (apparently that's what it's called), and I feel like I've learned a lot from repeatedly breaking my system three times in as many days. I can definitely see why Linux users get so superiorist about it - it's very easy to customize and I think I'll be trying to make this system as bespoke and shiny as I possibly can.

Any domain migration checklists?

Emilie Ma — Wed, 09 Sep 2020 18:41:11 +0000

I'm planning to migrate domains (switching from the github.io domain to my own), and I'm wondering what tips, pitfalls and checklists y'all have for migrating between domains.

Currently, I'm planning to just have the github.io domain redirect to my new one, so if people stumble on the old link ~~or I forget to change it somewhere~~ they'll still get to my actual site.

Also, if you have any places where I might forget to change my website link - please comment!

Hack the 6ix Experience

Emilie Ma — Thu, 03 Sep 2020 17:51:45 +0000

This post is edited from my original blog post. Find it, and other posts by me on kewbish.github.io/blog.

Introduction

A couple hours ago (as of the time of writing), Hack The 6ix had their closing ceremony, and finished 36 (and a bit-ish) hours of hacking away. Hack The 6ix markets itself as Toronto's biggest summer hackathon (evidently, it wasn't in person), and it was an amazing experience! I learned a lot of things regarding JavaScript and OAuth, and did a lot of classic Kewbish speedrunning in the meantime. HT6 is supposed to be 36 hours long, but I happened to be asleep for roundabout 20 hours of the entire thing, so I didn't have time to complete an elaborate project. That's the spirit of most hackathons anyhow - short and sweet. I had a lot of fun, and this blog post'll be a story of the 16 remaining hours of HT6.

The Project

Before I launch into my chronicle of HT6, I may as well describe my project briefly. In the beginning, it was supposed to be a one-click (but customizable) solution to get Hugo pages on GitHub and published on Netlify as well. The template'd include NetlifyCMS, a bunch of variables that can be tweaked (like all the colours, fonts, base images), and a script to automate deploys. I was supposed to get a login page up with Netlify and GitHub tokens, and then automatically create a repo and OAuth app in GitHub to pass into Netlify for automatic CMS and site generation.

However, it ended up being more on the GitHub side: didn't have time to integrate the Netlify. In the end, I created a GitHub action to replace all the variables and just have it set up for publication on Netlify without actually creating the site on Netlify for them. The app forks a template from my GitHub and runs a Python script to replace the variables. Very hacky, not very stable and probably doesn't work properly, given that you have to initiate workflows from forked repos manually. But then again: it's a hackathon project and it's a feature not a bug.

Day One

HT6 started at 6PM for me (I was eating dinner so I already missed out on like thirty minutes), and for the first night, I just spent time reading through GitHub and Netlify docs to obtain various OAuth tokens. I ran into a bunch of issues because I didn't know how to properly use fetch (if you're the MLH recruiter reading through my blog, I can assure you I actually do know how to use fetch, I just blanked a bunch of times during the hackathon; I was very tired, please still consider me). First, I messed up the Authorization headers, and then I had a bit of an issue exchanging tokens on the Netlify side. In this respect, I actually think that Netlify's docs (and forums) were a bit more useful - the GitHub OAuth issue took ages and ages to figure out, while the Netlify one at least showed up on the first page of Google search results.

Day Two

I woke up normally early, but didn't start working until quite late - 800. (I'd gone to sleep at a reasonable (read: 2130) hour - more time lost!) In the morning, I worked on setting up the customization form, which was challenging but still docs- and Google-able. Then, I started trying to get all the data into a bunch of API requests to start generating the template and hooking it up to GitHub's and Netlify's APIs.

The OAuth issue I was talking about earlier was that GitHub's OAuth web flow doesn't actually support client-only exchanging for tokens. I'm not sure if this is actually somewhere in the fine print or the OAuth docs that I just missed, but I ended up wasting a lot of time trying to debug a request that wouldn't work.

Luckily, the power of open source came in clutch: there was a repo for that! prose/gatekeeper provides an easily Heroku-deployable pre-configured app that I could use to exchange for tokens instead! If not for this, I would have wasted infinitely more time trying to set a server up and debugging that instead of properly working with my app (which got scope-chopped as is). I really don't understand why GitHub won't let client side apps call the API, and not even making it extremely clear in docs. Very annoying.

Then, after setting that gatekeeper instance up and managing to finally get a token, I started trying to pass the data to GitHub to copy in. The way I approached making a copy of a template was forking the repo from my GitHub, then trying to edit all the variables. By noon, I'd gotten the customization form to properly work, with nicely configured defaults, and I was planning to get the repos variable changing implemented right after lunch.

But then - I didn't. After lunch, I spent two hours trying to get a file's contents, replace a variable with the form data, and then POSTing a new page. I managed to get the contents fine, and managed to replace the variables fine. However, I had a lot of issues with trying to POST the new file contents back into the repository. It was returning a 404 not-found error - funny, because I'd already authorized my OAuth app with the scopes needed, according to the docs. After maybe an hour retrying the same request over and over again, I took my token from my app's storage, and tried making a request with Hoppscotch. The weird thing? It worked fine.

Now, long story short (it was a very long story, mind you), I ended up finally just trying to copy the generated fetch request out of Hoppscotch itself, and there was my issue. (Again, MLH recruiter, I promise I normally remember these things.) I'd forgotten to wrap my body in JSON.stringify. (Feel free to facepalm on my behalf.) With that, it worked, but by then, I'd already moved onto approach #2.

Approach #2 was, instead of getting, replacing, and posting the correct content (variable replaced) back to the GitHub repo, I'd just make a Python script to quickly activate with an Action. I have some experience with this (from my profile README, perhaps), and it was decently easy to implement. By then, I'd already cut the Netlify automation (generating the site, adding the OAuth client ID and secret, and setting build parameters) out from the project scope. Hopefully, I'll end up being able to work on that in the future.

By then, it was pretty late in the day. I ended up just writing a simple landing page, and cleaning up some of the UI. I removed the Netlify token and authentication logic, and just did some polishing. I wrote a simple README, took a couple screenshots, recorded a demo video, uploaded all my submission data, and sent it off. (By now, it was maybe 2100. Yeah, it was early. Could I have done more? Probably. Did I have the energy to? Probably, but I didn't really feel like staying up late for a hackathon).

Reflections

You might notice that I didn't attend any meetups. That was true (I did spend some time in the networking level of the event) - and I probably should have taken more time to explore and network, actually. Hackathons are probably the best way to meet like-minded people outside of proper jobs and conferences. However, if I did go out to a bunch of events, I wouldn't have any time to work on the hack. That's an aspect of hackathons that I'm not sure if I like: the balance between networking, events, fun time, and work. I'm not sure whether I'd value a proper project more, or a couple workshops.

And as well, this was the first short-term hackathon that I've done. (Does the Dev.to hackathon count?) Anyhow, this was the first very-short-term-panic-speedrun session that I attended, and I'd say it was a pretty good experience. Learning to get a project from 0 to somewhere in a very limited amount of time (which is even more limited when you value sleep instead of projects) is valuable, especially learning how to prioritize features and come up with a plan to work. For me, I'd written down a list of things I needed to do and attached them to time chunks I'd work. Managing that, as well as learning new things, was pretty fun. However, I don't think I want to constantly do a bunch of hackathons - they promote a very short-term, quick churn, small project mindset that I'd like to turn away from. I've done a lot of small projects in the past, and I'd like to get into a bigger, longer-term, more impressive project to dedicate myself to in the future.

Anyhow, here are my three biggest takeaways and things I'd like to do next hackathon:

network more! Set aside some time to go around sponsor booths and really connect.
limit scope - be realistic with what I can do, and then be more realistic and cut some things.
stringify my request bodies. I'm only half joking.

Conclusion

HT6 was a tiring, but engaging hackathon. In the end, I had fun - that's all that matters. Hackathons are really fun, but I think I'm too tired right now to do another 36 / 48 / 72 hour hackathon anytime in the near future.

(If you've taken the time to actually read through all these footnotes, you get a special link to my submission.)

CS50 - The End!

Emilie Ma — Tue, 25 Aug 2020 19:02:17 +0000

This post is edited from my original blog post. Find it, and other posts by me on kewbish.github.io/blog.

Introduction

And finally, after one and a half-ish months, I'm done CS50! From the very first blog post til now, I've been enjoying documenting my progress and learning, and it'll be a fun read to reference back to in a couple of years to see how I've built off this knowledge. This last week, I've been working on my final project, and here, I'll be going through that, along with my thoughts and reflections on the course and experience as a whole.

Final Project

My final project was a Chrome extension that added Web Monetization capabilities to GitHub. For context, Web Monetization is a proposed browser API that'll make it easy for developers and content creators to monetize their browser work. It works by streaming small micropayments over time to a payment pointer, specified by the creator. This pointer is located in a meta tag with the name monetization, and while this API hasn't officially been implemented in any major browsers, you can read more about it here.

Recently, there was also a Dev.to hackathon where Coil, Creative Commons, and Mozilla were calling for project proposals for their grants. I participated in this hackathon, and my project was an implementation of revenue sharing with payment pointers. Essentially, this implemented a custom HTML tag that had an attribute of an array of payment pointers, and my library, revshare.js made it easy for devs to add a probabilistic pointer choice for the payments to be streamed to.

Fast forward a couple months - to this week, in fact. I was looking for a CS50 project, and after brainstorming a couple random projects, a colleague suggested 'Revshare.js for GitHub'. I immediately liked the idea - it was relatively simple to implement, would teach me more about GitHub's GraphQL API and Chrome extensions, and I saw it as a logical extension of the earlier hackathon project. So with that - I jumped into creating the extension.

Features

The extension's pretty simple, but has three-ish main features.

One, it goes through the repository (through a check to see if you're on a github.com/*/* domain) and looks for a Sponsor button through some questionably implemented DOM checks. If found, it looks for the link to the fragment HTML, which I then scrape and select all the links, through more sketchy queries. Once I've retrieved all the links, I check if each link begins with a dollar sign, which is the syntax for a payment pointer. If that pointer's found, or if there are multiples, it chooses one randomly to add to the head of the page itself. With Coil, the extension then begins streaming payments to the payment pointer.

Two, it adds a starsOnly flag to Chrome's storage. If selected, it'll check that the repository is starred before inserting the monetization tag. If it's selected, and the repo isn't starred, then too bad - the repo owner won't be able to get any monetization. This is accomplished through some quite questionable queries to the GitHub links at the top of each repo.

Three, it adds a bunch of checks through each repository's dependencies, looking for the same funding links and $-indicated wallet pointers. If they're found, the extension performs another random selection between the existing repository funding pointer and these new links. It's a roughly 50/50 split between repository creator / owner (i.e. original funding pointer from repository itself) and a random dependency chosen from the list of valid wallet pointers found in each dependency's links. This is pretty cool, because a lot of repositories and dependencies are built for dev-only use, and they don't get a lot of love from supporters. We can even look at stars between a popular Python web microframework Flask and its dependency Jinja. Flask has 51.7k stars, and Jinja a meagre 7.2k in comparison. Given a similar percentage of starwatchers / sponsors, Jinja would get a lot less funding - though both libraries are equally important!

I found it really fun to learn a bit more JS (it's not as painful as it's made out to be, used in the appropriate contexts) and do a bit of DOM engineering. I was very intrigued at the little performance things that GitHub added, like only loading funding links when the button is hovered over, and loading both starred/unstarred versions of the top bar and switching between them client side (I believe). These little tricks were definitely annoying, but in the end, they probably make the GitHub website a ton faster, and hey, I eventually found ways around them. (And in the future, I'll be sure to use the same bypassing techniques I used, like getting fragment URLs instead of trying to force a button click, when I'm trying to manipulate something else.)

Oh, and no GitHub repo just yet. Will edit and insert here soon tho!

CS50 Experience

I found the first few weeks of CS50 extremely informative. Going from someone who'd basically only touched Python, it was sometimes difficult to understand lower-level details like pointers and such, but (while I'm still an extreme beginner and probably can't do much more complicated than Problem Set 5) it was great for what I was taking the course for: data structures, algorithms, and more low level knowledge.

However, I found the latter half of CS50 kind of glossed over a bunch of technologies instead of really diving deep into one for the entire time. I wouldn't have minded as much if it was just C / Python and we went equally (or not) deep into both. The way they have it now, we're just speedrunning a bunch of things that are slightly related but not entirely, and slapping them all together. I would have loved to spend more time with SQL and Flask - but then again, CS50 is an intro to programming, not a deep dive into whatever.

I guess there wasn't that much of an overlap between my initial purpose for taking CS50 and what it ended up being. It might have been due to the fact that I've already done Python and Web, really, but I ended up just speedrunning most of the latter half of the course. I would have enjoyed going into more DSA in C, for example, but that wouldn't have really helped newbie programmers, I suppose.

In the end, if you gave me a magic time machine, I would still have taken CS50, but maybe dropped after the first few weeks. I found it a great experience, and nothing's perfect. Taking a Python DSA (or even just C DSA) course, and then taking a few weeks to do a proper web project and learn SQL / Flask might have been a more beneficial route for me, but CS50 also gave me a lot of motivation. Look, a shiny Harvard certificate. Maybe the feeling of actually taking a class and getting that Harvard clickbait helped me through it. I don't know - but I'll be doing more research and more carefully considering any MOOCs / courses I take in the future.

It's been a fun experience - if you've done CS50, what're your thoughts on it?

CS50 Track - Web

Emilie Ma — Wed, 19 Aug 2020 18:40:11 +0000

This post is edited from my original blog post. Find it, and other posts by me on kewbish.github.io/blog.

Introduction

This post is going to be shorter than normal. I may or may not have forgotten to take proper notes for most of this track except Flask.

Web track, check. Homepage and Finance were pretty fun, and now, we have the final project ahead. This week (or track, I suppose) went through web technologies, going through HTML / CSS / JS and Flask. I've been planning on learning Flask for a while, and this was a great opportunity to try to use it! I know I said that Week 5 was the week I'd been looking forward to the most, but it was genuinely really nice this week to finally work on something I have experience with.

Notes

Each machine has an IP
- used to use IPv4 -> 32 bit addresses
- now IPv6 -> 128 bit addresses
- IPv4 overflowed (too many devices)
HTTP -> protocol to transfer information
- HyperText Transfer Protocol
DNS -> maps domain numbers to servers that can resolve that domain
Response codes -> return of request
Flask -> microframework to make web apps w/ Python
Flask templates -> render_template()
- allows us to interpolate variables, extend layouts
Flask templating uses Jinja syntax
- Like Vue mustache syntax, but with % signs
- specific end blocks for for and if
request.args.get() is useful for getting form inputs
- remember to sanitize input and make sure that inputs exist
  - otherwise, return error
In Finance, we make separate conditions to return different templates if post/get
use Flask sessions to maintain data like user cookies, etc.
- otherwise, use the sqlite database
  - run queries with db.execute

Problems

I did Finance first, because I thought that I wanted to redesign my own site for Homepage (spoiler alert: didn't end up doing that). I found their todo-list structure of teaching / explaining what to do quite interesting, and avoided the tedium of creating boilerplate.

Giving freedom to implement however we liked was a good idea in terms of implementation and research, but this comes at the cost of testing. (Or maybe this just wasn't in the CS50 check50 library and they didn't want to add a bunch of overhead for one problem set check) This might have been an interesting opportunity to explain TDD (hmm, another thing that I should learn) or even just explaining what automated testing was.

When I decided it was finally time to try to attempt redesigning my website, I quickly realized that it'd take ages, especially since I don't have any ideas right now. The requirements were a 4-page website, so I just slapped an extra page (that looks nothing like the rest of my site because it's entirely built with Bootstrap) onto my current site. For the JS requirement, I worked in one of those console header things; y'know, console.log()s in most big sites that remind you to apply for their jobs.

It's pretty simple, but I was feeling tired. Which is another discussion point - I think Homepage was probably the only big problem in CS50 where I didn't really feel challenged, besides the Python ones. Even with the Python ones I was trying to time myself in a self-run programming contest, whereas here I was just trying to speedrun. Despite this week's relative slow pace, I have a scheme for the final project.

Conclusion

I can't believe I'm already almost through with CS50 - as of writing this, I think it's only been a month and a couple days. I'm surprised how much I managed to speedrun, and how much I managed to learn in the meantime. It's been an great time, and I might consider doing the Web Programming course as well. (Or maybe not. We'll see.) Time to start brainstorming for a final project (I have too many ideas and I have a feeling I'm going to go with none of the above) - hopefully it'll turn out interesting.

What are your experiences with Flask? What Flask features should I check out?

Adding blog posts to your GitHub README with Python

Emilie Ma — Wed, 12 Aug 2020 20:42:34 +0000

This post is edited from my original blog post. Find it, and other posts by me on kewbish.github.io/blog.

Introduction

I'm working on the web track of CS50 at the moment, and I'm having a lot of fun, actually. In the meantime, I thought I'd take some time to investigate GitHub's new profile feature, and take a dive into GitHub Actions.

Yes, we have READMEs now.

I'm a bit late to the game, but hey, if you want these posts as I write them, here's my blog.

For a couple days, my Dev.to, Reddit, and dev Discord feeds were inundated with the shiny new GitHub profile README feature. All you need to know is that if you create a repo named your GitHub username (i.e. kewbish/kewbish) and make a README, it'll show on your profile, like so:

Hey, a cool README. ~~Yes, it's mine~~.

I'm not going to go through writing the copy / main text of the README much. After writing the first version, I started seeing lots of cool READMEs on Dev.to. Apparently, there's even an awesome list now. Why am I not surprised?

Anyhow, after reading through too many 'top 8 GitHub README' lists, I found SimonW's featured quite often, and I really liked the self-updating blog posts / TIL sections. So, I decided to implement a similar, albeit simpler version on my own README.

RSS with Hugo

Skip over this bit if you're not using Hugo - I'm just going over some changes to Hugo's default RSS that you can definitely ignore.

Hugo comes with a RSS template built in, so I had an RSS feed before I even knew I had one. However, you can also customize it just like all the other default layouts. This is the default template Hugo ships with - here are the changes I made.

Changing the description (line 18):

<description>Recent content {{ if ne  .Title  .Site.Title }}{{ with .Title }}in {{.}} {{ end }}{{ end }}on {{ .Site.Title }}</description>

This is pretty self-explanatory, just changed it to:

<description>Latest Yours, Kewbish posts</description>

Changing the date format (line 32):

<pubDate>{{ .Date.Format "Mon, 02 Jan 2006 15:04:05 -0700" | safeHTML }}</pubDate>

I prefer a cleaner date format (02 Jan 2006) instead of all this time info, so I changed this to:

<pubDate>{{ .Date.Format "02 Jan 2006" | safeHTML }}</pubDate>

Move from summary to description (line 35):

<description>{{ .Summary | html }}</description>

I wanted to use my descriptions instead of the first couple lines, so I used this:

<description>{{ .Description | html }}</description>

These are all just personal preference, but it makes the README bit a little more consistent with the actual blog.

Scripting with Python

The README update script is only 18 lines of Python, and uses the feedparser library to, well, parse the RSS feed.

Of course, let's start with installing and importing the library with pip install feedparser and:

from feedparser import parse

Next, we're going to get all our feed entries.

feed = parse("https://kewbish.github.io/blog/index.xml").entries
latest = [
    f"""- [{feed[i].title}]({feed[i].link})  \n{feed[i].description} - {feed[i].published}"""
    for i in range(3)]

feed contains all the entries of your RSS feed (you're going to want to change the URL to something other than my blog URL, obviously - try using your dev.to feed!). Then, we create a new list to store the first three entries, formatted as a two-line bullet point. The first line will have a link to the post and the title, and the second a description and publishing date. You can definitely play around with this, it's just plain markdown, and this is just how I decided to format my README.

farr = []
with open("README.md", "r", encoding='utf8') as x:
    for line in x:
        if line.strip() == "<!--bp-->":
            break
        farr.append(line)

We then open the README file and read each line into an array if it isn't this specific HTML comment. At this point, you might want to go back to your README and add the  comment at the end. (If you want it somewhere in the middle, you're going to have to modify the code by adding a new array and reading into that array after the comment is encountered, probably by setting a boolean value somewhere.)

with open("README.md", "w", encoding='utf8') as x:
    x.writelines(farr)
    x.write("<!--bp-->\n")
    li = [x.write(i + "\n") for i in latest]

And finally, we open the README, this time in write mode, and write all the lines back. Then, we rewrite our comment line, and then our latest list, which will be the list of formatted blog posts. (Again, if you want your widget somewhere in the middle of your README, you're going to have to write the new array you created after the blog post lines.)

The full script can be found on my GitHub.

You're also going to want to create a requirements.txt file with feedparser in it, so go ahead and do that.

Creating a GitHub Action

Note: SimonW's blog post was super helpful in figuring this out - much of my code was created after looking through theirs!

Now that we have our script and requirements, let's make our Action. There's a little Actions button on the main page of your repository, so click that and create a new workflow. Choose the 'by yourself' option, which will spit out a long YAML file. We're going to rewrite the file, so go ahead and delete it.

name: Add newest YK

on:
  workflow_dispatch:
  schedule:
    - cron: '0 */6 * * *'

First, we start with our Action name. Pretty self explanatory, call it whatever you want. Next, we have our on triggers. These define when our Action will run. workflow_dispatch lets me trigger one manually, and schedule uses familiar cron syntax. (In case you're wondering, this runs the Action every 6 hours. I highly recommend crontab.guru for figuring this out. GitHub does have a built-in tooltip though, so that can be helpful.)

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
    - name: Check out repo
      uses: actions/checkout@v2

Every Action also has a set of jobs that you can run on it. The build and runs-on line are pretty standard, just defining your Action to be run on the latest version of Ubuntu. Then, we have a set of steps, which are each individual tasks that can then run commands for us. Our first step will be checking out the repo. This is also pretty standard, as we just use one of GitHub's premade Actions.

- name: Set up Python
    uses: actions/setup-python@v2
    with:
    python-version: 3.8
- name: pip caches
    uses: actions/cache@v2
    with:
    path: ~/.cache/pip
    key: ${{ runner.os }}-pip-${{ hashFiles('**/requirements.txt') }}
    restore-keys: |
        ${{ runner.os }}-pip-

This part sets up Python, using another premade Action, and sets the default Python version. Next, we set up the pip cache so we won't have to download the dependencies each time. More information about this part can be found on the GitHub site.

- name: Install Python dependencies
    run: |
    python -m pip install -r requirements.txt

Here, we run one command to install the requirements from the requirements.txt file - here, just feedparser.

- name: Update README
    env:
        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
    run: |-
    python get_post.py
    cat README.md

With this, we get the GitHub secret that auto-generated when using GitHub Actions, and then run the script we created earlier. Then, we cat this to README.md. Now, in our Action, our README will have updated (or not - more on that with the next block.)

- name: Commit and push if changed
    run: |-
        git diff
        git config --global user.email "yourskewbot@notarealdomain.com"
        git config --global user.name "YoursKewbot"
        git add -A
        git commit -m "Update blog posts" || exit 0
        git push

We diff the two files. If they've changed, then we set a configuration for our committer bot. Here, I've just set it to some random information - this is what'll end up in Git history and in GitHub's contribution bar at the top of your repo. Then, as we normally do when committing code, we add all the files, commit them, and push them back to the repository. At this point, our README will have changed live.

See the full Action on my GitHub.

Conclusion

Now, every 6 hours, our Action will run and update our profile README. Hopefully, this was a good introduction to GitHub Actions, and now, you have a shiny new updating README! This was a really fun learning experience for me as well - now, I can be part of the cool GitHub Actions-powered README squad!

What are some other creative RSS-based README's you've seen?

CS50 Weeks 6 / 7 - Python and SQL

Emilie Ma — Wed, 05 Aug 2020 20:28:52 +0000

This post is edited from my original blog post. Find it, and other posts by me on kewbish.github.io/blog.

Introduction

Finally, we're getting into the more 'application' side on things - also known as the part where I'm more comfortable with. With my initial predictions, I'd predicted that Week 6 (where we are now) was where it'd get slightly easier. And I was right - Week 6 was a welcome week, an introduction to Python. I'm already somewhat decent at Python, so this week wasn't such a struggle, and Week 7 was a great mix of new tech / familiar tech.

I was actually pretty surprised how 'natural' C already felt, and I caught myself trying to printf("") instead of print(""). I guess I haven't been writing as much Python in the past few weeks - mainly focusing on C and some Javascript side things.

I was also pleasantly surprised how intuitive SQL was to write - nearly like plain English. SQL was one of those I'll learn it eventually things that I used to keep putting off, but I'm happy I finally took the time to learn it. In my projects that have required a backend, I've mostly used Firebase (and Mongo once, but that didn't pan out very well), so relational databases were a refreshing new experience.

Skip to my thoughts.

Week 6

Week 6 Notes

denote a raw string with r''
- raw means that special characters, like \n are entered as a slash and a n
- also can use formatted string inside it with rf''
. denotes any character
.* denotes 0 or more characters
.+ denotes 1 or more characters or matches
? denotes optionality
^ denotes a start
$ denotes the end of input
to use regex, usually have to import re
- comes with a couple useful functions
- .search takes two+ arguments
  - pattern, searching string, and options
to search for y or yes, you could do something like '^y(es)?$'
- ^ denotes the start of the string
- immediately after, there should be a y
- the brackets with the ? denotes the optional 'es'
- then, the string should end
to search for n or no:
- similar - '^n(o)?$'
re.IGNORECASE ignores the case of the input
used this in the DNA problem set
- used .findall, which returns all matches for a particular STR

Week 6 Problems

Finally being able to write implementations of the C problems in Python was extremely refreshing. I could actually use list comprehensions and not the clunky 3+ line for loop syntax, for example. Python's just a lot more concise and as people put it, it reads like pseudocode. That makes it a lot easier to translate my brainwaves into proper, functioning programs, unlike C, where I have to wrestle with pointers and such.

As a comparison, here are some of the line counts of the problems in problem set 6.

Hello: in C, 10 loc. In Python, 2 loc.
Mario Less: in C, 24 loc. In Python, 12 loc.
Mario More: in C, 29 loc. In Python, 15 loc.
Cash: in C, 21 loc. In Python, 15 loc.
Credit: in C, 65 loc. In Python, 21 loc.
Readability: in C, 41 loc. In Python, 15 loc. There's no correlation between lines of code and personal opinion, of course, but I personally prefer Python.

Skip to my thoughts.

Week 7

Week 7 Notes

SQLite3 is the management system, SQL is the language standard
- basically a system to query and select cells from Google Sheets or similar
- like more complicated CSV
use .schema to access the schema, or layout of the database
use .import [file] [name] to import a CSV or similar file
- need to set .mode csv first
once inside the database
- run SQL queries, usually involving select
* is a wildcard -> select all
- % -> some characters
- usually used with like
to select a count, use the count() function
to group, use group by x
- also can order by x
orders can be ascending or descending
to limit the number of results, use limit x
SQL queries can span multiple lines
- written in all caps by convention, but lowercase works too
- ends with a ;
All database operations fall under create, read, update, or delete
- in SQL, insert, select, update, delete
To initialize a table, need to use a create statement
- has x number of columns
- each column has a specific data type
These types can be:
- blob -> binary files
- integer -> several types
- numeric
- real -> real numbers
- text
- can also specify not null, to make sure it exists
where matches a specific condition
- can use and and or in conditions
- like matches substrings of text
join joins a second table to the first
- use an on bit to compare two pieces of data
- ex. stars.movie_id = movies.id
while using the CS50 library, use 'SQL'
- initialize a connection with db = SQL('path')
- run commands with db.execute('x')
- use wildcards with ? marks, specify in option arguments
multiple types of keys
- primary -> primary identifier
- foreign key -> another row in another table
- unique -> unique value in table
- index -> creates an index structure to make queries quicker
race conditions, where timing matters
- solve with transaction types
SQL injection attacks
- when special characters are put in and not escaped

The two problems were pretty intuitive to solve, and only minor Googling was needed. Definitely around as challenging as the Python bit.

Conclusion

We're almost done with the main chunk of CS50. I'm pretty surprised how fast these first 7 weeks went by - only the track (which I'm greatly looking forward to) and the final project (which I have a good idea for) to go! These'll probably take a while to complete as well, but I'm committed to writing a blog post a week describing my experiences.

Speaking of tracks, I'm planning on doing the Web track. I have experience with web technologies already, so this seemed like a logical choice. In past posts, people have commented that the Web track was a little outdated, so we'll see about that. I'm excited to get more acquainted with Flask, as well. Like SQL, getting more experience with Python in the web has been a 'one of those days' things, and hopefully, this will be a good reintroduction.

What are your recommendations for learning SQL better - SQL games, websites, or communities?

CS50 Week 5 - Speller

Emilie Ma — Wed, 29 Jul 2020 20:32:00 +0000

This post is edited from my original blog post. Find it, and other posts by me on kewbish.github.io/blog.

Introduction

Finally, I'm over the worst of CS50 (in my opinion, at least). Week 5 was a bit of a difficult lesson and problem set, but in the end, it actually wasn't as hard as I thought it'd be. Week 5 covers data structures - detailing hash tables, linked lists, and tries, which are a combination of both! This is the week I was looking the most forward to (my original purpose for taking CS50 was for data structures and algorithms after all), and dreading as well.

Skip to my thoughts.

Notes

Here we go:

you can't reassign something if it doesn't exist yet
- remember to initialize to a chunk of memory
also, you should remember malloc's effects if you reassign, and use free if you reassign a malloc
arrays are difficult to resize, because they're initialized to a certain amount of memory
- could move a copy of the array to a larger, free area
- then can delete old copy
we could also use realloc
- as its name implies, it reallocates memory
- give it the pointer of the old array
- will return address of new array
- remember to free variable
data structures are custom structures to store information
- made with structs
- builds off included data types
linked lists
- basically an array, but each element points to the next one
- elements are not together in memory
- each element includes a pointer to the next
can't access the middle of the list with just [x] notation
- there isn't a 'middle'
- need to navigate through the entire list first
also takes twice as much memory per element
- needs to store the next pointer
usually constructed of a struct
- one part to store the actual data, and the same struct pointing to the next struct
- initialize first to NULL, so you can assign
introduce a new notation, -> notation
- similar to dot notation of a pointer
- node->next = x;
need to use a while loop to iterate through the properties
- check if not NULL
- set the variable
if you want to add to the beginning
- set a pointer to point to the beginning
- then set the list to the last pointer
- inserts a node at the beginning
to insert in the middle
- do something similar
- need to create a temporary variable for the swap
linked lists are O(n) time, need to follow each node pointer to find the next
also introduces a tree
- each node points to two nodes, like the famous binary search tree
- makes binary search very easy, only compare two nodes
- makes insertion easy as well, only rearrange a small subset
- search is O(log n)
need to balance these though, or else may become reweighted
- also memory-expensive, but can search faster
hash table combines arrays and linked lists
- each element in the array is a linked list
- can add elements quickly, and the initial searching time is decreased
- however, they might all end up in the same element, in which case the time efficiency is negated
- get as close as possible to O(1) when the number of elements equals the possible values
retrieval tree provides O(1) searching, but at cost of space
- stores each level of element (here, letters) in a separate array
- in this example, 26x as much memory
more data structures
- stacks -> last in, first out, like email inbox
- queue -> first in, first out, like line in a store
- dictionary -> map keys to values, like Python!
these data structures can be implemented with arrays, linked lists, hashtables, and other structures

Problem Set

This week, we only had Speller to work through. But don't underestimate it either - it took long days of work to figure out. The logic wasn't too hard to implement, actually.

First, I split up the problem in its subparts:

hash -> I decided to use the simplest hash function - just the first character. Could this be optimized? Yes, but I just wanted to try the data structure out first, and not have to worry about copying a hash function from online that I didn't completely understand either.
load -> Made an array, and I'd put each word into its appropriate element. I just appended the current word to the end of the linked list, and lowercased the entire string as well.
size -> In load, I'd created a line to increment a global variable, which made size just a return count; statement.
check -> I hashed the current word to compare, and then used a while loop to iterate over the linked list and checked if it matched the current targeted word.
unload -> I iterated over each element in the array, and again iterated over the linked list to free each pointer in the list.

The first time I wrote the program, it worked as intended, so check. But, upon check50-ing, I got a bunch of valgrind errors. I had forgotten to free a bunch of malloc'ed variables, and to fix this, I tried to use a character array instead. Also, I finally learned to use valgrind properly - I'd kind of ignored it in the past week, given that there wasn't a check for a memory leak, just a reminder that it could exist. I also realized that my unload function was logically incorrect, and would always return true right away. After fixing this, I attempted to test it - but now, it didn't produce the intended output. Oops.

I stripped out the entire program, and rewrote it from scratch, including what I'd learned about valgrind and memory allocation in the first runthrough. (Now that I think about it, it might have been something to do with redirecting output to the wrong file, but hey, 🤦s aside, it was a good experience to rewrite.)

Now, I was memory leak free, and working with the correct output. Nice! In the process of scrolling dozens of Stack Overflow pages, I really learned to appreciate the full power of valgrind. It's a great resource for checking memory leaks, and despite its rather scary, confusing interface, it's an essential tool. I never had to keep memory management in mind with Python, but C made me more cognizant of the lower-level management that goes into C.

Conclusion

Am I going to continue pursuing C? I don't really think I will. Would I have ignored C if I were to take some version of CS50 where I didn't need C? Probably not. I've learned a lot about lower level things, like memory management, as well as getting a glimpse into how the easy-to-use features in Python were really implemented. As countless people have said before, working with C makes one really appreciate how nice higher level languages are to work with, and it was a great experience.

That said, I still can't wait to get into the later half of CS50 - Python, SQL, and web? Yes, thank you very much. I guess this would be the more application side of CS50, and I'm excited to get learning.

What are some recommendations you have to avoid memory management errors, or some areas of research that you'd recommend to understand memory (in Python or C) more?

CS50 Weeks 3 / 4 - Algorithms in C

Emilie Ma — Thu, 23 Jul 2020 20:44:02 +0000

This post is edited from my original blog post. Find it, and other posts by me on kewbish.github.io/blog.

Introduction

This post'll go through weeks 3 and 4 of CS50, and include my notes and comments for both, because I couldn't be bothered to write up two separate posts. I'll go back to one a (CS50)week when we hit Week 5.

Week 3 goes through several common sorting algorithms and Big-O notation, and week 4 goes through memory and files. Both problem sets are slightly unrelated, but that's fine - I learned a ton this week as well. We're in the thick of it now - weeks 3 - 5 are apparently the most difficult in CS50. (And I can finally say... I finished Tideman!)

Skip to my thoughts.

Week 3

Let's start with the usual notes:

there are many types of search and sort
- some are more efficient than others
for example, linear and binary search
- linear goes through all the elements
- binary effectively cuts the number of comparisons in half
linear search
- go through all elements one by one
- if it matches target, congrats!
binary search
- choose a midpoint
- if midpoint is the target, return index
- otherwise, search the left and right halves, depending on the target and the current midpoint
these two have different O times
- simple mathematical expressions to return the worst case
there are also lots of ways to do sorting
- insertion, bubble, and merge are among a few
- actually, I coded a bunch of these already on GitHub
merge sort introduces us to recursion
- a function calling itself with some arguments
- make sure there's a base case set, or else it'll infinitely run
- in merge sort, we check for the size of the array to check
- more efficient than selection sort, O-wise
C also allows us to declare custom structs
- a form of class / object is how I understand it
- used a lot in problem sets to simplify

Week 3 Problems

Personally, I found the lecture pretty unrelated to the problem set, which was all about voting. This was one week where I kind of regretted my idea to do both less/more comfortable versions of a problem - Runoff and Tideman were both super difficult. This was also the first week we got 'distribution code', or a template that takes care of most of the functions for us.

One thing that was a significant obstacle was the various structs and variables, and how we got the distribution code. I didn't spend much time poring over the given code, and as a result, it was a little difficult to remember the types and purposes of each variable. I kept having to refer to the walkthrough video to remember what each function was supposed to do. Runoff was clearer in this case, providing actual hints. I guess Tideman was supposed to be more difficult, but it would have been nice to have more hints along the way.

Tideman also involves a decent amount of graph theory, or at least knowledge of recursion. Nowhere in the lecture was the graph theory really covered, so I had to do a lot of figuring and drawing algorithms out on my own. It was helpful to draw dummy tables out and go through the algorithm step by step, at least.

Skip to my thoughts.

Week 4

Here are my notes:

we learn about a new counting system of hexadecimal
- what's used in colour codes
- denote hexadecimal with 0x
what even is the point of pointers?
- given a variable stored somewhere in memory, it has a pointer
- pointers give us the address of a variable
- denoted with *
the actual address can be found with &
- represented in hexadecimal as well
- &var gets the address
combine with *&var, which goes to the address and gets the value there
points at first character -> strings
- CS50 library had abstracted this away, but now we have to use pointers
- char * s
- can also access individual characters, which map to the pointer + x
week 4 also teaches us how to allocate memory
- malloc -> allocates a space in memory
- need to define a size that it needs to be allocated
- can't be changed
to copy a string, we can use strcpy or malloc and copy it with a loop
- remember to copy the \0 byte as well, or else things will crash
- doesn't know where the string ends
after we allocate memory, we need to remember to free it
- use the free function to free memory
if we don't, we're going to end up with a memory leak
- use valgrind to check for possible leaks
computer memory is split into several sections
- one is the code itself
- another is the global variables
- the heap is empty, where free memory gets pulled in
- the stack is used by functions that are currently being called
once a function is returned
- is freed from the stack
- any arguments get lost
this is also where pointers and addresses come into play
- by passing addresses into pointer arguments, you can actually change the variable itself
overflows (like the website)
- when there isn't enough memory in the heap to satisfy malloc, we get a heap overflow
- too many functions loaded? stack overflow!
- called buffer overflows -> might crash system, extremely fun
can assign a variable just to NULL
- doesn't point to anything
with pointers, we can start to manipulate files
- fopen opens a file with a character pointer (string), and a mode
  - like Python
- fwrite and fread, well, write and read
- fread is interesting, it takes the variable, block size, number of blocks to read, and finally the filepath
- remember to close all files with fclose

Week 4 Problems

This week's problem set was a lot less painful. I guess it's just because week 3 was more focused on algorithms and abstract programs, whereas this week was more hands-on and practical. The distribution code this week was also less related to what we actually had to implement, so was much easier to just skim through.

Having experience with Python's file operations definitely helped a lot, and made the entire file open/read/close flow easier to understand. I think the most annoying part of this week was definitely just learning that fwrite != fprintf (one is bytes and the other is strings). Besides that, the photo filtering and recover algorithms were really well explained in the problem brief, and didn't require too much mental gymnastics to figure out.

(Also, brief note that Filter Less and Filter More are basically the same program. More just replaces Sepia with Edges, but Edges builds off Blur, anyhow. Again, do both parts of the problem, I promise it's fun!)

In general, I think that I prefer practical projects rather than more abstract problems, which also explains my difficulties with algorithms. Well, I guess that was why I took this course - more algorithms!

Conclusion

In my predictions, I'd thought that weeks three through five were going to be the most painful, and I guess now that I'm here I kind of agree. Week 4 was tolerable, but Week 3 was absolute torture, and I'm not looking forward to Week 5 either. (On the bright side, Week 6 means the glory of Python once again.) Can't wait!

What are your tips for understanding pointers and algorithms, in C and in general?