DEV Community: Outvi V

Contributing assets is never easy, but we have GitHub Actions

Outvi V — Sat, 12 Sep 2020 02:16:04 +0000

Yeh, we have been building a site of soundbites for monthes.

Also yeh, collecting sound clips from videos is never easy. You need a good connection to YouTube. You need youtube-dl. You need ffmpeg. You also need to know how to use ffmpeg. And sure, many fans (whoever want to contribute to this project) are Windows users and they don't know about anything above. So how to do the audio collection?

Our idea is letting users contribute by simply commenting in a issue.

Procedure

A contributor making comments.
Our bot analyzes the comments, clip the audio and save it somewhere.
Our bot replies the link to the contributor.

This is a solution enough for the problem. After that, the contributor just downloads the file, fills in the metadata and makes a pull request to our project... wait, pull request?

They are not used to the terminal and now you ask them to make a PR

🤔

Well... that's a problem. To ease the process, we need to avoid writing code for merely submitting an audio clip. This is where the GitHub action come:

A More User-friendly Procedure

A contributor making comments.
Our bot analyzes the comments, clip the audio and save it somewhere.
Our bot replies the link to the contributor.
An action detects the audio link.
An action makes a PR with the audio link and the given metadata information.

That is our solution!

First, we tell the metadata:

And finally, the PR is made:

Tada! 🎉

Last but not least, recently we rebuilt the site with Svelte 3, making it faster to view and easier to maintain.

Have a try on the brand-new ☄️https://suisei.moe!

This post, as well as this post, is a part of the GitHub Actions Hackerthon.

Asset download, upload and metadata update all in one

Outvi V — Sat, 12 Sep 2020 01:00:56 +0000

My Workflow

This actions is designed to, with the help of a GitHub bot:

Accept clip and metadata input from a comment in issues
Clip the asset according to the comment
Download the asset and put it in the correct location (here begin the action part)
Fill in the metadata
Make a commit and push as a PR

Submission Category:

Maintainer Must-Haves

Yaml File or Link to Code

name: "Auto PR for new sound"

on:
  issue_comment:
    types: [created, edited]

jobs:
  push_to_branch_and_create_pr:
    name: Create a PR with the sound
    if: "(contains(github.event.comment.body, ' pr ')) && ((contains(github.event.comment.author_association, 'OWNER') || contains(github.event.comment.user.login, 'librehsbot')) && (github.event.issue.number == 1))"
    runs-on: ubuntu-18.04
    steps:
      - uses: actions/checkout@v2
        name: Check out current commit

      - uses: suisei-cn/actions-download-file@v1
        id: downloadfile
        name: Download the file
        with:
          url: ${{ github.event.comment.body }}
          target: assets
          auto-match: true

      - uses: suisei-cn/actions-update-metadata@v2
        id: updatemeta
        name: Update sounds.yml
        with:
          comment: ${{ github.event.comment.body }}
          target: sounds.yml
          default-username: librehsbot
          format: yaml

      - name: Create Pull Request
        uses: peter-evans/create-pull-request@v2.8.0
        with:
          committer: Suisei Bot <noreply@suisei.moe>
          author: ${{ steps.updatemeta.outputs.username }} <${{ steps.updatemeta.outputs.username }}@users.noreply.github.com>
          commit-message: "feat(sound): Add ${{ steps.downloadfile.outputs.filename }}"
          title: "feat(sound): Add ${{ steps.downloadfile.outputs.filename }}"
          body: |
            This is an automated sound addition PR. Netlify preview should be available soon, and a preview link should be shown.

            <sub>Having problems? Check your [workflow](https://github.com/suisei-cn/sbtn-assets/blob/master/.github/workflows/auto_pr.yml).</sub>
          branch: sound/new
          branch-suffix: "random"
          labels: new-sound

auto_pr.yml

suisei-cn / sbtn-assets

Starbuttons assets.

Starbuttons Assets

This repository saves assets of starbuttons since starbuttons@2.0.0.

Contributing

Sounds update: sounds.yml
Category update: categories.yml
Sound assets: assets/

View on GitHub

Additional Resources / Info

Two written actions with this action:
- actions-download-file
- actions-update-metadata
The GitHub bot: pvp-clipbot

Transparent proxy on Arch Linux, with iptables and systemd slice

Outvi V — Tue, 25 Feb 2020 12:16:24 +0000

Long story short. To build a transparent proxy, we need to redirect every outbound request to the proxy. It seems easy with iptables... wait a second, no. Outbound requests from the proxy should not be redirected, or nothing can be sent out. How to do that? Usually an exception on the proxy server IP is set. But what if you have multi proxy servers? What if your proxy servers are changing overtime? Wouldn't it be a pain editing iptables rules?

Well, we have cgroups. We can put our proxy program in a cgroup, and give an exception to this cgroup on iptables. On Arch Linux, the easist way to create cgroups is via systemd. Therefore, we are picking the systemd PANTS.

Here shows an example for a TCP IPv4 transparent proxy.

Create a slice, and put stuffs in

If your proxy program runs as a systemd service, you can add this "Slice=" line to its configurations:

[Service]
Slice=bypass.slice

After daemon-reloading and restarting, your proxy program will be in the "bypass.slice" group.

You can also see a new directory named bypass.slice in /sys/fs/cgroup/unified/:

# ls /sys/fs/cgroup/unified/bypass.slice/
cgroup.controllers  cgroup.max.descendants  cgroup.threads  io.pressure
cgroup.events       cgroup.procs            cgroup.type     memory.pressure
cgroup.freeze       cgroup.stat             cpu.pressure    run-u1925.scope/
cgroup.max.depth    cgroup.subtree_control  cpu.stat

If your proxy program is not a systemd service, run it with systemd-run：

sudo systemd-run --slice bypass.slice --scope clash -c /etc/clash/config.yaml

We use --scope so that you can see the stdouts and manipulate on stdins. It requires root permission (or not if you are using cgroups v2?). I would think you have got the root permission, since we're editing iptables later!

The same way if you want to start an application which bypasses the proxy:

sudo systemd-run --slice bypass.slice --scope firefox

Or a shell which bypasses the proxy:

sudo systemd-run --slice bypass.slice --scope -S

Have a try

Let's run a ping in a slice:

$ sudo systemd-run --slice test2.slice --scope -S
Running scope as unit: run-r7865e1749deb48e4bcf797f1f403f396.scope
$ ping 1.1.1.1

The ping is running here. Now we block all outbound packets from this slice:

iptables -A OUTPUT -m cgroup --path "test2.slice" -j DROP

The ping will start to fail:

ping: sendmsg: Operation not permitted

Okay. We see that iptables are working with our slice (cgroup). Now delete the rule:

iptables -L OUTPUT --line-numbers
# Find something like
#  "2 DROP   all  --  anywhere  anywhere  cgroup test2.slice"
# and remember its number
iptables -D OUTPUT 2

iptables' time!

Parts of this paragraph comes from Dreamacro/clash#158 and this.

# Create a chain on the nat table.
# Why nat table? Because we are doing redirects later.
iptables -t nat -N TP-TCP

# On this chain:
# 1. Everything from the bypass.slice should be where they were
iptables -t nat -A TP-TCP -m cgroup --path "bypass.slice" -j RETURN
# 2. Everything to local & loopback address should be where they were
iptables -t nat -A TP-TCP -d 0.0.0.0/8 -j RETURN
iptables -t nat -A TP-TCP -d 127.0.0.0/8 -j RETURN
iptables -t nat -A TP-TCP -d 10.0.0.0/8 -j RETURN
iptables -t nat -A TP-TCP -d 169.254.0.0/16 -j RETURN
iptables -t nat -A TP-TCP -d 172.16.0.0/12 -j RETURN
iptables -t nat -A TP-TCP -d 192.168.0.0/16 -j RETURN
iptables -t nat -A TP-TCP -d 224.0.0.0/4 -j RETURN
iptables -t nat -A TP-TCP -d 240.0.0.0/4 -j RETURN
# 3. Everything else should go thourh the proxy port
iptables -t nat -A TP-TCP -p tcp -j REDIRECT --to-ports 7892
# 4. All output packets should go through TP-TCP chain
iptables -t nat -A OUTPUT -p tcp -j TP-TCP

Do some curls and see if they are going through the proxy. Also, run

iptables -L TP-TCP -t nat -v -n

to see if any pkts and bytes is going through. If it works, we're done! Hooray! 🎉