DEV Community: Johannes Naylor

Discover Hidden Secrets in Git Repos with Rust

Johannes Naylor — Mon, 08 Nov 2021 15:22:32 +0000

Intro
Git Blobs
Secrets Test
Conclusion

Intro

It's no secret to we something, every once in a while, on occasion, accidently commit a password or api key into git. And it's not secret that instead of doing the right thing, we just delete the password or api key, recommit, and move on with our day only to forget it ever happened. Well, today we're going to find out how easy it is uncover that bad behavior and hopefully convince you to reconsider next time.

Git 101

For this task, we'll need to know a litle bit about how git works under the hood. While this isn't necessarily important for writing the code (you can 100% program this yourself by just Googling stuff and coping+pasting stackoverflow code), it'll for sure make the code easier to read and understand.

Git is a content-addressable filesystem. It means that at the core of Git is a simple key-value data store. What this means is that you can insert any kind of content into a Git repository, for which Git will hand you back a unique key you can use later to retrieve that content. (source)

What git really is under the hood is a database of objects (objects in this case refering to unstructured content like files). When you make changes to a project and commit those changes, the changed files are stored as objects along with a bunch of other fancy metadata git uses on top of that. Objects can be either tags, commits, trees, or blobs. Tags give labels to commits, commits contain metadata (e.g. timestamp, author, etc.), trees are like file and directory names that hold information about the project structure, blobs are the actual content of the files. Both of these object types are stored in git's object database (which you can see for yourself by looking in .git/objects )

Git in Rust

It's no secret that git is written in C which means there is plenty of support for C library like libgit2. Luckly, there are rust bindings for libgit2 in the crate git2 which gives us a nice high level API with types like Repository, Commit, Object, Odb, etc. that we can leverage to search for secrets.

Secrets

An important note, when I say secrets, I mean any sensitive information that you wouldn't want to commit to a public repo (e.g. passwords, api keys, mother's maiden name, etc.). Obviously, we can't check for everything so for this, we'll be checking for a subset of secrets, namely:

[
  "AWS API Key",
  "Facebook Oauth",
  "Generic API Key",
  "Generic Secret",
  "GitHub",
  "Google (GCP) Service-account",
  "Google Oauth",
  "Heroku API Key",
  "PGP private key block",
  "Password in URL",
  "RSA private key",
  "SSH (DSA) private key",
  "SSH (EC) private key",
  "SSH (OPENSSH) private key",
  "Slack Token",
  "Slack Webhook",
  "Twilio API Key",
  "Twitter Oauth"
]

Projects Overview

With all the background knowledge in place, we can combine to create the basic flow for the project. We're going to be giving our utility a git repo as an input, we'll need to get the object database from the git repo, get all of the objects that are blobs, and check each blob for secrets. To check for secrets, we'll use regex and check for matches in the blob's content.

# Pseudocode
secrets = [...]
repo = some_git_repo
objectDB = repo.objects

for object in objectDB
    for secret in secrets 
        if object.type is Blob && secret in object.content
            print secret.type
        end
    end
end

The final source code can be found here: https://github.com/jonaylor89/JAZ

Setup

Before starting, go ahead and create a new rust projects with cargo.

cargo new secret_catcher

Scanning Git Blobs

Looking at our pseudocode, it looks like the first thing we need to do it figure our how to ingest a git repository, read its object DB, and somehow iterate over it. As mentioned above, for this, we'll be using the git2 crate so the first set is adding that to our Cargo.toml

 [package]
name = "secret_catcher"
version = "0.0.1"
authors = ["John Naylor <my@email.con>"]
edition = "2021"
description = "Find secrets hidden in commits"
repository = "https://github.com/jonaylor89/JAZ"
license = "MIT"
readme = "README.md"

[dependencies]
git2 = "0.13"

Moving to the code, let's start by injesting the git repo:

use git2::Repository;

fn main() {

    // Get path to git repo via command line args or assume current directory
    let repo_root: String = std::env::args().nth(1).unwrap_or_else(|| ".".to_string());

    // Open git repo
    let repo = Repository::open(repo_root.as_str()).expect("Couldn't open repository");

    println!(
        "{} state={:?}",
        repo.path().display(),
        repo.state()
    );
}

To make things easier, we can test our code using the git repo for this project:

$ cargo run
/home/johannes/Repos/secrets_catcher/.git/ state=Clean

With that bit of code, we're using git2's Repository type to serial the git repo. What we need now it to extract the object DB from it. Fortunuately for us, git2 has some methods that'd make that trivial.

use git2::Repository;

fn main() {

    // Get path to git repo via command line args or assume current directory
    let repo_root: String = std::env::args().nth(1).unwrap_or(".".to_string());

    // Open git repo
    let repo = Repository::open(repo_root.as_str()).expect("Couldn't open repository");

    println!(
        "{} state={:?}",
        repo.path().display(),
        repo.state()
    );

        // Get object database from the repo
    let odb = repo.odb().unwrap();

    // Loop through objects in db
    odb.foreach(|oid| {
                println!("{}", oid);

        // Return true because the closure has to return a boolean
        true
    })
    .unwrap();
}

Gotta love when things are simple (can't beat Rust's no-cost abstraction)

The code above compiles to print all of the git objects to the console. By leveraging the method odb() for the Repository type, we can a git2 Odb type, that contains a very handle foreach() method. With the foreach() method, supply a closure that will soon contain the code for seeing if that git object contains any secrets.

For reabilitiy purposes, let's put the code for scanning the object into its own function:

odb.foreach(|oid| {
        // Get the object from the object's id
    let obj = repo.revparse_single(&oid.to_string()).unwrap();

    // Look for secrets in the object
    scan_object(&obj, &oid);

    // Return true because the closure has to return a boolean
    true
})
.unwrap();

With the function looking something like this:

use git2::{ObjectType, Object, Oid, Repository};
use std::str::from_utf8;

...
fn main() {...}
...

fn scan_object(obj: &Object, oid: &Oid) {

    if let Some(ObjectType::Blob) =  obj.kind() {
        let blob_str = match from_utf8(obj.as_blob().unwrap().content()) {
            Ok(x) => x,
            Err(_) => return,
        };
        // println!("{}",blob_str);

        // Check if the blob contains secrets
        // TODO: check for secrets in commit here!!!
    }
}

Raw git objects require a little bit of preprocessing before we can check if they contain secrets as you can see above. First we use obj.kind() to see if we're dealing with a Blob because that's the only type of git object we care about. Any other object type can be discarded. Next, to extract the raw Blob string, we use the as_blob() method in conjection with the from_utf8() , found in the standard library, to finally get a raw Rust string containing the Blob information.

Again, for readability, let's put the code for actually determining if a Blob has a secret in it, into its own function.

fn scan_object(obj: &Object, oid: &Oid) {

    if let Some(ObjectType::Blob) = obj.kind() {
        let blob_str = match std::str::from_utf8(obj.as_blob().unwrap().content()) {
            Ok(x) => x,
            Err(_) => return,
        };
        // println!("{}",blob_str);

        // Check if the blob contains secrets
        if let Some(secrets_found) = find_secrets(blob_str) {
            for bad in secrets_found {
                println!(
                    "object {} has a secret of type `{}`",
                    oid,
                    bad
                );
            }
        }
    }
}

// is_bad : if secrets are found in blob then they are returned as a vector, otherwise return None
fn find_secrets(blob: &str) -> Option<Vec<String>> {
    None
}

In the updated scan_objects() function, we're passing the Blob content to a newly created function find_secrets() , which we'll be filling in later, and printing a short message to the console if that Blob did, in fact, contain secrets.

For a little bit of flair, I went ahead and added some color to the output of the print statements. This is competely optional but it definitely makes it look cooler and is a tad more pleasing to the eyes.

use ...;

macro_rules! info {
    () => {
        format!("{}[INFO]{}", "\x1B[32m", "\x1B[0m")
    };
}

macro_rules! critical {
    () => {
        format!("{}[CRITICAL]{}", "\x1B[31m", "\x1B[0m")
    };
}

fn main() {
    ...
    println!(
      "{} {} state={:?}",
      info!(),
      repo.path().display(),
      repo.state()
  );
    ...
}

fn scan_object(...) {
    ...
    println!(
     "{} object {} has a secret of type `{}`",
      critical!(),
      oid,
      bad
  );
    ...
}

Finding Secrets with Regex

It seems that now, the only other thing that needs to be implemented is the find_secrets() function that gets called in out scan_objects() function. There are a few ways to go about doing this but for this project, let's use regex. We'll check every Blob we get against a set of regex which will tell us if there is a secret and what kind of secret it is. Every "hit" we get, we'll add that secret type to a Vec and return that Vec at the end of the function. First, for the exact regex, I found a few other similar projects that use regex to find secrets and frankensteined them into a HashMap which is our set of rules.

fn find_secrets(blob: &str) -> Option<Vec<String>> {
    let rules = HashMap::from([
      ("Slack Token", "(xox[p|b|o|a]-[0-9]{12}-[0-9]{12}-[0-9]{12}-[a-z0-9]{32})"),
      ("RSA private key", "-----BEGIN RSA PRIVATE KEY-----"),
      ("SSH (OPENSSH) private key", "-----BEGIN OPENSSH PRIVATE KEY-----"),
      ("SSH (DSA) private key", "-----BEGIN DSA PRIVATE KEY-----"),
      ("SSH (EC) private key", "-----BEGIN EC PRIVATE KEY-----"),
      ("PGP private key block", "-----BEGIN PGP PRIVATE KEY BLOCK-----"),
      ("Facebook Oauth", "[f|F][a|A][c|C][e|E][b|B][o|O][o|O][k|K].{0,30}['\"\\s][0-9a-f]{32}['\"\\s]"),
      ("Twitter Oauth", "[t|T][w|W][i|I][t|T][t|T][e|E][r|R].{0,30}['\"\\s][0-9a-zA-Z]{35,44}['\"\\s]"),
      ("GitHub", "[g|G][i|I][t|T][h|H][u|U][b|B].{0,30}['\"\\s][0-9a-zA-Z]{35,40}['\"\\s]"),
      ("Google Oauth", "(\"client_secret\":\"[a-zA-Z0-9-_]{24}\")"),
      ("AWS API Key", "AKIA[0-9A-Z]{16}"),
      ("Heroku API Key", "[h|H][e|E][r|R][o|O][k|K][u|U].{0,30}[0-9A-F]{8}-[0-9A-F]{4}-[0-9A-F]{4}-[0-9A-F]{4}-[0-9A-F]{12}"),
      ("Generic Secret", "[s|S][e|E][c|C][r|R][e|E][t|T].{0,30}['\"\\s][0-9a-zA-Z]{32,45}['\"\\s]"),
      ("Generic API Key", "[a|A][p|P][i|I][_]?[k|K][e|E][y|Y].{0,30}['\"\\s][0-9a-zA-Z]{32,45}['\"\\s]"),
      ("Slack Webhook", "https://hooks.slack.com/services/T[a-zA-Z0-9_]{8}/B[a-zA-Z0-9_]{8}/[a-zA-Z0-9_]{24}"),
      ("Google (GCP) Service-account", "\"type\": \"service_account\""),
      ("Twilio API Key", "SK[a-z0-9]{32}"),
      ("Password in URL", "[a-zA-Z]{3,10}://[^/\\s:@]{3,20}:[^/\\s:@]{3,20}@.{1,100}[\"'\\s]"),
  ]);

    None
}

Next, we'll need to iterate over those rules and see if we get a match from teh blob parameter passed. If there is a match, we'll push that secret type onto our secrets_found Vec that we defined at the start of the loop and return that Vec at the end (if it has any matches).

fn find_secrets(blob: &str) -> Option<Vec<String>> {
    let rules = HashMap::from([...]);

    let mut secrets_found = vec![];
    for (key, val) in rules {
        // Use regex from rules file to match against blob
        let re = Regex::new(val).unwrap();
        if re.is_match(blob) {
            secrets_found.push(key.to_string());
        }
    }
    if secrets_found.is_empty() {
        // Return bad commits if there are any
        return Some(secrets_found);
    }
    None
}

And with the find_secrets() function done, we can go ahead and test our code! Again, the easier way is to just test it against the git repo that you're using for the project:

Conclusion

At this point, we've succeeded at what we set out to create. I went ahead and scanned common testing repositories for this sort of thing like Plazmaz/leaky-repo and dijininja/leakyrepo. In general the program found all or most of the secrets. In the case of
dijininja/leakyrepo it found a lot of RSA private keys which is acceptable but technically a misidentification. For
Plazmaz/leaky-repo we find the majority of the keys although once again misidentify some. The decision to use rust makes performance really solid although still a little slow even for small repos. A couple good extensions to this to help with that could be adding a thread pool in order to scan objects in parallel. In more professional code, it seems more idiomatic for the scan_objects() function to return some objects of objects including their results rather than just printing the one containing secrets. For example, it could be formatted something like this:

{
    objectID1: [
        secrets1,
        ...
    ],
    ...
}

In the end, this tool could work as a good starting point for something more sophisticated.

The final source code can be found here: https://github.com/jonaylor89/JAZ

My Switch Away from Python

Johannes Naylor — Sun, 07 Nov 2021 00:04:11 +0000

From Beginner to Professional
Typing
GUIs
Deploying
Containers
Extendability
Typing
Rust

For some context for this article, python was one of the first languages I learned and became comfortable with. The incredible part about python is how accessible it makes programming to the world in comparison to other beginner languages like java. With that being said, the longer I've been in the software development sphere, the more I've started running into issues with using python in projects that I've outlined a bit here.

From Beginner to Professional

Undeniably, Python has made huge steps in program accessibility for not just the average computer science student, like myself, but to everyone around the world. It's given people the freedom to focus on being creative and not thinking about what's going on behind the scenes. And this accessibility is coming from a real honest-to-god programming languages with batteries included, not some toy language like MIT's scratch or whatever. Machine Learning is probably the biggest example of how much a field can benefit from having an accessible programming language behind it. If ML was still all done with something like C/C++ rather than in python atop numpy/tensorflow/matplotlib then you could make the argument that ML wouldn't have risen to popularity like it has. Looking at how the future is moving, I see two things that are sparking the end for python: the invention of the newer compiled languages (Swift, Golang, Rust) and the movement to NoCode.

Most of my projects I'm either writing something that is going to be fairly large in which case I want to use something that gives me a ton of power and performance (i.e. some compiled language that is easily containerized and not python) or I'm gluing a bunch of pieces together to make something using existing tools in which case I'm using NoCode stuff (Zapier, Airtable, IFTTT, etc.). With NoCode becoming so popular, the barrier to entry is almost nothing. Look, for example, at the apple shortcut app for iOS, I've seen just about everyone with an iPhone make something tiny with it and that's the essence of programming really. This space that python filled is being taken over by these tools and maybe they have a way to write extensions in python but let's be honest with ourselves, most people write the extensions in javascript (unfortunately 🤢). It'll be a bit till this way of programming has enough modules/libraries and support to take over Python's foothold in the accessibility space but it'll definitely continue to be a major reason to switch away from python when you just need something simple and rapidly developed.

Typing

Going along with the theme of accessibility, Python has been able to attact a wider audience by going with a dynamic typing system vs the tradition static typing of langauges like C or Java. This is great so beginners as I mentioned because of how forgiving it is and how easy it is to glue things together in code. For a big project stand point, dynamic typing is a huge pain in the ass: "Why am I getting this random type error in my code nowhere near where the actual value is being assigned!?" or "What the hell does this person's library take in as an input!?". This sucks, luckily, a lot of projects and libraries made for big projects use Mypy, Python's static extenal type checker, and proper type annotation (well at least I hope the big libraries have proper type annotations). And Personally, Python's type annotions along with Mypy is much better than typescript is for javascript. Although, just like how typescript isn't a complete fix for how terrible javascript is, so it Mypy for Python. I've seen too many people not use type annotations for things and if their aren't type stubs, which I've found for a lot of data science stuff, then it's really not that helpful. There are other options for types like for example the really incredible library Cython. Cython compiles python code into C, allowing you to add type annotations to your python that'll actually speed up your code. The major pain in the ass with this though is that Cython type annotations aren't the same as the normal Mypy type annotions and Cython types match directly to C types whereas Mypy type annotations do not.

Mypy

def f(x: int): int:
    y: int = 5
    return x + y

Cython

def int f(int x):
    cdef int y = 5
  return x + y

So it's kind of a mess to look at and deal with. Going forward, it really seems that Mypy type annotations are really cool but they'll never replace a really good statically typed language. Many of the new modern languages that I mentioned earlier like Swift, Go, and Rust are all statically typed, but the compiler is smart enough to infer the types of variables.

let x = 42 // x is inferred to be an Int

This is super accessible and the static type system doesn't get in the way unless you try and do something stupid.

GUIs

To be honest, there isn't a whole lot to say about GUIs in Python and that's the problem. My personal favorite GUI framework for Python is PyQt but it's really just a wrapper around the C++ Qt library and it shows. For a language that's so friendly to beginners, a really solid GUI framework should be essential. The undisputed leader of GUIs is javascript whether it be on the web or as an Electron desktop app. Compared to a quick Electron app, PyQt apps provide a less than optimal developer experience. From the mobile side of things the normal Swift for iOS, Kotlin for android, and javascript/dart for web apps dominate the field. Quick cross-platform GUI apps would be great to have but unfortunately Python just isn't there.

Client Side Deployment

Even if there was a good framework, it would have to address deployment. I recently tried deploying a PyQt app to some windows machines and it sucked. My team even tried compiling to an exe and that was a whole ass experience. Client side deployment for interpreted languages seems to always have a few issues and it seems to be especially prevalent with python. The operating system provided python interpreter always seems to cause problems without fail and particularly with beginners who aren't used/don't know who to set up a virtual environment for python. Going along with that, I've always found python dependencies to also be a huge mess when working with/deploying python code with setup.py being not very robust to errors and difficult to understand for beginners compared to other languages (rust's cargo being one in particular that I like). Fortunately, there are a few projects in the community that make dependency management easier like pipenv and poetry . Both tools make things pretty straight forward since pipenv feels like node's npm while poetry feels like rust's cargo .

Server Side Deployment

Back in dinosaur times, languages like C were running into an issue: they weren't as cross platform as the industry needed them to be. Through that, the magical thing known as the virtual machine was born. Java with it's huge, fancy virtual machine could sweep the market because it didn't care what architecture it was on as long as it had a java virtual machine. As the market expanded, interpreters and virtual machine were doing great except for one problem: dependencies. Java apps won't run if the machine they're running on is missing a dependency. Likewise, python apps will crash if the machine is missing a library. The answer to this and all the worlds problems were containers.

Luckily, server side, containers make deployment and dependency management easy even for the most difficult languages. With docker I can do the beautiful thing and pull in an image with a python interpreter that I can wrap everything in. I can't be too nice because even with containers, python seems to do worse than most other languages. Interpreted languages are known for having bloated container sizes (just the base python image is ~1Gb), which is a core reason that new compiled languages like Go can get away with not having an interpreter. Python, being such a high level language, has lots of important libraries that are actually just C code under the hood and when that gets factored into container builds it can lead to slow build and start up times. That's not just a python problem though, the JVM which used to be this really amazing thing now just created bloated containers with extremely slow start up times.

Rust

This final section is mostly just my opinion alone. I haven't researched much about what other people think of this matter but personally I strongly think that python should distance itself from C/C++. The main interpreter for python, CPython, is unsurprisingly written in C. This is great for speed but it means that contributing to the language is becoming more and more isolated from the majority of users. C (God's programming langauge) itself just is incredibly unsafe and should definitely not be used in any kind of production environment. Of course nobody is really arguing that C is unsafe which is why plenty of companies and organizations have moved to rust. Rust is the modern choice for things that need to be ultra efficient but also safe for production. Python is used globally in production environments and run by an unsafe interpreter. I think python should seriously be considered for a rust rewrite. The creator of nodejs is already attempting to do this with server side javascript by starting the project Deno. Deno is a safer version of nodejs written in rust. There are already community attempts to move python to rust just as the open source interpreter RustPython. One of the beauty of the python interpreter (especially compared to the JVM and V8 for node) is its amazing simplicity. People should be able understand the interpreter their code runs on and rust, being able to abstract things away from the bare metal while not sacrificing speed, would be a major step for this.

Of course a total rewrite of a code base like this is insanely hard and would be a lot of work but it would also have the unfortunate effect of breaking anything that relies on CPython's C API. CPython can't get rid of the Global Interpreter Lock (GIL) because it would require breaking some of the C API. A rust rewrite would do that times a thousand. The time and energy it would take to rewrite all the libraries that rely on the C API components of python could end up killing the language entirely. People may just end up deciding that it's not work the effort and more to a different programing language. It seems that with the addition of Tensorflow for swift and Python Interoperability in swift, the data science and machine learning community, where I think a majority of the C backed libraries come from, might move over to swift regardless.

Conclusion

Until something major happens, I don't see python going away that easily. My guess is that as time goes on, people will have less and less a need for it. It'll still likely be one of my go to languages for quick prototypes (there's nothing like spinning up a quick flask server for a hackathon). Plus as things stand now, everyone knows or has seen python before so demonstrations in it are safe. But when it comes down to making projects that I plan on keeping for more than a week, I now look at other choices for my language of choice.

Creating a Twitter Bot in Go

Johannes Naylor — Sun, 07 Nov 2021 00:03:08 +0000

Intro
Getting Twitter Credentials
Loading Credentials
Adding a Main
Putting It All Together
Deploying on Heroku

Intro

This is a short, step by step, tutorial for how to make a twitter bot. The bot I'll create in this tutorial is going to be called "Big Pineapple Guy" and it's purpose will be to comment on my friend's posts with facts about pineapples. I'm going to be making the bot in Go and I'm going to be deploying it onto Heroku so everything will be free. The end result of this will be a bot that comments stuff like this:

All the code for this can be found here

Getting Twitter Credentials

I want my bot to have its own Twitter account so step one for all of this is to go to Twitter create a new account for the bot.

Next I need api access to this account. To do that, I need to register as a developer at https://developer.twitter.com. From there, I create a new app and fill it in with basic info about what the bot will be doing.

Once the app is created, I can go in an get the keys and tokens I need.

Loading Credentials

I'm going to take the keys and tokens that Twitter gave me and put them in a yaml file for now to access them during development.

creds.yml

ConsumerKey: YOUR_CONSUMER_KEY
ConsumerSecret: YOUR_CONSUMER_SECRET
AccessToken: YOUR_ACCESS_TOKEN
AccessSecret: YOUR_ACCESS_SECRET

For the first bit of go code, I'm going to write something to get these credentials from the yaml file. There is a library for parsing yaml that I'll need first.

~$ go mod init // initializing a go module to keep track of dependencies
~$ go get gopkg.in/yaml.v2

Now that I have the library, I can write the basic function to get the credentials. I decided to put it in its own file to keep things clean.

creds.go

package main

import (
  "fmt"
    "os"
    "log"
    "io/ioutil"
    "gopkg.in/yaml.v2"
)

const (
    credsFile = "creds.yml"
)

type creds struct {
    ConsumerKey string `yaml:"ConsumerKey"`
    ConsumerSecret string `yaml:"ConsumerSecret"`
    AccessToken string `yaml:"AccessToken"`
    AccessSecret string `yaml:"AccessSecret"`
}

func getCreds() *creds {

    c := &creds{}

    // Read File
    yamlFile, err := ioutil.ReadFile(credsFile)
    if err != nil {
    log.Fatalf("[ERROR] %v", err)
    }

    // Unmarshall Creds
    err = yaml.Unmarshal(yamlFile, c)
    if err != nil {
        log.Fatalf("[ERROR] %v", err)
    }

    return c
}

So far the directory for the project should look like this:

.
├── creds.go
└── creds.yml

Adding a Main

Before programming a main, I have to find a good twitter API client library for go. Fortunately, there seems to be a really go one by a Github user named dghubble.

go get github.com/dghubble/go-twitter/twitter

I want my bot to listen for new tweets from specific twitter handles and run some code whenever that happens. This can be done by listening to twitter event streams. Twitter gives developers the ability to listen for events and filter out unwanted events. The documentation for it is here. In my case, I want to filter by specific twitter handles. The go twitter library I'm using has an example of how to listen to twitter streams (https://developer.twitter.com/en/docs/tweets/filter-realtime/overview). Going off this example, I can create the basic main function for my bot.

main.go

package main

import (
    "fmt"
    "log"
    "os"
    "os/signal"
    "syscall"

    "github.com/dghubble/oauth1"
    "github.com/dghubble/go-twitter/twitter"
)

func main() {

    creds := getCreds()

    if creds.ConsumerKey == "" || creds.ConsumerSecret == "" || creds.AccessToken == "" || creds.AccessSecret == "" {
        log.Fatal("Consumer key/secret and Access token/secret required")
    }

    config := oauth1.NewConfig(creds.ConsumerKey, creds.ConsumerSecret)
    token := oauth1.NewToken(creds.AccessToken, creds.AccessSecret)

    // OAuth1 http.Client will automatically authorize Requests
    httpClient := config.Client(oauth1.NoContext, token)

    // Twitter Client
    client := twitter.NewClient(httpClient)

    // Convenience Demux demultiplexed stream messages
    demux := twitter.NewSwitchDemux()
    demux.Tweet = func(tweet *twitter.Tweet) {
        fmt.Println("[INFO] ", tweet.Text)
    }
    demux.DM = func(dm *twitter.DirectMessage) {
        fmt.Println("[INFO] DM: ", dm.SenderID)
    }
    demux.Event = func(event *twitter.Event) {
        fmt.Printf("[INFO] Event: %#v\n", event)
    }

    fmt.Println("Starting Stream...")

  // FILTER
    filterParams := &twitter.StreamFilterParams{
        Track:         []string{"cat"},
        StallWarnings: twitter.Bool(true),
    }
    stream, err := client.Streams.Filter(filterParams)
    if err != nil {
        log.Fatal(err)
    }

    // Receive messages until stopped or stream quits
    go demux.HandleChan(stream.Messages)

    // Wait for SIGINT and SIGTERM (HIT CTRL-C)
    ch := make(chan os.Signal)
    signal.Notify(ch, syscall.SIGINT, syscall.SIGTERM)
    log.Println(<-ch)

    fmt.Println("Stopping Stream...")
    stream.Stop()
}

The demux is what is used to separate the different message types in the stream. For each message type (i.e. Tweet, DM, Event) I register a callback that is called every time one of those events happens. For now I'm just printing the messages out when I receive them.

Above the main function, I'm going to put a variable that is an array of all the Twitter handles I want to filter.

var victims = []string{
    "A_Chris_Kahuna",
  "georgiopizzeria",
  "HakaRoland",
}

func main() {
...

Unfortunately, I can't filter by just Twitter handles, only Twitter IDs. So I'll need to write something that converts the array of Twitter handles into an array of Twitter IDs.

import (
...
    "strconv"
...
)
...
// Twitter Client
client := twitter.NewClient(httpClient)

var ids = []string{}

    for _, name := range victims {

        // User Show
        user, _, err := client.Users.Show(&twitter.UserShowParams{
            ScreenName: name,
        })
        if err != nil {
            fmt.Println("[ERROR] on ", name)
            continue
        }

        ids = append(ids, strconv.FormatInt(user.ID, 10))
    }

// Convenience Demux demultiplexed stream messages
demux := twitter.NewSwitchDemux()
...

Using this new ids array, I can filter the stream.

...
fmt.Println("Starting Stream...")

// FILTER
    filterParams := &twitter.StreamFilterParams{
        Follow:        ids,
        StallWarnings: twitter.Bool(true),
    }
    stream, err := client.Streams.Filter(filterParams)
    if err != nil {
        log.Fatal(err)
    }

// Receive messages until stopped or stream quits
go demux.HandleChan(stream.Messages)
...

Now that I'm filtering by Twitter IDs and only getting the tweets I want, I can start writing some logic for what do when I receive a new Tweet message.

demux.Tweet = func(tweet *twitter.Tweet) {

        // Ignore RTs
        if tweet.Retweeted {
            return
        }

        // Ignore Replies
        if tweet.InReplyToStatusID != 0 || tweet.InReplyToScreenName != "" || tweet.InReplyToUserIDStr != "" {
            return
        }

        fmt.Println("[INFO] Tweet: ", tweet.Text)

    }

In the Twitter API, tweets have a ton of field. In the library I'm using, this file has the Tweet struct with all of the available fields. I only want to interact with tweets tweeted directly from my fields, no retweets or replies so that was the first thing I make sure to avoid. With this, I have everything I need to move onto the fun part.

Putting It All Together

Now I get to decide what I want to reply with. I want the reply to be a random pineapple fact so to keep things neat, I'm going to put the list of facts in a new file

facts.go

package main

var facts = [...]string{
    "Pineapple can reach 3.3 to 4.9 feet in height. Large specimens of this fruit can reach nearly 20 pounds of weight.", 
    "Pineapple is perennial herbaceous plant that has short and stocky stem. Its leaves are spiny and covered with wax on the surface.",
    "Fruit of pineapple is result of fusion of 100 to 200 individual flowers.",
    "Color of the fruit depends on the variety. Pineapples are usually red, purple or lavender in color.",
    "Christopher Columbus brought pineapple from South America to Europe. This fruit was first named \"pina\" because it looks like large pine cone. Englishmen later added name \"apple\" to denote it as a fruit.",
    "Two types of pineapples, called \"cayenne pineapple\" and \"red Spanish pineapple\" are currently cultivated and consumed throughout the world. They differ in color and size of the fruit.",
    "Birds (such as hummingbirds) and bats pollinate pineapples.",
    "One plant produces only one pineapple per season. Ripening process ends after removal of the fruit from the stem.",
    "Pineapple is rich source of fibers, manganese, vitamin C and vitamins of the B group. It can be consumed raw, in the form of juices, or as a part of various sweet and salty dishes. Pina colada is popular drink that is made of pineapples.",
    "Pineapple is used to alleviate symptoms of common cold, cough and nasal congestion (by decreasing the amount of mucus). It can reduce inflammation and prevent development of blood clots.",
    "Pineapple contains bromelain, a substance which decomposes proteins and facilitates digestion. People can use marinade made of pineapple juice to reduce toughness of the meat (bromelain acts as softener of meat).",
    "Pineapple can be easily cultivated using the crown of the fruit. Crown should be cut and partially dried before planting in the soil.",
    "South Asia is the biggest producer of pineapples. More than 2 million tons of pineapples are produced and exported from Philippines each year.",
    "Almost all parts of the pineapple can be used in the production of vinegar and alcohol. Inedible parts can be used as food for domestic animals.",
    "Pineapple can live and produce fruit for up to 50 years in the wild.",
    "Pineapples regenerate! You can plant pineapple leaves to grow a new plant.",
    "Hawaii produces about 1/3 of all pineapples in the world.",
    "Pineapples are a cluster of hundreds of fruitlets.",
    "Pineapples take about 18-20 months to become ready to harvest.",
    "Pineapple is the only edible fruit of its kind, the Bromeliads.",
    "One pineapple plant can produce one pineapple at a time.",
    "Pineapples ripen faster upside down.",
    "A pineapple is not an apple, or pine. It’s actually a berry!",
    "The world’s largest pineapple ever recorded was in 2011, grown by Christine McCallum from Bakewell, Australia. It measured 32cm long, 66cm girth and weighed a whopping 8.28kg!",
    "The Hawaiian word for pineapple is ‘halakahiki‘.",
    "The Dole Plantation’s Pineapple Garden Maze in Hawaii has the record for the largest maze in the world, stretching over three acres!",
}

Using a random number generator in Go's standard library, I'm going to pick a random item from the list to reply with. It's very important to seed the number generator with the current time because of how I'm going to be deploying it later. This'll just make sure I'm actually using a random number.

import (
    ...
    "math/rand"
    "time"
    ...
)

...

// Twitter Client
client := twitter.NewClient(httpClient)

rand.Seed(time.Now().UnixNano()) // Seed the random number generator

var ids = []string{}

...

    // Ignore Replies
    if tweet.InReplyToStatusID != 0 || tweet.InReplyToScreenName != "" || tweet.InReplyToUserIDStr != "" {
        return
    }

    choice := facts[rand.Intn(len(facts))] // Pick a random fact
    botResponse := fmt.Sprintf("@%s Pineapple Fact: %s", tweet.User.ScreenName, choice)

    fmt.Println("[INFO] Tweet: ", tweet.Text)

...

Finally, I'm going to add the logic for replying to the tweet.

demux.Tweet = func(tweet *twitter.Tweet) {

        ...

        // Reply to  Tweet
        reply, _, err := client.Statuses.Update(
            botResponse,
            &twitter.StatusUpdateParams{
                InReplyToStatusID: tweet.ID,
            },
        )
        if err != nil {
            fmt.Println(err)
        }
        fmt.Println("[INFO] ", reply)
    }

If everything is set up correctly and there are no bugs, I should be able to build and run this on my own machine.

~$ go build
~$ ./BigPineappleGuy

To test, I added my own personal Twitter account to the array of victims and tweeted some test tweets. With everything working, I can move onto deploying the bot so I don't have to run it on my own computer.

Deploying on Heroku

Before doing anything on Heroku, I'm going to have to make some changes to how I was running the bot. The first is how I'm getting the credentials. I really don't want to check my credentials into git and push them to Github so I'm going to modify the cred.go file to be able to accept credentials from environment variables.

creds.go

func getCreds() *creds {

    c := &creds{}

    // Read File
    yamlFile, err := ioutil.ReadFile(credsFile)
    if err != nil {
    fmt.Println("[INFO] No yml config, pulling from environment")
        c = &creds{
            ConsumerKey: os.Getenv("CONSUMER_KEY"),
            ConsumerSecret: os.Getenv("CONSUMER_SECRET"),
            AccessToken: os.Getenv("ACCESS_TOKEN"),
            AccessSecret: os.Getenv("ACCESS_SECRET"),
        }
    } else {
        // Unmarshall Creds
        err = yaml.Unmarshal(yamlFile, c)
        if err != nil {
            log.Fatalf("[ERROR] %v", err)
        }
    }

    return c
}

After this modification I can log into my Heroku account and make a new project. In this new project, I'm going to input all of my keys and tokens as environment variables through the web console.

Before actually deploying though, I'm going to add a Procfile to my project folder. This will tell Heroku that my project isn't actually a website and is just a bot.

Procfile

worker: BigPineappleGuy

At this point, my project folder looks like this:

.
├── Procfile
├── creds.go
├── facts.go
├── go.mod
├── go.sum
└── main.go

Now I'm ready to push all of my stuff to Github and point Heroku at my Github repo for deployment.

And with that I'm done! I have a Twitter bot written in go running on Heroku. This was a really fun project to make and it was really nice seeing my friends post pictures in our groupchat of a mysterious pineapple account commenting on their posts.

How to Manipulate Images on a GPU

Johannes Naylor — Sat, 06 Nov 2021 23:55:29 +0000

Intro
Project Setup
Creating the Kernel Functions
Async and CUDA Streams
Conclusion

Intro

Manipulating and analyzing images is one of the most useful and interesting topics in the field of computer and data science. With so much of the world being ingested or displayed as an image, the amount of data, the speed at which data is collected, and the use cases for the data are growing exponentially. One of the most common methods for image manipulation is image filtering. This technique involves operating on a pixel by pixel basis to transform one image to another. There are a wide variety of image filters, some very simple and straight forward while others are more complex.

One particularly popular and widely used image filter is called the median filter. This filter's primary purpose is to denoise and soften images. Specifically salt and pepper noise which looks like dark or corrupted pixels on the image. The way this filter operates on a pixel scale, is by looking at the surrounding neighboring pixels, specified by the size of the window desired, in an image, storing them in a list, and selecting the median value from that list. That selected median value is now the new pixel value for the pixel that was operated on. This is a fairly simple algorithm but there is an edge case. Border pixels that don't have enough neighboring pixels to satisfy the window size requirement are generally cropped out of the image, thus making the image smaller slightly in the process.

Median Filter Example

Median Filter Effects

Graphics Processing Units (GPUs) are specialized processing units made specifically are operating on images and graphics. Dedicated chips for graphics have been around since the 1970s starting with the first graphical user interfaces and video games but have become increasingly useful with the need for high performance computing in data science, machine learning, and even in digital currency. GPUs allow for doing billions of operating in parallel while also offering an advanced memory hierarchy to allow for varying level of memory I/O speeds and accessibility. It's clear that GPUs are the perfect hardware for performing image filtering. Even with high definition and pixel dense images, the GPU has the resources for one GPU thread per pixel. This means that for the majority of image filters, the median filter included, the operating can be done in near constant time.

There are plenty of methods, historically, to program on the GPU but one very common method is using the software development kit (sdk) released by GPU manufacturer Nvidia called CUDA. CUDA is a compiler and set of C/C++ libraries for programming directly on the GPU and accessing the resources it has available.

The feature particularly relevant to image filter is CUDA Streams. CUDA streams allow for an asynchronous programming design. Because the majority of the work done with images involves more than one image, being able to process them asynchronously massively increases the speed of the program. Rather than waiting for another image to finish before processing, the system is now able to process images from a queue whenever resources are available. This allows potentially multiple GPUs to be added to the system to increase the amount of resources available.

This blog post is formatted as a tutorial so everything is explicitly clear how it works and why it's there. By the end of the tutorial, if you follow along, you should have a working median filter that works on NVIDIA GPUS. Obviously, if you don't have an NVIDIA GPU or the compilers needed to compile CUDA code, then you won't be able to run this.

Completed code can be found here: https://github.com/jonaylor89/Median-Filter-CUDA

Project Setup

To get started, the first thing any C++ and CUDA project needs is imports and a main function. In this case, I've included an argument check in the main function since the program expects a directory of images as an argument.

#include <iostream>
#include <string>
#include <stdio.h>
#include <stdlib.h>
#include <dirent.h>
#include <unistd.h>
#include <sys/types.h>
#include <sys/stat.h>

// CUDA runtime library
#include <cuda_runtime.h>    

// Read/Write images to fs
#include <opencv2/opencv.hpp>
#include <opencv2/core.hpp>

using namespace cv;
using namespace std;

int main(int argc, char **argv) 
{
        if (argc < 2) 
    {
        cerr << "Usage: ./main inputDirectory" << endl;
        exit(1);
    }

        **//** Define Variables
    string inputDir  = string(argv[1]);
    string outputDir = string("motified") + inputDir;

        // **TODO**
        // Allocate memory
        // Read in images
        // Copy images to the GPU
        // Run Kernels (Greyscale + Median Filter)
        // Copy back to the CPU
        // Write filtered images to disk
        // Free memory 

        return 0;
}

Along with the main function, this program is going to need two kernels (functions that run on the GPU): one to convert images from RGBA to greyscale, and one to apply the median filter on the images. Having the greyscale kernel means we only have to deal with black and white images which reduces the complexity of things (although makes it a little more boring). To apply the median filter on colored images, rather than having a greyscale kernel, you would just run the median filter kernel on all color channels for the image (i.e. red, green, blue, alpha).

...

// Convert RGBA -> Greyscale : GPU functions are always void
__global__ void rgbaToGreyscaleGPU(
    uchar4 *rgbaImage, // pointer to rgba images
    uchar *greyImage,  // pointer to where the black/white image will be stored
    int rows,          // number of rows of pixels in the image
    int cols           // number of columns of pixels in the image
)      
{

}

// Apple Median Filter : GPU functions are always void
__global__ void medianFilterGPU(
    uchar4 *rgbaImage, // pointer to rgba images
    uchar *greyImage,  // pointer to where the black/white image will be stored
    int rows,          // number of rows of pixels in the image
    int cols           // number of columns of pixels in the image
)
{

}

int main(int argc, char **argv) { ... }

The last two functions needed to get everything set up are a function to read a directory to get all of the files inside, as well as a function to read an image file and, with the help of OpenCV, extract the pixel values into a matrix. These are fairly trivial steps so I won't go into detail with them, but linux syscalls are used to get the files in a particular directory and OpenCV is used to get a 3D of pixel values for an image (1D is rows, 2D is columns, 3D is color channel).

...

int readImage(
    string filename, // input filename
    Mat* inputImage, // matrix of rgba pixel values
    Mat* imageGrey   // matrix of the same size that'll be used later
)
{
        // Define Variables
    Mat image;
    Mat imageRGBA;
    Mat outputImage;

        // Use OpenCV to read the image file
    image = imread(filename.c_str(), IMREAD_COLOR);
    if (image.empty())
    {
        cerr << "[ERROR] Couldn't open file: " << filename << endl;
                return 1;
    }

    cvtColor(image, imageRGBA, COLOR_BGR2RGBA);

        // Allocate memory for identical greyscale image matrix
        // This'll help when it's time to copy things over from the GPU
    outputImage.create(image.rows, image.cols, CV_8UC1);

        // Return the image matrix by assigning it to the pointer arguments
    *inputImage = imageRGBA;
    *imageGrey = outputImage;

    return 0;
}

// Simple read_directory function you can find on stackoverflow
// ONLY WORKS ON LINUX
void read_directory(string name, vector<string> *v)
{
    DIR* dirp = opendir(name.c_str());
    struct dirent * dp;
    while ((dp = readdir(dirp)) != NULL) {
        v->push_back(dp->d_name);
    }
    closedir(dirp);
}

int main(int argc, char **argv) { ... }

Finally, putting everything together for the setup, read_directory and read_image are called from the main function. The result of all of this is two vectors, one which is a list of image matrices corresponding to the images we want to have filtered, and the other is a list of empty image matrices of the same size, which'll be where we will save the new images after they get filtered on the GPU.

int main(int argc, char **argv) 
{

        // Define Variables
        ...
        vector<string> inputFilenames;
    read_directory(inputDir, &inputFilenames);

    vector<Mat> outputImages;
    vector<Mat> inputImages;

        ...

        // Read in images from the fs
    for (int i = 0; i < inputFilenames.size(); i++)
    {
        Mat imageMat;
                Mat outputMat;
        string curImage = inputFilenames[i]; 
                string filename = inputDir + curImage;

        // Read in image
        int err = readImage(
            filename, 
            &imageMat, 
                    &outputMat
        );

                if (err != 0) { continue; }

        inputImages.push_back(imageMat);
                outputImages.push_back(outputMat);
    }
}

And, naturally, we can't forget a Makefile which you'll notice uses the NVIDIA CUDA compiler.

todo: main
main:
    nvcc main.cu `pkg-config --libs --cflags opencv` -o main
clean:
    rm main && rm -rf motifiedImages

Creating Kernel Functions

Kernel functions in CUDA are easy to spot because they all begin with a "dunder" identifier (e.g. __global__ ). In this case, the identifier is __global__ because it's a kernel that can be called by the CPU but there are others in the CUDA developer docs that change the behavior and scope of the kernels.

GreyScale

Kernel functions operate on one pixel value at a time. This is how they're so efficient - every pixel on every image can be operated on at the same time in parallel. The question becomes, how does the kernel know which pixel it's operating on? Luckily, CUDA has variables created under the hood that the kernel can use to get the context. The variables in the case that'll be used are blockIdx , blockDim , and threadIdx represent the ID for the current block, the dimensions for each block, and the ID for the current thread. To know more about what those variables are, it might be useful to read through the CUDA developer docs. The important thing to know is that using these variables, you can get the x and y value for the pixel the kernel should work on.

__global__ void rgbaToGreyscaleGPU(
    uchar4 *rgbaImage, 
    uchar *greyImage, 
    int rows, 
    int cols
)
{

        // Formula to get the X and Y value from the context
        int x = blockIdx.x * blockDim.x + threadIdx.x;
    int y = blockIdx.y * blockDim.y + threadIdx.y;

    if (x > cols || y > rows)
    {
        return;
    }

    return;
}

The last thing to add is simply the formula for getting the greyscale value from the RGBA.

🥇 An important note: the rgbaImage array, and the greyImage are both 1D arrays. That's why the formula for getting the x and y are weird and why indexing looks like uchar4 rgba = rgbaImage[y * cols + x];.

__global__ void rgbaTogreyscaleGPU(
    uchar4 *rgbaImage, 
    uchar *greyImage, 
    int rows, 
    int cols
)
{
        ...

        uchar4 rgba = rgbaImage[y * cols + x];
    unsigned char greyValue =  (0.299f * rgba.x) + (0.587f * rgba.y) + (0.114f * rgba.z);
    greyImage[y * cols + x] = greyValue;

        return;

}

Median Filter

The kernel for the median filter started exactly the same as for the greyscale kernel.

__global__ void medianFilterGPU(
    unsigned char* greyImageData, 
  unsigned char *filteredImage, 
  int rows, 
  int cols
)
{

        // Formula to get the X and Y value from the context
        int x = blockIdx.x * blockDim.x + threadIdx.x;
    int y = blockIdx.y * blockDim.y + threadIdx.y;

    if (x > cols || y > rows)
    {
        return;
    }

    return;
}

From there, the next step is to get the adjacent pixel values since they will be used to determine the new, filtered pixel value.

Median Filter Graphic

__global__ void medianFilterGPU(
    unsigned char* greyImageData, 
  unsigned char *filteredImage, 
  int rows, 
  int cols
)
{

        ...

        // 3x3 window (optionally you could use a bigger window)
        int windowSize = 3;

        // Filter out the diagonals (OPTIONAL)
    int filter[9] {
        0, 1, 0,
        1, 1, 1,
        0, 1, 0
    };

        // Allocate memory to save the adjecent pixel values
    unsigned char pixelValues[9] = {0, 0, 0, 0, 0, 0, 0, 0, 0};

        // Bounds checking
    if (
        x > cols - windowSize + 1 ||
        y > rows - windowSize + 1 ||
        x < windowSize - 1 ||
        y < windowSize - 1
    )
    {
        return;
    }

        // Loop through vertical and horizonally to get adjecent values
    for (int hh = 0; hh < windowSize; hh++) 
    {
        for (int ww = 0; ww < windowSize; ww++) 
        {
            if (filter[hh * windowSize + ww] == 1)
            {
                int idx = (y + hh - 1) * cols + (x + ww - 1);
                pixelValues[hh * windowSize + ww] = greyImageData[idx];
            }
        }
    }

        return;

}

With a list of all pixel values in the window, the last thing to do is determine the median value and make that the new pixel value.

🥇 You have to do this all manually because you can't use std lib inside the Kernel functions ☹️

__global__ void medianFilterGPU(
    unsigned char* greyImageData, 
  unsigned char *filteredImage, 
  int rows, 
  int cols
)
{

        ...

        // Sort the pixel values
    for (int i = 0; i < (windowSize * windowSize); i++) {
        for (int j = i + 1; j < (windowSize * windowSize); j++) {
            if (pixelValues[i] > pixelValues[j]) {
                //Swap the variables.
                char tmp = pixelValues[i];
                pixelValues[i] = pixelValues[j];
                pixelValues[j] = tmp;
            }
        }
    }

        // Get the middle of the list for the median
    unsigned char filteredValue = pixelValues[(windowSize * windowSize) / 2];
    filteredImage[y * cols + x] = filteredValue;

        return;
}

Async and CUDA streams

Now to tie it all together and make it work.

Programming anything on the GPU is a 5 step process:

Allocate Memory
Copy to the GPU
Run Kernels
Copy back to the CPU
Free memory

Allocating Memory

Beginning with Allocating memory correctly, we first have to go back and define a few new variables and structs. At the top of the file I create my own FakeMat struct which help clean things up later on. In the main function, we have to create and initialize the array of CUDA streams. we also need to define the pointers which'll store the location of the images after they're copied to the GPU (prefixed with d_).

#define THREAD_DIM 256
#define NUM_STREAMS 32
#define MAX_IMAGE_SIZE (1920 * 1080)

typedef struct FakeMat_ {
    unsigned char *Ptr;
    int rows;
    int cols;
} FakeMat;

...

int main(int argc, char **argv) 
{

        // Define Variables
        ...

        cudaStream_t streams[NUM_STREAMS];
        for (int i = 0; i < NUM_STREAMS; i++) { cudaStreamCreate(&streams[i]); }

    uchar4 *d_rgbaImage;
    unsigned char *d_greyImage;
    unsigned char *d_filteredImage;

        ...

}

To actually allocate memory, the functions cudaMalloc() and cudaMallocHost() have to be used as opposed to the normal malloc() because memory is being allocated on both the local machine (i.e. Host) and on the GPU.

int main(int argc, char **argv) 
{       

        ...

        // Allocate Memory
    cudaMalloc(&d_rgbaImage, sizeof(uchar4) * MAX_IMAGE_SIZE * NUM_STREAMS);
    cudaMalloc(&d_greyImage, sizeof(unsigned char) * MAX_IMAGE_SIZE * NUM_STREAMS);
    cudaMalloc(&d_filteredImage, sizeof(unsigned char) * MAX_IMAGE_SIZE * NUM_STREAMS);

    // Read in images from the fs
    for (int i = 0; i < inputFilenames.size(); i++) { ... }

    FakeMat *outputImagesArray;
    cudaMallocHost(&outputImagesArray, sizeof(FakeMat) * inputImages.size());

}

Copying to the GPU

Once memory has been allocated on the GPU, the image data collected can be copied over. The functions CUDA provides for copying and settings data on the GPU are cudaMemsetAsync() and cudaMemcpyAsync() . You'll notice in the code below, to make everything more efficient, all of the images are being squished to 1D arrays and those arrays are concatenated together. This means that we're basically just copying one huge 1D array full of all of our image data over to the GPU and although this seems like a lot of boilerplate, under the hood it's making things much more efficient.

🥇 btw - CUDA also provides synchronous function cudaMemset() and cudaMemcpy() but because we're operating on many images at once using streams, we use the async versions.

int main(int argc, char **argv)
{

        ...

        // Loop through all the images
        for (int i = 0; i < inputImages.size(); i++)
        {
                // Get the current stream
            const int curStream = i % NUM_STREAMS; 
            Mat curImageMat = inputImages[i];

            int rows = curImageMat.rows;
            int cols = curImageMat.cols;
            int size = rows * cols;

                if (size >= MAX_IMAGE_SIZE) { continue; }

                // Clear any old data from the GPU
            cudaMemsetAsync(d_rgbaImage + MAX_IMAGE_SIZE * curStream, 0, sizeof(uchar4) * MAX_IMAGE_SIZE, streams[curStream]);
            cudaMemsetAsync(d_greyImage + MAX_IMAGE_SIZE * curStream, 0, sizeof(unsigned char) * MAX_IMAGE_SIZE, streams[curStream]);
            cudaMemsetAsync(d_filteredImage + MAX_IMAGE_SIZE * curStream, 0, sizeof(unsigned char) * MAX_IMAGE_SIZE, streams[curStream]);

                // Special CUDA functions to set the gridSize and blockSize 
                // Check the developer docs for more info on what they do and why they're important
                // In short: GPUs are very complex
            dim3 gridSize (ceil(cols / (float)THREAD_DIM), ceil(rows / (float)THREAD_DIM));
            dim3 blockSize (THREAD_DIM, THREAD_DIM);

                // Get the pointer for the image data
                uchar4 *curImagePtr = (uchar4 *)curImageMat.ptr<unsigned char>(0);
                if (curImagePtr == NULL) { continue; }

            // Copy data to GPU
            cudaMemcpyAsync(
                d_rgbaImage + MAX_IMAGE_SIZE * curStream, // HUGE 1D array of all the pixels
                curImagePtr,             // Pointer to the image
                sizeof(uchar4) * size,   // Size of the image (total number of pixels
                cudaMemcpyHostToDevice,  // Enum value provided by CUDA
                streams[curStream]       // Which stream is in charge of this image
            );
        }
}

Running Kernels

To run the kernels, you use almost the exact same syntax as calling a function in C but with extra <<<>>> which is where you short context about the GPU (grid size, block size, and the current stream, check out the CUDA docs for more info). Note too that because we copied the data over to the GPU as a big 1D array, the arguments that get past have to accommodate by doing some pointer math and offsetting the pointer value.

for (int i = 0; i < inputImages.size(); i++) 
{

        ...

        // Run kernel(s)
        rgbaToGreyscaleGPU<<< gridSize, blockSize, 0, streams[curStream] >>>(
            d_rgbaImage + (MAX_IMAGE_SIZE * curStream), 
            d_greyImage + (MAX_IMAGE_SIZE * curStream), 
            rows, 
            cols
        );

        medianFilterGPU<<< gridSize, blockSize, 0, streams[curStream] >>>(
            d_greyImage + (MAX_IMAGE_SIZE * curStream), 
            d_filteredImage + (MAX_IMAGE_SIZE * curStream), 
            rows, 
            cols
        );

}

Copying back to the CPU

After the kernel is run, the new, filtered images have to be brought back to the CPU. This is done almost identically to copying the images to the GPU. (i.e. with cudaMemcpyAsync())

for (int i = 0; i < inputImages.size(); i++) 
{

...

        // Create a pointer to which the output image is saved   
        unsigned char *outputImagePtr = outputImages[i].ptr<unsigned char>(0);

        // Copy results to CPU
    cudaMemcpyAsync(
        outputImagePtr,
        d_filteredImage + MAX_IMAGE_SIZE * curStream, 
        sizeof(unsigned char) * size, 
        cudaMemcpyDeviceToHost,  // Enum value provided by CUDA
        streams[curStream]
    );

        // Store the image in an array
        outputImagesArray[i] = (FakeMat){ outputImagePtr, rows, cols };       
}

Freeing Memory and Writing Results

As you could probably guess from the word Async being on all of the CUDA functions, those operations aren't synchronized and destroyed until you explicitly tell them to, therefore, right after the for loop for handling each image, the streams have to be synchronized.

int main(int argc, char **argv) 
{

        ...

        for (int i = 0; i < inputImages.size(); i++) { ... }

        // sync and destroy streams
    for (int i = 0; i < NUM_STREAMS; i++)
    {
        cudaStreamSynchronize(streams[i]);
        cudaStreamDestroy(streams[i]);
    }

}

With the streams synced up, we can be sure that everything is finished and you're safe to write the new, filtered images to disk. Once again, OpenCV helps with this by making things fairly straight forward. The 1D images are converted to a type that OpenCV understands (i.e. Mat) then written to disk with cv::imwrite().

...

void writeImage(string dirname, string filename, string prefix, Mat outputImage)
{
    string outFile = dirname + string("/") + prefix + filename;

    cv::imwrite(outFile.c_str(), outputImage);
}

int main(int argc, char **argv)
{

        ...

        // Create a directory for the new images
        struct stat st = {0};
    if (stat(outputDir.c_str(), &st) == -1) {
        mkdir(outputDir.c_str(), 0700);
    }

    // Write modified images to the fs
    for (int i = 0; i < inputImages.size(); i++)
    {

                if (outputImagesArray[i].Ptr == NULL) { continue; }

                // Convert images to OpenCV Mat type
                Mat outputImageMat = Mat(
                        outputImagesArray[i].rows, 
                        outputImagesArray[i].cols, 
                        CV_8UC1, 
                        outputImagesArray[i].Ptr
                );

        // Write Image
        writeImage(outputDir, to_string(i) + string(".jpg"), "modified_", outputImageMat);
    }
}

Last and definitely not least, any allocated memory on both the host and GPU must be freed.

int main(int argc, char **argv)
{

        ...

        // Free Memory
    cudaFreeHost(&outputImages);
    cudaFree(&d_rgbaImage);
    cudaFree(&d_greyImage);

    return 0;
}

Conclusion

We finally have a working program 😤 and it really only took around ~300 lines of code. With those 300 lines though it's now possible to run image filters on A LOT of images VERY FAST. How fast depends on your GPU and the number of images but it should be able to do many images a second.

In the future, it would be great to implement something involving a reduction set such as histogram segmentation which would require reducing the image to a histogram before calculating pixel values. This great complicates the parallelization process and might cause some difficulty on the GPU side of things. Along with that, another field of study to investigate further with this is real time image processing. The speeds achieved with the GPU accelerated version were plenty fast to be considered real time but perhaps doing something a bit more complicated to parallelize as well as moving to more resource scare hardware like internet of things devices or drones might stress the system to its limits.

Completed code can be found here: https://github.com/jonaylor89/Median-Filter-CUDA

DEV Community: Johannes Naylor

Discover Hidden Secrets in Git Repos with Rust

Table of Contents

Intro

Git 101

Git in Rust

Secrets

Projects Overview

Setup

Scanning Git Blobs

Finding Secrets with Regex

Conclusion

Further Reading

My Switch Away from Python

Table Of Contents

From Beginner to Professional

Typing

GUIs

Client Side Deployment

Server Side Deployment

Rust

Conclusion

Creating a Twitter Bot in Go

Table of Contents

Intro

Getting Twitter Credentials

Loading Credentials

Adding a Main

Putting It All Together

Deploying on Heroku

How to Manipulate Images on a GPU

Table of Contents

Intro

Project Setup

Creating Kernel Functions

GreyScale

Median Filter

Async and CUDA streams

Allocating Memory

Copying to the GPU

Running Kernels

Copying back to the CPU

Freeing Memory and Writing Results

Conclusion

Further Reading & Links