DEV Community: Donald Feury

How to create the content hash for a Dropbox file upload in Elixir

Donald Feury — Sat, 05 Aug 2023 22:43:01 +0000

Dropbox has a specific way they want you to calculate the content hash for a file you are uploading. You can find the instructions here but to summarize:

Split the file into blocks of 4 MB (4,194,304 or 4 * 1024 * 1024 bytes). The last block (if any) may be smaller than 4 MB.
Compute the hash of each block using SHA-256.
Concatenate the hash of all blocks in the binary format to form a single binary string.
Compute the hash of the concatenated string using SHA-256. Output the resulting hash in hexadecimal format.

We can convert each of these steps into parts of a pipeline!

First we need to start reading in a file, split into chunks of the specified size. File.stream! is a good way of achieving this.

chunk_size = 4 * 1024 * 1024

File.stream!("my_file.txt", [], chunk_size)

Next, we need to create a new hash from each chunk. Iterating over the stream chunks with Enum.map and passing those chunks into the Erlang module :crypto.hash will get us our chunk hashes.

chunk_size = 4 * 1024 * 1024

File.stream!("my_file.txt", [], chunk_size)
|> Enum.map(&:crypto.hash(:sha256, &1))

After that, we'll combine all individual hashes into one big chunky string of hashes. Enum.join will let us that list of strings and slap them together

chunk_size = 4 * 1024 * 1024

File.stream!("my_file.txt", [], chunk_size)
|> Enum.map(&:crypto.hash(:sha256, &1))
|> Enum.join()

Getting warmer, time to take that big boy and get one final hash using :crypto.hash again. We'll have to wrap it inside an anonymous function since we need to pass the string in as the second argument.

chunk_size = 4 * 1024 * 1024

File.stream!("my_file.txt", [], chunk_size)
|> Enum.map(&:crypto.hash(:sha256, &1))
|> Enum.join()
|> (&:crypto.hash(:sha256, &1)).()

Home stretch! Lastily, base 16 encode the final hash using Base.encode16. Don't forget to lowercase it by passing in case: :lower.

chunk_size = 4 * 1024 * 1024

File.stream!("my_file.txt", [], chunk_size)
|> Enum.map(&:crypto.hash(:sha256, &1))
|> Enum.join()
|> (&:crypto.hash(:sha256, &1)).()
|> Base.encode16(case: :lower)

There ya go! You should now be able to create the content hash for a file that you intend to upload to Dropbox.

How to perform a JOIN in MongoDB

Donald Feury — Mon, 31 Jan 2022 00:45:32 +0000

For a full overview of MongoDB and all my posts on it, check out my overview.

If you have relational data in your MongoDB instance, you can perform an operation similar to a JOIN typically done in SQL queries.

Consider the following data set:

db.podcasts.insertMany([
    {id: 1, name: "Off The Clock", category: "Technology", rss: "https://anchor.fm/s/76aafa5c/podcast/rss"},
    {id: 2, name: "Tech Over Tea", category: "Technology", rss: "https://anchor.fm/s/149fd51c/podcast/rss"}
]);

db.episodes.insertMany([
    {podcast_id: 1, title: "Resume Tips", published_on: "2022-01-11"},
    {podcast_id: 2, title: "#75 Welcome Our Hacker Neko Waifu | Cyan Nyan", published_on: "2021-08-04"},
    {podcast_id: 2, title: "Square Enix Refuses To Take My Money | Solo", published_on: "2022-01-26"},
    {podcast_id: 1, title: "Find the Right Job", published_on: "2022-01-25"}
]);

If you want to get every podcast with all of its associated episodes, you can accomplish this using the $lookup aggregation stage.

The aggregation would look something like this:

db.podcasts.aggregate([
    { $lookup: {
        from: "episodes", // Name of the other collection to "join" from
        localField: "id", // Name of the field your current documents contain to compare with
        foreignField: "podcast_id", // Name of field to compare to in the "from" collection's documents
        as: "episodes" // What to call the field that contains the array of sub documents that matched
    }}
]);

Any documents in the episodes collection that had a podcast_id that matched one of the podcasts documents would have been added to an array of sub-documents called episodes.

The result would look like this:

[
    {
        id: 1,
        name: "Off The Clock",
        category: "Technology",
        rss: "https://anchor.fm/s/76aafa5c/podcast/rss",
        episodes: [
            {podcast_id: 1, title: "Resume Tips", published_on: "2022-01-11"},
            {podcast_id: 1, title: "Find the Right Job", published_on: "2022-01-25"}
        ]
    },
    {
        id: 2,
        name: "Tech Over Tea",
        category: "Technology",
        rss: "https://anchor.fm/s/149fd51c/podcast/rss",
        episodes: [
            {podcast_id: 2, title: "#75 Welcome Our Hacker Neko Waifu | Cyan Nyan", published_on: "2021-08-04"},
            {podcast_id: 2, title: "Square Enix Refuses To Take My Money | Solo", published_on: "2022-01-26"},
        ]
    }
]

This would be the equivalent of the SQL query:

SELECT
    *
FROM
    podcasts
LEFT JOIN episodes ON episodes.podcast_id = podcasts.id;

I specified LEFT JOIN in the SQL example because if no documents match the $lookup aggregation stage, the podcasts would still be returned with the episodes field being an empty array.

Did you find this information useful? If so, consider heading over to my donation page and drop me some support.

Want to ask a question or just chat? Contact me here

When to use MongoDB aggregations instead of queries

Donald Feury — Mon, 24 Jan 2022 01:54:50 +0000

When it comes to reading data out of your MongoDB instance you have two options, a query using the standard query language or an aggregation pipeline. While you can often use either for many use cases, there are differences in how they are used and what use cases they are a good fit for.

For use with the following examples you can insert the following documents into your MongoDB instance:

💡 If you haven't inserted multiple documents into MongoDB before, you can read about that here.

db.podcasts.insertMany([
    {id: 1, name: "Podcast 1", category: "Business", rss: "https://mypodcast1.com/podcast/rss"},
    {id: 2, name: "Podcast 2", category: "Business", rss: "https://podcast2.net/rss"},
    {id: 3, name: "Podcast 3", category: "Health & Wellness", rss: "https://podcast3.xyz/feed/rss"},
    {id: 4, name: "Podcast 4", category: "Technology", rss: "https://podcast4.com/rss"}
]);

db.episodes.insertMany([
    {podcast_id: 3, title: "Anabolic Recipes", published_on: "2020-01-02"},
    {podcast_id: 4, title: "Getting Going with Go", published_on: "2021-02-24"},
    {podcast_id: 1, title: "Talk About Money Early", published_on: "2022-01-01"},
    {podcast_id: 1, title: "Charge More by Saying No", publshed_on: "2021-12-01"},
    {podcast_id: 2, title: "Candidate Interview Strategies", published_on: "2020-05-01"},
    {podcast_id: 4, title: "The Magic of Pattern Matching", published_on: "2022-01-01"}
]);

Ease of use

MongoDB's query language and its related methods are meant to be straightforward to use. Compared to aggregations they are easier to read and are less prone to error.

Take the example of wanting to filter out and return only certain documents. Using the query language is as simple as passing in an object describing how to filter the objects to the find method.

💡 You can read more about using the find method here.

db.podcasts.find({category: "Business"});

To do the same thing with an aggregation would require the use of $match aggregation stage.

db.podcasts.aggregate([
 { $match: {
    category: "Business"
 }}
]);

Use Case

While the strength of the standard query language is its simplicity, that also limits what problems it can solve. Aggregations are essentially the opposite, harder to use but much more powerful, allowing them to accomplish certain use cases that are unable to be done by simple queries.

The standard query language is a good solution for:

Returning/affecting only certain documents that match provided filter
When reading data back out, only including certain fields

Anything that falls outside the scope of those scenarios is either better done or required to be done by aggregations.

One such example and the earliest one I personally needed to use aggregations was when wanting to perform an operation similar to a join in a SQL query.

This simply cannot be done using the available query methods and language but CAN be accomplished using the $lookup aggregation stage.

What if we wanted to get all of our podcasts including each episode associated with that podcast:

db.podcasts.aggregate([
    { $lookup: {
        from: "episodes",
        localField: "id",
        foreignField: "podcast_id",
        as: "episodes"
    }}
]);

How to do conditional execution in Bash

Donald Feury — Sat, 23 Oct 2021 23:15:14 +0000

In Bash scripts, any command following a && will only execute if the previous conditional or command succeeded (returned 0)

The following script will only print out the variable if the conditional succeeds:

[[-z "$1"]] && echo "$1"

On the other hand, any command following a || will only execute if the previous conditional or command failed (returned a non zero)

The following script will only print out the variable if the conditional fails:

[[-z "$1"]] || echo "$1"

Posts about Bash

Did you find this information useful? If so, consider heading over to my donation page and drop me some support.

Want to ask a question or just chat? Contact me here

How to check if a string is empty in Bash

Donald Feury — Fri, 22 Oct 2021 16:36:15 +0000

To check if a string is empty in a Bash script, use the -z conditional which returns 0 (true) if the length of the string is 0 and 1 (false) if it is greater than 0.



myVar="hello"

if [[-z "$myVar"]]; then
    echo "myVar is empty"
else
    echo "myVar is not empty"
fi

Alternatively, you can check if a bash string is NOT empty using -n and take the opposite using !.



myVar=""

if [[! -n "$myVar"]]; then

    echo "myVar is empty"

else

    echo "myVar is not empty"

fi

All my posts on Bash

Did you find this information useful? If so, consider heading over to my donation page and drop me some support.

Want to ask a question or just chat? Contact me here

Use the filter positional operator in MongoDB to update specific array elements

Donald Feury — Wed, 20 Oct 2021 16:00:00 +0000

For a full overview of MongoDB and all my posts on it, check out my overview.

MongoDB provides a filter positional operator $[<name_of_filter>] to only update some elements in an array. If you want to update all the elements in the array, use the all positional operator in MongoDB to update all elements in an array.

You must define the filters to use in a property called arrayFilters.

With the following data inserted into a collection called games:

db.games.insertMany(
    {
        name: "Genshin Impact",
        reviewScores: [8, 6, 9, 5]
    },
    {
        name: "Factorio",
        reviewScores: [7, 7, 10, 8]
    },
    {
        name: "Bloodborne",
        reviewScores: [9, 8, 9, 9]
    }
)

For the game Genshin Impact, increase all the reviews by 1 that are less than 9.

db.games.updateMany(
    { name: "Genshin Impact" },
    { $inc: { "reviewScores.$[reviews]": 1 } },
    { arrayFilters: [
        { "reviews": { $lt: 9 } }
    ]}
)

Notice in the $inc update operator, we reference the array filter defined called reviews to tell MongoDB which elements to update.

Use the find method to read data back out of MongoDB to view the results:

{
    name: "Genshin Impact",
    reviewScores: [9, 7, 9, 6]
},
{
    name: "Factorio",
    reviewScores: [7, 7, 10, 8]
},
{
    name: "Bloodborne",
    reviewScores: [9, 8, 9, 9]
}

Did you find this information useful? If so, consider heading over to my donation page and drop me some support.

Want to ask a question or just chat? Contact me here

Use the all positional operator in MongoDB to update all elements in an array

Donald Feury — Tue, 19 Oct 2021 13:22:00 +0000

For a full overview of MongoDB and all my posts on it, check out my overview.

MongoDB provides a special update operator $[] to apply an update operation to all elements in an array belonging to matching documents.

With the following data inserted into a collection called games:

db.games.insertMany(
    {
        name: "Genshin Impact",
        reviewScores: [8, 6, 9, 5]
    },
    {
        name: "Factorio",
        reviewScores: [7, 7, 10, 8]
    },
    {
        name: "Bloodborne",
        reviewScores: [9, 8, 9, 9]
    }
)

For every game that does NOT have a score of 10 in its reviews, increase all the review scores by 1.

db.games.updateMany(
    { reviewScores: { $ne: 10 } },
    { $inc: { "reviewScores.$[]": 1 } }
)

Use the find method to read all the data back out to view the results:

{
    name: "Genshin Impact",
    reviewScores: [9, 7, 10, 6]
},
{
    name: "Factorio",
    reviewScores: [7, 7, 10, 8]
},
{
    name: "Bloodborne",
    reviewScores: [10, 9, 10, 10]
}

All posts on MongoDB

Did you find this information useful? If so, consider heading over to my donation page and drop me some support.

Want to ask a question or just chat? Contact me here

This is how to do upserting in MongoDB

Donald Feury — Sun, 22 Aug 2021 18:25:14 +0000

For a full overview of MongoDB and all my posts on it, check out my overview.

Upserting is a database concept that combines inserting and updating that MongoDB supports. To perform an upsert during an update operation, you must pass an additional argument to the update, updateOne, or updateMany methods.

With the given data inserted in a collection called users:

db.users.insertMany([
    {
        _id: 1,
        name: "John Doe",
        email: "doe@doe.com",
        admin: true
    },
    {
        _id: 2,
        name: "Jane Doe",
        email: "jane@doe.com",
        admin: true
    },
    {
        _id: 3,
        name: "Billy Bob",
        email: "billy@bob.com",
        admin: false
    },
    {
        _id: 4
        name: "Steve Stevenson",
        email: "steve@test.com",
        admin: true
    },
])

If the following command is used:

db.users.updateOne({_id: 5}, {$set: {admin: true}})

It won't do anything as there is no document matching the query of _id = 5.

If upserting is enabled:

db.users.updateOne(
    {_id: 5},
    { $set: { admin: true } },
    { upsert: true }
)

Since there is no document matching the query, a new document is inserting using the given _id and the result of the update operators.

Use the find method to read data back out of MongoDB to check the collection:

db.users.find()

Will yield:

{
    _id: 1,
    name: "John Doe",
    email: "doe@doe.com",
    admin: true
},
{
    _id: 2,
    name: "Jane Doe",
    email: "jane@doe.com",
    admin: true
},
{
    _id: 3,
    name: "Billy Bob",
    email: "billy@bob.com",
    admin: false
},
{
    _id: 4
    name: "Steve Stevenson",
    email: "steve@test.com",
    admin: true
},
{
    _id: 5
    admin: true
}

How to replicate the output of the FullScreen template from StreamLadder

Donald Feury — Tue, 17 Aug 2021 23:37:46 +0000

StreamLadder is a nifty little tool for creating vertical versions of your Twitch clips for platforms like Tiktok and YouTube Shorts. However, I like tinkering with FFmpeg so I replicated the output with a simple command.

If we look at the Fullscreen template, all it's essentially doing is cropping the original clip down into a 9:16 ratio.

To do the same with FFmpeg we can just do the following:

#!/usr/bin/env sh

# the source video (in this case, a twitch clip)
input=$1

# where you want to save the processed version
output=$2

ffmpeg \
    -i "$input" \
    -vf "crop=w=in_h*9/16:h=in_h,scale=1080x1920" \
    -vcodec libx264 \
    -crf 23 \
    -preset veryfast \
    -c:a copy \
    -s 1080x1920 \
    "$output"

The main part of this command doing all the work is the video filter, denoted with -vf. We do w=in_h*9/16 to set the width of the video to the 9:16 ratio compared to what the height of the video is while we keep the height the same with h=in_h. We finally scale the video up to 1080x1920 with the scale filter.

If the video quality looks a bit pixelated from scaling up too much, anywhere you see 1080x1920 replace with a smaller equivalent like 720x1280

How to do bulk updates in MongoDB

Donald Feury — Mon, 16 Aug 2021 18:59:17 +0000

For a full overview of MongoDB and all my posts on it, check out my overview.

MongoDB provides several ways to update specifically one document and bulk updating is done in almost the same way.

Using the following data:

db.podcasts.insertMany([
    {
        "name": "Tech Over Tea",
        "episodeName": "#75 Welcome Our Hacker Neko Waifu | Cyan Nyan",
        "dateAired": ISODate("2021-08-02"),
        "listenedTo": true,
    },
    {
        "name": "Tech Over Tea",
        "episodeName": "Neckbeards Anonymous - Tech Over Tea #20 - feat Donald Feury",
        "dateAired": ISODate("2020-07-13"),
        "listenedTo": true
    },
    {
        "name": "Tech Over Tea",
        "episodeName": "#34 The Return Of The Clones - feat Bryan Jenks",
        "dateAired": ISODate("2020-10-19"),
        "listenedTo": false
    },
    {
        "name": "Cinemassacre Podcast",
        "episodeName": "AVGN Fan Q&A, Starting a Band, and the Last Year - Cinemassacre Podcast",
        "dateAired": ISODate("2021-08-10"),
        "listenedTo": true
    }
])

Let's update all the "Tech Over Tea" podcasts entry to be marked as listened to using update. In order to update multiple documents using update, we have to pass in an optional argument to tell MongoDB to do that.

db.podcasts.update(
    {name: "Tech Over Tea"},
    {$set: { listenedTo: true} },
    {multi: true}
)

We can achieve the same result using updateMany.

db.podcasts.updateMany(
    {name: "Tech Over Tea"},
    {$set: { listenedTo: true} }
)

How to drop a collection from MongoDB

Donald Feury — Sun, 15 Aug 2021 14:20:51 +0000

For a full overview of MongoDB and all my posts on it, check out my overview.

WARNING! This is a dangerous command that will result in the loss of data

If you want to completely drop a collection, which will result in the loss of all documents in that collection, there is one simple command to do that called drop.

To drop a collection in your MongoDB instance called podcasts:

db.podcasts.drop()

Only do this if you're sure you want to completely reset that collection.

How to replace an existing document in MongoDB

Donald Feury — Sat, 14 Aug 2021 15:41:40 +0000

For a full overview of MongoDB and all my posts on it, check out my overview.

MongoDB provides several ways to update specifically one document that works great when doing partial updates.

If you want to completely replace an existing document with a new one, you can use replaceOne.

First, let's insert some data into a collection called podcasts:

db.podcasts.insertMany([
    {
        "name": "Tech Over Tea",
        "episodeName": "#75 Welcome Our Hacker Neko Waifu | Cyan Nyan",
        "dateAired": ISODate("2021-08-02"),
        "listenedTo": true,
    },
    {
        "name": "Tech Over Tea",
        "episodeName": "Neckbeards Anonymous - Tech Over Tea #20 - feat Donald Feury",
        "dateAired": ISODate("2020-07-13"),
        "listenedTo": true
    },
    {
        "name": "Tech Over Tea",
        "episodeName": "#34 The Return Of The Clones - feat Bryan Jenks",
        "dateAired": ISODate("2020-10-19"),
        "listenedTo": false
    }
])

Let's completely replace the podcast that aired on 2020-07-13 with a new podcast. Unlike update and updateOne, replaceOne does not use update operators.

db.podcasts.replaceOne(
    {dateAired: ISODate("2020-07-13")},
    {
        "name": "Tech Over Tea",
        "episodeName": "#73 Is This A Gaming Podcast Now | Solo",
        "dateAired": ISODate("2021-07-19"),
        "listenedTo": false
    }
)

The arguments are similar to update and updateOne where the first argument is the query to match a document to replace. However, the second argument is a new document that will completely replace the first matched document.

DEV Community: Donald Feury

How to create the content hash for a Dropbox file upload in Elixir

How to perform a JOIN in MongoDB

When to use MongoDB aggregations instead of queries

Ease of use

Use Case

How to do conditional execution in Bash

Related

How to check if a string is empty in Bash

Related

Use the filter positional operator in MongoDB to update specific array elements

Related

Use the all positional operator in MongoDB to update all elements in an array

Related

This is how to do upserting in MongoDB

How to replicate the output of the FullScreen template from StreamLadder

How to do bulk updates in MongoDB

How to drop a collection from MongoDB

How to replace an existing document in MongoDB