DEV Community: Bernd Verst

From Cloud Security Alert to Open Source Bugfix

Bernd Verst — Wed, 08 Apr 2020 17:06:00 +0000

Perhaps you've seen vulnerability reports in your CI/CD pipeline or tools like NPM. Cloud infrastructure has these too and I was surprised to get an alert recently. Naturally, I had to investigate to see where I went wrong... (and of course mitigate the problem in the process). Little did I know about the journey I was embarking on...

The security alert

It all started a few weeks ago when I received an email from a colleague with the following content.

Secure transfer to storage accounts should be enabled

Subscription ID: devrel-berndverst-demo-test  
Resource: /subscriptions/XXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXXX/
    resourcegroups/kubeflowrelease/providers/
    microsoft.storage/storageaccounts/ > fairingXXXXXXXXXXXXXXXXX  
[1-Click mitigation link]

***Note: several similar results have been omitted*

This was my gut reaction:

😱 This cannot be. What did I do wrong? Did I mess something up? 😰

My colleague had used Azure Security Center to generate a report of security issues. Since I had never received such a message before (note I could have set up email alerts in Azure Security Center myself) I was a bit skeptical and first verified the authenticity of the email: Email domain, Sender Police Framework (SPF) check, Domain Keys Identified Mail (DKIM) all indicated the email was legitimate.

The 1-Click mitigation link (which I verified went to the Azure web portal domain) took me directly to the recommendations section in the Azure Security Center for my Azure account and jumped to the relevant entries. The Security Center recommendations documentation also had more information on the alert I had received.

What I did wrong (supposedly)

Azure Blob Storage accounts should be configured to only serve traffic over https.

The 1-step mitigation in Azure Security Center resolved the issue, but I could also have followed these instructions in the docs.

🤔 Why would I ever not want secure blob transfers? This does not sound like me.

Taking a closer look

I noticed that all affected blob storage accounts had programmatically generated names. Furthermore they all resided in my Azure resource group kubeflowrelease and contained the string fairing in the account names.

Days earlier I had ported an end to end tutorial for Kubeflow using the MNIST training set to Azure. Of course in the process I deployed Kubeflow to my Kubernetes cluster and went through the tutorial I wrote.

The Kubeflow project is dedicated to making deployments of machine learning (ML) workflows on Kubernetes simple, portable and scalable.

The culprit: Kubeflow

💡Kubeflow somehow created the storage accounts in question

The Kubeflow Fairing helper library created the storage account without forcing secure transport.

How to correctly create secure storage accounts

Looking at the relevant documentation I discovered something very interesting:

By default, the Secure transfer required property is enabled when you create a storage account in Azure portal. However, it is disabled when you create a storage account with SDK.

In other words, the SDKs default to doing that for which the Azure Security Center alerted me, creating insecure storage accounts.

❓Kubeflow Fairing is written in Python. I wonder how to create secure storage accounts with the Azure Python SDK.

The Python SDK sample code at github.com/Azure-Samples/storage-python-manage does not explicitly set the secure option. The sample code uses the insecure default [Edit: it turns out that this depends on the SDK version used]. I made this pull request to fix this.

Use secure https-only access for newly created storage accounts. #11

berndverst posted on Mar 11, 2020

Addresses #10

View on GitHub

Fixing the sample code led to several unrelated issues with the tests, all of which I fixed:

The Azure Python SDK version was not locked and the API output had changed, no longer matching the mocked responses
Travis was running tests for a version of Python that was no longer supported
Travis was not running tests for current versions of Python 3.7 and 3.8.

It took me hours, but the above pull request fixed it all.

What did Kubeflow Fairing do?

Taking a close look we can see that Kubeflow Fairing basically copied from the Azure Python sample code.

I would have done the same. Good thing that is fixed now!

Only use secure transport mode for new Azure Storage Accounts #477

berndverst posted on Mar 11, 2020

What this PR does / why we need it:

Currently Fairing creates new Storage Accounts on Azure (if necessary) that do not use best practices of enabling strict transport security. Azure Security Center will flag these storage accounts.

This PR uses the recommended default (which unfortunately isn't the default in the Python SDK).

References: https://github.com/Azure-Samples/storage-python-manage#create-account https://docs.microsoft.com/en-us/azure/storage/common/storage-require-secure-transfer https://docs.microsoft.com/en-us/python/api/azure-mgmt-storage/azure.mgmt.storage.v2018_07_01.models.storageaccountcreateparameters?view=azure-python

Release note: -->

NONE

This change is

View on GitHub

Success?

- ✅ Kubeflow Fairing now creates secure storage accounts.
- ✅ Anyone finding the Azure Python SDK samples
  for storage account creation will create secure accounts.

What about the API / server-side default?

Since API version 2019-04-01 the secure mode is the default. All current SDKs will be using this API version or a newer one which defaults to the secure setting.

So why the Kubeflow Fairing issue afterall?

Kubeflow Fairing was using the deprecated azure library. Instead azure-mgmt-storage. As such it was consuming an outdated version of the API with the insecure default setting. API and SDK versioning is hard!

The edits to the Azure Python sample documentation weren't strictly necessary. But at least I fixed the tests.

Trim Videos Instantly

Bernd Verst — Fri, 03 Apr 2020 22:04:08 +0000

Is your video too long for upload to Twitter or Instagram? Or do you want to export just a specific clip? Using video editors to trim videos requires reencoding the entire video which takes a long time. Instead I will show you how to instantly trim your videos and even maintain embedded subtitle tracks.

Background

A few days ago I created a post on easily adding subtitles to your videos. Today I wanted to verify that Twitter correctly shows these subtitles, but I discovered that my video was too long. Having played with FFmpeg I suspected that there may be a more efficient way for trimming videos than using a video editor. Be prepared to have your mind blown! 🤯

Requirements

FFmpeg

You will need the open-source FFmpeg tool.

FFmpeg is the leading multimedia framework, able to decode, encode, transcode, mux, demux, stream, filter and play pretty much anything that humans and machines have created.

Please follow the installation instructions here.
On Mac, simply use Homebrew with brew install ffmpeg

Instantly trim videos

The following command reads the source video and copies all streams contained within (video, audio, subtitles) to the output. The various time parameters ensure only a specific part of the original video is written to the output. The video and audio streams are not reencoded. The time required for reencoding would directly depend on the duration of the video, however we are simply copying parts of the video. For this reason the following operation is near instant. A huge time saving over workflows involving video editors.

Adding subtitles to your videos the easy way

Bernd Verst — Wed, 01 Apr 2020 00:00:00 +0000

For accessibility reasons all video content you create should be captioned but adding subtitles can be a daunting task. In this article we cover how to quickly and easily add subtitles to your existing videos without any video editing software.

Requirements

Inputs

Video Format: This approach will only work for MP4 videos. Both H.264 and H.265/HEVC are supported.

Caption / Subtitle file: Either SRT or WebVTT format are supported. Don't have a caption file? See instructions below.

FFmpeg

The instructions rely on the open-source FFmpeg tool.

FFmpeg is the leading multimedia framework, able to decode, encode, transcode, mux, demux, stream, filter and play pretty much anything that humans and machines have created.

Please follow the installation instructions here.
On Mac, simply use Homebrew with brew install ffmpeg

Creating a caption file automatically (optional)

If you do not have a captions file with subtitles for your video you can easily auto-caption your video with one of two methods.

Upload your video to YouTube. YouTube can automatically caption your video. Captions can be downloaded in SRT file format.
If your company has Microsoft Stream, upload your video to enjoy auto-captioning. Once the captions have been generated you can easily make edits through the web interface before downloading your captions in WebVTT format.

Making subtitles selectable

These subtitles can be toggled on or off in any video player that supports subtitles.

# Replace with your input video and subtitle file
INFILE=video.mp4
SUBTITLES=subtitles.vtt

ffmpeg -i $INFILE -i $SUBTITLES -c copy -c:s mov_text \
  -metadata:s:s:0 language=eng outfile_selectable.mp4

Multiple languages

You can even have multiple subtitles in different languages all within the same video.

# Replace with your input video and subtitle files
INFILE=video.mp4
SUBTITLESENG=english.vtt
SUBTITLEGER=german.vtt
SUBTITLESSPA=spanish.vtt

ffmpeg -i $INFILE -i $SUBTITLESENG -i $SUBTITLESGER -i $SUBTITLESSPA \
  -map 0 -map 1:s -map 2:s -map 3:s -c copy -c:s mov_text -c:s mov_text c:s mov_text \
  -metadata:s:s:0 language=eng -metadata:s:s:1 language=ger \
  -metadata:s:s:2 language=spa outfile_selectable_multi.mp4

Burned in subtitles

These subtitles are part of the video itself and cannot be toggled off.

We will need to render the video again with the subtitles permanently added into the video stream. Note that this will take some time depending on the video duration, quality and the quality of your CPU and GPU.

Tradeoff: While hardware accelerated encoding using GPUs is generally significantly faster, software encoding (CPU only) produces smaller files and better quality.

Tradeoff: H.264 has the most compatibility. You likely have hardware support for encoding and decoding. H.265/HEVC files are 25-50% more efficient but decoding is much more compute intensive. Only recent GPUs have hardware support for encoding in this format. Nevertheless, H.265/HEVC is the format of the future.

For output in H.264 with hardware encoding on MacOS

# Replace with your input video and subtitle file
INFILE=video.mp4
SUBTITLES=subtitles.vtt

ffmpeg -i $SUBTITLES temp.ass && \
BITRATE=$(ffprobe -v error -select_streams v:0 -show_entries stream=bit_rate \
  -of default=noprint_wrappers=1:nokey=1 $INFILE) && \
ffmpeg -i $INFILE -vf ass=temp.ass -vcodec h264_videotoolbox \
  -b:v $BITRATE -c:a copy output_burnedin.mp4

For H.264 with hardware encoding on Windows/Linux with NVIDIA GPU

# Replace with your input video and subtitle file
INFILE=video.mp4
SUBTITLES=subtitles.vtt

ffmpeg -i $SUBTITLES temp.ass && \
BITRATE=$(ffprobe -v error -select_streams v:0 -show_entries stream=bit_rate \
  -of default=noprint_wrappers=1:nokey=1 $INFILE) && \
ffmpeg -i $INFILE -vf ass=temp.ass -vcodec h264_nvenc \
  -b:v $BITRATE -c:a copy output_burnedin.mp4

For H.264 with software encoding on any operating system

# Replace with your input video and subtitle file
INFILE=video.mp4
SUBTITLES=subtitles.vtt

ffmpeg -i $SUBTITLES temp.ass && \
BITRATE=$(ffprobe -v error -select_streams v:0 -show_entries stream=bit_rate \
  -of default=noprint_wrappers=1:nokey=1 $INFILE) && \
ffmpeg -i $INFILE -vf ass=temp.ass -vcodec libx264 \
  -b:v $BITRATE -c:a copy output_burnedin.mp4

For H.265/HEVC with hardware encoding on MacOS

# Replace with your input video and subtitle file
INFILE=video.mp4
SUBTITLES=subtitles.vtt

ffmpeg -i $SUBTITLES temp.ass && \
BITRATE=$(ffprobe -v error -select_streams v:0 -show_entries stream=bit_rate \
  -of default=noprint_wrappers=1:nokey=1 $INFILE) && \
ffmpeg -i $INFILE -vcodec hevc_videotoolbox -tag:v hvc1 \
  -b:v $BITRATE -c:a copy -vf ass=temp.ass output_burnedin.mp4

For H.265/HEVC with hardware encoding on Windows/Linux with NVIDIA GPU

# Replace with your input video and subtitle file
INFILE=video.mp4
SUBTITLES=subtitles.vtt

ffmpeg -i $SUBTITLES temp.ass && \
BITRATE=$(ffprobe -v error -select_streams v:0 -show_entries stream=bit_rate \
  -of default=noprint_wrappers=1:nokey=1 $INFILE) && \
ffmpeg -i $INFILE -vcodec hevc_nvenc -tag:v hvc1 \
  -b:v $BITRATE -c:a copy -vf ass=temp.ass output_burnedin.mp4

For H.265/HEVC software encoding on any operating system

# Replace with your input video and subtitle file
INFILE=video.mp4
SUBTITLES=subtitles.vtt

ffmpeg -i $SUBTITLES temp.ass && \
BITRATE=$(ffprobe -v error -select_streams v:0 -show_entries stream=bit_rate \
  -of default=noprint_wrappers=1:nokey=1 $INFILE) && \
ffmpeg -i $INFILE -vcodec libx265 -tag:v hvc1 \
  -b:v $BITRATE -c:a copy -vf ass=temp.ass output_burnedin.mp4

Note: If the source video was in H.264 format you can also use the H.265 command to produce a final video in H.265/HEVC format. Since H.265 is 25-50% more efficient you should be able to reduce the Bitrate and still maintain the same video quality.