DEV Community: David Liman

Don't use clj-time, use clojure.java-time instead

David Liman — Wed, 10 Jan 2024 03:41:00 +0000

clj-time

A date and time library for Clojure, wrapping the Joda Time library. DEPRECATED

A big warning sign on a project's README should have deterred anyone from using the library but my goldfish memory can't remember why

I have used clj-time in my previous project before. I did not recall anything bad about it. Time, just like a logger, in Java has a long history. For the most part, when you have to deal with time, you google around, find which method to call, and move on.

Until this one fine afternoon, I saw lots of new error logs. The stacktrace looks like this:

#error {
 :cause "The datetime zone id 'America/Ciudad_Juarez' is not recognised"
 :via
 [{:type java.lang.IllegalArgumentException
   :message "The datetime zone id 'America/Ciudad_Juarez' is not recognised"
   :at [org.joda.time.DateTimeZone forID "DateTimeZone.java" 234]}]
 :trace
 [[org.joda.time.DateTimeZone forID "DateTimeZone.java" 234]
  [clj_time.core$time_zone_for_id invokeStatic "core.clj" 411]
  [clj_time.core$time_zone_for_id invoke "core.clj" 406]
  [app.domain.hotel$time_zone invokeStatic "hotel.clj" 129]
   ...
  [clojure.core$reduce1 invokeStatic "core.clj" 944]
  [clojure.core$set invokeStatic "core.clj" 4101]
  [org.eclipse.jetty.util.thread.QueuedThreadPool$Runner run "QueuedThreadPool.java" 1034]
  [java.lang.Thread run nil -1]]}

It turns out a partner system sends an update with timezone record like 'America/Ciudad_Juarez'

Wait, what is wrong with that time zone? Let's fire up a REPL and check that:

(require '[clj-time.core :as time])

(time/time-zone-for-id "America/Ciudad_Juarez')
Execution error (IllegalArgumentException) at org.joda.time.DateTimeZone/forID (DateTimeZone.java:234).
The datetime zone id 'America/Ciudad_Juarez' is not recognised

Uh oh, wait that can't be right. How about:

(time/time-zone-for-id "America/Chicago")
;; => #<org.joda.time.tz.CachedDateTimeZone@23749549 America/Chicago>

That one is fine. Quick Google-ing tells me Ciudad Juarez was only recently added to a list of timezones back in 2022

Okay, how do we upgrade our Clojure / Java system with that latest timezone? Does Java reads the timezone database somewhere? That leads me to Oracle TZUpdater app

Fair enough. Let's update our Dockerfile:

FROM debian:bullseye-slim as final
RUN apt-get update && apt-get install -y curl wget unzip

ENV JAVA_HOME=/usr/local/openjdk-17

RUN wget --header "Cookie: oraclelicense=accept-securebackup-cookie" https://download.oracle.com/otn-pub/java/tzupdater/2.3.2/tzupdater-2.3.2.zip && \
    unzip tzupdater-2.3.2.zip && \
    $JAVA_HOME/bin/java -jar tzupdater-2.3.2/tzupdater.jar -f -l https://www.iana.org/time-zones/repository/tzdata-latest.tar.gz

ENTRYPOINT ["java", "-jar", "app-standalone.jar"]

Not a big deal. That should be it, right? Not too soon! Turns out clj-time is a wrapper of Joda Time and the project's README says:

If you are using Java 8 or later, please use the built-in Java Time instead of Joda Time -- or look at clojure.java-time if you want a Clojure wrapper for that, or cljc.java-time for a thin Clojure(Script) wrapper, or juxt/tick for another cross-platform option. See Converting from Joda Time to java.time for more details about the similarities and differences between the two libraries.

Okay so clj-time is deprecated, latest version is 0.15.2. Let's try dm3/clojure.java-time. Sure enough:

(jt/local-date (jt/instant) "America/Ciudad_Juarez")
;; => #<java.time.LocalDate@5bfc8967 2024-01-09>

It picks up the new timezone. Now why wouldn't Joda pick up the latest timezone? To be fair, Joda time does. Here is the release report. It bundles new Global Tz which is derived from IANA Time Zone database

But clj-time wouldn't. It is marked as deprecated. Now my only option is to wholesale adopt java-time, change hundreds of lines of code, or fork clj-time and upgrade to the latest Joda time. Note to myself, pick the newer library for new project guys.

Building a Live Streaming app in Clojure

David Liman — Tue, 01 Mar 2022 03:13:56 +0000

I want to echo John Carmack’s tweet that all giant companies use open-source FFmpeg in the backends. FFmpeg is a core piece of technology that powers our live-streaming and recording system at Inspire Fitness. It certainly is high-quality open-source software that we use to record and stream countless hours of workout videos.

It looks like this:

Users can:

watch live-streaming content, or
playback on-demand videos from our content library

High level

Behind the scene, we have:

IP cameras are wired up in each studio room.
The cameras support the RTSP protocol.
A software pipeline integrated with our custom CMS to broadcast (stream) our cameras feed to the internet while simultaneously recording and storing the content to AWS S3 storage for on-demand playback.

How it all works together:

We configure classes (the recordings) to start at a particular time in our dashboard. The time aligns with our studio schedules, where gym members would often join our classes to work out alongside the instructors.
We kick off a dedicated ec2 instance with FFmpeg baked in an AMI image when the class starts. We call this our encoder/transcoder.

;; vm.clj

(aws/invoke state/ec2
  {:op       :RunInstances
   :request {:InstanceType      "c5d.2xlarge"
             :MaxCount          1
             :MinCount          1
             :SubnetId          "subnet-id"
             :ImageId           "ami-id"
             :SecurityGroupIds  ["sg-id"]
             :UserData          (build-user-data class-id)
             :TagSpecifications [{:ResourceType "instance"
                                  :Tags         [{:Key "Name" :Value (make-class-name class-id)}]}]}})

As soon as the ec2 boots up, it runs the cloud-init script, which starts the Clojure process and mount (a state management library) that would then starts the dependencies:

;; core.clj

(defn run-encoder []
  (mount/start #'state/s3
               #'state/db
               #'encoder/encoder))

(defn -main [& args]
  (run-encoder))

This would in turn calls start-stream:

;; encoder.clj

(defstate encoder
  :start (try
           (start-stream)
           (catch Exception e))
             ;; error-handling here
  :stop (stop-stream))

The start-stream logic is actually pretty simple. It pulls feed from our camera and egress out our CDN partner

(defn start-stream []
  (record :encoder :connect-to-camera (make-event-details ...))
  (manager/register :connect-to-camera
    (sh/proc "ffmpeg" "-hide_banner" "-re" "-rtsp_transport" "tcp" "-i"
      config/encoder-rtsp-endpoint
      "-c:a" "aac" "-ar" "48000" "-b:a" "128k"
      "-c:v" "h264" "-profile:v" "high"
      "-g" "48" "-keyint_min" "48" "-sc_threshold" "0" "-b:v" "3072k"
      "-maxrate" "3500k" "-vcodec" "libx264" "-bufsize" "3072k"
      "-hls_time" "6"
      "-hls_playlist_type" "event"
      "-hls_segment_filename" segment-file-pattern
      manifest-file-path))

  (redirect-stdout-stderr :connect-to-camera hls-file-path)

  ;; it takes some time to pull from the RTSP stream and write to the m3u8 file
  ;; the HLS egress looks at the m3u8 file; if it can't find it, the process will exit - so wait for the file
  (record :encoder :wait-for-manifest-file (make-event-details ...))

  (record :encoder :egress (make-event-details ...))
  (manager/register :egress
    (sh/proc "ffmpeg" "-hide_banner" "-re" "-i" manifest-file-path
      "-c:v" "copy" "-c:a" "aac" "-ar" "48000" "-b:a" "128k" "-f" "flv"
      config/encoder-ingress-endpoint))

  (redirect-stdout-stderr :egress egress-file-path))

The output would be a playback URL that our video player would be pulling from.

Notice we are essentially invoking the FFmpeg that we bundled earlier to:

take input from RTSP transport protocol
argument flags for the video/audio codec

Flag	What it does
-re	read input at native frame rate. Mainly used to simulate a grab device i.e if you wanted to stream a video file, then you would want this, otherwise it might stream too fast
-c:a aac	transcode to AAC codec
-ar 48000	set the audio sample rate
-b:a 128k	set the audio bitrate
-c:v h264 or copy	transcode to h264 codec or simply send the frame verbatim to output
-hls_time	the duration for video segment length
-f flv	says to deliver the output stream in an flv wrapper
rtmp://	is where the transcoded video stream get pushed to

The code is essentially a shell wrapper to FFmpeg command-line arguments. FFmpeg is the swiss-army tool for all video/audio codecs

The whole encoder.clj is about 300 lines long with error handling. It handles file uploads (video segment files, FFmpeg logs for debugging), egress to primary and secondary/fallback RTMP slot, shutdown processes, and the ec2 instance when we are done with the recording.

Lesson learned

This was a rescue project from Go to Clojure. The previous architecture had too many moving pieces, making HTTP requests across multiple micro-services. The main server would crash daily due to improper handling of WebSocket messages, causing messages to be lost and encoder instances not starting up on time.

The rewrite reduced the complexities. Simple Made Easy as Rich Hickey.

Rewriting a project is never a good approach considering the opportunity cost. I evaluated a few offerings: mux.com, Cloudflare Stream, Amazon IVS. On paper, they have all the building blocks we need. In addition, some have features like video analytics, policing/signing playback URL, which would be useful for us.

Ultimately, the fact that we still had 2 years contract with the CDN company was why we still manage our encoder.

In hindsight, if we consider the storage costs and S3 egress bandwidth fee on top of the CDN costs, I would probably go for ready-made solutions for our company stage. I would start optimizing when we have more traffic.

The good thing is that this system works really well for live-streaming workload with programmatic access. (Sidenote: barring occasional internet hiccups in our studio).

If you have a video production pipeline that involves heavy video editing, going with prebuilt software could be more flexible until you solidify the core functionalities.

Special thanks to:

Daniel Fitzpatrick and Vincent Ho. My coworkers helped proofread this article and maintain the broadcast system. I enjoy working with you both ❤️. We even have automated tests to prove the camera stream is working end to end!
Neil, our product manager, understands tech trade-offs and works with me to balance the product roadmap.
Daniel Glauser, who hired me for this Clojure gig