DEV Community

Vee Satayamas
Vee Satayamas

Posted on

3 1

Reading lines from a Zstd archive in Common Lisp

Reading lines from a Zstd archive in Common Lisp

I have a big line-separated JSON file, compressed in Zstd format. I will call this file data.ndjson.zst.

I want my program to read the file line by line. I can use https://github.com/glv2/cl-zstd to make a binary stream from the file. Still, I can run the function read-line of a binary stream. So I need to wrap the binary stream with Flexi-stream.

(ql:quickload 'flexi-streams)
(ql:quickload 'zstd)

(defpackage #:ex1
  (:use #:cl #:flexi-streams :zstd))
(in-package :ex1)

(with-open-file (f #P"data.ndjson.zst"
           :element-type '(unsigned-byte 8)
           :direction :input)
  (with-decompressing-stream (zstd-stream f)
    (let ((s (make-flexi-stream zstd-stream
                :external-format (make-external-format :utf-8))))
      (loop for line = (read-line s nil nil)
        until (null line)
        do (print line)))))
Enter fullscreen mode Exit fullscreen mode

Printing those lines is a realistic example. So you can replace (print line) with your practical applications, for example, parse a JSON line and extract a book title.

AWS Security LIVE!

Join us for AWS Security LIVE!

Discover the future of cloud security. Tune in live for trends, tips, and solutions from AWS and AWS Partners.

Learn More

Top comments (0)

A Workflow Copilot. Tailored to You.

Pieces.app image

Our desktop app, with its intelligent copilot, streamlines coding by generating snippets, extracting code from screenshots, and accelerating problem-solving.

Read the docs