DEV Community: poloxue

Colly: A Comprehensive Guide to High-Performance Web Crawling in Go

poloxue — Sat, 20 Jan 2024 11:00:49 +0000

Colly is a well-known web crawling framework implemented in Go, which is particularly suitable for high-concurrency and distributed scenarios - areas where web crawling technology thrives. Its primary attributes include being lightweight, fast, elegantly designed, and simple to distribute and extend.

This article, based on Colly's official documentation, provides a guide to learning Colly as well as my personal insights into the framework.

How to Learn

When it comes to web crawling frameworks, Python's Scrapy is probably the most famous. It's often the first encounter many have with web crawling, and I'm no exception. Scrapy boasts comprehensive documentation and a rich set of extension components. When designing a web crawling framework, it's common to draw inspiration from Scrapy's design. I've read articles mentioning implementations in Go similar to Scrapy.

In comparison, learning resources for Colly are unfortunately scarce. Initially, I couldn't help but try to apply my experience with Scrapy to Colly, only to find that this approach doesn't work.

The natural next step was to seek out articles for guidance. However, articles on Colly are quite rare, and those available are mostly from official sources and seem somewhat incomplete. The solution? Bite the bullet and dive into the official learning materials, which generally consist of three parts: documentation, case studies, and source code.

Let's start with the official documentation today.

Official Documentation

The official documentation focuses primarily on usage. If you have experience with web crawlers, a quick read through the documentation should suffice. I spent some time organizing the official documentation according to my understanding.

The main content isn't extensive and covers topics like installation, getting started, configuration, debugging, distributed crawling, storage, using multiple collectors, configuration optimization, and extensions.

Each of these documents is concise, to the point where scrolling is barely necessary.

How to Install

Installing Colly is as simple as installing any other Go library. Just run:

go get -u github.com/gocolly/colly

One command and you're done. So easy!

Getting Started

Let's quickly dive into using Colly with a simple hello world example. Here are the steps:

First, import Colly.

import "github.com/gocolly/colly"

Second, create a collector.

c := colly.NewCollector()

Third, set up event listeners and execute event handling through callbacks.

// Find and visit all links
c.OnHTML("a[href]", func(e *colly.HTMLElement) {
    link := e.Attr("href")
    // Print link
    fmt.Printf("Link found: %q -> %s\n", e.Text, link)
    // Visit link found on page
    // Only those links are visited which are in AllowedDomains
    c.Visit(e.Request.AbsoluteURL(link))
})

c.OnRequest(func(r *colly.Request) {
    fmt.Println("Visiting", r.URL)
})

Let's also list the types of events Colly supports:

OnRequest: Called before a request is executed
OnResponse: Called after a response is received
OnHTML: Listens for and executes a selector
OnXML: Listens for and executes a selector
OnHTMLDetach, stops listening, with the selector string as a parameter
OnXMLDetach, stops listening, with the selector string as a parameter
OnScraped, executed after scraping is complete, after all work is done
OnError, callback for errors

Finally, c.Visit() starts the actual web page visit.

c.Visit("http://go-colly.org/")

The complete code for this example can be found in the basic directory under the _examples folder in the Colly source code.

How to Configure

Colly is a flexible framework that provides a plethora of configuration options for developers. By default, each option is set to a reasonable default value.

Here's how you can create a collector with the default configuration:

c := colly.NewCollector()

And here's how to configure a collector, for example by setting a user agent and allowing repeated visits. Code is as follows:

c2 := colly.NewCollector(
    colly.UserAgent("xy"),
    colly.AllowURLRevisit(),
)

You can also change the configuration after creating a collector.

c2 := colly.NewCollector()
c2.UserAgent = "xy"
c2.AllowURLRevisit = true

Collector configuration can be changed at any stage of the crawl. A classic example is randomly changing the user-agent, which can help us implement simple anti-crawling techniques.

const letterBytes = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ"

func RandomString() string {
    b := make([]byte, rand.Intn(10)+10)
    for i := range b {
        b[i] = letterBytes[rand.Intn(len(letterBytes))]
    }
    return string(b)
}

c := colly.NewCollector()

c.OnRequest(func(r *colly.Request) {
    r.Headers.Set("User-Agent", RandomString())
})

As mentioned earlier, the collector's default configuration is already optimized, but it can also be changed through environment variables. This way, you don't have to recompile every time you want to change a configuration. The environment variable configuration takes effect when the collector is initialized, but it can be overridden after the collector is officially started.

The supported configuration options are as follows:

ALLOWED_DOMAINS (string slice), allowed domains, e.g., []string{"segmentfault.com", "zhihu.com"}
CACHE_DIR (string), cache directory
DETECT_CHARSET (y/n), whether to detect response encoding
DISABLE_COOKIES (y/n), disable cookies
DISALLOWED_DOMAINS (string slice), prohibited domains, same type as ALLOWED_DOMAINS
IGNORE_ROBOTSTXT (y/n), whether to ignore ROBOTS protocol
MAX_BODY_SIZE (int), maximum response size
MAX_DEPTH (int - 0 means infinite), visit depth
PARSE_HTTP_ERROR_RESPONSE (y/n), parse HTTP response errors
USER_AGENT (string)

These are all very straightforward options.

Let's take a look at the HTTP configuration. These are all common configurations, such as proxies, various timeout times, etc.

c := colly.NewCollector()
c.WithTransport(&http.Transport{
    Proxy: http.ProxyFromEnvironment,
    DialContext: (&net.Dialer{
        Timeout:   

 30 * time.Second,          // Timeout
        KeepAlive:  30 * time.Second,          // keepAlive timeout
        DualStack:  true,
    }).DialContext,
    MaxIdleConns:          100,               // Maximum number of idle connections
    IdleConnTimeout:       90 * time.Second,  // Idle connection timeout
    TLSHandshakeTimeout:   10 * time.Second,  // TLS handshake timeout
    ExpectContinueTimeout: 1 * time.Second,  
}

Debugging

When using Scrapy, it provides a very convenient shell that helps us debug easily. Unfortunately, Colly doesn't have a similar feature. The debugger here mainly refers to the collection of runtime information.

Debugger is an interface. As long as we implement the two methods in it, we can complete the collection of runtime information.

type Debugger interface {
    // Init initializes the backend
    Init() error
    // Event receives a new collector event.
    Event(e *Event)
}

There's a typical example in the source code, LogDebugger. We only need to provide the corresponding io.Writer type variable. How do we use it?

Here's an example:

package main

import (
    "log"
    "os"

    "github.com/gocolly/colly"
    "github.com/gocolly/colly/debug"
)

func main() {
    writer, err := os.OpenFile("collector.log", os.O_RDWR|os.O_CREATE, 0666)
    if err != nil {
        panic(err)
    }

    c := colly.NewCollector(colly.Debugger(&debug.LogDebugger{Output: writer}), colly.MaxDepth(2))
    c.OnHTML("a[href]", func(e *colly.HTMLElement) {
        if err := e.Request.Visit(e.Attr("href")); err != nil {
            log.Printf("visit err: %v", err)
        }
    })

    if err := c.Visit("http://go-colly.org/"); err != nil {
        panic(err)
    }
}

After running, open collector.log to see the output content.

Distributed

Distributed web crawling can be considered at several levels: the proxy level, the execution level, and the storage level.

Proxy Level

By setting up a proxy pool, we can assign download tasks to different nodes for execution, which helps improve the webpage download speed of the crawler. At the same time, this can effectively reduce the possibility of IP bans due to excessive crawl rates.

The code for implementing proxy IPs in Colly is as follows:

package main

import (
    "github.com/gocolly/colly"
    "github.com/gocolly/colly/proxy"
)

func main() {
    c := colly.NewCollector()

    if p, err := proxy.RoundRobinProxySwitcher(
        "socks5://127.0.0.1:1337",
        "socks5://127.0.0.1:1338",
        "http://127.0.0.1:8080",
    ); err == nil {
        c.SetProxyFunc(p)
    }
    // ...
}

proxy.RoundRobinProxySwitcher is a function built into Colly that implements proxy switching through polling. Of course, we can also fully customize it.

For example, here's a case of randomly switching proxies:

var proxies []*url.URL = []*url.URL{
    &url.URL{Host: "127.0.0.1:8080"},
    &url.URL{Host: "127.0.0.1:8081"},
}

func randomProxySwitcher(_ *http.Request) (*url.URL, error) {
    return proxies[random.Intn(len(proxies))], nil
}

// ...
c.SetProxyFunc(randomProxySwitcher)

However, it's worth noting that the crawler is still centralized at this point, with tasks executed on a single node.

Execution Level

This approach distributes tasks to different nodes for execution, achieving true distributed crawling.

To implement distributed execution, the first issue we face is how to distribute tasks to different nodes and ensure coordinated work among different task nodes.

First, we choose an appropriate communication scheme. Common communication protocols include HTTP, TCP (one is a stateless text protocol, the other is a connection-oriented protocol). In addition, there's a rich variety of RPC protocols to choose from, such as Jsonrpc, Facebook's Thrift, Google's gRPC, etc.

The document provides an HTTP service example code that is responsible for receiving requests and executing tasks. As follows:

package main

import (
    "encoding/json"
    "log"
    "net/http"

    "github.com/gocolly/colly"
)

type pageInfo struct {
    StatusCode int
    Links      map[string]int
}

func handler(w http.ResponseWriter, r *http.Request) {
    URL := r.URL.Query().Get("url")
    if URL == "" {
        log.Println("missing URL argument")
        return
    }
    log.Println("visiting", URL)

    c := colly.NewCollector()

    p := &pageInfo{Links: make(map[string]int)}

    // count links
    c.OnHTML("a[href]", func(e *colly.HTMLElement) {
        link := e.Request.AbsoluteURL(e.Attr("href"))
        if link != "" {
            p.Links[link]++
        }
    })

    // extract status code
    c.OnResponse(func(r *colly.Response) {
        log.Println("response received", r.StatusCode)
        p.StatusCode = r.StatusCode
    })
    c.OnError(func(r *colly.Response, err error) {
        log.Println("error:", r.StatusCode, err)
        p.StatusCode = r.StatusCode
    })

    c.Visit(URL)

    // dump results
    b, err := json.Marshal(p)
    if err != nil {
        log.Println("failed to serialize response:", err)
        return
    }
    w.Header().Add("Content-Type", "application/json")
    w.Write(b)
}

func main() {
    // example usage: curl -s 'http://127.0.0.1:7171/?url=http://go-colly.org/'
    addr := ":7171"

    http.HandleFunc("/", handler)

    log.Println("listening on", addr)
    log.Fatal(http.ListenAndServe(addr, nil))
}

This example doesn't provide the code for the scheduler, but the implementation is not complicated. After the task is completed, the service returns the corresponding link to the scheduler, which is responsible for sending new tasks to the worker nodes for execution.

If it's necessary to decide the task execution node based on the node's load, additional service monitoring APIs are needed to obtain node performance data to assist the scheduler in decision-making.

Storage Level

We have already achieved distributed crawling by distributing tasks to different nodes for execution. However, some data, such as cookies and visited URL records, need to be shared between nodes. By default, these data are saved in memory, meaning each collector has its own separate data.

We can achieve data sharing between nodes by saving data in storage systems like Redis or MongoDB. Colly supports switching between any storage as long as the corresponding storage implements the colly/storage.Storage interface methods.

In fact, Colly has already built-in implementations for some storage types, which you can view in the storage section. We'll also touch on this topic in the next section.

Storage

As we just mentioned, let's take a closer look at the storage options already supported by Colly.

InMemoryStorage, the default storage in Colly, stores data in memory. You can replace it by using collector.SetStorage().

RedisStorage is also available. Perhaps because Redis is more commonly used in distributed scenarios, the official website provides a usage example.

Other options include Sqlite3Storage and MongoStorage.

Multiple Collectors

The spiders we've demonstrated so far are relatively simple, with similar processing logic. For a complex spider, we can create different collectors responsible for handling different tasks.

How should we understand this? Let's take an example.

If you've been writing spiders for a while, you must have encountered the issue of fetching parent-child pages. Typically, the processing logic for parent pages is different from that of child pages, and there's usually a need for data sharing between parent and child pages. Those who have used Scrapy will know that Scrapy manages different page logic by binding callback functions to requests, and data sharing is achieved by binding data to requests to pass data from parent pages to child pages.

After research, we find that Colly does not support Scrapy's method. So, what should we do? This is the problem we need to solve.

For different page processing logic, we can define and create multiple collectors, i.e., different collectors responsible for handling different page logic.

c := colly.NewCollector(
    colly.UserAgent("myUserAgent"),
    colly.AllowedDomains("foo.com", "bar.com"),
)
// Custom User-Agent and allowed domains are cloned to c2
c2 := c.Clone()

Typically, the collector for the child pages is the same as for the parent pages. In the example above, the collector c2 for the child pages clones the configuration of the parent collector c.

For data passing between parent and child pages, we can use Context to transfer data between different collectors. Note that this Context is just a data-sharing structure implemented by Colly, not the Context from the Go standard library.

c.OnResponse(func(r *colly.Response) {
    r.Ctx.Put("Custom-header", r.Headers.Get("Custom-Header"))
    c2.Request("GET", "https://foo.com/", nil, r.Ctx, nil)
})

This way, we can get the data passed from the parent in the child pages through r.Ctx. For this scenario, you can refer to the official example coursera_courses.

Configuration Optimization

Colly's default configuration is optimized for a small number of sites. If you're crawling a large number of sites, some improvements are needed.

Persistent Storage

By default, cookies and URLs in Colly are saved in memory. We need to switch to a persistent storage. As mentioned earlier, Colly has already implemented some common persistent storage components.

Enable Asynchronous Execution to Speed Up Task Execution

By default, Colly will block and wait for a request to complete, leading to an increasing number of waiting tasks. We can set the Async option of the collector to true to handle requests asynchronously and avoid this problem. If you use this method, remember to add c.Wait(), otherwise, the program will exit immediately.

Disable or Limit KeepAlive Connections

Colly enables KeepAlive by default to increase the crawling speed. However, this requires open file descriptors, and for long-running tasks, it's very easy to reach the maximum descriptor limit.

Here's an example code to disable KeepAlive in HTTP:

c := colly.NewCollector()
c.WithTransport(&http.Transport{
    DisableKeepAlives: true,
})

Extensions

Colly provides some extensions mainly related to common web crawling functionalities, such as referer, random_user_agent, url_length_filter, etc. The source code is located under colly/extensions/.

Let's understand how to use them through an example:

import (
    "log"

    "github.com/gocolly/colly"
    "github.com/gocolly/colly/extensions"
)

func main() {
    c := colly.NewCollector()
    visited := false

    extensions.RandomUserAgent(c)
    extensions.Referrer(c)

    c.OnResponse(func(r *colly.Response) {
        log.Println(string(r.Body))
        if !visited {
            visited = true
            r.Request.Visit("/get?q=2")
        }
    })

    c.Visit("http://httpbin.org/get")
}

You just need to pass the collector into the extension function. It's that simple.

But can we implement an extension ourselves?

When using Scrapy, if you want to implement an extension, you need to understand quite a few concepts and read its documentation carefully. But Colly doesn't even mention this in the documentation. So what should we do? It seems we can only look at the source code.

Let's open the source code of the referer plugin:

package extensions

import (
    "github.com/gocolly/colly"
)

// Referer sets valid Referer HTTP header to requests.
// Warning: this extension works only if you use Request.Visit
// from callbacks instead of Collector.Visit.
func Referer(c *colly.Collector) {
    c.OnResponse(func(r *colly.Response) {
        r.Ctx.Put("_referer", r.Request.URL.String())
    })
    c.OnRequest(func(r *colly.Request) {
        if ref := r.Ctx.Get("_referer"); ref != "" {
            r.Headers.Set("Referer", ref)
        }
    })
}

By adding some event callbacks to the collector, you can implement an extension. The source code is so simple that you don't need a documentation explanation to implement your own extension. Of course, if you look closely, you'll find that its approach is similar to that of Scrapy, both extending request and response callbacks, and Colly's simplicity is largely due to its elegant design and Go's simple syntax.

Conclusion

After reading Colly's official documentation, you'll find that although the documentation is rudimentary, it covers everything that

should be introduced. If there are parts that weren't covered, I've supplemented them in this article. Previously, when using Go's elastic package, I also found the documentation to be scarce, but a simple read of the source code made it clear how to use it.

Perhaps this is the simplicity of Go.

Finally, if you encounter any problems while using Colly, the official examples are the best practice. I suggest taking the time to read them.

My blog post: Colly: A Comprehensive Guide to High-Performance Web Crawling in Go.

Mastering Audio Editing in MoviePy: Extraction and Manipulation

poloxue — Thu, 04 Jan 2024 14:22:49 +0000

In the preceding text, we delved into the installation of MoviePy and utilized the VideoFileClip class to access video files, performing simple edits like resizing, rotation, and segment clipping.

This article's focus lies on understanding MoviePy's audio capabilities.

Creating AudioClip

MoviePy offers two methods to create audio clips: extracting audio from video files and generating audio clips based on existing audio files.

Extracting Audio from Video Files

Assuming we possess a video file named input_video.mp4, we can directly extract its audio with the following code:

from moviepy.editor import *

video = VideoFileClip("input_video.mp4")
audio = video.audio
audio.write_audiofile("output_audio.mp3")

Generating Audio Clips from Audio Files

By using AudioFileClip, we can read the audio file extracted from the video. The code appears as follows:

audio = AudioFileClip("output_audio.mp3")

We can embed this audio clip into another video segment.

Directly create a video using TextClip and set the video duration using audio.duration.

video = TextClip("Hello MoviePy", color="orange", size=(100, 100))
video = video.set_duration(audio.duration).set_fps(1)

Let's load the audio onto this video.

Here's the code:

video.audio = audio
video.write_audio("text_video.mp4")

When opening the video file using QuickTime Player on a Mac, remember to add the parameter audio_codec="aac" when writing:

video.write_audio("text_video.mp4", audio_codec="aac")

Upon completion, you can open the video and review the effects.

Audio Operations

This section discusses fundamental operations on AudioClips, including volume control, audio segment clipping, and removing audio from videos.

Volume Control

Adjusting the volume of audio clips is accomplished directly through the volumex method of AudioClip.

To decrease the volume by 50%, use the code below:

audio = audio.volumex(0.5)

Segment Clipping

Similar to trimming videos, subclip enables us to extract specific portions of audio clips.

For instance, to extract the audio segment from 5 to 15 seconds, use the code:

audio = audio.subclip(5, 15)

Removing Audio from Videos

Here's a handy trick: to remove audio from a video, set the audio parameter to False when reading the video file with VideoFileClip.

video = VideoFileClip("input_video", audio=False)

Alternatively, you can utilize the without_audio method of VideoFileClip to remove audio from video segments.

video = video.without_audio()

Isn't it simple?

Conclusion

This article primarily covers fundamental MoviePy operations. For more MoviePy tutorials, subscribe to our blog. Thank you!

MoviePy: Basic Usage

poloxue — Tue, 02 Jan 2024 17:24:42 +0000

In an era dominated by short-form videos, numerous user-friendly editing software options exist.

However, for specialized video formats like eBook presentations or comic readings, streamlining the video creation process through automation stands as a crucial enhancement for efficiency.

In this tutorial, we'll delve into MoviePy, a Python library designed to automate video production processes. We'll explore its capabilities and demonstrate how it simplifies video editing tasks through simple Python code.

Installation

To utilize MoviePy, the initial requirement is installation. Python's package manager - pip, can install it.

pip install moviepy

This automatically installs essential dependencies like ffmpeg. For text clips, imagemagick is necessary, which can be installed via HomeBrew:

brew install imagemagick

Additionally, for previewing work in progress, pygame is required:

pip install pygame

Getting Started with MoviePy

To access all necessary functions and classes in MoviePy, import them:

from moviepy.editor import *
video = VideoFileClip("input_video.mp4")

We initialize a video variable that represents the underlying video clip we are operating on.

Resizing and Rotating Videos

Let's begin by resizing a video:

video.resize(width=404, height=240).write_videofile("resized_video.mp4")

Similarly, we can rotate videos:

video.rotate(180).write_videofile("rotated_video.mp4")

Cutting and Extracting Segments

Trimming segments from a video is effortless with MoviePy:

video.subclip(5, 15).write_videofile("subclip_5_15_video.mp4")

Exporting a GIF

Creating a GIF from a video segment is easy:

video.subclip(5, 15).resize(width=404).rotate(180).to_gif("output_video.gif")

Conclusion

In this tutorial, we covered various features of MoviePy, showcasing its simplicity and efficiency in video editing tasks. For more advanced techniques with MoviePy, consider subscribing to our channel for future tutorials.

Explore these examples, experiment, and enjoy the creative possibilities MoviePy offers in simplifying video editing!

Setup Your Own iTerm2 - Install iTerm2 and Configure its Color Presets

poloxue — Fri, 13 Oct 2023 12:06:48 +0000

Hi guys, This is POLO X.

Today, I plan to talk about how to install iTerm2 and configure iTerm2 color presets. At the end of the post, there will be some tips about iTerm2 to share with you.

What is iTerm2?

Quote from the official site.

iTerm2 is a replacement for Terminal and the successor to iTerm. It works on Macs with macOS 10.14 or newer. iTerm2 brings the terminal into the modern age with features you never knew you always wanted.

Shortly, it is a better terminal than the terminal built in MacOS and provides more powerful features.

Installation

Now, let's start to install it.

There are two ways to install iTerm2.

One is download the installer from the official site.
Another is use homebrew command.

In this tutorial, I'm going to demonstrate how to install it using homebrew.

Install Homebrew

Homebrew is not a function built in MacOS, so it must be installed in advance.

Let's open the homebrew official site.

Copy the command from this site.



/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

Open the built-in terminal, and paste it into our terminal. Input the password and confirm.

Wait until it is completed.

After all done, type the command brew to see if it works.

Install iTerm2

Now , let us run



brew install --cask iterm2

Wait for a while.

Now, we succeed in installing iTerm2.

Configure iTerm2's color presets?

It's very easy.

To teach you how to configure it. Three color presets, Material Design Colors, Snazzy and Dracula, will be installed.

Install Material Design Colors

Firstly, let's use Material Design Colors as iTerm2 color preset. Download the iterm colors file using curl command.



curl -Ls https://raw.githubusercontent.com/mbadolato/iTerm2-Color-Schemes/master/schemes/MaterialDesignColors.itermcolors > /tmp/material-design-colors.itermcolors
open /tmp/material-design-colors.itermcolors

Open the iterm colors file.

It will be installed.

Now let's use Command + , to open the preferences of iTerm2.

Select profile.
Select colors tab.
Click the color presets.

You will see the color preset we just installed. Select it, and then you will see a change in iterm2.

Install Snazzy and Dracula

Now, we can install more color presets into our iTerm2, such as Snazzy and Dracula.

Install Snazzy.



curl -Ls https://raw.githubusercontent.com/sindresorhus/iterm2-snazzy/main/Snazzy.itermcolors > /tmp/Snazzy.itermcolors
open /tmp/Snazzy.itermcolors

Install Dracula.



curl -Ls https://raw.githubusercontent.com/dracula/iterm/master/Dracula.itermcolors > /tmp/Dracula.itermcolors
open /tmp/Dracula.itermcolors

Now, all color presets have been installed.

More Color Presets

Do you wanna more choices for color presets?

Let me share you a web site: iterm2colorsschemes.

Here you can get more color presets. Let's chose one from the site - 3024 Night.

Download its itermcolors file, and install it.



curl -Ls https://raw.githubusercontent.com/mbadolato/iTerm2-Color-Schemes/master/schemes/3024%20Night.itermcolors > /tmp/3024Night.itermcolors
open /tmp/3024Night.itermcolors

Open the preferences -> Profiles -> Colors -> Click color presets, and check whether 3024 Night has been installed.

iTerm2 Tips

At last, let me introduce three tips about how to use iTerm2.

Split Panes

Use multiple panes in iTerm2.

Compared to Terminal, iTerm2 supports multiple panels. You can work in different panels in the same window.

How?

We can use shortcut CMD+D to split panes horizontally. And use shift command D to split panes vertically.

Now, we can do different tasks in different panels.

Toggle a Pane Full Screen.

iTerm2 allows you to maximize and minimize a pane on demand, without messing with existing open panes.

This shortcut for this function is Shift+Cmd+Enter.

Press Shift+Cmd+Enter, and then it will maximize or minimize the panel where your cursor is now.

Search text.

Press Command+F. Input the text we want to search.

It looks more comfortable than the default Terminal.

That's all for this post.

Originally published at https://en.poloxue.com on October 12, 2023.

How to Autostart Tmux in iTerm2

poloxue — Sun, 17 Sep 2023 09:05:06 +0000

This post will introduce how to make iTerm2 start into Tmux mode by default.

By default, every time you start iTerm2, you need to enter tmux attach to enter tmux mode.

I use Tmux to manage workspaces for different projects. Common IDEs generally provide an interface for users to select projects. Naturally, can iTerm2 + Tmux have a similar function?

Very simple!

Solution 1: A single-line shell script

First, let's see a bash script:

tmux ls && read -p "Select a session<default>:" tmux_session && tmux attach -t ${tmux_session:-default} || tmux new -s ${tmux_session:-default}

The description of this script is as follows:

tmux ls, output the currently available sessions;
read xxx, read user input and store the name of the session you want to open into variable tmux_session;
tmux attach, try to open the session, if tmux_session is empty, open the default session;
tmux new, if opening fails, try to create a new session;

Note: Because of the read command option -p, you must use bash to run this script.

Configure iTerm2 startup loading

Choose 'Preference -> Profile -> General -> Command -> Select "Login Sell", "Send text at start"' and input the above script.

Restart your iTerm2, and what you will see is like this:

hello: 1 windows (created Tue Sep 12 17:13:54 2023) (group hello) (attached)
default: 1 windows (created Wed Sep 13 19:54:36 2023) (group default)
Select a session<default>:

Input "hello" and enter , iTerm2 will enter the "hello" session.

It's simple and efficient, and a bit like the IDE that lets us select a project after it's started.

The session list is not concise enough. You can configure the output formatting of tmux ls by the command tmux -F '\#{session_name}'.

Output:

hello
default

Solution 2: A python script

Can I just enter the session index to select? This single line shell script is not easy to implement. I wrote a Python script as follows:

#!/usr/bin/env python
import os

output = os.popen("tmux ls -F \\#{session_name}").read()
sessions = output.strip().split("\n")

print("Sessions:")
for index in range(len(sessions)):
    print(f"{index} - {sessions[index]}")

input_value = input("Please select a session <Index or Name>(default):")
if input_value.isdigit():
    sess_name = sessions[int(input_value)]
elif not input_value:
    sess_name = "default"
else:
    sess_name = input_value

if sess_name not in sessions:
    answer = input(f"New a session `{sess_name}`?(Y/N)")
    answer == "Y" and os.system(f"tmux new -s {sess_name}")  # pyright: ignore
else:
    os.system(f"tmux attach -t {sess_name}")

Suppose this file's name is tmux_selector and put it under the home directory.

Let's configure this script as iTerm2 startup script. Restart our iTerm2.

We will see:

❯ ~/tmux_selector
Sessions:
0 - hello
1 - default
Please select a session <Index or Name>(default):

This is just a demo. The input of tmux ls has other formatting variables. If you are interested, you can continue to expand. Code snippet github address.

It's good, but wouldn’t it be better if it uses the built-in menu?

Let's continue!

Solution 3: Use tmux built-in menu tree

After entering tmux mode, you can use the shortcut key prefix-key + s to start the tmux selection menu by default. One strength is the built-in menu can be moved by keys jk.

hello: 1 windows (created Tue Sep 12 17:13:54 2023) (group hello) (attached)
default: 1 windows (created Wed Sep 13 19:54:36 2023) (group default)

Is there a way to start this menu directly when iTerm2 starts? What a great idea! Actually it's not difficult to implement.

First, use tmux list-keys to check which command the prefix-key + s is bound.

$ tmux list-keys
...
bind-key    -T prefix       s                    choose-tree -Zs
...

We can display the menu by choose-tree -Zs. In tmux mode, use tmux choose-tree -Zs to test whether it works.

How to configure it as the iTerm2 startup command?

This command must be run in tmux mode. How to do? We can use the format tmux attach\; <commands running on tmux mode> to achieve this goal.

The command is as follows:

$ tmux attach\; choose-tree -zS

Try testing this code in non-tmux mode to see if you can successfully show the menu.

Finally, in order to prevent when the first entry without a session, I further optimized the script. It will create a default session window when no session exists.

The final script:

$ tmux attach\; choose-tree -Zs || read -p "Create a default session?(Y/N)" anwser && [[ "${anwser}" == "Y" ]] && tmux new -t default

But there is another disadvantage here. Pressing q can only exit the menu and the terminal is still in the tmux mode. One more step of detach is required to exit. This is a difference from the steps we generally think of.