DEV Community: Marvin Zhang

Introducing LeanSpec: A Lightweight SDD Framework Built from First Principles

Marvin Zhang — Thu, 27 Nov 2025 14:34:15 +0000

Earlier this year, I was amazed by agentic AI coding with Claude Sonnet 3.7. The term "vibe coding" hadn't been coined yet, but that's exactly what I was doing—letting AI generate code while I steered the conversation. It felt magical. Until it didn't.

After a few weeks, I noticed patterns: code redundancy creeping in, intentions drifting from my original vision, and increasing rework as the AI forgot context between sessions. The honeymoon was over. I needed structure, but not the heavyweight processes that would kill the speed I'd gained.

That search led me through several existing tools—Kiro, Spec Kit, OpenSpec—and eventually to building LeanSpec, a lightweight Spec-Driven Development framework that hits v0.2.7 today with 10 releases in under three weeks. This post shares why I built it, what makes it different, and how you can try it yourself.

The Problem: Vibe Coding's Hidden Costs

The Vibe Coding Trap
AI coding assistants are incredibly productive—until they're not. Without structured context, AI generates plausible but inconsistent code, leading to technical debt that compounds session after session.

If you've used AI coding tools extensively, you've likely encountered these patterns:

Symptom	Root Cause	Impact
Code redundancy	AI doesn't remember previous implementations	Duplicate logic scattered across files
Intention drift	Context lost between sessions	Features that don't quite match your vision
Increased rework	No persistent source of truth	Circular conversations explaining the same thing
Inconsistent architecture	No structural guidance	Components that don't fit together cleanly

The industry's answer has been Spec-Driven Development (SDD)—writing specifications before code to give AI (and humans) persistent context. But when I explored the existing tools, I found a gap.

Related Reading
New to SDD? Start with my foundational article Spec-Driven Development: A Systematic Approach to Complex Features for methodology basics, or dive into the 2025 SDD Tools Landscape for a comprehensive comparison of industrial tools. Want to try the methodology without installing anything? The Practice SDD Without the Toolkit tutorial has you covered.

Why I Built LeanSpec

My journey through the SDD landscape revealed three categories of tools, each with trade-offs that didn't fit my needs:

Vendor lock-in: Kiro (Amazon's SDD IDE) offers tight integration but requires abandoning my existing workflow. I like my tools—switching IDEs wasn't an option.

Cognitive overhead: Spec Kit provides comprehensive structure, but its elaborate format creates significant cognitive load. Even with AI-assisted writing, parsing and maintaining those specs demands mental bandwidth that feels excessive for solo and small-team work.

Missing project management: OpenSpec came closest to my ideal—lightweight and flexible—but lacked the project management capabilities I needed to track dozens of specs across multiple projects.

I wanted something different: a methodology, not just a tool. Something like Agile—a set of principles anyone can adopt, with lightweight tooling that gets out of the way.

So I built LeanSpec. And then I used LeanSpec to build LeanSpec.

First Principles: The Foundation

LeanSpec isn't just tooling—it's built on five first principles that guide every design decision:

Context Economy: Specs must fit in working memory—both human and AI. Target under 300 lines. If you can't read it in 10 minutes, it's too long.

Signal-to-Noise Maximization: Every line must inform decisions. No boilerplate, no filler, no ceremony for ceremony's sake.

Intent Over Implementation: Capture why, not just how. Implementation details change; intentions persist.

Bridge the Gap: Specs serve both humans and AI. If either can't understand it, the spec has failed.

Progressive Disclosure: Start simple, add structure only when pain is felt. No upfront complexity.

These principles aren't just documentation—LeanSpec's validate command enforces them automatically.

Key Features

Web UI for Visual Management

The feature I'm most excited about: lean-spec ui launches a full web interface for managing your specs visually.

# Launch the web UI
npx lean-spec ui

The UI provides Kanban-style board views, spec detail pages with Mermaid diagram rendering, and dependency visualization—all without leaving your browser. Perfect for planning sessions or reviewing project status.

First Principles Validation

LeanSpec doesn't just store specs—it validates them against first principles:

# Check your specs against first principles
lean-spec validate

# Output:
# specs/045-user-auth/README.md
#   ⚠️  warning  Spec exceeds 300 lines (342)  context-economy
#   ⚠️  warning  Missing overview section      structure
# 
# ✖ 2 warnings in 1 spec

This keeps specs lean and meaningful, preventing the specification bloat that plagues heavyweight SDD tools.

Smart Search & Project Management

Finding relevant specs shouldn't require remembering exact names:

# Semantic search across all specs
lean-spec search "authentication flow"

# Advanced queries
lean-spec search "status:in-progress tag:api"
lean-spec search "created:>2025-11-01"

The Kanban board gives you instant project visibility:

lean-spec board

# 📋 LeanSpec Board
# ─────────────────────────────────────
# 📅 Planned (12)     🚧 In Progress (3)     ✅ Complete (47)
# ─────────────────────────────────────

MCP Server for AI Integration

LeanSpec includes an MCP (Model Context Protocol) server, enabling AI assistants to directly interact with your specs:

{
  "mcpServers": {
    "leanspec": {
      "command": "npx",
      "args": ["@leanspec/mcp"]
    }
  }
}

Works with Claude Code, Cursor, GitHub Copilot, and other MCP-compatible tools. AI agents can search specs, read context, and update status—all programmatically.

Example Projects for Quick Start

New to SDD? Start with a working example:

# Scaffold a complete tutorial project
npx lean-spec init --example dark-theme

Three examples available: dark-theme, dashboard-widgets, and api-refactor—each demonstrating different SDD patterns.

The Journey: Building LeanSpec with LeanSpec

The most meta aspect of this project: after the initial release, LeanSpec has been developed entirely using LeanSpec.

Milestone	Date	Notes
First line of code	Oct 23, 2025	Started with basic spec CRUD
v0.1.0 (First release)	Nov 2, 2025	10 days from scratch to release
v0.2.0 (Production-ready)	Nov 10, 2025	First principles validation, comprehensive CLI
v0.2.7 (Current)	Nov 26, 2025	10 releases in 24 days

Over 120 specs have been created within LeanSpec itself—covering features, architecture decisions, reflections, and even marketing strategy. The feedback loop is tight: identify friction → write spec → implement → validate with real use.

I've also applied LeanSpec to other projects:

Crawlab (12k+ stars) — web crawler management platform
This blog (marvinzhang.dev)
Upcoming projects under the codervisor org

The pattern holds across all of them: specs provide context that survives between sessions, AI stays aligned with my intentions, and I spend less time re-explaining.

What Makes LeanSpec Different

If you've read my SDD Tools analysis, you know I evaluated six major tools in this space. Here's where LeanSpec fits:

Aspect	Heavyweight Tools	LeanSpec
Learning curve	Days to weeks	Minutes
Spec overhead	Extensive upfront work	Write as you go
Token cost	Often >2,000 per spec	<300 lines target
Flexibility	Rigid structure	Adapt to your workflow
Vendor lock-in	Often required	Works anywhere
Philosophy	Tool-first	Methodology-first

LeanSpec is "lean" in multiple senses:

Methodology: Like Agile, it's principles you can adopt regardless of tooling
Cognitive load: Low overhead, quick to learn
Token economy: Specs stay small, fitting in AI context windows
Flexibility: Adapt to your workflow, not the other way around

Getting Started

Try LeanSpec in under 5 minutes:

# Install globally
npm install -g lean-spec

# Initialize in your project
lean-spec init

# Create your first spec
lean-spec create user-authentication

# Launch the web UI
lean-spec ui

Or try an example project:

npx lean-spec init --example dark-theme

Already using Spec Kit or OpenSpec? Check out the migration guide—the transition is straightforward.

What's Next

LeanSpec is actively evolving. Current development focuses on:

VS Code extension for inline spec management (Spec 17)
AI chatbot UI for interactive spec assistance (Spec 94)
Comprehensive internationalization support (Spec 91)
GitHub multi-project ingration (Spec 98)

I built LeanSpec to solve my own problems—code quality degradation from vibe coding, context loss between AI sessions, the cognitive overhead of heavyweight SDD tools. If you face similar challenges, I hope it helps you too.

Links:

Questions, feedback, or feature requests? Open an issue or start a discussion. I read everything.

Web Crawler in Action: How to use Webspot to implement automatic recognition and data extraction of list web pages

Marvin Zhang — Sun, 09 Apr 2023 05:17:16 +0000

Introduction

Using web crawling programs to extract list web pages is a one of those common web data extraction tasks. For engineers to write web crawlers, how to efficiently code and generate extraction rules is quite necessary, otherwise most of the time can be wasted on writing CSS selectors and XPath data extraction rules of web crawling programs. In light of this issue, this article will introduce an example of using open source tool Webspot to automatically recognize and extract data of list web pages.

Webspot

Webspot is an open source project aimed at automating web page data extraction. Currently, it supports recognition and crawling rules extraction of list pages and pagination. In addition, it provides a web UI interface for users to visually view the identified results, and allows developers to use APIs to obtain recognition results.

Installation of Webspot is quite easy, you can refer to the official documentation for the installation tutorial with Docker and Docker Compose. Execute the commands below to install and start Webspot.

# clone git repo
git clone https://github.com/crawlab-team/webspot

# start docker containers
docker-compose up -d

Wait for it to start up, which might take half a minute to initialize the application.

After the initialization, we can visit the web page http://localhost:9999, and should be able to see the user interface below, which means it has started successfully.

Now, we can create a new request for page recognition. Click "New Request" and enter https://quotes.toscrape.com. Then click "Submit". Wait for a while and we should be able to see the page below.

Use API to auto extract data

We are now using Python to call the API of Webspot to auto extract data.

The whole process is as below.

Call the Webspot API to obtain the extraction rules for list pages and pagination. The extraction rules are CSS Selectors.
Define the retrieval target based on the extraction rules of the list page, that is, each item and its corresponding field on the list page.
Determine the target for crawling the next page based on the extraction rules of pagination, and let the crawler automatically crawl the data of the next page.

Call API

Calling the API is very simple, just pass the URL to be recognized into the body. The code is as follows.

import requests
from bs4 import BeautifulSoup
from pprint import pprint

# API endpoint
api_endpoint = 'http://localhost:9999/api'

# url to extract
url = 'https://quotes.toscrape.com'

# call API to recognize list page and pagination elements
res = requests.post(f'{api_endpoint}/requests', json={
    'url': 'https://quotes.toscrape.com'
})
results = res.json()
pprint(results)

Running the code above in Python Console can obtain the recognition result data similar to the following.

{...
 'method': 'request',
 'no_async': True,
 'results': {'pagination': [{'detector': 'pagination',
                             'name': 'Next',
                             'score': 1.0,
                             'scores': {'score': 1.0},
                             'selectors': {'next': {'attribute': None,
                                                    'name': 'pagination',
                                                    'node_id': 120,
                                                    'selector': 'li.next > a',
                                                    'type': 'css'}}}],
...
             'plain_list': [{...
                                                'fields': [{'attribute': '',
                                         'name': 'Field_text_1',
                                         'node_id': None,
                                         'selector': 'div.quote > span.text',
                                         'type': 'text'},
                                         ...],
                                        ...}],
                        },
...}

The recognition results include CSS Selectors for list pages and pagination, as well as the corresponding fields for each item on the list page.

List page and fields extraction logics

Next, we will write the logic for extracting list pages and fields.

First, we can obtain list page selectors and fields through results.

# list result
list_result = results['results']['plain_list'][0]

# list items selector
list_items_selector = list_result['selectors']['full_items']['selector']
print(list_items_selector)

# fields
fields = list_result['fields']
print(fields)

Then, we can write the logic for parsing list page items.

def get_data(soup: BeautifulSoup) -> list:
    # data
    data = []

    # items
    items_elements = soup.select(list_items_selector)
    for el in items_elements:
        # row data
        row = {}

        # iterate fields
        for f in fields:
            # field name
            field_name = f['name']

            # field element
            field_element = el.select_one(f['selector'])

            # skip if field element not found
            if not field_element:
                continue

            # add field value to row
            if f['type'] == 'text':
                row[field_name] = field_element.text
            else:
                row[field_name] = field_element.attrs.get(f['attribute'])

        # add row to data
        data.append(row)

    return data

In the function get_data in the above code, we pass a BeautifulSoup instance as parameter, and use list_items_selectors and fields to parse and obtain the list data which is then returned the function caller.

Request list page and pagination logics

Then, we need to write the logics of requesting list pages and pagination, that is, to request a given URL and parse its pagination, then call the function above get_data.

We need to first obtain the pagination's CSS Selector.

# pagination next selector
next_selector = results['results']['pagination'][0]['selectors']['next']['selector']
print(next_selector)

Then, we write crawler logic, which continuously crawls data from website list pages.

def crawl(url: str) -> list:
    # all data to crawl
    all_data = []

    while True:
        print(f'requesting {url}')

        # request url
        res = requests.get(url)

        # beautiful soup of html
        soup = BeautifulSoup(res.content)

        # add parsed data
        data = get_data(soup)
        all_data += data

        # pagination next element
        next_el = soup.select_one(next_selector)

        # end if pagination next element not found
        if not next_el:
            break

                # url of next page
        url = urljoin(url, next_el.attrs.get('href'))

    return all_data

So we have completed all the coding parts.

Putting them all together

The following is the complete code for the entire crawl logic.

from urllib.parse import urljoin

import requests
from bs4 import BeautifulSoup
from pprint import pprint


def get_data(soup: BeautifulSoup) -> list:
    # data
    data = []

    # items
    items_elements = soup.select(list_items_selector)
    for el in items_elements:
        # row data
        row = {}

        # iterate fields
        for f in fields:
            # field name
            field_name = f['name']

            # field element
            field_element = el.select_one(f['selector'])

            # skip if field element not found
            if not field_element:
                continue

            # add field value to row
            if f['type'] == 'text':
                row[field_name] = field_element.text
            else:
                row[field_name] = field_element.attrs.get(f['attribute'])

        # add row to data
        data.append(row)

    return data


def crawl(url: str) -> list:
    # all data to crawl
    all_data = []

    while True:
        print(f'requesting {url}')

        # request url
        res = requests.get(url)

        # beautiful soup of html
        soup = BeautifulSoup(res.content)

        # add parsed data
        data = get_data(soup)
        all_data += data

        # pagination next element
        next_el = soup.select_one(next_selector)

        # end if pagination next element not found
        if not next_el:
            break

        # url of next page
        url = urljoin(url, next_el.attrs.get('href'))

    return all_data


if __name__ == '__main__':
    # API endpoint
    api_endpoint = 'http://localhost:9999/api'

    # url to extract
    url = 'https://quotes.toscrape.com'

    # call API to recognize list page and pagination elements
    res = requests.post(f'{api_endpoint}/requests', json={
        'url': 'https://quotes.toscrape.com'
    })
    results = res.json()
    pprint(results)

    # list result
    list_result = results['results']['plain_list'][0]

    # list items selector
    list_items_selector = list_result['selectors']['full_items']['selector']
    print(list_items_selector)

    # fields
    fields = list_result['fields']
    print(fields)

    # pagination next selector
    next_selector = results['results']['pagination'][0]['selectors']['next']['selector']
    print(next_selector)

    # start crawling
    all_data = crawl(url)

    # print crawled results
    pprint(all_data[:50])

Run the code and we can obtain the following result data.

[{'Field_link_url_6': '/author/Albert-Einstein',
  'Field_link_url_8': '/tag/change/page/1/',
  'Field_text_1': '“The world as we have created it is a process of our '
                  'thinking. It cannot be changed without changing our '
                  'thinking.”',
  'Field_text_2': '“The world as we have created it is a process of our '
                  'thinking. It cannot be changed without changing our '
                  'thinking.”',
  'Field_text_3': '\n'
                  '            Tags:\n'
                  '            \n'
                  'change\n'
                  'deep-thoughts\n'
                  'thinking\n'
                  'world\n',
  'Field_text_4': 'Albert Einstein',
  'Field_text_5': '(about)',
  'Field_text_7': 'change'},
  ...

Now, we have achieved the crawling task of automatically extracting lists using Webspot. In this way, there is no need to explicitly define CSS Selectors or XPaths. Just call the Webspot API and we can obtain the list page data.

Talking Algorithm: Exploration of Intelligent Web Crawlers

Marvin Zhang — Sat, 25 Mar 2023 06:57:36 +0000

Introduction

"If I had asked people what they wanted, they would have said faster horses" -- Henry Ford

Today is the era of artificial intelligence. Whether it is ChatGPT or the various intelligent applications that follow, many people see the upcoming sci-fi world that was almost unimaginable a few years ago. However, in the field of reptiles, artificial intelligence does not seem to be involved too much. It is true that crawlers, as an "ancient" technology, have created many technical industries such as search engines, news aggregation, and data analysis in the past 20 years, but we have not seen obvious technological breakthroughs yet: crawler engineers still mainly rely on technologies such as XPath and reverse engineering to automatically obtain web data. However, with the development of artificial intelligence and machine learning, crawler technology can theoretically achieve "automatic driving". This article will introduce the current status and possible future development direction of the so-called intelligent crawler (intelligent, automated data extraction crawler technology) from multiple perspectives.

Current Web Crawling Technology

A web crawler is an automated program used to obtain data from the Internet or other computer networks. They usually use automated scraping techniques to automatically visit the website and collect, parse and store information on the website. This information can be structured or unstructured data.

Crawler technology in the traditional sense mainly includes the following modules or systems:

Network request : initiate an HTTP request to a website or web page to obtain data such as HTML;
Web page parsing : parse HTML to form a structured tree structure, and obtain target data through XPath or CSS Selector;
Data storage : store the parsed structured data, which can be in the form of a database or a file;
URL management : manage the URL list to be crawled and the URL list that has been crawled, such as URL resolution and request for paging or list pages.

The above is the basic crawler technology modules. For a large crawler system, it is also necessary to have the necessary modules for the production environment such as task scheduling, error management, and log management. The author's crawler management platform Crawlab is a crawler management platform for enterprise-level production environments. In addition, for some anti-crawling measures, such as verification code or IP blocking, additional modules are usually required, such as verification code identification, IP proxy, etc.

However, at present, the development of crawler programs is mainly focused on webpage parsing, which consumes a lot of manpower. Although HTML needs to be parsed into the webpage data, the layout, format, style, and content of different websites are different, so each website and webpage needs separate parsing logic, which greatly increases the cost of manual coding. Although some general-purpose crawlers such as search engine crawlers do not need to write too much parsing logic, such crawlers usually cannot focus on data extraction of specific topics. Therefore, in order to reduce the cost of manual writing, it is best to automatically extract web page data without writing or writing a small amount of parsing logic, which is the main goal of intelligent crawlers.

Known Implementations

It is not easy to implement intelligent web page extraction, but there are already some attempts to develop intelligent crawlers. Among them, GNE (GeneralNewsExtractor) developed by Kingname is an open source implementation of webpage text extraction, based on text and punctuation density extraction algorithms. GerapyAutoExtractor developed by Cui Qingcai implemented webpage list page recognition based on list cluster and SVM algorithm. Octoparse, a commercial client software, has developed an automatic list recognition module. Diffbot is an API-based intelligent web page recognition platform, with a very high recognition accuracy rate, claiming to be 99%. Known smart crawler implementations are currently based mainly on the HTML structure and content of web pages, such as GNE and GerapyAutoExtractor. For commercial software such as Octopus and Diffbot, we cannot know the specific implementation method.

Explore List Page Recognition

Now the accuracy of text recognition is very high, and there are many technical implementations and applications. Here we mainly focus on the identification of list pages, which is the web page parsing work of many crawlers.

We can infer from experience how to automatically identify desired content. Humans are visual animals. When we see a web page with a list of articles, we will immediately recognize the list of articles without any surprises, as shown in the figure below. But how exactly do we recognize it? In fact, we naturally group the article list items of the same category into one category. So, we'll quickly realize that this is actually a list page. Of course, why are these list items similar? We can see that the child elements in these lists are also similar, so it's natural to tie them together. The individual sub-elements add up to a single list item, which our brains automatically group them together. This is the process of listing page recognition.

Based on such an analysis, it is actually very easy to think of the clustering algorithm in machine learning. All we need to do is to extract the characteristics of each node on the webpage, and then use the clustering algorithm to filter out the nodes of the same category. Of course, the feature selection here needs to be considered. Instead of simply looking at a single node of HTML, we need to associate it with other nodes to extract features, so that we can get some nodes of different categories. Then, we can filter out the desired list page according to the overall information of the node cluster.

Of course, it is not an easy task to actually implement such an algorithm with code. It is necessary to model and vectorize each node of HTML, and build a tree-like graph based on them. This is very tedious thing. Fortunately, the author has used sklearn, networkx and other libraries to implement a basic list page recognition system Webspot, which can automatically recognize list elements on a list page, and can visually display the recognition results, as shown in the figure below.

For most listings, Webspot's ability to identify is good. Although it is not as accurate as Diffbot, it can still be accurately identified for pages that are not very complicated.

So, why invent a new wheel when there is already a list page identification solution like Diffbot? One of the most important reasons is that commercial and high-accuracy software such as Diffbot cannot directly provide reusable extraction rules such as XPath and CSS Selector. Extraction rules for automatic identification are all we need. Then, by integrating into open source crawlers such as Scrapy and Colly, the cost of data capture can be greatly reduced. This is also a feature that Webspot can currently bring to users. It is not only able to identify list page elements and corresponding fields, but also provide extraction rules, as shown in the figure below.

With such an extraction rule, data can be automatically extracted from similar web pages only by automatic identification once.

Currently, Webspot is still in the early stages of development, and there should be more new features and algorithm development and optimization in the future.

Future Development

Intelligent crawlers are equivalent to autopilot on web pages, allowing crawlers to obtain the desired data or information as required without too many manual operations. This is an ideal technology for many data demanders and crawler engineers. However, intelligent crawlers are not yet mature at present, and the existing implementations and technologies are relatively simple. In the future, technologies such as deep learning and reinforcement learning may be used to improve the recognition ability of intelligent crawlers. In addition, the combination of graph theory and artificial intelligence, along with visual technology, may allow intelligent crawlers to achieve breakthroughs in accuracy. The author will continue to explore on intelligent crawlers through the Webspot project to solve the cost problem of data extraction. Those who are interested in the development of intelligent crawlers please feel free to contact me on GitHub "tikazyq".

Talking Architecture: What skills should architects have apart from drawing architecture diagrams?

Marvin Zhang — Mon, 14 Nov 2022 10:37:25 +0000

Introduction

"Architecture is about the important stuff... whatever it is." -- Ralph Johnson

Architect is a position with powerfulness and respect. When you hear that someone is an architect of a company, will you feel a sense of awe? Architects are usually believed to be relevant to system design, technical strength, leadership and influence. It is precisely for this reason that many of the positions of architects in the enterprise are held by experienced and skilled senior software engineers. However, the definition of an architect in the software industry is not quite clear: cloud service providers such as Amazon and Alibaba Cloud have their own teams of architects, but most of them provide after-sales service to customers under the name "architect"; Some architects are nothing more than using his rich experience and excellent strength to solve technical problems, which is equivalent to what a senior software engineer can do. These are very different from the omniscient architect who designs the architecture diagram.

The concepts related to architects in this article mainly come from a book Fundamentals of Software Architecture (authored by Mark Richards, Neal Ford) that I have been reading recently. This article will briefly introduce about what a pragmatic architect should do and required skills.

Chief Engineer

First of all, an architect should be a chief engineer of the entire software project, who is responsible for the overall design, implementation and quality of the software project. Therefore, for software architects, they need to have excellent programming skills, good understanding of the software project development process, and a certain breadth and depth in various technical areas. Not only that, because the architect is the chief person in charge of the technology side, he (or she) usually needs to think it through, from a perspective of the system as a whole, about how the various modules interact, whether the division of functional services is reasonable, where the bottleneck of the whole system will be, and so on. These are all in the technical area.

When a software engineer has been engaged in many projects, he or she should have some deep understanding of software engineering and architecture, and with some system learning, he or she could ultimately become a qualified architect. Therefore, with technical consideration only, a software architect is equivalent to a senior software engineer.

Is Architecture Diagram Necessary?

You may have seen more or less some architecture diagrams: hierarchical architecture diagram, physical network topology diagram, flow diagram, interaction logic diagram, etc. But to answer the question that whether the architecture diagram is necessary, we need to understand the reason behind it. The main purpose of the architecture diagram is to help internal engineers or external technicians quickly understand the system module information contained in it. Therefore, those seemingly cool, professional and pleasing architecture diagrams, if lacking in meaningful content, would not be helpful for understanding. Instead, it is the architecture diagram which is simple, clear and easy to understand, even if looking ugly, that will be helpful for software developers. Therefore, the architecture diagram is necessary, but on condition that it reveals the system module information with simpleness, conciseness, and clearness.

Not Only Tech

As mentioned earlier, the architect, as the chief engineer of software engineering, will take technical responsibilities. But in reality, few architects can work on a project smoothly if they only focus on the technical bits. A successful architect, or an influential architect, needs at least good leadership or team management skills.

Half of the eight skill requirements for architects in Software Architecture Foundation are not directly related to technology. Let's take a look at the technical and non-technical requirements mentioned in this book.

Technical Requirements
- Make architecture decisions
- Continually analyze the architecture
- Keep current with latest trends
- Diverse exposure and experience
Non-Technical Requirements
- Ensure compliance with decisions
- Have business domain knowledge
- Process interpersonal skills
- Understand and navigate politics

You may be surprised at the non-technical requirements for architects, such as why architects need to consider politics? As a young junior developer, I did not understand at first, but with the continuous accumulation of project experience, I found that many architectural decisions were not only from the technical rationality of the architecture itself, but also related to corporate politics. The politics we mentioned here is not bureaucracy in the traditional sense, but more referring to the cooperation or competition between various departments. This is particularly important for large companies. I will not elaborate in detail here, but may write an article later for this topic.

Continuous Improvement

Unfortunately, many architects promoted from software engineers continue to stick with their technical skills, ignoring non-technical things. Therefore, it is emphasized in this article that the responsibilities and requirements of architects should not only be restricted to technology and architecture themselves, but also more importantly focused on non-technical skills and experience accumulation. For example, how do we balance compliance, resources and development efficiency? What business value does your software project provide and what impact it will have? How to explain professional terms to non-technical personnel (especially the boss)? When external departments refuse to cooperate, how do we navigate to ensure the project success? Such issues require non-technical skills and experience significantly, and architects who would like to pursue their career goal should take them as necessity. When you step out of your comfort zone, your are probably more like to enjoy continuous learning and growing.

Go Project Source Code Analysis: Schedule Job Library "cron"

Marvin Zhang — Sat, 12 Nov 2022 03:22:06 +0000

Introduction

There are many excellent open-source projects on GitHub, where the code is transparent and available to everyone. As software developer, we can learn a lot from them including software engineering, unit testing, coding style standardization, etc. We can even find issues by looking into their code, and submit pull requests to contribute to tech communities. Today we are going to dig into the source code of a popular Golang open-source project on GitHub, robfig/cron, which is small and with clear annotations, and is very suitable for new developers to learn how to read and analyze source code.

Environment Preparation

First, we are going to fork the project into our personal repo. We can do so by clicking the Fork button and enter the project name. After it's forked, we can either download to our local, or click Create codespace on master to create a Codespace on the GitHub repo's home page. Codespace is a Azure-based GitHub service that allows developers to develop remotely, and it is now available to individuals. Let's give it a try!

After clicking, a new page will be opened in the browser, where there is a web interface of VS Code, displaying the directory, code and terminal of the project. See the figure below.

As our objective is to analyze source code, we will use it to display code for reading, instead of executing it.

Now, we can start reading and analyzing.

Entry File

A good way of analyzing source code is find the entry file, which is like a book's introduction, where the overall project structure will show up.

We can find in README.md file that the usage of this library is something like cron.New(cron.WithSeconds()), or the method cron.New. Therefore, we reckon that this method is in the file cron.go. Let's open it and take a look.

A quick go-through can allow us to locate the New method at Line 113, as below.

If we take a close look, it is basically a pointer to the instance of the Cron struct. The parameters opts ...Option is called Functional Option. The actual implementation is to build an instance pointer c of the struct Cron, applied by functional options, and return it.

With above analysis, we can conclude that the main logic is inside the Cron struct.

Regardless to say, we can be sure that the entry file is cron.go. What we can do next is analyze the core modules of this file to fully understand the source code.

Core Struct/Class

Now let's take a look at the structure of the core struct Core to find some enlightenment.

We can locate the struct Cron at Line 13.

There are quite many fields in the struct Cron, including the lower-case private attributes entries, chain, parser, but we can hardly know their actual meanings and functionalities. According to the comments at Line 10-12, we can deduct that it is used for tracking entries and executing functions defined by schedule. However, we are still not quite sure about what it does exactly, so only through reading the source code further, we can understand more details.

We can also find 3 interfaces just under beneath with their descriptions:

ScheduleParser: parser of the schedule, which parses specs and returns Schedule instance.
Job: submitted scheduled jobs.
Schedule: used for describing running cycles of jobs.

In fact, these 3 interfaces are very important, given their locations in the source code.

Entry Method

Before we explore further, let's recall the usage of the library apart from cron.New: it takes effect after calling c.Start(). Therefore, we need to dig into the Start method in Cron, which is actually the entry method of the core struct.

We can locate the Start method at Line 215 in cron.go as below.

An experienced Go developer could notice this is an atomic operation. c.runningMu is an instance of sync.Mutex, which can lock at the beginning, and unlock after the function execution finishes. This technique avoids data racing in the repeated calls of this method. if c.running { return } denotes that the method won't go further and return immediately if it's already started. c.running is set as true. The last line is an essential statement, go c.run(), meaning it starts a goroutine to asynchronously execute c.run. Now we have found the real core method, and the rest of the work is to analyze it through.

Is it similar to we playing RPG games and unlocking all the milestones and entering the next episode?

Conclusion

Wait! Is it over?

The reason why we stopped here is that I want to keep the article short. Reading and analyzing source code a tedious and boring process, which might prevent readers from going further. As a result, the objective of this article is to introduce the right and basic techniques of analyzing source code to readers, allowing you guys to do the analysis by yourselves. This way can make you learn faster and with more interest.

Now let's summarize the techniques for analyzing source code:

Find the entry file
Locate the core struct/class
Analyze the entry method

Talking Data: Why data governance is so important in digital transformation?

Marvin Zhang — Sun, 30 Oct 2022 03:15:40 +0000

Introduction

"Organisms are living upon negentropy."--What is Life? --by Erwin Schrödinger

In today's Internet era, data is an important asset for enterprises. We generate data all the time: every time we open a mobile app, place an order online, or even drive through traffic lights, data will be generated. Data is everywhere. This is especially true in enterprises. With so much raw data and increasingly mature data analysis technology, entrepreneurs are excited, because this is the gold mine piled up for companies. However, things are not as simple as imagined. It is not easy to extract valuable treasures from these messy so-called "gold mines" that look like garbage dumps. In my previous article Talking Data: What do we need for engaging data analytics?, the concept of Data Governance was mentioned. This article will introduce how data governance creates value in the chaos of enterprise data from the perspective of enterprise data management.

Isolated Data Island

Large and medium-sized enterprises (generally more than 100 people with multiple departments) will encounter the problem of management chaos while their business grows fast. The sales department has its own sales statistics, typically in the form of large and scattered Excel files; The IT department manages the inventory systems by themselves; The HR department maintains an entire personnel statistics list. This situation will lead to many headaches: the boss often complains that he or she can only receive management reports every week; Managers looked at the up and down data in the report, doubting the data integrity; employees work overtime to sort out the data for the reports, but they are often questioned about data quality. Sounds familiar? These are very common issues in companies. The direct cause is the so-called Isolated Data Island problem.

The main reason for isolated data island issues is that data from various departments or teams is disconnected. Many times, because of the rapid growth of business, some teams need to quickly set up a data system, but are incapable of developing it in time, so they can only use Excel or some fast and effective data entry tools to ensure the efficiency of business operations. With the growth of business, these hacking workarounds continue to derive new internal processes, and bottlenecks will gradually emerge after reaching a certain scale, especially when integration with other departments or external systems is required.

When bosses become aware of the isolated data island issue and decide to make a change, they will find it difficult to move forward most of the time. This is because they have many problems to solve, such as:

Format and Standard: The monthly sales data of the marketing department may be stored in various Excel files, while the product department is stored in the SQL Server database.
Statistical Caliber: The minimum statistical caliber of the accounting and financial department is by month, but the marketing department is by a single order.
Association: The marketing department uses the order number, while the product department uses the database order ID.
Scale: Thousands of Excel files with different formats are difficult to be efficiently processed.

Data Governance

The so-called data governance is actually to solve the problem of data islands through data integration tools. As shown in the figure below, the data governance team need to integrate, clean, transform, and store the business data of each department in the Data Warehouse, which will eventually be used by the entire organization. Through data governance and data warehouse, data islands of various business departments will be connected. After some sort of association and integration of the data, various data sources can generate many insights. For example, the marketing team will be surprised to find that their main customers rarely use APP!

However, the above figure is a simplified process. In the real world, data governance in a large company or organization will be more complex, which normally involve organizational, cultural, political, business nature and other factors. This is why many large and medium-sized enterprises, especially those accustomed to traditional Excel reports, will encounter obstacles in digital transformation.

For the data governance team, the key to success is not the technology, but the familiarity and understanding of business process and enterprise culture. There are ready-made tools and techniques for data cleaning and data transformation, but business processes and corporate culture are often quite subtle. This requires the data governance team to have good communication skills and business knowledge. The success of data governance requires not only practical tools and technologies, but also wisdom in dealing with various business teams.

Keep improving

The business development cannot live without data insights, but to achieve data driven decision making, communication and collaboration between teams and departments are often necessary. This is why many companies have started to adopt Agile. Here you can refer to the author's previous article Talking Agile: Are you sure your team is practicing Agile properly?. What data governance needs to do more is to understand the business, think about how to use data integration to optimize the business, and create more value from data insights.

Golang in Action: How to implement a simple distributed system

Marvin Zhang — Wed, 26 Oct 2022 05:25:43 +0000

Introduction

Nowadays, many cloud-native and distributed systems such as Kubernetes are written in Go. This is because Go natively supports not only asynchronous programming but also static typing to ensure system stability. My open-source project Crawlab, a web crawler management platform, has applied distributed architecture. This article will introduce about how to design and implement a simple distributed system.

Ideas

Before we start to code, we need to think about what we need to implement.

Master Node: A central control system, similar to a troop commander to issue orders
Worker Node: Executors, similar to soldiers to execute tasks

Apart from the concepts above, we would need to implement some simple functionalities.

Report Status: Worker nodes report current status to the master node.
Assign Task: Client makes API requests to the master node which assign tasks to worker nodes.
Execute Script: Worker nodes execute scripts from tasks.

The overall architectural diagram is as below.

Action

Node Communication

The communication between nodes are very important in distributed systems. If each node runs individually, it will be not necessary to use a distributed system. Therefore, the communication module is an essential part in distributed systems.

gRPC Protocol

First, let's think about how to make nodes to communicate with each other. The most common way is API, which yet has a drawback that it requires nodes to expose their IP addresses and ports to others, which is very insecure particularly in the public network. In light of that, we chose gRPC, a popular Remote Procedure Call (RPC) framework. We won't go deep into its technical details, but it is actually a mechanism to allow remote machines to execute commands by RPC callers.

To use gRPC, let's first create a file named go.mod and enter below content, then execute go mod download.

module go-distributed-system

go 1.17

require (
    github.com/golang/protobuf v1.5.0
    google.golang.org/grpc v1.27.0
    google.golang.org/protobuf v1.27.1
)

Then we create a Protocol Buffers file node.proto, a gRPC protocol file, and enter below content.

syntax = "proto3";

package core;
option go_package = ".;core";

message Request {
  string action = 1;
}

message Response {
  string data = 1;
}

service NodeService {
  rpc ReportStatus(Request) returns (Response){};       // Simple RPC
  rpc AssignTask(Request) returns (stream Response){};  // Server-Side RPC
}

Here we have created two RPC service methods: one for reporting current status with a Simple RPC, another for assigning tasks with a Server-Side RPC. The difference between Simple RPC and Server-Side RPC is that Server-Side RPC can allow the server (master node in our case) to send data to the client (worker node) through a stream, but Simple RPC can only allow clients to call servers.

After .proto file is created, we need to compile it into .go code file so that it can be used by the Go program. Let's execute the command as below. (Note: the compiler protocol is not built-in and needs to be downloaded, please refer to https://grpc.io/docs/protoc-installation/)

mkdir core
protoc --go_out=./core \
    --go-grpc_out=./core \
    node.proto

After it is executed, you can see two Go code files under the directory core, node.pb.go and node_grpc.pb.go respectively, which serve as the gRPC library.

gRPC Server

Now let's start writing server-side code.

Firstly create a new file core/node_service_server.go, and enter the content below. It implemented the two gRPC service methods created before. The channel CmdChannel in particular would transfer commands to be executed on worker nodes.

package core

import (
    "context"
)

type NodeServiceGrpcServer struct {
    UnimplementedNodeServiceServer

    // channel to receive command
    CmdChannel chan string
}

func (n NodeServiceGrpcServer) ReportStatus(ctx context.Context, request *Request) (*Response, error) {
    return &Response{Data: "ok"}, nil
}

func (n NodeServiceGrpcServer) AssignTask(request *Request, server NodeService_AssignTaskServer) error {
    for {
        select {
        case cmd := <-n.CmdChannel:
            // receive command and send to worker node (client)
            if err := server.Send(&Response{Data: cmd}); err != nil {
                return err
            }
        }
    }
}

var server *NodeServiceGrpcServer

// GetNodeServiceGrpcServer singleton service
func GetNodeServiceGrpcServer() *NodeServiceGrpcServer {
    if server == nil {
        server = &NodeServiceGrpcServer{
            CmdChannel: make(chan string),
        }
    }
    return server
}

gRPC Client

We don't have to care too much about the implementation of gRPC client. Instead, we only need to call the methods in the gRPC client, and the rest of the service requests and response will be handled automatically by the program.

Master Node

After we implemented the node communication part, we can now write the master node, which is the core of the distributed system.

Let's create a new file node.go and enter the content below.

package core

import (
    "github.com/gin-gonic/gin"
    "google.golang.org/grpc"
    "net"
    "net/http"
)

// MasterNode is the node instance
type MasterNode struct {
    api     *gin.Engine            // api server
    ln      net.Listener           // listener
    svr     *grpc.Server           // grpc server
    nodeSvr *NodeServiceGrpcServer // node service
}

func (n *MasterNode) Init() (err error) {
  // TODO: implement me
  panic("implement me")
}

func (n *MasterNode) Start() {
  // TODO: implement me
  panic("implement me")
}

var node *MasterNode

// GetMasterNode returns the node instance
func GetMasterNode() *MasterNode {
    if node == nil {
        // node
        node = &MasterNode{}

        // initialize node
        if err := node.Init(); err != nil {
            panic(err)
        }
    }

    return node
}

There are two placeholder methods Init and Start to be implemented.

In the initialization method Init, we will two things:

Regster gRPC services
Register API services

Now, we can add below code in Init method.

func (n *MasterNode) Init() (err error) {
    // grpc server listener with port as 50051
    n.ln, err = net.Listen("tcp", ":50051")
    if err != nil {
        return err
    }

    // grpc server
    n.svr = grpc.NewServer()

    // node service
    n.nodeSvr = GetNodeServiceGrpcServer()

    // register node service to grpc server
    RegisterNodeServiceServer(node.svr, n.nodeSvr)

    // api
    n.api = gin.Default()
    n.api.POST("/tasks", func(c *gin.Context) {
        // parse payload
        var payload struct {
            Cmd string `json:"cmd"`
        }
        if err := c.ShouldBindJSON(&payload); err != nil {
            c.AbortWithStatus(http.StatusBadRequest)
            return
        }

        // send command to node service
        n.nodeSvr.CmdChannel <- payload.Cmd

        c.AbortWithStatus(http.StatusOK)
    })

    return nil
}

Here we created a gRPC server, and registered NodeServiceGrpcServer created before. We then use API framework gin to create a simple API service, which can allow POST request to /tasks to send commands to the channel CmdChannel and pass to NodeServiceGrpcServer. All pieces have been put together!

The starting method Start is quite simple, which is simplely to start the gRPC server and API server.

func (n *MasterNode) Start() {
    // start grpc server
    go n.svr.Serve(n.ln)

    // start api server
    _ = n.api.Run(":9092")

    // wait for exit
    n.svr.Stop()
}

下一步，我们就要实现实际做任务的工作节点了。

Worker Node

现在，我们创建一个新文件 core/worker_node.go，输入以下内容。

package core

import (
    "context"
    "google.golang.org/grpc"
    "os/exec"
)

type WorkerNode struct {
    conn *grpc.ClientConn  // grpc client connection
    c    NodeServiceClient // grpc client
}

func (n *WorkerNode) Init() (err error) {
    // connect to master node
    n.conn, err = grpc.Dial("localhost:50051", grpc.WithInsecure())
    if err != nil {
        return err
    }

    // grpc client
    n.c = NewNodeServiceClient(n.conn)

    return nil
}

func (n *WorkerNode) Start() {
    // log
    fmt.Println("worker node started")

    // report status
    _, _ = n.c.ReportStatus(context.Background(), &Request{})

    // assign task
    stream, _ := n.c.AssignTask(context.Background(), &Request{})
    for {
        // receive command from master node
        res, err := stream.Recv()
        if err != nil {
            return
        }

        // log command
        fmt.Println("received command: ", res.Data)

        // execute command
        parts := strings.Split(res.Data, " ")
        if err := exec.Command(parts[0], parts[1:]...).Run(); err != nil {
            fmt.Println(err)
        }
    }
}

var workerNode *WorkerNode

func GetWorkerNode() *WorkerNode {
    if workerNode == nil {
        // node
        workerNode = &WorkerNode{}

        // initialize node
        if err := workerNode.Init(); err != nil {
            panic(err)
        }
    }

    return workerNode
}

In the above code, we created the gRPC client and connected it to the gRPC server in Init method.

In Start method, we have done several things:

Report status with a Simple RPC method.
Assign tasks to acquire a stream with a Server-Side RPC method.
Continuously receive data from the server (master node) via the acquired stream and execute commands.

Now we have completed all core logics in the distributed systems.

Putting them all together

Finally, we need to encapsulate these core functionalities.

Create the main entry file main.go and enter the content below.

package main

import (
    "go-distributed-system/core"
    "os"
)

func main() {
    nodeType := os.Args[0]
    switch nodeType {
    case "master":
        core.GetMasterNode().Start()
    case "worker":
        core.GetWorkerNode().Start()
    default:
        panic("invalid node type")
    }
}

Now the simple distributed system is all done!

Final Results

We can then test the code.

Open two command prompts. Enter go run main.go master in one prompt to start the master node, and enter go run main.go worker to start the worker node in another.

If the master code starts successfully, you should be able to see the logs below.

[GIN-debug] [WARNING] Creating an Engine instance with the Logger and Recovery middleware already attached.

[GIN-debug] [WARNING] Running in "debug" mode. Switch to "release" mode in production.
 - using env:   export GIN_MODE=release
 - using code:  gin.SetMode(gin.ReleaseMode)

[GIN-debug] POST   /tasks                    --> go-distributed-system/core.(*MasterNode).Init.func1 (3 handlers)
[GIN-debug] [WARNING] You trusted all proxies, this is NOT safe. We recommend you to set a value.
Please check https://pkg.go.dev/github.com/gin-gonic/gin#readme-don-t-trust-all-proxies for details.
[GIN-debug] Listening and serving HTTP on :9092

For worker node, you can see logs like this.

worker node started

After the master node and worker node have all started, we can open another command prompt to execute the command below to make an API call.

curl -X POST \
    -H "Content-Type: application/json" \
    -d '{"cmd": "touch /tmp/hello-distributed-system"}' \
    http://localhost:9092/tasks

In the worker node logs, you should be able to see received command: touch /tmp/hello-distributed-system.

Then let's check if the file has been created by executing ls -l /tmp/hello-distributed-system.

-rw-r--r--  1 marvzhang  wheel     0B Oct 26 12:22 /tmp/hello-distributed-system

The file was successfully created, which means the worker node has executed the task successfully. Hooray!

Conclusion

This article introduced a way to develop a simple distributed system written in Golang, with gRPC and built-in Go channel.

Core libraries and techniques:

The code of the whole project is on GitHub: https://github.com/tikazyq/codao-code/tree/main/2022-10/go-distributed-system

CI/CD in Action: Manage auto builds of large open-source projects with GitHub Actions?

Marvin Zhang — Fri, 21 Oct 2022 05:55:40 +0000

Introduction

In the previous article about CI/CD in Action: How to use Microsoft's GitHub Actions in a right way?, we introduced how to use GitHub Actions workflows with a practical Python project. However, this is quite simple and not comprehensive enough for large projects.

This article introduces practical CI/CD applications with GitHub Actions of my open-source project Crawlab. For those who are not familiar with Crawlab, you can refer to the official site or documentation. In short, Crawlab is a web crawler management platform for efficient data collection.

Overall CI/CD Architecture

The new version of Crawlab v0.6 split general functionalities into separated modules, so that the whole project is consisted of a few dependent sub-projects. For example, the main project crawlab depends on the front-end project crawlab-ui and back-end project crawlab-core. Higher decoupling and maintainability are the benefits.

Below is the diagram of the overall CI/CD architecture.

The building process of the whole Crawlab project is a little bit trivial. The ultimate deliverable or the Docker image crawlabteam/crawlab depends on the main repository, which depends on the sub-projects of front-end, back-end, base images and plugins. They are come from their own repos, which again depend on upstream core-module repos. Here we have simplified the dependencies of front-end and back-end modules.

Front-End Building

We start with the front-end part.

The front-end repo crawlab-ui is distributed through NPM. Let's take a look at the CI/CD workflow.

name: Publish to NPM registry

on:
  pull_request:
    branches: [ main ]
  push:
    branches: [ main ]
  release:
    types: [ created ]

jobs:
  publish:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - uses: actions/setup-node@v2
        with:
          node-version: '12.22.7'
          registry-url: https://registry.npmjs.com/
      - name: Get version
        run: echo "TAG_VERSION=${GITHUB_REF#refs/*/}" >> $GITHUB_ENV
      - name: Install dependencies
        run: yarn install
      - name: Build
        run: yarn run build
        env:
          TAG_VERSION: ${{env.TAG_VERSION}}
      - if: ${{ github.event_name == 'release' }}
        name: Publish npm
        run: npm publish --registry ${REGISTRY}
        env:
          NODE_AUTH_TOKEN: ${{secrets.NPM_PUBLISH_TOKEN}}
          TAG_VERSION: ${{env.TAG_VERSION}}
          REGISTRY: https://registry.npmjs.com/

There are some important parts:

Set up Node.js environment uses: actions/setup-node@v2 and its version node-version: '12.22.7'
Install dependencies run: yarn install
Build the package yarn run build
Publish the package to NPM registry npm publish --registry ${REGISTRY}

The token for publishing NPM package is ${{secrets.NPM_PUBLISH_TOKEN}}, which is a GitHub secret configured by the repo owner, and private to the public for security reasons.

After the workflow is set up, a GitHub Actions workflow job will be automatically triggered once any code commit is push to crawlab-ui.

We barely need to take care of anything for NPM package publishing, because it is fully automated. Awesome!

Base Image Building

Let's see another special workflow: base image building. The GitHub repo is docker-base-images.

As the new published base image needs to be integrated into the final Docker image, we need to re-trigger a workflow job in crawlab once it is built. Let's see how this workflow is configured.

name: Docker crawlab-base

on:
  push:
    branches: [ main ]
  release:
    types: [ published ]
  workflow_dispatch:
  repository_dispatch:
    types: [ crawlab-base ]

env:
  IMAGE_PATH: crawlab-base
  IMAGE_NAME: crawlabteam/crawlab-base

jobs:

  build:
    name: Build Image
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2

      - name: Get changed files
        id: changed-files
        uses: tj-actions/changed-files@v18.7

      - name: Check matched
        run: |
          # check changed files
          for file in ${{ steps.changed-files.outputs.all_changed_files }}; do
            if [[ $file =~ ^\.github/workflows/.* ]]; then
              echo "file ${file} is matched"
              echo "is_matched=1" >> $GITHUB_ENV
              exit 0
            fi
            if [[ $file =~ ^${IMAGE_PATH}/.* ]]; then
              echo "file ${file} is matched"
              echo "is_matched=1" >> $GITHUB_ENV
              exit 0
            fi
          done

          # force trigger
          if [[ ${{ inputs.forceTrigger }} == true ]]; then
              echo "is_matched=1" >> $GITHUB_ENV
              exit 0
          fi

      - name: Build image
        if: ${{ env.is_matched == '1' }}
        run: |
          cd $IMAGE_PATH
          docker build . --file Dockerfile --tag image

      - name: Log into registry
        if: ${{ env.is_matched == '1' }}
        run: echo ${{ secrets.DOCKER_PASSWORD}} | docker login -u ${{ secrets.DOCKER_USERNAME }} --password-stdin

      - name: Push image
        if: ${{ env.is_matched == '1' }}
        run: |
          IMAGE_ID=$IMAGE_NAME

          # Strip git ref prefix from version
          VERSION=$(echo "${{ github.ref }}" | sed -e 's,.*/\(.*\),\1,')

          # Strip "v" prefix from tag name
          [[ "${{ github.ref }}" == "refs/tags/"* ]] && VERSION=$(echo $VERSION | sed -e 's/^v//')

          # Use Docker `latest` tag convention
          [ "$VERSION" == "main" ] && VERSION=latest

          echo IMAGE_ID=$IMAGE_ID
          echo VERSION=$VERSION

          docker tag image $IMAGE_ID:$VERSION
          docker push $IMAGE_ID:$VERSION

          if [[ $VERSION == "latest" ]]; then
            docker tag image $IMAGE_ID:main
            docker push $IMAGE_ID:main
          fi

      - name: Trigger other workflows
        if: ${{ env.is_matched == '1' }}
        uses: peter-evans/repository-dispatch@v2
        with:
          token: ${{ secrets.WORKFLOW_ACCESS_TOKEN }}
          repository: crawlab-team/crawlab
          event-type: docker-crawlab

As you can see in the workflow, the last step name: Trigger other workflows will trigger another GitHub Actions workflow job in another GitHub repo crawlab-team/crawlab through peter-evans/repository-dispatch@v2, a re-usable action. That means, if we make modifications in the base image code and push the commits, the base image will be built automatically before it triggers another workflow job in the repo crawlab to build the final image.

This is so great! We can sit down and take a coffee, waiting for the job to finish, instead of doing any manual work.

Conclusion

Today we introduced the use of GitHub Actions in the large open-source project Crawlab along with its automatic building process and overall CI/CD architecture. Overall, GitHub Actions supports the CI/CD integration of large projects quite well.

Techniques used:

Automatic triggers to build
Publish NPM packages
Repo secrets
Trigger workflows in other repos

The code of the whole project is in the repos of Crawlab on GitHub and publicly available.

Talking Algorithm: The hidden secret of nature in the divide-and-conquer algorithm

Marvin Zhang — Tue, 18 Oct 2022 06:57:01 +0000

Introduction

"The empire, long divided, must unite; long united, must divide. " -- The Romance of the Three Kingdoms

It is hard to deny the importance of algorithms in our modern IT industry. A well designed algorithm can run software programs with the minimal resources in the most efficient way. Therefore, algorithms are important, so that IT companies set high standards in the hiring process. Think about the algorithm tests in the technical interviews. Many people might feel that algorithms are quite distant from us, but I think its efficiency enhancement comes naturally, as the reason behind can be found in nature.

From a snowflake

As we all know, a snowflake is beautiful, because of not only its shining appearing, but also the shape. It looks like a polished hexagon, and each branch is a snowflake-like sub-hexagon. The term for this structure with recursive self similarity is Fractal. Fractals are so common in nature, such as tree roots and tree leaf branches, pulmonary capillaries, natural coastal lines, or even broken glasses.

So, why? What is the reason that fractals are so common in nature? Is it the design from Gods, or some fundamental mathematical laws behind it? Academic researchers believe in the latter. According the thermodynamics, snowflakes are formed when water vapor encounters an abrupt drop in temperature; according to fluid mechanics, the tree-like structure of pulmonary capillaries would allow the oxygen to be absorbed by red blood cells. In conclusion, fractals exist for efficiencies.

Divide-and-conquer algorithm

Now let's go back to the algorithm. For those who are familiar with algorithms should have known the techniques using the divide-and-conquer methodology. Its fast speed comes from the process where a large piece is recursively divided into the most granular parts and then merged back, or the self-similar recursion. It is just like the fractal theory. Another example of fractal principles is the balanced tree (B-tree) database index, which transform unstructured data layer-by-layer into a tree-like structure, and ultimately make database querying much faster. The process of optimizing B-tree indexes is identical to that of optimizing the querying efficiency. The math expression LogN you often see in the complexity analysis, comes from the sum of arithmetic sequence in fractals. You can try deriving the equations through apply your mathematical knowledge from middle school.

Is the world a fractal?

The fractal theory is a new scientific research area, and many scientists associate it with the theory of complex systems. Many complex systems are formed from some simple matters or laws. For example, ant colonies are like intelligent living individuals who can do loads of smart things, but only replying on very little information hormones. Life is made of cells, and cells come from DNAs, and DNAs are formed by micro particles including molecules, protons, and neutrons. Everything in the world is made up by these physical models. Just like the English poet put it, "To see a World in a Grain of Sand, And a Heaven in a Wild Flower".

Although the fractal theory explains most of the formation in nature, it is still imperfect. As we are amazed by the powerfulness of the fractal theory, we should realize that it is not ideal, and we should explore as we see.

CI/CD in Action: How to use Microsoft's GitHub Actions in a right way?

Marvin Zhang — Fri, 14 Oct 2022 03:32:17 +0000

Introduction

GitHub Actions is the official CI/CD workflow service provided by GitHub. It is aimed at making it easy for open-source project contributors to manage operational maintenance, and enable open-source communities to embrace cloud-native DevOps. GitHub Actions is integrated into most of my open-source projects including Crawlab and ArtiPub. As a contributor, I think GitHub Actions is not only easy to use, but also free (which is the most important). Therefore, I hope this article will allow open-source project contributors who are not familiar with GitHub Actions, to really get ideas on how to utilize it and make an impact.

Starting from documentation

For those who are not familiar with GitHub Actions, it is strongly recommended that you read the official documentation first, where you can find Introduction Video, Quick Start, Examples, concepts, how it works, etc. If you read through the docs, you can easily do GitHub DevOps with your own experience in CI/CD. References of all codes in this article can be found on the official documentation,

Ideas

Let's first figure out what we would like to implement, i.e. using GitHub Actions to run a web crawler to get daily ranking from GitHub Trending.

Steps of implementation:

Upload crawler code to GitHub repo.
Create and commit a GitHub Actions workflow.
Trigger the GitHub Actions workflow to run the crawler.
Check out the status of crawling tasks.

Crawler code

Now we can start coding!

Let's first upload the code to GitHub repo. As our topic today is focused on GitHub Actions, we are not going to dig deep into the code details.

The code is as below:

# main.py
import requests
from bs4 import BeautifulSoup


def main():
    res = requests.get('https://github.com/trending')
    sel = BeautifulSoup(res.content)
    rows = sel.select('article.Box-row')
    for row in rows:
        repo_name = '/'.join(map(lambda x: x.strip(), row.select('h1 a')[0].text.strip().split('/')))
        description = row.select('p')[0].text.strip()
        stars = row.select('a.Link--muted')[0].text.strip()
        print(f'{repo_name} ({stars} stars): {description}')


if __name__ == '__main__':
    main()

The first workflow

We can click on the Actions tab in the GitHub repo page.

Then you can see the welcome page as above, which indicates there is no workflows in your repo yet. You may also find there are some introductions as well as popular template workflows, where you can click the Configure button to create a new workflow.

We can search the keyword Python, then we can find a workflow that runs Python programs. Let's use it.

After clicking the Configure button, you will enter the page as below.

A GitHub Actions workflow is actually a YAML configuration file, similar to popular concepts like PaaS, IaaS, cloud-native applications, which automate configurations through codes.

There is already some default workflow code, and we can slightly modify it.

name: GitHub Trending Crawler

on:
  push:
    branches: [ "main" ]
  pull_request:
    branches: [ "main" ]

permissions:
  contents: read

jobs:
  build:

    runs-on: ubuntu-latest

    steps:
    - uses: actions/checkout@v3
    - name: Set up Python 3.10
      uses: actions/setup-python@v3
      with:
        python-version: "3.10"
    - name: Install dependencies
      run: |
        cd 2022-10/github-actions-python
        python -m pip install --upgrade pip
        if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
    - name: Run
      run: |
        cd 2022-10/github-actions-python
        python main.py

Click the Commit button on the web UI, and you can find that the workflow configuration file github-trending-crawler.yml is generated on the directory .github/workflows .

Overall, the workflow above has done several things:

Checkout the repo's code
Set up Python environment
Install dependencies
Run the program

View task status

As you submit the workflow commit, a CI/CD task will automatically be triggered given the default triggering conditions.

on:
  push:
    branches: [ "main" ]
  pull_request:
    branches: [ "main" ]

It means the workflow will be auto-triggered once we push commits or create pull requests.

Now if go to the Actions tab to check out the running status, you can find a list of workflow runs.

Let's click the latest one and click the build job, you can see relevant logs. Now click to expand the details of the step Run.

As you can see, the top repos on GitHub Trending are already printed out in the logs, as expected. The only thing left is to validate if it is in line with the actual ranking.

The content on the page is exactly the same as the output results are exactly. That's awesome, all done!

Scheduled workflow

How can we run crawlers without schedules? And here we are, GitHub Actions supports that just fine. Let's add it with pleasure.

Open the edit panel of the created workflow, and add some code in the trigger part as below.

on:
  push:
    branches: [ "main" ]
  pull_request:
    branches: [ "main" ]
  schedule:
    - cron:  '0 * * * *'

The cron expression 0 * * * * means triggering at 0 every hour. For those who are unfamiliar with cron expression, please refer to Cron Guru.

After the edit is committed, we can see the hourly executions of the crawler's scheduled tasks.

Conclusion

We introduced how to use GitHub Actions workflows to deploy a simple web crawling schedule task, with techniques as below.

GitHub repo
GitHub Actions workflows, including scheduled triggering
Python crawler development, including dependencies requests and bs4

The code of the whole project is on GitHub: https://github.com/tikazyq/codao-code/tree/main/2022-10/go-task-scheduler

Talking Testing: the love and hate of Unit Tests

Marvin Zhang — Tue, 11 Oct 2022 10:13:05 +0000

Introduction

"No code is the best way to write secure and reliable applications."--Kelsey Hightower

Many developers have probably more or less heard of Unit Tests, and even written one and are familiar with it. However, in the volatile and fast changing environment, unit tests seem to be in an embarrassed situation. Developers know it is useful, but treat it with neglect. "The schedule is tight. What time do we have for unit tests?" Does it sound familiar?

What is Unit Test?

Unit Test is some lines of code written by developers to validate whether their own functional codes can run as expected. If the code is not passed, it means the functional codes are problematic.

This self-testing method looks self-deceiving, similar to taking an exam with official answers. In validation area, this term is called White Box Test. The counterpart of White Box Test is Black Box Test which uses other methods to validate things. Unit Test is White Box Test, and higher-level testing methods such as Integration Test, End to End Test, and UI Test are all Black Box Tests. Unit Tests only test code itself.

What are the benefits of unit testing?

Unit testing is a very useful tool in Agile Development. Some agile frameworks, such as eXtreme Programming (XP), requires that every feature must be covered by unit test cases. My previous article Talking Agile: Are you sure your team is practicing Agile properly mentioned the importance of unit tests.

Overall, unit tests have the main benefits as below:

Sustainable Quality Assurance: Unit tests can make sure functional codes working as expected after being changed or refactored.
Automatic Integration: Unit tests are normally integrated in CI/CD pipelines, and will be triggered once the code is committed.
Feature Documentation: Unit test cases can help new maintainers to get familiar with features and validation criteria.

Therefore, it looks like unit tests can bring a decent amount of benefits to software development.

Why give up unit testing practice?

Unit testing can improve our product quality and test efficiency, so why so many developers are not fond of writing unit tests? According to JetBrains, only 57% of respondents write unit tests, and only 35% will integrate automatic tests in most of their projects.

So why is that? There are several possible reasons:

More work: "I need to write 100 lines of code to test a feature with only 50. How can I get it done without overtime?"
Self-confidence: "As a senior developer, can you imagine there are bugs in my code?"
Not useful: "Do I really need to write unit tests for a limited number of pages?"
Dedicated testers: "Here you have QAs. Aren't they the guys to find bugs?"

However, they are not strong enough to be correct. Firstly, the amount of codes for bug fixing is way more larger than that. Apart from that, statistically speaking, there is a chance for a highly skilled developer to make mistakes if they write enough code. Furthermore, simple modules referenced by many other ones can become too important to fail. Finally, the responsibility of QA is ensure the overall quality instead of the locality of robustness, and have you seen any high-quality products full of buggy components?

To be farsighted

In fact, the main reason for the issue is the mindset. Most of the time developers tend to think from an individual and short-term perspective, instead of considering long-term benefits. As a good developer, he or she should develop something that is of most value in a most efficient way. Therefore, the fact that unit tests cannot bring visible benefits in a short period, makes it neglected by the majority.

Unit testing is like reading books, which cannot quickly make people knowledgeable, wealthy or famous, but can be significantly effective in the long run. If unit testing becomes the corporate culture, it is much easier to deliver high-quality products and services. Promoting it requires something more: either fundamental agile processes like XP or TDD, or high-level management such as CTO or architects.

Talking Data: What do we need for engaging data analytics?

Marvin Zhang — Thu, 06 Oct 2022 15:58:46 +0000

Introduction

"According to incomplete statistics, the proportion of white hair of data workers is higher than the average of the same age group." by a data worker

Data, a familiar but mysterious word, has become a totem pursued by everyone. Managers love fancy data reports, data analysts are keen on building complicated statistical models, and salesmen take dashboards as compasses to see whether they can complete their KPIs. Since over ten years ago, the data industry has been developing fast, and there have been some novel yet formidable jargons, such as Big Data, Data Science, Data Lake, Data Mesh, Data Governance. Yet the "traditional" terms are still abstruse: Data Warehouse, Business Intelligence, Data Mart, Data Mining. What is more headachy is that many people are still unable to understand their relationship with recently popular concepts such as Artificial Intelligence, Machine Learning, and Deep Learning. These hot buzzwords are the results of aggressive development in the data area.

Professional Doctor or Fortune Teller?

Years ago, with the rapid development of the Internet industry, the bubble of the data industry was getting larger. Data, the by-product of the Internet applications, has large volumes and diversities. Data owners would like to get the most out of it and regard it as the gold mine. Therefore, data mining engineers became one of the most popular professionals. Later, a brand new yet more popular position Data Scientist emerged as "the sexiest job in the 21st century".

The popularity of data scientists is its requirement for abilities and experience in various areas:

Programming Skills: at least able to use Python or R to do data cleansing, analysis and modeling.
Mathematics and Statistics: familiar with probability theory, calculus, and discrete mathematics.
Business Knowledge: deep understanding of market, process and macro trends in related areas.
Communication Skills: able to convey insights and analysis results in a human-friendly way.

Therefore, the sexiness of data scientist comes from its high barriers, because people who are excellent in all above are quite rare. However, even for the versatile talents, many data science projects would ultimately fail. The 2 major issues are scale and quality. According to CrowdFlower, for data science projects in 2016, 80% of the time was spent on data collection and data cleansing, whilst only 20% was spent on analysis and modeling. This is a huge waste.

As most system architectures of enterprises are unable to support large-scale and high-quality data processing pipelines, so some work has to be manually done, just like the so called "human intelligence". There are their data models with low prediction accuracy, so their data scientists are labelled as "quacks" or "fortune tellers". To become a true "professional doctor", you need not only the "professional medical knowledge" (core abilities), but also the support from "professional medical equipment" (architecture and process).

Where is the way？

Many data workers are complaining about the fierce competition in the data area. Fortunately, the situation seems to be improving. Data analysts had to manually analyze distribution charts for deep insights, but now they can use smart machine learning models to automate this process. Traditional data analysis and modeling skills have been gradually becoming easy. For instance, Power BI or Tableau allow users to use a drag-and-drop low-code fashion to generate visual charts and models, whilst the old way is to import Python libraries such as pandas, matplotlib and sklearn to do the same in Jupyter Notebook. Open-source projects Apache Superset and Metabase allow users to easily analyze data on the web pages. This is quite similar to the development of digital cameras, from the film cameras to digital cameras and to smartphone cameras used by everyone. With lower and lower technical barriers, the whole industry can be developing fast. "Everyone can be data analyst" will no longer be a fantasy.

However, data quality is still an issue. Although we can automatically fill missing data and correct wrong data with intelligent machine learning models, most of the time manual interactions are still needed. The powerful AI models based on deep learning are trained from a large amount of manually annotated data. As a result, many organizations are promoting data standardization, an essential part of data governance. Garbage in ,garbage out.

No Silver Bullet

Data automation and data standardization is the mega trend in the future development. However, we should not regard it as the only solution to deal with data issue, given its wide range of application areas. Apart from fundamental professional data skills, the more important skills for current data workers would data sensitivity and logical thinking, which would not be taught in textbooks or courses. They have to come from project experience. Some seemingly high-profile terms, may not be as useful as those simple and practical methodologies.