DEV Community: Semaphore

Best GitHub Actions Alternatives in 2026

Pete Miloravac — Thu, 23 Jul 2026 12:51:23 +0000

GitHub Actions became the default CI/CD choice for millions of repositories simply because it’s built into GitHub. But “default” and “best” are not the same thing — and in 2026, the gap between the two has gotten harder to ignore. A ten-hour outage in July, a supply-chain attack that backdoored over 5,500 repositories in May, and a steady stream of developer complaints about pricing, runner management, and log usability have sent teams looking for alternatives.

This guide breaks down the best GitHub Actions alternatives in 2026, why GitHub Actions is falling short for a growing number of teams, and how to choose the right replacement for your workflow.

TL;DR: Best GitHub Actions Alternatives in 2026

Semaphore — Best overall. Agent-native CI/CD that’s roughly 2x faster and cheaper than GitHub Actions, with self-maintaining pipelines built for teams using coding agents (Claude Code, Codex, Cursor, etc.).
CircleCI — Best for teams that need deep parallelism and orchestration at enterprise scale.
GitLab CI/CD — Best if you want CI/CD bundled into a broader all-in-one DevOps platform.
Jenkins — Best (only) if you specifically need full self-hosted control and don’t mind the maintenance overhead — more a legacy fallback than a modern alternative.

Where GitHub Actions Falls Short

GitHub Actions works fine for small projects with simple workflows. Problems tend to show up as soon as a team scales, needs predictable reliability, or starts running more automated, agent-driven work through its pipelines.

Reliability incidents are no longer rare. On July 9, 2026, GitHub Actions suffered a major outage — GitHub’s own status page confirmed delayed and failed job starts on GitHub-hosted runners for over ten hours, from 03:29 to 13:39 UTC, caused by an unhealthy backend provisioning service. Developers on Hacker News and Reddit have documented a pattern of similar incidents throughout the year, with one engineer summing up the sentiment bluntly: teams “cannot deploy for days because GitHub Actions is down.”

Supply-chain security is now a real, demonstrated risk. In May 2026, a campaign dubbed “Megalodon” pushed malicious GitHub Actions workflows to 5,561 repositories in under six hours, using workflow_dispatch triggers to plant dormant backdoors that could be activated later via stolen tokens. This came on top of a documented class of workflow_run privilege-escalation vulnerabilities that let attackers exploit elevated permissions to tamper with releases, tags, and artifacts. GitHub published its own 2026 security roadmap acknowledging these exact exploit categories: untrusted code execution, unobserved malicious workflows, and over-permissioned credential exposure.

Concurrency limits and runner management get painful fast. Free and lower tiers cap concurrent jobs, and once you outgrow GitHub-hosted runners, self-hosting your own comes with real infrastructure and maintenance overhead — the opposite of what most teams want from a “managed” CI/CD product.

Usage-based pricing is hard to forecast. Costs scale with runner minutes and job complexity, and teams frequently report surprise bills once test suites, matrix builds, or agent-driven runs multiply the number of jobs.

Log and debugging UX wasn’t built for today’s workflows. Parallel jobs and increasingly automated pipelines (including those driven by coding agents) produce log output that’s genuinely hard to parse — a recurring complaint across developer forums.

You’re locked into the GitHub ecosystem. Workflows, secrets, and runner configuration are tightly coupled to GitHub itself, which makes any future migration — to a different Git host, a different CI provider, or a hybrid setup — more expensive than it should be.

Real support is expensive and hard to get to. GitHub’s included Enterprise support is 24/5, web-ticket-only, with no guaranteed response times. Actual SLA-backed support — phone callbacks, screen-share troubleshooting, and a named Customer Reliability Engineer — only exists behind the Premium and Premium Plus add-ons, and GitHub does not publish pricing for either; you have to contact sales. Customers who have gone through that process report quotes in the tens of thousands of dollars annually just to get a real SLA on Actions issues.

Comparison Table: GitHub Actions vs. the Alternatives

.semaphore-comparison-embed {
--font-down-1: 0.889rem;
--font-down-2: 0.79rem;
--font-down-3: 0.702rem;

--color-cream: #F7F3EC;
--color-blue: #2865DE;
--color-ink: #0D1421;
--color-green: #16A34A;
--color-red: #B91C1C;
--color-border: #c8cbd0;
--color-soft-border: #e5e7eb;

box-sizing: border-box;
color: var(--color-ink);
container-type: inline-size;
font-family: inherit;
width: 100%;

}

.semaphore-comparison-embed *,
.semaphore-comparison-embed *::before,
.semaphore-comparison-embed *::after {
box-sizing: inherit;
}

.semaphore-comparison-embed .comparison-card {
min-width: 0;
}

.semaphore-comparison-embed .comparison-table-wrap {
overflow-x: auto;
width: 100%;
}

.semaphore-comparison-embed .comparison-table {
border: 1px solid var(--color-border);
border-radius: 14px;
border-collapse: separate;
border-spacing: 0;
font-size: var(--font-down-1);
min-width: 0;
overflow: hidden;
table-layout: fixed;
width: 100%;
}

.semaphore-comparison-embed col:first-child {
width: 25%;
}

.semaphore-comparison-embed col:not(:first-child) {
width: 18.75%;
}

.semaphore-comparison-embed th {
background: var(--color-cream);
color: var(--color-ink);
font-size: var(--font-down-2);
font-weight: 800;
letter-spacing: 0;
overflow-wrap: anywhere;
padding: 0.875rem;
text-align: left;
text-transform: uppercase;
vertical-align: top;
}

.semaphore-comparison-embed th:first-child {
border-top-left-radius: 14px;
}

.semaphore-comparison-embed th:last-child {
border-top-right-radius: 14px;
}

.semaphore-comparison-embed td {
border-bottom: 1px solid var(--color-soft-border);
font-weight: 700;
line-height: 1.35;
overflow-wrap: anywhere;
padding: 0.875rem;
vertical-align: top;
}

.semaphore-comparison-embed tbody tr:last-child th,
.semaphore-comparison-embed tbody tr:last-child td {
border-bottom: 0;
}

.semaphore-comparison-embed .feature-cell {
background: #fff;
border-bottom: 1px solid var(--color-soft-border);
color: rgba(13, 20, 33, 0.78);
font-size: var(--font-down-2);
line-height: 1.3;
text-transform: none;
}

.semaphore-comparison-embed td:nth-child(2) {
background: rgba(40, 101, 222, 0.06);
box-shadow: inset 3px 0 0 var(--color-blue);
}

.semaphore-comparison-embed .status {
border-radius: 999px;
display: inline-block;
font-size: var(--font-down-3);
font-weight: 800;
line-height: 1;
margin-right: 0.375rem;
padding: 0.3125rem 0.5rem;
text-transform: uppercase;
vertical-align: 0.0625rem;
white-space: nowrap;
}

.semaphore-comparison-embed .status-yes {
background: rgba(22, 163, 74, 0.12);
color: var(--color-green);
}

.semaphore-comparison-embed .status-no {
background: rgba(185, 28, 28, 0.1);
color: var(--color-red);
}

.semaphore-comparison-embed .status-limited {
background: var(--color-cream);
color: rgba(13, 20, 33, 0.72);
}

@container (max-width: 57.5rem) {
.semaphore-comparison-embed th,
.semaphore-comparison-embed td {
padding: 0.75rem;
}
}

@container (max-width: 42rem) {
.semaphore-comparison-embed .comparison-table-wrap {
overflow-x: visible;
}

.semaphore-comparison-embed .comparison-table {
  border: 0;
  border-radius: 0;
}

.semaphore-comparison-embed .comparison-table,
.semaphore-comparison-embed thead,
.semaphore-comparison-embed tbody,
.semaphore-comparison-embed tr,
.semaphore-comparison-embed th,
.semaphore-comparison-embed td {
  display: block;
  width: 100%;
}

.semaphore-comparison-embed thead {
  clip: rect(0 0 0 0);
  height: 1px;
  overflow: hidden;
  position: absolute;
  white-space: nowrap;
  width: 1px;
}

.semaphore-comparison-embed tbody {
  display: grid;
  gap: 0.75rem;
}

.semaphore-comparison-embed tr {
  border: 1px solid var(--color-soft-border);
  border-radius: 14px;
  display: grid;
  gap: 0.5rem;
  grid-template-columns: repeat(2, minmax(0, 1fr));
  padding: 0.625rem;
}

.semaphore-comparison-embed .feature-cell {
  border-bottom: 1px solid var(--color-soft-border);
  font-size: var(--font-down-1);
  grid-column: 1 / -1;
  padding: 0 0 0.5rem;
}

.semaphore-comparison-embed td {
  background: var(--color-cream);
  border: 0;
  border-radius: 10px;
  min-width: 0;
  padding: 0.625rem;
}

.semaphore-comparison-embed td:nth-child(2) {
  background: rgba(40, 101, 222, 0.08);
  box-shadow: inset 3px 0 0 var(--color-blue);
}

.semaphore-comparison-embed td::before {
  color: rgba(13, 20, 33, 0.62);
  content: attr(data-provider);
  display: block;
  font-size: var(--font-down-3);
  font-weight: 800;
  margin-bottom: 0.25rem;
  text-transform: uppercase;
}

}

@container (max-width: 30rem) {
.semaphore-comparison-embed tr {
gap: 0.4375rem;
padding: 0.5rem;
}

.semaphore-comparison-embed td {
  padding: 0.5rem;
}

.semaphore-comparison-embed .status {
  margin-bottom: 0.25rem;
}

}

@media (max-width: 57.5rem) {
.semaphore-comparison-embed th,
.semaphore-comparison-embed td {
padding: 0.75rem;
}
}

@media (max-width: 42rem) {
.semaphore-comparison-embed .comparison-table-wrap {
overflow-x: visible;
}

.semaphore-comparison-embed .comparison-table {
  border: 0;
  border-radius: 0;
}

.semaphore-comparison-embed .comparison-table,
.semaphore-comparison-embed thead,
.semaphore-comparison-embed tbody,
.semaphore-comparison-embed tr,
.semaphore-comparison-embed th,
.semaphore-comparison-embed td {
  display: block;
  width: 100%;
}

.semaphore-comparison-embed thead {
  clip: rect(0 0 0 0);
  height: 1px;
  overflow: hidden;
  position: absolute;
  white-space: nowrap;
  width: 1px;
}

.semaphore-comparison-embed tbody {
  display: grid;
  gap: 0.75rem;
}

.semaphore-comparison-embed tr {
  border: 1px solid var(--color-soft-border);
  border-radius: 14px;
  display: grid;
  gap: 0.5rem;
  grid-template-columns: repeat(2, minmax(0, 1fr));
  padding: 0.625rem;
}

.semaphore-comparison-embed .feature-cell {
  border-bottom: 1px solid var(--color-soft-border);
  font-size: var(--font-down-1);
  grid-column: 1 / -1;
  padding: 0 0 0.5rem;
}

.semaphore-comparison-embed td {
  background: var(--color-cream);
  border: 0;
  border-radius: 10px;
  min-width: 0;
  padding: 0.625rem;
}

.semaphore-comparison-embed td:nth-child(2) {
  background: rgba(40, 101, 222, 0.08);
  box-shadow: inset 3px 0 0 var(--color-blue);
}

.semaphore-comparison-embed td::before {
  color: rgba(13, 20, 33, 0.62);
  content: attr(data-provider);
  display: block;
  font-size: var(--font-down-3);
  font-weight: 800;
  margin-bottom: 0.25rem;
  text-transform: uppercase;
}

}

@media (max-width: 30rem) {
.semaphore-comparison-embed tr {
gap: 0.4375rem;
padding: 0.5rem;
}

.semaphore-comparison-embed td {
  padding: 0.5rem;
}

.semaphore-comparison-embed .status {
  margin-bottom: 0.25rem;
}

}

  <table>
    <colgroup>
      <col>
      <col>
      <col>
      <col>
      <col>
    </colgroup>
    <thead>
      <tr>
        <th>Criteria</th>
        <th>Semaphore</th>
        <th>CircleCI</th>
        <th>GitLab CI/CD</th>
        <th>Jenkins</th>
      </tr>
    </thead>
    <tbody>
      <tr>
        <th>Avg. build time (benchmark)<sup>6</sup></th>
        <td>5m 01s</td>
        <td>13m 18s</td>
        <td>11m 15s</td>
        <td>Not benchmarked (self-hosted, hardware-dependent)</td>
      </tr>
      <tr>
        <th>Cost per job (benchmark)</th>
        <td>$0.04</td>
        <td>$0.08</td>
        <td>$0.11</td>
        <td>Infrastructure cost only (no per-job pricing)</td>
      </tr>
      <tr>
        <th>Agent-native / AI coding agent support</th>
        <td><span>Yes</span>Built-in (sem-ai)</td>
        <td><span>Limited</span></td>
        <td><span>Limited</span></td>
        <td><span>No</span></td>
      </tr>
      <tr>
        <th>Self-hosted option</th>
        <td><span>Yes</span>Open source</td>
        <td>Enterprise only</td>
        <td><span>Yes</span></td>
        <td><span>Yes</span>Default</td>
      </tr>
      <tr>
        <th>Open source</th>
        <td><span>Yes</span>Fully</td>
        <td><span>No</span></td>
        <td>Core only</td>
        <td><span>Yes</span></td>
      </tr>
      <tr>
        <th>Enterprise security certs</th>
        <td>SOC 2 Type II, ISO 27001</td>
        <td>SOC 2</td>
        <td>SOC 2, ISO 27001</td>
        <td>Depends on self-managed setup</td>
      </tr>
      <tr>
        <th>Setup / migration effort from GHA</th>
        <td>Low (guided migration)</td>
        <td>Medium</td>
        <td>Medium</td>
        <td>High</td>
      </tr>
      <tr>
        <th>Monorepo support</th>
        <td><span>Yes</span></td>
        <td><span>Limited</span></td>
        <td><span>Limited</span></td>
        <td>Manual/plugin-dependent</td>
      </tr>
      <tr>
        <th>Built-in flaky test detection</th>
        <td><span>Yes</span>Cloud</td>
        <td><span>No</span></td>
        <td><span>No</span></td>
        <td><span>No</span></td>
      </tr>
      <tr>
        <th>SSO (SAML / Okta / LDAP)</th>
        <td><span>Yes</span>Cloud</td>
        <td>Enterprise only</td>
        <td>Enterprise only</td>
        <td>Plugin-dependent</td>
      </tr>
    </tbody>
  </table>

The Best GitHub Actions Alternatives, Ranked

1. Semaphore — Best Overall, Best for AI Coding Agents

Semaphore has repositioned itself as agent-native CI: “Tell your agent what you want. It runs your CI.” Instead of hand-writing YAML, developers (or their coding agents) can tell Semaphore what they need, and it configures and runs the pipeline — including pre-push validation in production-identical sandboxes and self-maintaining pipelines that identify and fix flaky tests as your codebase grows.

This matters more than it might sound. In a recent interview, Semaphore’s Head of Product Marko Gaćeša made the case that testing — not writing code — is now the real bottleneck in software delivery, especially as teams run multiple coding agents in parallel: “I see the biggest bottleneck being testing… [pipelines] are going to look significantly different” as agents need feedback in minutes, not the 20-30 minute cycles teams tolerated before. That’s a direct response to the workflows GitHub Actions wasn’t designed for.

On raw performance, Semaphore’s own third-party-reproducible benchmark (10 runs, identical Ruby on Rails workload, cache warm) shows an average build time of 5m 01s versus GitHub Actions’ 9m 44s — a 94.48% speed advantage — at $0.04 per job versus $0.06 for GitHub Actions.

Key capabilities: Agent-native pipeline setup (sem-ai), self-hosted or cloud, fully open source and self-hostable, SOC 2 Type II + ISO 27001 certified, transparent pay-as-you-go pricing with $15/month free usage and no seat or idle costs, guided migration tooling from GitHub Actions, GitLab CI, CircleCI, and Jenkins.

Support is also a real differentiator, not just a feature checkbox. Where GitHub keeps its real support pricing behind a sales call, Semaphore publishes every tier on its pricing page: a free tier for basic help, $50/month for email support, $250/month for email plus Slack, and up to $750/month for SLA-backed plans with response times as fast as 1 hour on urgent issues (with optional 24/7 coverage) — plus add-on customer success and engineering hours for teams that need deeper, hands-on help. You pick the level your team actually needs and see the exact price upfront, instead of negotiating a five-figure contract just to get someone on the phone.

Best for: Teams already using (or planning to use) AI coding agents, teams that got burned by a GitHub Actions outage or security incident, and teams that want faster feedback loops without a corresponding cost increase.

Considerations: Smaller ecosystem of pre-built community actions/integrations than GitHub’s marketplace, since Semaphore is a newer entrant to public awareness despite being in the market since 2014.

What Teams Who Switched From GitHub Actions Are Saying

“Since moving to Semaphore from GitHub Actions, our CI pipeline has been stable and had more consistent run times, this allowed us to detect failures and flaky specs faster.” — Amin Ben Slimen, Senior Software Engineer at klarx
“It’s cheap, it’s easy to understand, and the support team is quick to assist if you need help. Honestly, after trying GitHub Actions, CircleCI, Octopus, Jenkins, and more, this clearly stands out to me as the better option.” — Fredrik August Madsen-Malmo, Head of DevOps at Kvist
“Semaphore is an incredibly easy-to-use CI/CD platform. Integrating my Ruby on Rails applications to use it is always very easy to do.” — Lorenzo Zabot, Backend Engineer at Vox Group

2. CircleCI — Best for Enterprise-Scale Parallelism

CircleCI remains one of the most established cloud CI/CD platforms, particularly for teams running large test suites that benefit from aggressive parallelization and orchestration features like dynamic config and matrix jobs.

Key capabilities: Strong parallelism and caching, orchestration for complex multi-service pipelines, self-hosted runners for enterprise plans, broad language/framework support.

Best for: Larger engineering orgs with complex, multi-repo build graphs that need fine-grained control over job orchestration.

Considerations: In the same benchmark referenced above, CircleCI was the slowest and most cost-variable of the tools tested — averaging 13m 18s per build (165% slower than Semaphore) at $0.08 per job, with individual run times swinging as high as 17 minutes. Pricing and plan tiers can also get complex at scale.

3. GitLab CI/CD — Best All-in-One DevOps Platform

If your team already runs on GitLab, or wants source control, CI/CD, security scanning, and project management under one roof, GitLab CI/CD is a natural alternative. It’s configured with a familiar .gitlab-ci.yml file and integrates tightly with merge requests and built-in runners.

Key capabilities: Native integration with GitLab’s broader DevSecOps suite, built-in container registry, tight merge-request integration, both SaaS and self-managed deployment options.

Best for: Teams willing to consolidate their whole toolchain onto GitLab, or teams already there who want to reduce the number of vendors they manage.

Considerations: Migrating your Git hosting itself (not just CI) is a bigger lift than swapping CI providers alone. Build times in the same benchmark averaged 11m 15s, and cost per job was the highest of all tools tested at $0.11 — nearly 3x Semaphore’s cost.

4. Jenkins — The Legacy Self-Hosted Option

Jenkins predates all of the above and remains the most customizable, plugin-extensible CI server on the market — entirely because it’s open source and infinitely scriptable. Some teams with very specific compliance or air-gapped infrastructure requirements still choose it deliberately.

Key capabilities: Enormous plugin ecosystem, full control over infrastructure and build environment, no vendor lock-in, free to run (excluding your own hosting costs).

Best for: Organizations with strict on-premises requirements and dedicated platform engineering resources to maintain it.

Considerations: This isn’t really a “GitHub Actions alternative” so much as a different category of tool — you’re trading GitHub’s reliability and security problems for the maintenance burden, Groovy scripting complexity, and plugin management overhead Jenkins is well known for. Most teams migrating away from GitHub Actions are looking for less operational overhead, not more.

How to Choose the Right GitHub Actions Alternative

If you’re already using (or planning to use) AI coding agents for development, prioritize a CI platform built around that workflow rather than retrofitting one designed for manual pushes. Semaphore is purpose-built for this.
If your primary pain is speed and unpredictable cost, compare benchmarked build times and per-job pricing directly rather than relying on marketing claims — the numbers above are independently reproducible.
If your primary pain is a recent outage or security incident, weigh how each vendor handles incident transparency, and whether the platform is open source enough for you to self-host as a fallback.
If you need to stay inside a single platform for compliance or simplicity, GitLab CI/CD is the more natural move than introducing a new best-of-breed tool.
If you have hard on-premises requirements and the engineering headcount to support it, Jenkins remains viable — just budget for the maintenance cost.

Ready to see the difference for yourself? Start free with Semaphore — no credit card required, $15 of usage included every month.

Frequently Asked Questions

What is the best alternative to GitHub Actions in 2026?

Semaphore is the strongest overall alternative for most teams — it’s benchmarked as roughly 2x faster and cheaper than GitHub Actions, and it’s purpose-built for teams using AI coding agents, which is where CI/CD workflows are heading. CircleCI and GitLab CI/CD are also strong options depending on whether you prioritize enterprise-scale parallelism or an all-in-one platform.

Why are teams moving off GitHub Actions?

The three most common reasons are reliability (including the ~10-hour outage on July 9, 2026), security incidents like the Megalodon supply-chain attack that backdoored over 5,500 repositories, and cost/performance — GitHub Actions is measurably slower and more expensive per job than newer alternatives in independent benchmarks.

Is switching from GitHub Actions to Semaphore difficult?

No — Semaphore provides guided migration tooling for teams moving from GitHub Actions, GitLab CI, CircleCI, or Jenkins, and most teams are able to migrate core pipelines in well under an hour.

Is Semaphore open source?

Yes. Semaphore is fully open source and self-hostable, in addition to being available as a managed cloud service.

Does Semaphore support the same features as GitHub Actions?

Semaphore covers the core CI/CD workflow — build, test, deploy — plus additions like pre-push sandbox validation and self-maintaining pipelines. Its community action/integration marketplace is smaller than GitHub’s, since GitHub Actions has a multi-year head start, but core CI/CD functionality and migration paths are well covered.

The post Best GitHub Actions Alternatives in 2026 appeared first on Semaphore.

What CI/CD strategies work for embedded or IoT projects that require hardware testing?

Pete Miloravac — Thu, 30 Apr 2026 10:48:53 +0000

Embedded and IoT teams face a very different CI/CD reality than traditional SaaS teams. While most continuous integration and continuous delivery pipelines assume everything can run in the cloud, embedded systems depend on physical hardware, constrained environments, and real world signals.

If you search forums like Reddit (r/embedded, r/devops), Stack Overflow, or vendor communities, the same questions keep appearing:

How do I run CI tests when I need actual hardware?
How do I scale hardware testing across teams?
How do I avoid flaky tests caused by devices?
Can I still use modern CI/CD tools or do I need something custom?

This guide walks through practical CI/CD strategies used by engineering teams building firmware, IoT platforms, and hardware dependent systems. It focuses on approaches that scale, reduce cost, and maintain reliability.

Why CI/CD is harder for embedded and IoT projects

Unlike pure software systems, embedded pipelines must deal with:

Physical device availability
Hardware state and reset issues
Long flashing and boot cycles
Non deterministic behavior (timing, sensors, connectivity)
Limited parallelization

This creates a mismatch with traditional CI/CD platforms that expect fast, stateless, fully virtualized execution.

For engineering leaders, this often results in slower pipelines, higher costs, and fragile automation. The goal is not to force cloud native assumptions onto hardware, but to design a hybrid pipeline.

The core strategy: split your pipeline by test layers

The most effective approach seen across teams is to separate tests into layers, minimizing hardware usage to only what is necessary.

1. Fast feedback layer (no hardware)

Run as much as possible without devices:

Unit tests for firmware logic
Static analysis (clang tidy, cppcheck)
Build validation
Simulation or emulation (QEMU, Renode)

Example Semaphore pipeline block:

version: v1.0
name: Firmware CI

blocks:
  - name: Build and unit tests
    task:
      jobs:
        - name: Build
          commands:
            - checkout
            - make build
            - make test

This layer should cover 70 to 90 percent of your test surface. It keeps pipelines fast and cost effective.

2. Hardware in the loop testing (HIL)

Only after passing fast checks should jobs use real devices.

Typical setup discussed in forums:

USB connected device farms
Network controlled power switches
Raspberry Pi or similar acting as device controllers

Example structure:

  - name: Hardware tests
    task:
      prologue:
        commands:
          - checkout
      jobs:
        - name: Run on device farm
          commands:
            - ./scripts/flash_device.sh
            - ./scripts/run_integration_tests.sh

With Semaphore, you can integrate self hosted agents that have direct access to hardware, allowing your pipeline to orchestrate physical tests while keeping control centralized.

This hybrid model is critical for teams that have outgrown default CI tools that cannot reliably interface with hardware.

Designing a reliable device farm

A common pain point across discussions is flaky hardware tests. The root cause is usually poor device management.

Best practices:

Isolate devices per job

Avoid shared hardware when possible. If sharing is required, implement locking mechanisms.

Automate reset and recovery

Use:

Smart power switches
USB relay boards
Watchdog scripts

Example reset script:

#!/bin/bash
# Power cycle device
curl http://power-switch.local/off
sleep 2
curl http://power-switch.local/on
sleep 5

Make tests idempotent

Each test run should not depend on previous device state.

Collect logs externally

Stream logs via serial or network so failures can be diagnosed post run.

Orchestrating hardware with CI/CD

One of the biggest questions teams ask is: how do I connect CI/CD pipelines to real devices?

The most robust approach is:

Use cloud CI/CD for orchestration
Use self hosted runners or agents for hardware interaction

With Semaphore, this means:

Running standard pipeline steps in the cloud
Routing hardware jobs to agents inside your lab

This keeps pipelines fast while maintaining control over physical infrastructure.

Handling test flakiness and timing issues

Forum discussions consistently highlight flaky tests as a major blocker.

Strategies that work:

Add retry logic at the pipeline level

retry:
  limit: 2

Add health checks before tests

Ensure device is reachable and responsive before running tests.

Use time budgets instead of strict timing

Avoid asserting exact timing unless necessary.

Tag and isolate flaky tests

Run them separately to avoid blocking the main pipeline.

Parallelizing hardware testing

Scaling is a key concern for engineering managers.

Options include:

Expanding device farms
Sharding tests across devices
Prioritizing critical test subsets

Example:

jobs:
  - name: Device 1
    commands:
      - ./run_tests.sh --group=1
  - name: Device 2
    commands:
      - ./run_tests.sh --group=2

Semaphore allows parallel job execution, which is especially valuable when coordinating multiple hardware nodes.

Cost optimization strategies

Hardware pipelines can become expensive quickly.

Key tactics:

Maximize non hardware test coverage
Use on demand hardware instead of always on
Optimize pipeline duration
Avoid rerunning full hardware suites unnecessarily

Engineering teams often switch CI/CD platforms when costs become unpredictable or tied to inefficient pipelines. A system that allows fine grained control over execution and resource usage is critical.

When to consider custom tooling vs CI/CD platforms

Many teams ask whether they should build their own system.

Build custom only if:

Your workflow is highly specialized
You require deep hardware orchestration beyond standard pipelines

Otherwise, modern CI/CD platforms like Semaphore can handle orchestration while letting you customize the hardware layer.

This balance avoids reinventing core CI/CD capabilities while still supporting embedded workflows.

Putting it all together

A production ready embedded CI/CD pipeline typically looks like:

Code commit triggers pipeline
Build and unit tests run in parallel
Simulation tests validate behavior
Hardware tests run on device farm via self hosted agents
Results aggregated and reported
Deployment triggered if all checks pass

This structure aligns with how high performing engineering teams reduce risk while maintaining delivery speed.

FAQs

How do I run CI tests on real hardware?

Use a device farm connected to self hosted CI agents. The pipeline triggers scripts that flash firmware and execute tests on physical devices.

What tools are commonly used for hardware testing in CI?

Teams commonly use QEMU or Renode for simulation, and custom scripts combined with device controllers for real hardware testing.

How do I reduce flaky tests in IoT CI/CD?

Focus on device reset automation, idempotent tests, retries, and proper health checks before execution.

Can CI/CD pipelines scale with hardware constraints?

Yes, by parallelizing across device farms and minimizing reliance on hardware through simulation and unit testing.

Is CI/CD worth it for embedded systems?

Yes. Teams that invest in CI/CD for embedded systems see improvements in reliability, faster debugging, and more predictable releases.

The post What CI/CD strategies work for embedded or IoT projects that require hardware testing? appeared first on Semaphore.

How Does CI/CD Differ for Machine Learning Pipelines (MLOps)?

Pete Miloravac — Thu, 23 Apr 2026 15:24:09 +0000

For most engineering teams, CI/CD is already a solved problem—at least on the surface. You commit code, run tests, build artifacts, and deploy.

But when teams start introducing machine learning into production systems, that familiar pipeline begins to break down.

Across forums like Reddit (r/MachineLearning, r/devops), Stack Overflow, and Hacker News, the same questions come up repeatedly:

“How do I version datasets in CI/CD?”
“Why does my model degrade after deployment even though tests pass?”
“How do I test something that learns from data?”
“Should I deploy models the same way as application code?”

This tutorial answers those questions with a practical lens. More importantly, it explains what engineering leaders need to rethink when adapting CI/CD pipelines for MLOps.

Why Traditional CI/CD Breaks Down for Machine Learning

In traditional software delivery, your pipeline is built around code determinism.

Given the same input, your application produces the same output. Your CI/CD pipeline enforces this through:

Unit tests
Integration tests
Build reproducibility
Static artifacts

Machine learning systems violate this assumption in three key ways:

Data is a first-class dependency
Outputs are probabilistic, not deterministic
Performance degrades over time (data drift)

This fundamentally changes how you design continuous integration and continuous deployment.

For engineering managers and CTOs, this is where pipelines often become fragile, slow, and expensive—especially when built on top of tools that were not designed for these workflows.

Key Differences Between CI/CD and MLOps Pipelines

1. What You Version: Code vs Code + Data + Models

In a standard CI/CD pipeline:

You version application code
Dependencies are managed via package managers
Builds are reproducible

In MLOps, you must version:

Training data
Feature engineering logic
Model artifacts
Hyperparameters

A typical approach is to combine Git with a data versioning tool like DVC.

Example:

# Track dataset
dvc add data/training.csv

# Push data to remote storage
dvc push

# Commit metadata
git add data/training.csv.dvc .gitignore
git commit -m "Track training dataset"

Your CI/CD pipeline now needs to fetch not just code, but also the correct dataset version.

2. What You Test: Logic vs Behavior

Traditional CI focuses on correctness:

assert(add(2, 2) === 4)

In machine learning, you test behavior:

Accuracy thresholds
Precision and recall
Model drift
Bias detection

Example test step in a pipeline:

assert model_accuracy > 0.87, "Model accuracy below threshold"

This introduces a new challenge: tests can fail even when code hasn’t changed.

3. What You Build: Binaries vs Experiments

In traditional pipelines:

Build once
Deploy artifact

In MLOps:

Train model
Evaluate multiple experiments
Select best candidate

Your pipeline becomes iterative and branching.

Example workflow:

blocks:
  - name: Train models
    task:
      jobs:
        - name: train-xgboost
        - name: train-random-forest

  - name: Evaluate
    task:
      jobs:
        - name: compare-metrics

  - name: Deploy best model
    task:
      jobs:
        - name: deploy

4. Deployment: Static Releases vs Continuous Retraining

Traditional deployment:

Triggered by code changes
Releases are versioned and stable

MLOps deployment:

Triggered by new data
Models may be retrained daily or hourly
Performance must be monitored continuously

This is where many teams struggle. They try to force data-driven workflows into code-driven pipelines.

Designing a CI/CD Pipeline for Machine Learning

Let’s walk through a practical pipeline using Semaphore.

Semaphore is particularly well-suited here because it allows you to orchestrate complex workflows without introducing unnecessary pipeline overhead—critical for compute-heavy ML workloads.

Step 1: Reproducible Environment

version: v1.0
name: ML Pipeline

agent:
  machine:
    type: e1-standard-4
    os_image: ubuntu2004

Pin dependencies:

pip install -r requirements.txt

For ML, reproducibility is everything. Use Docker or pinned environments to avoid failures.

Step 2: Fetch Data and Dependencies

blocks:
  - name: Setup
    task:
      jobs:
        - name: Fetch data
          commands:
            - checkout
            - dvc pull

This step is often missing in traditional pipelines—and is one of the main sources of confusion discussed in forums.

Step 3: Train Model

  - name: Train
    task:
      jobs:
        - name: Train model
          commands:
            - python train.py

Step 4: Evaluate Model

  - name: Evaluate
    task:
      jobs:
        - name: Evaluate model
          commands:
            - python evaluate.py

Example evaluation script:

if accuracy < 0.87:
    raise Exception("Model did not meet quality threshold")

Step 5: Conditional Deployment

  - name: Deploy
    task:
      jobs:
        - name: Deploy model
          commands:
            - python deploy.py

In Semaphore, you can gate this step using promotions, approvals, or conditions—important for controlling risk in ML deployments.

Common Pitfalls

Treating Models Like Code Artifacts

Models are not static. If you deploy them once and forget them, they will degrade.

Fix: Add monitoring and retraining triggers.

Ignoring Data Versioning

Without versioned data, debugging becomes impossible.

Fix: Use DVC, feature stores, or data snapshots.

Overloading CI with Training Jobs

Training jobs can be expensive and slow.

Fix: Separate lightweight CI from heavy training workflows.

Lack of Observability

Traditional CI/CD tools focus on build logs—not model performance.

Fix: Integrate monitoring and metrics.

Strategic Implications for Engineering Leaders

For decision makers, the shift to MLOps is not just technical—it affects:

Cost structure
Reliability
Tooling decisions

Teams that succeed treat CI/CD for ML as a first-class system, not an extension of existing pipelines.

This is where platforms like Semaphore position themselves differently:

Flexible pipeline orchestration for complex workflows
Predictable performance at scale
Cost efficiency compared to legacy tools

When Should You Adapt Your Pipeline?

You likely need to rethink your CI/CD if:

You are deploying models to production
Your pipelines are slowing down due to training workloads
You cannot reproduce model results reliably
CI/CD costs are increasing unpredictably

FAQs

What is the main difference between CI/CD and MLOps pipelines?

Traditional CI/CD focuses on deterministic code, while MLOps pipelines must handle data, probabilistic outputs, and continuous retraining.

Can I use standard CI/CD tools for machine learning?

Yes, but most teams need to extend them significantly to support data versioning, model evaluation, and retraining workflows.

How do you test machine learning models in CI/CD?

By validating metrics such as accuracy, precision, recall, and monitoring for drift.

Should model training run in CI?

Not always. Many teams separate training pipelines from CI to control cost and runtime.

How do you deploy machine learning models safely?

Use staged rollouts, approval gates, and continuous monitoring.

The post How Does CI/CD Differ for Machine Learning Pipelines (MLOps)? appeared first on Semaphore.

What is MCP Really?

Tomas Fernandez — Mon, 03 Nov 2025 13:02:00 +0000

Photo by Pavel Danilyuk: https://www.pexels.com/photo/a-robot-holding-a-wine-8439094/

The Model Context Protocol (MCP) has been described as “the USB-C for AI”. It’s a fitting analogy, but what does it really mean? What is MCP?

Large language models (LLMs) are incredibly capable, but they only know what they know. Once trained, an LLM can’t access real-time information or specialized systems until it connects to external tools.

MCP provides a communication protocol that lets models like GPT or Claude interact with any compatible tool or service. Instead of relying on proprietary APIs or one-off integrations, MCP introduces a shared language for interaction between AIs (as clients) and software (as servers).

How the MCP Works

At its core, MCP is a simple client–server model. The large language model acts as the client, while a server provides one or more tools the AI can use. Communication between the two happens through JSON-RPC.

During initialization, the AI and server negotiate capabilities. Then the AI sends a client a tools/list request.

{
  "jsonrpc": "2.0",
  "id": 1,
  "method": "tools/list",
  "params": {
    "cursor": "optional-cursor-value"
  }
}

And the server responds with a manifest of available tools:

{
  "jsonrpc": "2.0",
  "id": 1,
  "result": {
    "tools": [
      {
        "name": "get_weather",
        "description": "Get current weather information for a location",
        "inputSchema": {
          "type": "object",
          "properties": {
            "location": {
              "type": "string",
              "description": "City name or zip code"
            }
          },
          "required": ["location"]
        }
      }
    ],
    "nextCursor": "next-page-cursor"
  }
}

The AI now knows what tools or functions are available and picks one for a user request. So, in our example, the AI would call the get_weather tool.

{
  "jsonrpc": "2.0",
  "id": 2,
  "method": "tools/call",
  "params": {
    "name": "get_weather",
    "arguments": {
      "location": "Lisbon"
    }
  }
}

And the MCP server responds with structured output, in this case, how's the weather in Lisbon:

{
  "jsonrpc": "2.0",
  "id": 2,
  "result": {
    "content": [
      {
        "type": "text",
        "text": "Current weather in Lisbon:\nTemperature: 32°C\nConditions: Partly cloudy"
      }
    ],
    "isError": false
  }
}

Local vs Remote MCP Servers

The easiest way to implement the MCP server is to run it on the same host as the client. For example, if I'm using OpenAI Codex or Claude Desktop, the AI can spawn an MCP server locally and communicate over standard input and output (stdio).

For more complex setups, the MCP allows communication over HTTP and provides mechanisms for authentication and authorization. These servers can require credentials, API keys, or tokens, depending on how sensitive their capabilities are.

The State of the Standard

MCP is still an emerging standard. It was introduced in 2024 as an open specification. MCP is being developed collaboratively by several players in the AI ecosystem.

The initial specification was published at modelcontextprotocol.info, and the work is happening in the open with input from AI companies, open-source developers, and infrastructure providers.

Conclusion

MCP represents a quiet but fundamental shift in how AI systems interact with the world.

It offers a shared, open standard; a common language that any model and any tool can use to talk to each other.

For developers, this means fewer one-off connectors and more reusable, interoperable systems. For users, it means AI assistants that can reach beyond their training data and tap into live information, files, or applications with precision and context.

Thanks for reading, and happy building!

Lightweight Containers With Docker and WebAssembly

Tomas Fernandez — Mon, 18 Dec 2023 14:14:22 +0000

🐳Did you know you can now build and run WebAssembly with Docker Desktop? That's the only tool you need to run the same binary file at native speed on any platform: Linux, Windows, Mac or any browser.

WebAssembly, or Wasm, is increasingly relevant in software development. It's a portable binary code format designed for efficient and fast execution on any platform, including web browsers.

Watch the hands-on tutorial:

WebAssembly is so essential for the web that Solomon Hykes, the founder of Docker, announced that if Wasm and WASI were available in 2008, Docker might not have been developed.

Despite its advantages, WebAssembly has yet to reach the same level of adoption as Docker. Part of the challenge lies in the complexity of the tooling for Wasm, particularly in building, running, and debugging across different languages. For example, creating a Wasm binary involves installing a language-specific compiler toolchain.

Docker offers a solution here, providing a reproducible and isolated build environment. Let's see how it works.

Docker and WebAssembly

In July 2023, Docker introduced experimental support for WebAssembly, adding a new dimension to running Wasm applications. This integration brings several advantages:

Simplified Process: Using Docker for building and running Wasm applications reduces the learning curve by minimizing the required tools.
Enhanced Portability: Wasm containers don't require different builds for different machine architectures, simplifying deployment.
Consistent Builds: Docker's isolated environment ensures consistent builds across various platforms.
Integration with Existing Tools: Docker's compatibility with Docker Compose and Kubernetes facilitates complex deployments and scaling.

Building a Wasm container

Let's start by creating a new project:



$ cargo new docker-wasm-demo-rust

The default project is a "Hello, World" application printing to console, perfect as a first step.

To build and run it, we use cargo:



$ cargo build --release
$ cargo run
Hello World!

But this is a native binary, not WebAssembly. To compile a Wasm target, let's install the WebAssembly toolchain. This command requires that you have installed rustup:



$ rustup target add wasm32-wasi

Next, we can build a Wasm target with:



$ cargo build --target wasm32-wasi --release

We can't run this directly from the command line unless we install some runtime like wasmtime:



$ was time target/wasm32-wasi/release/docker-wasm-demo-rust.wasm
Hello World!

This works, but there's a lot of tooling involved. We need the Rust compilers, the Wasm toolchain, and some runtime to test it. We can streamline all this with Docker.

Running Wasm in Docker

Before using Docker with WebAssembly, we need to ensure we have the latest version of Docker Desktop installed and then enable containerd and Wasm support in the options:

Then we need to create a simple Dockerfile that copies the wasm inside the container:



# Dockerfile
FROM scratch
COPY target/wasm32-wasi/release/docker-wasm-demo-rust.wasm /hello.wasm
ENTRYPOINT [ "/hello.wasm" ]

We need to build the image, but we need to switch to the wasi/wasm platform for this case.



docker build --platform wasi/wasm -t hello-wasm .

If we check the Docker Desktop image tab, we should see the new image with the WASM badge. This means the image is actually WebAssembly instead of a regular container:

To run the image, we use the familiar docker run with two extra flags:

--runtime=io.containerd.wasmedge.v1 one of the possible runtimes supported by Docker Desktop
--platform=wasi/wasm to tell Docker we want to run the Wasm image. Otherwise, Docker will fail to find the image.



docker run --runtime=io.containerd.wasmedge.v1 --platform=wasi/wasm hello-wasm

Building Wasm with Docker

We can step the process further by using Docker to build the image. This allows us to run the build in a clean, shareable environment, making it easier to run the build stage in CI/CD.

The following 2-stage Dockerfile builds and creates the Wasm image:

The first stage uses the Rust base image to build the Wasm inside the container
The second stage copies the Wasm binary from the first stage and creates the Wasm image.



# Dockerfile
FROM --platform=$BUILDPLATFORM rust:1.74 AS build
RUN rustup target add wasm32-wasi
RUN mkdir -p /build
WORKDIR /build
COPY Cargo.toml .
COPY src ./src
RUN cargo build --target wasm32-wasi --release
RUN chmod a+x /build/target/wasm32-wasi/release/docker-wasm-demo-rust.wasm

FROM scratch
COPY --link --from=build /build/target/wasm32-wasi/release/docker-wasm-demo-rust.wasm /hello.wasm
ENTRYPOINT [ "/hello.wasm" ]

Let's build the image again:



$ docker build --platform wasi/wasm -t hello-wasm .

And then run it:



$ docker run --runtime=io.containerd.wasmedge.v1 --platform=wasi/wasm hello-wasm

As you can see, this new Docker feature lets us work with Wasm without needing anything other than Docker Desktop on our machines. Neat!

Conclusion

WebAssembly and Docker do make an interesting combination. At the end of the day, having more options for developers, it's always good news. Be sure to check the YouTube tutorial video above for a more in-depth explanation.

Thanks for reading, and happy building!

State of DevOps 2023 Report: Key Findings and Insights

Tomas Fernandez — Thu, 14 Dec 2023 17:39:36 +0000

The much-anticipated State of DevOps 2023 report is finally out. Developer by the DORA team at Google, this annual report is a synthesis of trends and insights collected from professionals and organizations working in the realm of DevOps.

Key takeaways:

The ramifications of AI on productivity
What factors contribute to job satisfaction and burnout.
What matters when choosing a cloud.
The non-linear nature of investing in reliability and SRE practices.

I've published a video covering all the key insights. Scroll down to see a summary.

Here’s a summarized look at some of the major takeaways. You can download the report for free from Google:

📕 2023 State of DevOps Report

Key findings

From the outset, the report presented several new areas of focus. On page 4, the spotlight was on:

The pivotal role documentation plays in the ecosystem.
How underrepresented groups are at a higher risk of burnout.
The growing emphasis on prioritizing user feedback and needs.
The role of the cloud in boosting productivity and job satisfaction.

Page 6 of the report outlines key organizational outcomes like enhancing productivity and job satisfaction while curbing burnout.

Prioritizing Users

One of the standout findings is how tuning into users during the developmental stages can amplify performance metrics. Wether they are customers, developers, analysts or testers, when we focus on the needs of our users there is a substantial increase in performance and reliability across the board.

Taking a user-centric approach reaps multifold benefits. It not only fosters an environment conducive to enhanced performance, CI/CD, job satisfaction, and productivity, but also plays a role in mitigating burnout.

The technical capabilities that matter

New technologies call for a refreshed set of metrics. The results show that the impact of AI on team performance was less pronounced than expected. Instead, more foundational capabilities like CI/CD, robust documentation, trunk-based development, and rapid code reviews emerged as stronger pillars of performance.

The power of flexibility

The flexibility of infrastructure directly correlates with performance, with most organizations turning to the cloud for this adaptability. However, the type of cloud chosen holds significance:

Private clouds improve operational and team performance.
Public clouds enhance organizational and team performance at the expense of operational and software delivery effectiveness. This dip in performance may be due teams shifting to the cloud and encountering unfamiliar tools.

Documentation: an amplifier of technical capabilities

State of DevOps 2023 underscored the immense value of quality documentation. Beyond being foundational, it magnifies other technical capabilities, with the potential to boost performance by nearly 13x when paired with trunk-based development.

In addition, well-maintained documentation also positively influences the human side of development, elevating job satisfaction and productivity and reducing burnout tendencies.

Cultivating a Generative Culture

The idea of a generative culture has been a recurring theme in the State of DevOps reports. Rooted in the studies of American sociologist Ron Westrum, a generative culture paves the way for innovation, higher productivity, and a more fulfilling work environment.

The data confirms that a generative culture is a robust indicator of high performance.

Wrapping Up

I have only scratched the surface on the insights that State of DevOps 2023 reveals. I highly recommend downloading the report and checking it our by yourself.

Thanks for reading!

Function Calling: Integrate Your GPT Chatbot With Anything

Tomas Fernandez — Wed, 09 Aug 2023 14:02:14 +0000

Yes we do!

Imagine creating an AI assistant to which you can say something like, "Book me the earliest reservation for the nearest Thai restaurant and update my calendar." Language models continue to push boundaries and evolve. OpenAI, the company behind ChatGPT, recently introduced a powerful new feature called function calling in their GPT models. Function calling simplifies the creation of chatbots that communicate with external tools and APIs, opening up a new realm of possibilities for AI-powered applications.

In this article, we will delve into the concept of function calling, its implications, and its transformative impact on how we interact with AI systems by creating NewsGPT, a chatbot that brings you breaking news worldwide.

Example run of NewsGPT. A command line interface shows the user asking, "What's the latest news in the US?" and receiving a list of five articles with links to the journals or sources.

What is Function Calling?

Function calling is a new feature in OpenAI's GPT-4-0613 and GPT-3.5 Turbo-0613 models. These AI models are trained to detect the need for function calling based on the user's prompt and respond with a structured call request instead of regular text.

Function calling allows chatbots to interact with other systems, enabling the GPT models to respond to questions they otherwise could not, such as those requiring real-time information or data not included in their training set. In other words, function calling provides another way to teach AI models how to interact with the external world.

What is the Purpose of Function Calling?

Before function calling, there were only two ways of augmenting the capabilities of a GPT language model:

Fine-tuning: further training the language model by providing example responses. Fine-tuning is a powerful technique, but it requires significant work (and cost) to prepare the training data. In addition, only a few older models can be fine-tuned until OpenAI enables this feature in GPT-3.5 and GPT-4 models.
Embeddings: enriching the prompt with context data can extend the bot's knowledge and create more accurate responses. The downside is that this context can take up a lot of tokens, increasing the cost and leaving fewer tokens free for building complex responses.

Function calling adds a third way of extending the GPT capabilities by allowing it to ask us to run functions on its behalf. The model can then take the function's result and build a human-readable response that fits seamlessly into the current conversation.

How to Use Function Calling

The introduction of function calling changes how we interact with the GPT API. Before these functions, the interaction was simple:

Send a prompt to the API.
Receive a response.
Repeat.

A diagram showing the basic interaction with the GPT API before function calling.

With function calling, the sequence becomes more involved:

Send the user prompt along with a list of callable functions.
The GPT model responds with either a regular text response or a function call request.
If the model requests a function call, your chatbot's job is to execute it and return the results to the API.
Using the supplied data, the model then forms a coherent text response. However, in some cases, the API may request a new function call.

A diagram showing the more complex interaction with the GPT API using function calling.

Function Calling with the Chat Completions API

To allow the model to call functions, we must use the Chat Completions API. The API takes a POST request with a JSON payload containing a list of messages to process. A typical prompt sent to the API looks like the following:

{
 "model": "gpt-3.5-turbo",
 "messages": [
 {
 "role": "user",
 "content": "How many planets does the solar system have?"
 }
 ]
}

The role: user informs the API that the content is user-generated. The GPT API might reply with something along these lines:

{
   "id":"chatcmpl-7WVo3fYwerpAptzeqU46JamOvgBzh",
   "object":"chat.completion",
   "created":1687983115,
   "model":"gpt-3.5-turbo-0613",
   "choices":[
      {
         "index":0,
         "message":{
            "role":"assistant",
            "content":"There are eight planets in the solar system. They are:\n\n1. Mercury\n2. Venus\n3. Earth\n4. Mars\n5. Jupiter\n6. Saturn\n7. Uranus\n8. Neptune"
         },
         "finish_reason":"stop"
      }
   ],
   "usage":{
      "prompt_tokens":15,
      "completion_tokens":44,
      "total_tokens":59
   }
}

The role: assistant corresponds to messages generated by the GPT model. To keep the conversation flow, we must supply the entire message history back to the API on each request. For example, if we want to delve deeper into our previous question, the corresponding JSON payload would be:

{
   "model":"gpt-3.5-turbo",
   "messages":[
      {
         "role":"user",
         "content":"How many planets does the solar system have?"
      },
      {
         "role":"assistant",
         "content":"There are eight planets in the solar system. They are:\n\n1. Mercury\n2. Venus\n3. Earth\n4. Mars\n5. Jupiter\n6. Saturn\n7. Uranus\n8. Neptune"
      },
      {
         "role":"user",
         "content":"Tell me more about the second planet."
      }
   ]
}

To let the language model know it can call functions, we need to add a list of them to the payload. For example:

{
   "model":"gpt-3.5-turbo-0613",
   "messages":[
      {
         "role":"user",
         "content":"How is the weather in NYC?"
      }
   ],
   "functions":[
      {
         "name":"get_current_weather",
         "description":"Get the current weather in a given location",
         "parameters":{
            "type":"object",
            "properties":{
               "location":{
                  "type":"string",
                  "description":"The city and state, e.g. San Francisco, CA"
               },
               "unit":{
                  "type":"string",
                  "enum":[
                     "celsius",
                     "fahrenheit"
                  ]
               }
            },
            "required":[
               "location"
            ]
         }
      }
   ]
}

You may have noticed that we switched the model to "gpt-3.5-turbo-0613" because it supports function calling. If the model decides to call the function, we will receive a response of type role: assistant with a function_call property defined like this:

{
   "id":"chatcmpl-7WWG94C1DCFlAk5xmUwrZ9OOhFnOq",
   "object":"chat.completion",
   "created":1687984857,
   "model":"gpt-3.5-turbo-0613",
   "choices":[
      {
         "index":0,
         "message":{
            "role":"assistant",
            "content":null,
            "function_call":{
               "name":"get_current_weather",
               "arguments":"{\n \"location\": \"New York, NY\"\n}"
            }
         },
         "finish_reason":"function_call"
      }
   ],
   "usage":{
      "prompt_tokens":81,
      "completion_tokens":19,
      "total_tokens":100
   }
}

Our task is to execute get_current_weather with the provided arguments. OpenAI does not execute the function. Instead, it's the job of our chatbot to run it and parse the returned data.

Once we retrieve the weather data, we send it back to the model using a new type of role called function. For example:

{
   "model":"gpt-3.5-turbo-0613",
   "messages":[
      {
         "role":"user",
         "content":"How is the weather in NYC?"
      },
      {
         "role":"assistant",
         "content":null,
         "function_call":{
            "name":"get_current_weather",
            "arguments":"{\n \"location\": \"New York, NY\"\n}"
         }
      },
      {
         "role":"function",
         "name":"get_current_weather",
         "content":"Temperature: 57F, Condition: Raining"
      }
   ],
   "functions":[
      {
         "name":"get_current_weather",
         "description":"Get the current weather in a given location",
         "parameters":{
            "type":"object",
            "properties":{
               "location":{
                  "type":"string",
                  "description":"The city and state, e.g. San Francisco, CA"
               },
               "unit":{
                  "type":"string",
                  "enum":[
                     "celsius",
                     "fahrenheit"
                  ]
               }
            },
            "required":[
               "location"
            ]
         }
      }
   ]
}

Note that we passed the entire message history to the API, including our original prompt, the function call from the model, and the result of executing the weather function in our code. This enables the language model to understand the context in which the function was called.

Finally, the model may reply with a properly formatted answer, responding to our initial question:

{
   "id":"chatcmpl-7WWQUccvLUfjhbIcuvFrj2MDJVEiN",
   "object":"chat.completion",
   "created":1687985498,
   "model":"gpt-3.5-turbo-0613",
   "choices":[
      {
         "index":0,
         "message":{
            "role":"assistant",
            "content":"The weather in New York City is currently raining with a temperature of 57 degrees Fahrenheit."
         },
         "finish_reason":"stop"
      }
   ],
   "usage":{
      "prompt_tokens":119,
      "completion_tokens":19,
      "total_tokens":138
   }
}

Building NewsGPT

To learn more about function calling, we will build NewsGPT, a Python chatbot capable of accessing breaking news in real time. The bot will use function calling to determine what kind of data to fetch from NewsAPI.org.

To build the bot, you will need the following:

An OpenAI API key. This requires a credit card because API requests have a cost. However, new accounts receive \$5 credit for the first three months.
A NewsAPI API key. Register at NewsAPI.org and get a starter key for free.
Python 3.

Setting Up the Project

Install the required dependencies:

$ pip install openai tiktoken

The project consists of only one file; let's name it newsgpt.py. I will begin by adding all the necessary imports:

import openai
import tiktoken
import json
import os
import requests

Next, I will define a few constants:

The GPT model to use. I will use gpt-3.5-turbo-16k as it has a 16k token limit, allowing me to process longer conversations with more context.
The system prompt that instructs the model on its basic purpose.
The encoding used to count tokens in strings and messages; required to ensure we do not exceed the language model limits.
The maximum number of functions to call in a chain (more on this later).

llm_model = "gpt-3.5-turbo-16k"
llm_max_tokens = 15500
llm_system_prompt = "You are an assistant that provides news and headlines to user requests. Always try to get the lastest breaking stories using the available function calls."
encoding_model_messages = "gpt-3.5-turbo-0613"
encoding_model_strings = "cl100k_base"
function_call_limit = 3

All OpenAI models have a token limit. If this limit is exceeded, the API will throw an error instead of responding to our request. So, we need a function to count the number of tokens. I will use this function from the official example documentation:

def num_tokens_from_messages(messages):
 """Returns the number of tokens used by a list of messages."""
 try:
 encoding = tiktoken.encoding_for_model(encoding_model_messages)
 except KeyError:
 encoding = tiktoken.get_encoding(encoding_model_strings)

 num_tokens = 0
 for message in messages:
 num_tokens += 4
 for key, value in message.items():
 num_tokens += len(encoding.encode(str(value)))
 if key == "name":
 num_tokens += -1
 num_tokens += 2
 return num_tokens

Defining a Function to Call

Now, I will define a function to query the NewsAPI.org API to get the breaking news:


def get_top_headlines(query: str = None, country: str = None, category: str = None):
 """Retrieve top headlines from newsapi.org (API key required)"""

    base_url = "https://newsapi.org/v2/top-headlines"
    headers = {
    "x-api-key": os.environ['NEWS_API_KEY']
    }
    params = { "category": "general" }
    if query is not None:
    params['q'] = query
    if country is not None:
    params['country'] = country
    if category is not None:
    params['category'] = category

    # Fetch from newsapi.org - reference: https://newsapi.org/docs/endpoints/top-headlines
    response = requests.get(base_url, params=params, headers=headers)
    data = response.json()

    if data['status'] == 'ok':
    print(f"Processing {data['totalResults']} articles from newsapi.org")
    return json.dumps(data['articles'])
    else:
    print("Request failed with message:", data['message'])
    return 'No articles found'

To inform GPT about this function, we need to describe using a specific JSON structure. The format is described in the official documentation as follows:

signature_get_top_headlines = {
   "name":"get_top_headlines",
   "description":"Get top news headlines by country and/or category",
   "parameters":{
      "type":"object",
      "properties":{
         "query":{
            "type":"string",
            "description":"Freeform keywords or a phrase to search for."
         },
         "country":{
            "type":"string",
            "description":"The 2-letter ISO 3166-1 code of the country you want to get headlines for"
         },
         "category":{
            "type":"string",
            "description":"The category you want to get headlines for",
            "enum":[
               "business",
               "entertainment",
               "general",
               "health",
               "science",
               "sports",
               "technology"
            ]
         }
      },
      "required":[

      ]
   }
}

Using the Chat Completions API with Function Calling

Next, I will define the complete function, which performs several tasks:

Adds a system prompt at the end of the messages. This system message helps define the role that the GPT model will fulfill.
Removes old messages if the total token count exceeds the model's limit.
Sends the request to the GPT API.
Removes the system message from the end of the list.

def complete(messages, function_call: str = "auto"):
 """Fetch completion from OpenAI's GPT"""

 messages.append({"role": "system", "content": llm_system_prompt})

 # delete older completions to keep conversation under token limit
 while num_tokens_from_messages(messages) >= llm_max_tokens:
 messages.pop(0)

 print('Working...')
 res = openai.ChatCompletion.create(
 model=llm_model,
 messages=messages,
 functions=[signature_get_top_headlines],
 function_call=function_call
 )

 # remove system message and append response from the LLM
 messages.pop(-1)
 response = res["choices"][0]["message"]
 messages.append(response)

 # call functions requested by the model
 if response.get("function_call"):
 function_name = response["function_call"]["name"]
 if function_name == "get_top_headlines":
 args = json.loads(response["function_call"]["arguments"])
 headlines = get_top_headlines(
 query=args.get("query"),
 country=args.get("country"),
 category=args.get("category") 
 )
 messages.append({ "role": "function", "name": "get_top_headline", "content": headlines})

To complete the bot, we will create the main loop that continuously prompts the user for input and provides the answers:

print("\nHi, I'm a NewsGPT, a breaking news AI assistant. I can give you news for most countries over a wide range of categories.")
print("Here are some example prompts:\n - Tell me about the recent science discoveries\n - What is the latest news in the US?\n - What has Elon Musk been up to recently?")

messages = []
while True:
    prompt = input("\nWhat would you like to know? => ")
    messages.append({"role": "user", "content": prompt})
    complete(messages)

    # the LLM can chain function calls, this implements a limit
    call_count = 0
    while messages[-1]['role'] == "function":
    call_count = call_count + 1
    if call_count < function_call_limit:
    complete(messages)
    else:
    complete(messages, function_call="none")

    # print last message
    print("\n\n==Response==\n")
    print(messages[-1]["content"].strip())
    print("\n==End of response==")

Next, we need to define environment variables for the OpenAI and NewsAPI API keys:

export OPENAI_API_KEY=YOUR_API_KEY
export NEWS_API_KEY=YOUR_API_KEY

Finally, load the file and run the chatbot.

$ source .env
$ python newsgpt.py

Testing NewsGPT

We can inquire about a specific topic or person. For example:

=> What is Elon Musk up these days?
Working...
Processing 1 articles from newsapi.org
Working...


==Response==

Elon Musk has been in the news recently regarding a feud with Mark Zuckerberg, the CEO of Meta. According to an article from The Washington Post, Mark Zuckerberg is trying to make himself more relevant to the tech elite, and there seems to be a rivalry between him and Elon Musk. The article discusses their ongoing fight and how Zuckerberg is trying to compete with Musk's online antagonism and offline antics. You can read more about it [here](https://www.washingtonpost.com/technology/2023/06/24/elon-musk-mark-zuckerberg-fight-meta-twitter-rival/).

==End of response==

The bot is also capable of recognizing categories, allowing us to request health-related news:

Are there important developments in health?
Working...
Processing 1000 articles from newsapi.org
Working...


==Response==

Based on the latest headlines in the health category, here are some important developments:

1. [A Case of Herpes Zoster Ophthalmicus in a Recently Transplanted Renal Patient](https://www.cureus.com/articles/164429-a-case-of-herpes-zoster-ophthalmicus-in-a-recently-transplanted-renal-patient?score_article=true) - This case study presents the clinical course of a 51-year-old male who underwent a renal transplant and developed Herpes Zoster Ophthalmicus.

2. [Cuáles son los principales consejos a seguir para mejorar el estado de ánimo a base de una buena alimentación](https://news.google.com/rss/articles/CBMilQFodHRwczovL3d3dy5jYW5hbDI2LmNvbS9nZW5lcmFsL2N1YWxlcy1zb24tbG9zLXByaW5jaXBhbGVzLWNvbnNlam9zLWEtc2VndWlyLXBhcmEtbWVqb3Jhci1lbC1lc3RhZG8tZGUtYW5pbW8tYS1iYXNlLWRlLXVuYS1idWVuYS1hbGltZW50YWNpb24tLTM0NTMwM9IBAA?oc=5) (in Spanish) - This article provides tips on improving mood through a good diet.

3. [Как да стопите коремните мазнини за лятото](https://news.google.com/rss/articles/CBMiXmh0dHBzOi8vbS5hei1qZW5hdGEuYmcvYS81LXpkcmF2ZS1pLWtyYXNvdGEvNjM1Mzkta2FrLWRhLXN0b3BpdGUta29yZW1uaXRlLW1hem5pbmktemEtbGlhdG90by_SAWJodHRwczovL20uYXotamVuYXRhLmJnL2EvNS16ZHJhdmUtaS1rcmFzb3RhLzYzNTM5LWthay1kYS1zdG9waXRlLWtvcmVtbml0ZS1tYXpuaW5pLXphLWxpYXRvdG8vYW1wLw?oc=5) (in Bulgarian) - This article provides tips on losing belly fat for the summer.

4. [Recap: From RCC Diagnosis to Treatment, Toxicity Management, and Beyond](https://news.google.com/rss/articles/CBMiZ2h0dHBzOi8vd3d3LmNhbmNlcm5ldHdvcmsuY29tL3ZpZXcvcmVjYXAtZnJvbS1yY2MtZGlhZ25vc2lzLXRvLXRyZWF0bWVudC10b3hpY2l0eS1tYW5hZ2VtZW50LWFuZC1iZXlvbmTSAQA?oc=5) - This article discusses the diagnosis, treatment, and management of renal cell carcinoma.

5. [Review Highlights Progress, Challenges With CRS in Cancer Immunotherapies](https://news.google.com/rss/articles/CBMiYmh0dHBzOi8vd3d3LmFqbWMuY29tL3ZpZXcvcmV2aWV3LWhpZ2hsaWdodHMtcHJvZ3Jlc3MtY2hhbGxlbmdlcy13aXRoLWNycy1pbi1jYW5jZXItaW1tdW5vdGhlcmFwaWVz0gEA?oc=5) - This review highlights the progress and challenges in cancer immunotherapies targeting CRS (cytokine release syndrome).

These are just a few of the recent developments in the field of health. For more detailed information, you can read the full articles by clicking on the provided links.

==End of response==

It can countries as well, enabling us to query the bot for news about a particular region:

What is the latest news from France?
Working...
Processing 34 articles from newsapi.org
Working...


==Response==

Here are some of the latest news headlines in France:

1. "Diplôme national du brevet session 2023" - The Ministry of National Education announces the national diploma for the 2023 session. [Read more](https://news.google.com/rss/articles/CBMiTGh0dHBzOi8vd3d3LmVkdWNhdGlvbi5nb3V2LmZyL2RpcGxvbWUtbmF0aW9uYWwtZHUtYnJldmV0LXNlc3Npb24tMjAyMy0zNzg1NjDSAQA?oc=5)

2. "Cyclisme: la Nordiste Victoire Berteau sacrée championne de France après sa victoire sur les routes de Cassel" - Victoire Berteau from Nord wins the championship in cycling in France. [Read more](https://news.google.com/rss/articles/CBMiiQFodHRwczovL3d3dy5mcmFuY2V0dmluZm8uZnIvc3BvcnRzL2N5Y2xpc21lL2N5Y2xpc21lLXZpY3RvaXJlLWJlcnRlYXUtc2FjcmVlLWNoYW1waW9ubmUtZGUtZnJhbmNlLWFwcmVzLXNhLXZpY3RvaXJlLWEtY2Fzc2VsXzU5MDg4NDcuaHRtbNIBAA?oc=5)

3. "Guerre en Ukraine: comment les capitales étrangères réagissent-elles à la rébellion de la milice Wagner en Ru" - Foreign capitals' reactions to the rebellion of the Wagner militia in Ukraine. [Read more](https://news.google.com/rss/articles/CBMiwAFodHRwczovL3d3dy5mcmFuY2V0dmluZm8uZnIvbW9uZGUvZXVyb3BlL21hbmlmZXN0YXRpb25zLWVuLXVrcmFpbmUvZ3VlcnJlLWVuLXVrcmFpbmUtY29tbWVudC1sZXMtY2FwaXRhbGVzLWV0cmFuZ2VyZXMtcmVhZ2lzc2VudC1lbGxlcy1hLXJlYmVsbGlvbi1kZS1sYS1taWxpY2Utd2FnbmVyLWVuLXJ1c3NpZV81OTA4NzY2Lmh0bWzSAQA?oc=5)

4. "Marche des fiertés LGBT+: six jeunes mineurs interpellés pour homophobie" - Six minors arrested for homophobia during the LGBT+ Pride March. [Read more](https://news.google.com/rss/articles/CBMifmh0dHBzOi8vd3d3LnJ0bC5mci9hY3R1L2p1c3RpY2UtZmFpdHMtZGl2ZXJzL21hcmNoZS1kZXMtZmllcnRlcy1sZ2J0LXNpeC1qZXVuZXMtbWluZXVycy1pbnRlcnBlbGxlcy1wb3VyLWhvbW9waG9iaWUtNzkwMDI3Nzg4M9IBAA?oc=5)

5. "ATP 500 Queen's - De Minaur a dominé Rune avec autorité: le film de la demi-finale" - Alex de Minaur dominates Rune in the ATP 500 Queen's semifinals. [Read more](https://news.google.com/rss/articles/CBMimwFodHRwczovL3d3dy5ldXJvc3BvcnQuZnIvdGVubmlzL2F0cC1sb25kcmVzLzIwMjMvYXRwLTUwMC1xdWVlbi1zLXN1aXZlei1sYS1kZW1pLWZpbmFsZS1lbnRyZS1hbGV4LWRlLW1pbmF1ci1ldC1ob2xnZXItcnVuZS1lbi1kaXJlY3Rfc3RvOTY3MTM4My9zdG9yeS5zaHRtbNIBAA?oc=5)

These are just a few of the latest news headlines in France. Let me know if you want more information about any specific news article.

==End of response==

Ideas for Improvement

This simple bot is quite capable, even with a single function call. Now, imagine the possibilities if we integrate more features. Here are a few ideas to augment NewsGPT:

Retrieve the original articles to get summaries and analyze the news. We would need to navigate paywalls, perform web scraping, or check if RSS feeds or APIs provide content.
Add more endpoints. NewsAPI offers endpoints for searching news by date, categories, and filtering through sources.
Incorporate extra integrations, such as obtaining real-time data from sources like weather or finance.

Conclusion

Function calling is a powerful feature in OpenAI's GPT models, enabling them to interact with external tools and APIs in a more deterministic and structured manner. This feature lays the groundwork for more dynamic and responsive AI applications capable of providing current information and executing tasks beyond what was previously possible.

Happy building!

Word Embeddings: Giving Your ChatBot Context For Better Answers

Tomas Fernandez — Tue, 25 Jul 2023 22:19:52 +0000

Photo by Mojahid Mottakin on Unsplash

There is no doubt that OpenAI's ChatGPT is exceptionally intelligent — it has passed the lawyer's bar test, it possesses knowledge akin to a doctor, and some tests have clocked its IQ at 155. However, it tends to fabricate information instead of conceding ignorance. This tendency, coupled with the fact that its knowledge ceases in 2021, poses challenges in building specialized products using the GPT API.

How can we surmount these obstacles? How can we impart new knowledge to a model like GPT-3? My goal is to address these questions by constructing a question-answering bot employing Python, the OpenAI API, and word embeddings.

What I Will Be Building

I intend to create a bot that generates continuous integration pipelines from a prompt, which, as you may know, are formated with YAML in Semaphore CI/CD.

Here is an example of the bot in action:

Screenshot of the running program. On the screen, the command python query.py "Create a CI pipeline that builds and uploads a Docker image to Docker Hub" is executed, and the program prints out YAML corresponding to a CI pipeline that performs the requested action.

In the spirit of projects like DocsGPT, My AskAI, and Libraria, I plan to "teach" the GPT-3 model about Semaphore and how to generate pipeline configuration files. I will achieve this by leveraging the existing documentation.

I will not assume prior knowledge of bot building and will maintain clean code so that you can adapt it to your requirements.

Prerequisites

You do not need experience in coding a bot or knowledge of neural networks to follow this tutorial. However, you will need:

Python 3.
A Pinecone account (sign up for the Starter plan for free).
An OpenAI API Key (paid, requires a credit card); new users can experiment with $5 in free credit during the first 3 months.

But ChatGPT Can't Learn, Can It?

ChatGPT, or more accurately, GPT-3 and GPT-4, the Large Language Models (LLMs) powering them, have been trained on a massive dataset with a cutoff date around September 2021.

In essence, GPT-3 knows very little about events beyond that date. We can verify this with a simple prompt:

ChatGPT doesn't know who won the World Cup in 2022.

While some OpenAI models can undergo fine-tuning, the more advanced models, such as the ones were interested in, cannot; we cannot augment their training data.

How can we get answers from GPT-3 beyond its training data? One method involves exploiting its text comprehension abilities; by enhancing the prompt with relevant context, we can likely obtain the correct answer.

In the example below, I provide context from FIFA's official site, and the response differs significantly:

With the supplied context, ChatGPT can answer accurately.

We can deduce that the model can respond to any prompt if given enough relevant context. The question remains: how can we know what's relevant given an arbitrary prompt? To address this, we need to explore what word embeddings are.

What Are Word Embeddings?

In the context of language models, an embedding is a way of representing words, sentences, or entire documents as vectors or lists of numbers.

To calculate embeddings, we will need a neural network such as word2vec or text-embedding-ada-002. These networks have been trained on massive amounts of text and can find relationships between words by analyzing the frequencies with which specific patterns appear in the training data.

Let’s say we have the following words:

Cat
Dog
Ball
House

Imagine we use one of these embedding networks to calculate the vectors for each word. For example:

Word	Vector	Context
Cat	[0.1, 0.2, 0.3, 0.4, 0.5]	Animals, objects, small things
Dog	[0.6, 0.7, 0.8, 0.9, 1.0]	Animals, objects, large things
Ball	[0.2, 0.4, 0.6, 0.8, 1.0]	Objects, toys, small things
House	[0.3, 0.6, 0.9, 1.2, 1.5]	Buildings, homes, large things

Once we have the vectors for each word, we can use them to represent the meaning of the text. For example, the sentence “The cat chased the ball” can be represented as the vector [0.1, 0.2, 0.3, 0.4, 0.5] + [0.2, 0.4, 0.6, 0.8, 1.0] = [0.3, 0.6, 0.9, 1.2, 1.5]. This vector represents a sentence that is about an animal chasing an object.

Word embeddings can be visualized as multidimensional spaces where words or sentences with similar meanings are close together. We can compute the "distance" between vectors to find similar meanings for any input text.

3D representation of embeddings as vector spaces. In reality, these spaces can have hundreds or thousands of dimensions. Source: Meet AI’s Multitool: Vector Embeddings

The actual mathematics behind all this is beyond the scope of this article. However, the key takeaway is that vector operations allow us to manipulate or determine meaning using mathematics. Take the vector that represents the word “queen,” subtract the “woman” vector from it, and add the “man” vector. The result should be a vector in the vicinity of “king.” If we add “son,” we should get somewhere close to “prince.”

Embedding Neural Networks with Tokens

So far, we have discussed embedding neural networks taking words as inputs and numbers as outputs. However, many modern networks have moved from processing words to processing tokens.

A token is the smallest unit of text that can be processed by the model. Tokens can be words, characters, punctuation marks, symbols, or parts of words.

We can see how words are converted to tokens by experimenting with the OpenAI online tokenizer, which uses Byte-Pair Encoding (BPE) to convert text to tokens and represent each one with a number:

There is often a 1-to-1 relationship between tokens and words. Most tokens include the word and a leading space. However, there are special cases like "embedding," which consists of two tokens, "embed" and "ding," or "capabilities," which consists of four tokens. If you click "Token IDs," you can see the model's numeric representation of each token.

Designing a Smarter Bot Using Embeddings

Now that we have an understanding of what embeddings are, the next question is: how can they help us build a smarter bot?

First, let's consider what happens when we use the GPT-3 API directly. The user issues a prompt, and the model responds to the best of its ability.

However, when we add context to the equation, things change. For example, when I asked ChatGPT about the winner of the World Cup after providing context, it made all the difference.

So, the plan to build a smarter bot is as follows:

Intercept the user's prompt.
Calculate the embeddings for that prompt, yielding a vector.
Search a database for documents near the vector, as they should be semantically relevant to the initial prompt.
Send the original prompt to GPT-3, along with any relevant context.
Forward GPT-3's response to the user.

Let's begin like most projects, by designing the database.

Creating a Knowledge Database with Embeddings

Our context database must include the original documentation and their respective vectors. In principle, we can employ any type of database for this task, but a vector database is the optimal tool for the job.

Vector databases are specialized databases designed to store and retrieve high-dimensional vector data. Instead of employing a query language such as SQL for searching, we supply a vector and request the N closest neighbors.

To generate the vectors, we will use text-embedding-ada-002 from OpenAI, as it is the fastest and most cost-effective model they offer. The model converts the input text into tokens and uses an attention mechanism known as Transformer to learn their relationships. The output of this neural network is vectors representing the meaning of the text.

To create a context database, I will:

Collect all the source documentation.
Filter out irrelevant documents.
Calculate the embeddings for each document.
Store the vectors, original text, and any other relevant metadata in the database.

Converting Documents into Vectors

First, I must initialize an environment file with the OpenAI API key. This file should never be committed to version control, as the API key is private and tied to your account.

export OPENAI_API_KEY=YOUR_API_KEY

Next, I'll create a virtualenv for my Python application:

$ virtualenv venv
$ source venv/bin/activate
$ source .env

And install the OpenAI package:

bash
$ pip install openai numpy


Let's try calculating the embedding for the string "Docker Container". You can run this on the Python REPL or as a Python script:

python
$ python

import openai

embeddings = openai.Embedding.create(input="Docker Containers", engine="text-embedding-ada-002")

embeddings

JSON: {
"data": [
{
"embedding": [
-0.00530336843803525,
0.0013223182177171111,

... 1533 more items ...,

-0.015645816922187805
],
"index": 0,
"object": "embedding"
}
],
"model": "text-embedding-ada-002-v2",
"object": "list",
"usage": {
"prompt_tokens": 2,
"total_tokens": 2
}
}


As you can see, OpenAI's model responds with an `embedding` list containing 1536 items — the vector size for text-embedding-ada-002.

## Storing the embeddings in Pinecone

While there are multiple vector database engines to choose from, like [Chroma](https://www.trychroma.com/) which is open-source, I chose [Pinecone](https://www.pinecone.io/) because its a managed database with a free tier, which makes things simpler. Their [Starter plan](https://www.pinecone.io/pricing/) is more than capable of handling all the data I will need.

After creating my Pinecone account and retrieving my API key and environment, I add both values to my `.env` file.


![Pinecone API Key generation screenshot](https://raw.githubusercontent.com/tomfern/blog-assets/master/images/aa79dbcadd804a2b1aefe607553f50e960619717590f573da6ca04e967668802.jpg)

Now `.env` should contain my Pinecone and OpenAI secrets.

bash
export OPENAI_API_KEY=YOUR_API_KEY

Pinecone secrets

export PINECONE_API_KEY=YOUR_API_KEY
export PINECONE_ENVIRONMENT=YOUR_PINECONE_DATACENTER


Then, I install the Pinecone client for Python:

bash
$ pip install pinecone-client


I need to initialize a database; these are the contents of the `db_create.py` script:

python

db_create.py

import pinecone
import openai
import os

index_name = "semaphore"
embed_model = "text-embedding-ada-002"

api_key = os.getenv("PINECONE_API_KEY")
env = os.getenv("PINECONE_ENVIRONMENT")
pinecone.init(api_key=api_key, environment=env)

embedding = openai.Embedding.create(
input=[
"Sample document text goes here",
"there will be several phrases in each batch"
], engine=embed_model
)

if index_name not in pinecone.list_indexes():
print("Creating pinecone index: " + index_name)
pinecone.create_index(
index_name,
dimension=len(embedding['data'][0]['embedding']),
metric='cosine',
metadata_config={'indexed': ['source', 'id']}
)


The script can take a few minutes to create the database.

bash
$ python db_create.py


Next, I will install the [tiktoken](https://github.com/openai/tiktoken) package. I'll use it to calculate how many tokens the source documents have. This is important because the embedding model can only handle up to 8191 tokens.

bash
$ pip install tiktoken


While installing packages, let's also install `tqdm` to produce a nice-looking progress bar.

bash
$ pip install tqdm


Now I need to upload the documents to the database. The script for this will be called `index_docs.py`. Let's start by importing the required modules and defining some constants:

python

index_docs.py

Pinecone db name and upload batch size

index_name = 'semaphore'
upsert_batch_size = 20

OpenAI embedding and tokenizer models

embed_model = "text-embedding-ada-002"
encoding_model = "cl100k_base"
max_tokens_model = 8191


Next, we'll need a function to count tokens. There is a [token counter example](https://platform.openai.com/docs/guides/embeddings/how-can-i-tell-how-many-tokens-a-string-has-before-i-embed-it) on OpenAI page:

python
import tiktoken
def num_tokens_from_string(string: str) -> int:
"""Returns the number of tokens in a text string."""
encoding = tiktoken.get_encoding(encoding_model)
num_tokens = len(encoding.encode(string))
return num_tokens


Finally, I'll need some filtering functions to convert the original document into usable examples. Most examples in the documentation are between code fences, so I'll just extract all YAML code from every file:

python
import re
def extract_yaml(text: str) -> str:
"""Returns list with all the YAML code blocks found in text."""
matches = [m.group(1) for m in re.finditer("


", text)]
 return matches

I'm done with the functions. Next, this will load the files in memory and extract the examples:


 python

from tqdm import tqdm
import sys
import os
import pathlib

repo_path = sys.argv[1]
repo_path = os.path.abspath(repo_path)
repo = pathlib.Path(repo_path)

markdown_files = list(repo.glob("**/*.md")) + list(
 repo.glob("**/*.mdx")
)

print(f"Extracting YAML from Markdown files in {repo_path}")
new_data = []
for i in tqdm(range(0, len(markdown_files))):
 markdown_file = markdown_files[i]
 with open(markdown_file, "r") as f:
 relative_path = markdown_file.relative_to(repo_path)
 text = str(f.read())
 if text == '':
 continue
 yamls = extract_yaml(text)
 j = 0
 for y in yamls:
 j = j+1
 new_data.append({
 "source": str(relative_path),
 "text": y,
 "id": f"github.com/semaphore/docs/{relative_path}[{j}]"
 })

At this point, all the YAMLs should be stored in the new_data list. The final step is to upload the embeddings into Pinecone.


 python

import pinecone
import openai

api_key = os.getenv("PINECONE_API_KEY")
env = os.getenv("PINECONE_ENVIRONMENT")
pinecone.init(api_key=api_key, enviroment=env)
index = pinecone.Index(index_name)

print(f"Creating embeddings and uploading vectors to database")
for i in tqdm(range(0, len(new_data), upsert_batch_size)):

 i_end = min(len(new_data), i+upsert_batch_size)
 meta_batch = new_data[i:i_end]
 ids_batch = [x['id'] for x in meta_batch]
 texts = [x['text'] for x in meta_batch]

 embedding = openai.Embedding.create(input=texts, engine=embed_model)
 embeds = [record['embedding'] for record in embedding['data']]

 # clean metadata before upserting
 meta_batch = [{
 'id': x['id'],
 'text': x['text'],
 'source': x['source']
 } for x in meta_batch] 

 to_upsert = list(zip(ids_batch, embeds, meta_batch))
 index.upsert(vectors=to_upsert)

As a reference, you can find the full index_docs.py file in the demo repository

Let's run the index script to finish with the database setup:


 bash
$ git clone https://github.com/semaphoreci/docs.git /tmp/docs
$ source .env
$ python index_docs.py /tmp/docs

Testing the database

The Pinecone dashboard should show vectors in the database.

We can query the database with the following code, which you can run as a script or in the Python REPL directly:


 python
$ python

>>> import os
>>> import pinecone
>>> import openai

# Compute embeddings for string "Docker Container"
>>> embeddings = openai.Embedding.create(input="Docker Containers", engine="text-embedding-ada-002")


# Connect to database
>>> index_name = "semaphore"
>>> api_key = os.getenv("PINECONE_API_KEY")
>>> env = os.getenv("PINECONE_ENVIRONMENT")
>>> pinecone.init(api_key=api_key, environment=env)
>>> index = pinecone.Index(index_name)

# Query database
>>> matches = index.query(embeddings['data'][0]['embedding'], top_k=1, include_metadata=True)

>>> matches['matches'][0]
{'id': 'github.com/semaphore/docs/docs/ci-cd-environment/docker-authentication.md[3]',
 'metadata': {'id': 'github.com/semaphore/docs/docs/ci-cd-environment/docker-authentication.md[3]',
 'source': 'docs/ci-cd-environment/docker-authentication.md',
 'text': '\n'
 '# .semaphore/semaphore.yml\n'
 'version: v1.0\n'
 'name: Using a Docker image\n'
 'agent:\n'
 ' machine:\n'
 ' type: e1-standard-2\n'
 ' os_image: ubuntu1804\n'
 '\n'
 'blocks:\n'
 ' - name: Run container from Docker Hub\n'
 ' task:\n'
 ' jobs:\n'
 ' - name: Authenticate docker pull\n'
 ' commands:\n'
 ' - checkout\n'
 ' - echo $DOCKERHUB_PASSWORD | docker login '
 '--username "$DOCKERHUB_USERNAME" --password-stdin\n'
 ' - docker pull /\n'
 ' - docker images\n'
 ' - docker run /\n'
 ' secrets:\n'
 ' - name: docker-hub\n'},
 'score': 0.796259582,
 'values': []}

As you can see, the first match is the YAML for a Semaphore pipeline that pulls a Docker image and runs it. It's a good start since it's relevant to our "Docker Containers" search string.

Building the bot

We have the data, and we know how to query it. Let's put it to work in the bot.

The steps for processing the prompt is:

Take the user's prompt.
Calculate its vector.
Retrieve relevant context from the database.
Send the user's prompt along with context to GPT-3.
Forward the model's response to the user.

As usual, I'll start by defining some constants in complete.py, the bot's main script:


 python
# complete.py

# Pinecone database name, number of matched to retrieve
# cutoff similarity score, and how much tokens as context
index_name = 'semaphore'
context_cap_per_query = 30
match_min_score = 0.75
context_tokens_per_query = 3000

# OpenAI LLM model parameters
chat_engine_model = "gpt-3.5-turbo"
max_tokens_model = 4096
temperature = 0.2 
embed_model = "text-embedding-ada-002"
encoding_model_messages = "gpt-3.5-turbo-0301"
encoding_model_strings = "cl100k_base"

import pinecone
import os

# Connect with Pinecone db and index
api_key = os.getenv("PINECONE_API_KEY")
env = os.getenv("PINECONE_ENVIRONMENT")
pinecone.init(api_key=api_key, environment=env)
index = pinecone.Index(index_name)

Next, I'll add functions to count tokens as shown in the OpenAI examples. The first function counts tokens in a string, while the second counts tokens in messages. We'll see messages in detail in a bit. For now, let's just say it's a structure that keeps the state of the conversation in memory.


 python
import tiktoken

def num_tokens_from_string(string: str) -> int:
 """Returns the number of tokens in a text string."""
 encoding = tiktoken.get_encoding(encoding_model_strings)
 num_tokens = len(encoding.encode(string))
 return num_tokens


def num_tokens_from_messages(messages):
 """Returns the number of tokens used by a list of messages. Compatible with model """

 try:
 encoding = tiktoken.encoding_for_model(encoding_model_messages)
 except KeyError:
 encoding = tiktoken.get_encoding(encoding_model_strings)

 num_tokens = 0
 for message in messages:
 num_tokens += 4 # every message follows {role/name}\n{content}\n
 for key, value in message.items():
 num_tokens += len(encoding.encode(value))
 if key == "name": # if there's a name, the role is omitted
 num_tokens += -1 # role is always required and always 1 token
 num_tokens += 2 # every reply is primed with assistant
 return num_tokens

The following function takes the original prompt and context strings to return an enriched prompt for GPT-3:


 python
def get_prompt(query: str, context: str) -> str:
 """Return the prompt with query and context."""
 return (
 f"Create the continuous integration pipeline YAML code to fullfil the requested task.\n" +
 f"Below you will find some context that may help. Ignore it if it seems irrelevant.\n\n" +
 f"Context:\n{context}" +
 f"\n\nTask: {query}\n\nYAML Code:"
 )

The get_message function formats the prompt in a format compatible with API:


 python
def get_message(role: str, content: str) -> dict:
 """Generate a message for OpenAI API completion."""
 return {"role": role, "content": content}

There are three types of roles that affect how the model reacts:

User: for the user's original prompt.
System: helps set the behavior of the assistant. While there is some controversy regarding its effectiveness, it appears to be more effective when sent at the end of the messages list.
Assistant: represents past responses of the model. The OpenAI API does not have a "memory"; instead, we must send the model's previous responses back during each interaction to maintain the conversation.

Now for the engaging part. The get_context function takes the prompt, queries the database, and generates a context string until one of these conditions is met:

The complete text exceeds context_tokens_per_query, the space I reserved for context.
The search function retrieves all requested matches.
Matches that have a similarity score below match_min_score are ignored.


 python
import openai

def get_context(query: str, max_tokens: int) -> list:
 """Generate message for OpenAI model. Add context until hitting `context_token_limit` limit. Returns prompt string."""

 embeddings = openai.Embedding.create(
 input=[query],
 engine=embed_model
 )

 # search the database
 vectors = embeddings['data'][0]['embedding']
 embeddings = index.query(vectors, top_k=context_cap_per_query, include_metadata=True)
 matches = embeddings['matches']

 # filter and aggregate context
 usable_context = ""
 context_count = 0
 for i in range(0, len(matches)):

 source = matches[i]['metadata']['source']
 if matches[i]['score'] < match_min_score:
 # skip context with low similarity score
 continue

 context = matches[i]['metadata']['text']
 token_count = num_tokens_from_string(usable_context + '\n---\n' + context)

 if token_count < context_tokens_per_query:
 usable_context = usable_context + '\n---\n' + context 
 context_count = context_count + 1

 print(f"Found {context_count} contexts for your query")

 return usable_context

The next and final function, complete, issues the API request to OpenAI and returns the model's response.


 python
def complete(messages):
 """Query the OpenAI model. Returns the first answer. """

 res = openai.ChatCompletion.create(
 model=chat_engine_model,
 messages=messages,
 temperature=temperature
 )
 return res.choices[0].message.content.strip()

That's all; now I only have to deal with the command line arguments and call the functions in the correct order:


 python
import sys

query = sys.argv[1]

context = get_context(query, context_tokens_per_query)
prompt = get_prompt(query, context)

# initialize messages list to send to OpenAI API
messages = []
messages.append(get_message('user', prompt))
messages.append(get_message('system', 'You are a helpful assistant that writes YAML code for Semaphore continuous integration pipelines and explains them. Return YAML code inside code fences.'))

if num_tokens_from_messages(messages) >= max_tokens_model:
 raise Exception('Model token size limit reached') 

print("Working on your query... ")
answer = complete(messages)
print("Answer:\n")
print(answer)
messages.append(get_message('assistant', answer))

It's time to run the script and see how it fares:


 bash
$ python complete.py "Create a CI pipeline that builds and uploads a Docker image to Docker Hub"

The result is:


 yaml
version: v1.0
name: Docker Build and Push
agent:
 machine:
 type: e1-standard-2
 os_image: ubuntu1804

blocks:
 - name: "Build and Push Docker Image"
 task:
 jobs:
 - name: "Docker Build and Push"
 commands:
 - checkout
 - docker build -t /: .
 - echo "$DOCKERHUB_PASSWORD" | docker login -u "$DOCKERHUB_USERNAME" --password-stdin
 - docker push /:

promotions:
 - name: Deploy to production
 pipeline_file: deploy-production.yml
 auto_promote:
 when: "result = 'passed' and branch = 'master'"

This is the first good result. The model has inferred the syntax from the context examples we provided.

Thoughts on Expanding the Bot's Capabilities

Remember that I started with a modest goal: creating an assistant to write YAML pipelines. With richer content in my vector database, I can generalize the bot to answer any question about Semaphore (or any product — remember cloning the docs into /tmp?).

The key to obtaining good answers is — unsurprisingly — quality context. Merely uploading every document into the vector database is unlikely to yield good results. The context database should be curated, tagged with descriptive metadata, and be concise. Otherwise, we risk filling the token quota in the prompt with irrelevant context.

So, in a sense, there is an art — and a great deal of trial and error — involved in fine-tuning the bot to meet our needs. We can experiment with the context limit, remove low-quality content, summarize, and filter out irrelevant context by adjusting the similarity score.

Implementing a Proper Chatbot

You may have noticed that my bot does not enable us to have actual conversation like ChatGPT. We ask one question and get one answer.

Converting the bot into a fully-fledged chatbot is, in principle, not too challenging. We can maintain the conversation by resending previous responses to the model with each API request. Prior GPT-3 answers are sent back under the "assistant" role. For example:


 python

messages = []

while True:

 query = input('Type your prompt:\n')

 context = get_context(query, context_tokens_per_query)
 prompt = get_prompt(query, context)
 messages.append(get_message('user', prompt))
 messages.append(get_message('system', 'You are a helpful assistant that writes YAML code for Semaphore continuous integration pipelines and explains them. Return YAML code inside code fences.'))

 if num_tokens_from_messages(messages) >= max_tokens_model:
 raise Exception('Model token size limit reached') 

 print("Working on your query... ")
 answer = complete(messages)
 print("Answer:\n")
 print(answer)

 # remove system message and append model's answer
 messages.pop() 
 messages.append(get_message('assistant', answer))

Unfortunately, this implementation is rather rudimentary. It will not support extended conversations as the token count increases with each interaction. Soon enough, we will reach the 4096-token limit for GPT-3, preventing further dialogue.

So, we have to find some way of keeping the request within token limits. A few strategies follow:

Delete older messages. While this is the simplest solution, it limits the conversation's "memory" to only the most recent messages.
Summarize previous messages. We can utilize "Ask the model" to condense earlier messages and substitute them for the original questions and answers. Though this approach increases the cost and lag between queries, it may produce superior outcomes compared to simply deleting past messages.
Set a strict limit on the number of interactions.
Wait for the GPT-4 API general availability, which is not only smarter but has double token capacity.
Use a newer model like "gpt-3.5-turbo-16k" which can handle up to 16k tokens.

Conclusion

Enhancing the bot's responses is possible with word embeddings and a good context databse. To achieve this, we need good quality documentation. There is a substantial amount of trial and error involved in developing a bot that seemingly possesses a grasp of the subject matter.

I hope this in-depth exploration of word embeddings and large language models aids you in building a more potent bot, customized to your requirements.

Happy building!

Breaking Free From the Cloud With MRSK: Just Enough Orchestration for Your Apps

Tomas Fernandez — Wed, 19 Jul 2023 21:07:32 +0000

A lightweight alternative to Kubernetes. Powering the likes of Hey.com and Basecamp.

Breaking Free From the Cloud With Mrsk: Just Enough Orchestration for Your Apps

There is no doubt that the cloud has changed the way we run our software. Startups, for instance, can get started without buying expensive hardware and scale flexibly. Also, the cloud has enabled novel solutions such as serverless, managed Kubernetes and Docker, or edge functions. For a time, cloud-native applications seemed to be the way forward for most teams, big or small.

But in exchange for all this power, we pay a cost. And it can be a steep one. 37signals — the company behind HEY.com and Basecamp.com — has calculated that by buying a few servers and moving from the cloud to on-premise, they can save 7 million dollars over 5 years.

And it is not an isolated case. Recently, Amazon Prime — the poster child of serverless architectures — moved part of its infrastructure from serverless into a monolith and cut costs by 90%.

Does this mean that we should we go back to bare metal servers like in the old days? Not quite. We can still enjoy many of the benefits of the cloud, like horizontal scalability and no-outage upgrades using containers in combination with an orchestration platform. On that note, 37signals recently released MRSK, a tool that allowed them to to completely leave the cloud, saving them money and improving performance for their users.

What is MRSK?

MRSK is a deployment tool for containerized applications. It uses Docker containers to run the software and Traefik, a reverse proxy and load manager, to perform rolling updates.

The application container runs behind an instance of Traefik running as a reverse proxy. User requests are routed to the active application container.

MRSK keeps things simple by:

Using standard Docker images.
Connecting to the servers via SSH.
Giving each server a single role.

The last point gives us the first clue about how MRSK works. It assumes a multi-server setup, where every machine fulfills only one role. In other words, if your application requires a database, MRSK expects to have at least two servers: one for the application and one for the database.

Deploying the application and database with a single MRSK command.

MRSK cares about your servers and little else. Load balancing is limited to the server level. MRSK uses Traefik to forward HTTP traffic to every container running on a machine. If you want to do horizontal scaling, you'll need to put a separate load balancer in front of everything.

MRSK does not manage multi-server load balancing. You need to add your own.

Because MRSK doesn't care where the application runs. You can use a cloud server, on-premise machines, or even run VMs on your laptop to test drive the tool.

Getting started with MRSK

You'll need the following to build and deploy an application with MRSK:

Docker.
A Docker Hub account or similar.
A software project with its Dockerfile.
Two servers with SSH access.

Note: will need at least one server for each role. A deployment will typically need at least two machines: one for the application and one for the database. You cannot host the database on the same machine as the application.

Ensure you have SSH access

Before starting, ensure that you have SSH access to the deployment server. It can be any cloud, bare metal server, or even a VM on your laptop. What matters is that it already has your SSH key authorized for root access.

$ ssh root@SERVER_IP_ADDRESS

If that doesn't work, but you know the root's password, you can add your key with the following command:

$ ssh-copy-id root@SERVER_IP_ADDRESS

Installing MRSK

So, now that we have our server, we can install MRSK on your local machine. Either as a Ruby Gem:

$ gem install mrsk -v 0.13.2

Or, you can try the ready-made official Docker image by aliasing the command in your shell like this:

$ alias mrsk='docker run --rm -it -v $HOME/.ssh:/root/.ssh -v /var/run/docker.sock:/var/run/docker.sock -v ${PWD}/:/workdir ghcr.io/mrsked/mrsk:v0.13.2'

Since MRSK is in active development and quite new, I suggest picking a version and sticking with it to prevent updates from breaking your deployments.

Configuring MRSK

For this next step, we will need a project to deploy. MRSK works with any codebase as long as it includes a Dockerfile. For this tutorial, I will be using the following demo project, which exposes an API-based addressbook with PostgreSQL for persistence:

TomFern/dockerizing-nodejs

In the project's root folder, run:

$ mrsk init

This will create three files:

config/deploy.yml: the main configuration file. It declares how to deploy the application to all your servers.
.env: for sensitive environment variables, including passwords and tokens. It should be added to .gitignore so it's never checked in the repository.
.mrsk/hooks: contains Git-like hooks. We won't be using any of these features in this tutorial.

The building block of MRSK deployments is called applications, which are containers running in one or more servers. They are connected to the Internet via Traefik, a reverse proxy.

Let's configure our deployment. To do that, open deploy.yml and edit the following values near the top of the file. Uncomment the lines as needed. You will, at the very least, need to define the following:

A name for the application.
The server IP address. This will be your application server.
The Docker image name without the tag.
Your Docker Hub username (you can also use a different registry).
In the registry.password add "DOCKER_PASSWORD". This is a reference to a variable defined in .env.
Environment variables for the application. The values under clear are stored in plaintext. Passwords should be stored in .env, and their variable name should be listed under secret in the config file.

I'm configuring an "addressbook" application in the example below. Its image name is TomFern/addressbook (notice the lack of tags, which are handled by MRSK). My application uses a PostgreSQL database, for which I set DB_HOST to the IP address of the db and reference the variable containing the password in .env.

service: addressbook
image: tomfern/addressbook

# Deploy to these servers.
servers:
 - 216.238.101.228

# Credentials for your image host.
registry:
 username: tomfern
 password:
 - DOCKER_PASSWORD

# Environment variables
env:
 clear:
 DB_HOST: 216.238.113.141
 secret:
 - DB_PASSWORD

MRSK can handle containerized dependencies like databases, caches, or search services. MRSK calls them accessories. If you need to define an accessory, scroll down the config file until you find the accessories section. Then, set the following values:

The db engine image name with its tag, e.g. postgres:14.
The db server IP address and the database port.
Any secrets or environment variables needed for database initialization.
One or more volume mappings for data persistence — so data is not wiped out when the container stops.
You can also define startup scripts to run during database initialization.

The following snippet shows how to configure a PostgreSQL database accessory for our demo app:

accessories:
 db:
 image: postgres:14
 host: 216.238.113.141
 port: 5432
 env:
 secret:
 - POSTGRES_PASSWORD
 volumes:
 - /var/lib/postgresql/data:/var/lib/postgresql/data

Now, open the .env file. Fill in the password of your Docker Hub account and for the "Postgres" user in your PostgreSQL server. The value of every secret environment variable defined in deploy.yml must be set here.

DOCKER_PASSWORD=YOUR_DOCKER_HUB_PASSWORD
POSTGRES_PASSWORD=THE_POSTGRES_ADMIN_PASSWORD
DB_PASSWORD=THE_ADDRESSBOOK_APP_PASSWORD

You may use the same password for POSTGRES_PASSWORD and DB_PASSWORD or create a dedicated user for the app in the database. If you do so, you'll also need to define DB_USER in the config file like this:

service: addressbook
image: tomfern/addressbook

# ...

env:
 clear:
 DB_HOST: 216.238.113.141
 DB_USER: my_app_username
 secret:
 - DB_PASSWORD

Add a healthcheck route

By default, MRSK checks that the application container is up by running curl on the /up route. You can change the healthcheck endpoint in the config file.

Since the demo does not have a healthcheck route, let's add one. In app.js, add the following lines:

// app.js

// ....

const healthRouter = require('./routes/health');
app.use('/up', healthRouter);

// ...

And create a new file called routes/health.js with the following content, which checks if the app can connect with the database:

// routes/health.js


const express = require('express');
const router = express.Router();
const db = require('../database');

router.get("/", function(req, res) {
 db.sequelize.authenticate()
 .then(() => res.status(200).send(JSON.stringify({ ok: true })))
 .catch( err => res.status(500).send(JSON.stringify(err)));
});

module.exports = router;

MRSK uses curl inside the container to perform the healthcheck, so ensure that your Dockerfile installs the tool. For example:

FROM node:18.16.0-alpine3.17
RUN apk update && apk add curl

...

Prepare servers and deploy

We're set to begin with the deployment. MRSK can do everything with a single command:

$ mrsk setup

Acquiring the deploy lock
Ensure curl and Docker are installed...
Log into image registry...
Build and push app image...
Ensure Traefik is running...
Start container with version c439617 using a 7s readiness delay (or reboot if already running)...
Releasing the deploy lock

This command does the following:

Installs Docker on all machines.
Starts Traefik in the app server.
Starts the PostgreSQL container in the database server.
Builds the Docker image for your application on your laptop and uploads it to Docker Hub.
Pulls the image and starts the application on the server.
Routes inbound traffic into the application container.

MRSK runs a health check to verify that the application is ready to work before being exposed to the Internet. Once the check passes, Traefik routes traffic into the app's container.

By default, the health check is a GET request to the /up route (expecting status code 200), but you can change that in deploy.yml.

MRSK deployment mechanism. The image is built into the developer's machine, uploaded to Docker Hub, and pulled into the server. Then, a health check ensures it started correctly. Once verified, Traefik routes traffic into the application container.

We can check what containers are running with the following:

$ mrsk details

Traefik Host: 216.238.101.228
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
5d08d56b760b traefik:v2.9 "/entrypoint.sh --pr…" 2 minutes ago Up 2 minutes 0.0.0.0:80->80/tcp, :::80->80/tcp traefik

App Host: 216.238.101.228
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
28acfd2cda02 tomfern/addressbook:3ecd87430ab7ab6cc30a1542784ddb75fbfd8e74 "docker-entrypoint.s…" 2 minutes ago Up 2 minutes 3000/tcp addressbook-web-3ecd87430ab7ab6cc30a1542784ddb75fbfd8e74


CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
a843852686b6 postgres:14 "docker-entrypoint.s…" 2 minutes ago Up 2 minutes 0.0.0.0:5432->5432/tcp, :::5432->5432/tcp addressbook-db

As you can see, MRSK tags every built image with a unique identifier, allowing us to track changes, rolling back and forwards as needed.

Before testing the application, we should run any database migration/setup scripts to initialize its contents. MRSK allows us to run commands in running containers with the following:

$ mrsk app exec "npm run migrate"

Finally, we can test the application, which should be ready to receive user requests.

$ curl -X PUT http://216.238.101.228/persons -H 'Content-Type: application/json' -d '{
"id": 1,
"firstName": "David",
"lastName": "Bowie"
}'

$ curl 216.238.101.228/persons/all
{
 "firstName": "David",
 "lastName": "Bowie",
 "id": 1,
 "updatedAt": "2023-04-30T22:44:29.115Z",
 "createdAt": "2023-04-30T22:44:29.115Z"
}

Deploying updates with MRSK

Let's make a change to the application so we can see how MRSK handles updates. In the demo code, we have a /persons API endpoint; it would be a good idea to version it. So let's change this line in app.js:

app.use('/persons', personsRouter);

Change the line so the base endpoint route is /persons/v1/.

app.use('/persons/v1', personsRouter);

The fastest way to update the container is with mrsk redeploy, which does away with a few tasks like checking whether Docker is installed in all servers.

$ mrsk redeploy

Acquiring the deploy lock
Running docker buildx build
Running docker image rm --force tomfern/addressbook:7db892 on 216.238.101.228
Running docker pull tomfern/addressbook:latest on 216.238.101.228
Health check against /up succeeded with 200 OK!
Finished all in 35.8 seconds
Releasing the deploy lock

The redeploy command will rebuild the image, upload it to Docker Hub and start it on the application server. As soon as the health check passes, the traffic is routed to the new version, and the old container shuts down.

MRSK redeploy mechanism. A new instance of the application is built and deployed. Once its health check passes, traffic is routed to the new instance, and the old one is shut down.

After deployment, we can check that the new route is working:

$ curl 216.238.101.228/persons/v1/all

[
 {
 "id": 1,
 "firstName": "David",
 "lastName": "Bowie",
 "createdAt": "2023-05-07T17:41:45.580Z",
 "updatedAt": "2023-05-07T17:41:45.580Z"
 }
]

Rolling back updates with MRSK

MRSK gives us a safe path for rolling back updates. If the new version is causing trouble, we can return to the last working version with a single command.

To rollback, first, we need to find which images are available on the server with the following:

$ mrsk app containers

App Host: 216.238.101.228

CONTAINER ID   IMAGE                                                          COMMAND                  CREATED         STATUS                     PORTS      NAMES

f648fec5f604   tomfern/addressbook:66347a86f8a123e35492dd43463540c23f7db892   "docker-entrypoint.s…"   4 minutes ago   Up 4 minutes               3000/tcp   addressbook-web-66347a86f8a123e35492dd43463540c23f7db892

b3b1d13b8a1c   9500e07b6387                                                   "docker-entrypoint.s…"   9 minutes ago   Exited (1) 4 minutes ago              addressbook-web-66347a86f8a123e35492dd43463540c23f7db892_05d41f3ba39d2b1b

ce4a5c31e6fc   tomfern/addressbook:f043325e3984ec245a94b21bd236afcc537a9739   "docker-entrypoint.s…"   3 hours ago     Exited (1) 9 minutes ago              addressbook-web-f043325e3984ec245a94b21bd236afcc537a9739

We can see that the previous version was tagged as "f043325e3984ec245a94b21bd236afcc537a9739". We can run mrsk rollback to go back to it. For example:

$ mrsk rollback f043325e3984ec245a94b21bd236afcc537a9739

If the container does not exist on the server — MRSK routinely prunes older images — you can always look up the last good version in Docker Hub or by checking the logs in your CI/CD platform.

MRSK rollback mechanism. The old version is restarted and health-checked. Once working, traffic is switched over to the old version, and the new container stops.

Once you're done with your application, you can remove everything, including the database, with mrsk remove. This will destroy the database, remove Docker and stop all applications.

MRSK’s limitations

MRSK development is ongoing, so we may expect behavior changes and new breaking features in the future. But do not mistake its newness for immaturity; 37signals has reportedly begun using it to move away from the cloud with great success already.

That being said, there are a few things MRSK cannot do, and that makes it a bad fit for some use cases:

The default behavior is to expose the application's HTTP port without SSL. You are supposed to set up an SSL terminator, a load balancer, or a CDN in front of the whole deployment.
Containers cannot communicate between themselves within the same server. This is by design. You can always configure Docker networks server-side manually. But it's better to stick to one server per role, so putting the application and the database on the same machine is a bad idea.
You will need to configure a firewall in front of your whole setup to ensure users can only access the web application.
If you want to distribute the load among many servers, you should set up a load balancer in front of your servers. MRSK does not provide this feature.

MRSK is designed for each server to have one role and does not provide load balancing or SSL termination. You need to handle it by yourself.

At the end of the day, MRSK offers a simplified workflow — especially when compared with beasts like Kubernetes — by not trying to handle every aspect of the system.

Deploying applications with MRSK and CI/CD

MRSK can be easily integrated into Continuous Integration and Delivery to get automated deployments. This section will show how to configure a Semaphore CI/CD workflow for continuous deployment. If you've never used Semaphore, check out our getting started guide.

First, we'll need to configure a couple of secrets. Secrets are Semaphore's way of storing sensitive data such as passwords or API keys.

Log in or create an account on Semaphore. Then, go to your organization in the top right corner and select Settings > Secrets.

Use the New Secret button to create a secret called "mask-deploy". Set the values as follows. - POSTGRES_PASSWORD: with the database "postgres" password. - Upload your SSH keys (public and private) to the path /home/semaphore/.ssh/ as shown in the picture below. These are needed for Semaphore to access your servers via SSH.

Next, create a second secret called "dockerhub" with your Docker Hub credentials, as shown below. This is required to push images into the Docker Registry.

Now open or add your project in Semaphore. This will connect Semaphore to your repository. If you don't already have a CI pipeline, you can select one of the examples or create your own.

Select +Add Promotion to create a continuous deployment pipeline. Select the new pipeline and type the following commands in the Prologue. This pins the MRSK and Ruby versions, so updates keep our automated deployment intact.

Now click on Add Block and type the following commands in the job. Then, enable the "dockerhub" and "mask-deploy" secrets.

echo "${DOCKER_PASSWORD}" | docker login -u "${DOCKER_USERNAME}" --password-stdin
checkout
mrsk build create
mrsk redeploy

Finally, we'll create a job to migrate the database, in case the new version includes schema changes. Select Add Block and type the following commands in the job.

checkout
mrsk app exec --reuse "npm run migrate"

Enable the "mask-deploy" secret and select Run the workflow. Wait for the CI pipeline to be done. Then, press the Deploy to Prod button to run the deployment pipeline.

The final continuous deployment pipeline.

And we're done! Congratulations on automating your deployment with CI/CD and MRSK. You can see the final state of the code and pipeline in the "mrsk" branch of the demo repository:

TomFern/dockerizing-nodejs

Troubleshooting

If you experience issues with deployment try the following:

If the healthcheck fails, ensure tht curl is installed in the application container image.
If it still fails after ensuring curl is installed, check that connectivity works between the application and the database. You may need to configure a VPS or set firewall rules.
You can try starting the images manually by running the docker exec commands in the machines to see the output of the logs. These will help you find the root cause of the problem.

If you get an error message stating that there is a lock, run mrsk lock release. This can happen when MRSK fails during deployment and the admin lock is not released.

Conclusion

MRSK is minimalistic to the point of elegance. By combining traditional servers with the flexibility of containers, it allows us to run our containerized services with ease on any server.

MRSK presents a compelling solution if you're seeking to optimize your software deployments, reduce vendor lock-in, and maintain flexibility in choosing deployment environments while enjoying the benefits of cloud-native architectures.

A First Look at Neon: A Postgres Database That Branches

Tomas Fernandez — Mon, 06 Mar 2023 21:25:05 +0000

Photo by Damir Kopezhanov on Unsplash

Relational databases have a long, long history. The first appeared in the 1970s, and while the technology has certainly evolved, the relational model has proved the most popular over the last 5 decades.

Is there room for innovation after 50 years of history? The folks at Neon are showing us that one can, in fact, teach an old dog new tricks.

What is Neon?

Neon is an open-source (Apache 2.0) alternative to AWS Aurora or Google's Cloud SQL for Postgres. Neon is a serverless, scalable implementation of PostgreSQL that you can run on-premise or try through its managed service.

Neon decomposes the PostgreSQL architecture into two layers: compute and storage. The compute layer consists of stateless PostgreSQL running on Kubernetes, allowing pods to be scaled on demand — even to zero.

Persistence is achieved with the Neon storage engine, a custom-made layer that handles both transactions and data. The transaction log is processed through a set of redundant safekeeper services, while data pages are managed on disk by the pageserver.

Neon architecture splits the database into scalable compute and storage planes. Compute nodes can be started and stopped depending on demand.

Currently, the managed service is running a free tech preview with some limitations that we'll discuss later.

Neon's killer feature: branches

Neon was launched in June 2021. Being a new project, the managed service may have fewer features than the competition. But Neon has one feature that, to my knowledge, no one else has: branches.

Every developer is familiar with branches. In Neon, branches work pretty much the same as in Git, except they cannot be merged (although there are plans to add schema-based merging in the future). You can, at any point, branch off the main trunk, effectively creating an “alternate timeline".

Since branches in Neon are writable, this feature allows us to do things no other database engine can do. For instance:

Freely experiment without impacting the main branch.
Instantly back up the database. So, if data is lost by mistake, we can switch to the last good branch.
Simplify integration testing. Developers can run tests in disposable test-specific branches.
Safely try out automated database migrations on production.
Run analytics or machine learning workloads in isolation.

Instantly duplicate all the databases that serve to a specific cluster of microservices.

You can't do any of these things on traditional database engines. Not easily at least. Some database engines like SQL Server have snapshots, which indeed can create instant copies of a database. But snapshots are read-only and this limits their utility. On most database engines, we have to resort to clunkier mechanisms like backup and restore or replication.

A few use cases for Neon branches.

Branches are per-project. And a project can have multiple databases. That means that creating a branch duplicates all the databases in that project. We can take advantage of this project > database hierarchy to clone a group of related databases in one operation.

Getting started with Neon

Let's try out Neon's managed service. To create a tech preview free account, just follow these steps:

Sign up at neon.tech/sign_in.
Click on Create a project.
Click on Download env.txt. This file contains everything you need to connect to the database instance.
Click on Settings and copy the project id.

We also need to generate an API Key, as shown below:

Click on your avatar and select Account > Developer Settings > Create new API key
Edit env.txt and add the following lines:
- export NEON_API_KEY=Your-API-Key
- export PROJECT_ID=Your-Project-ID
Add the keyword export before every variable.

The final env.txt file should look like this example:

# Connection details
export PGHOST=ep-random-name.us-east-2.aws.neon.tech
export PGDATABASE=neondb
export PGUSER=Tommy
export PGPASSWORD=sekret1

# Connection string
export DATABASE_URL=postgres://Tommy:sekret1@ep-random-name.us-east-2.aws.neon.tech/neondb

# Neon config
export NEON_API_KEY=MyApiKey
export NEON_PROJECT_ID=random-name-140532

We’ll need this file to connect to the Neon database and API.

The Neon UI

The Neon dashboard includes an SQL editor to run commands and controls for creating branches or endpoints.

The managed service dashboard.

On the Branches page, we’ll find options for creating a new branch. Here, you can select the what and when. You must choose the parent branch and how much data to include:

Head: the new branch is a copy of the current database.
Time: the branch has the parent's data up to a specified date and time.
LSN: the branch has the parent's data up to a specified log sequence number.

The create branch UI.

Endpoints for the branches can be created on the same screen or on the Endpoints page.

Connecting to Neon

Neon is a PostgreSQL database, so we'll need to install the client tools. Check which version yours is running with:

$ psql --version
psql (PostgreSQL) 15.1 (Ubuntu 15.1-1.pgdg20.04+1)

Neon works best with versions 14 and 15 of the client tools. So, if needed, head to postgresql.org/download to get the latest release.

Now, let's source the env.txt and try connecting.

$ source env.txt
$ psql

psql (15.1 (Ubuntu 15.1-1.pgdg20.04+1), server 14.6)

neondb=> SELECT version();
 version
---------------------------------------------------------------------------------------------------
 PostgreSQL 14.6 on x86_64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit
(1 row)

You can also do browser-based authentication by running the command shown below. This will open a browser window and let you select the project you want to connect to.

$ psql -h pg.neon.tech

We may also test the Neon API key with curl:

$ curl -s 'https://console.neon.tech/api/v2/projects' \
 -H 'accept: application/json' \
 -H "Authorization: Bearer $NEON_API_KEY"

{
 "projects": [
 {
 "id": "calm-guy-140532",
 "platform_id": "aws",
 "region_id": "aws-us-east-2",
 "name": "test-project",
 "provisioner": "k8s-pod",
 "pg_version": 14,
 "locked": false,
 "created_at": "2022-12-22T18:16:13Z",
 "updated_at": "2022-12-22T18:16:13Z",
 "proxy_host": "us-east-2.aws.neon.tech"
 }
 ]
}

Automated branching with CI/CD

One of the most delicate parts of deployment is the database migration step because there is always the possibility of data loss. There are several techniques to make this process safer. But with Neon, we can leverage branches to make the process infallible.

You can check out a live example for running branches with continuous integration and delivery by checking out this tutorial.

Technical preview limitations

At the time of writing Neon is in a technical preview stage. While the managed service is entirely free, it comes with some limitations:

You can only have one project per user. But a project can have multiple databases.
A project can have up to nine branches in addition to the main branch.
You can have up to three endpoints. One is always reserved for the main database. That leaves only two endpoints accessible for two other branches.
The size limit is 3GB per branch on the free tier.
Point-in-time branches can only go up to seven days into the past.
There are no backup or restore options on the UI. The only alternative seems to run pg_dump neondb and take a remote backup.
There are a few observations around importing data from another PostgreSQL instance or a backup.
There is a 100 concurrent connections limit. You can enable connection pooling to raise the limit to 1,000 connections.

The good news is that if you like the database, you can always run it on-premise or in your cloud of choice to remove these limitations.

Conclusion

Neon's branching feature presents new options for development and database management. Even in its current technical preview stage, I can see great potential for this engine. Of course, we'll have to see how the project evolves, especially how the final pricing model will turn out.

I hope you found this project interesting, and if so, you might want to consider contributing to Neon.

Thanks for reading!

Getting Started With Astro

Tomas Fernandez — Fri, 03 Mar 2023 22:03:38 +0000

Photo by Ju Guan on Unsplash

Getting Started With Astro

Web development, like most technologies, moves in waves and cycles. Static websites were all we had in the beginning. But pretty soon developers were hacking together the first server-generated sites thanks to Perl and PHP. This was the advance that would eventually kickstart frameworks like Laravel, Django, or Rails.

Mobile devices would come to change how people consume the web. So long server-generated websites, hello client-rendered applications. The next wave brought frameworks that could give users a more app-like experience — without reloads — like React or AngularJS.

But while single-page applications did provide a smoother experience, they have their drawbacks. Namely, longer page loads caused by all the extra JavaScript that has to be parsed and executed. Not to mention all the work it took to optimize for search engines.

Astro is a prime example of the current wave that we have been in since Next.js, taking us full circle: a web framework that combines server and client rendering to get the best of both worlds.

What is Astro?

Astro is an open-source framework for generating web applications on top of popular UI frameworks like React, Preact, Vue, or Svelte. An Astro page is composed of several independent components. To improve load times, Astro strips away all JavaScript and prerenders pages in the server unless developers mark a component as interactive, in which case Astro will send the minimum amount of JavaScript required for interactivity.

Astro favors generating static and server-rendered sites without JavaScript in the browser. Developers can opt-in to client-side rendering for given routes or parts of the page.

Thanks to this strategy, Astro pages load quickly, as no JavaScript needs to execute for the first render. In a process called hydration, Astro will “pour” JavaScript into components to make them dynamic.

Why Astro?

Astro stands out from the other web frameworks in many ways:

Speed: Astro is statically and server-rendered when possible, making it an excellent fit for content-first, SEO-friendly websites like blogs, courseware, or marketing sites.
UI agnostic: unlike Next.js, which only works with React, or Nuxt.js–which requires Vue expertise, Astro doesn't force any UI framework on you. You can use none, roll your own, or use any of the many supported integrations.
Themes: Astro has an impressive array of ready-made themes and templates to get started quickly.
Easy to pick up: you don't need to learn React or Vue to get started with Astro. Its templating engine looks like plain HTML, and the code is cleanly separated from the presentation. You can pick an empty project and build it up at your own pace.
Batteries included: Astro comes with the features we expect from a modern JavaScript/TypeScript framework like top-level awaits, Markdown and MDX support, and ESM includes.
Islands: the island architecture allows us to mix static content, server-rendered, and client-rendered components on the same page without conflicts, allowing us to build interactive sites with minimal overhead. We can even mix different frameworks in the same page, giving us yet another way of building microfrontends. We'll see more about how islands work in a bit.

Getting started with Astro

To get started with Astro, install Node version 16.12.0 or higher and run the following command. Follow the on-screen wizard and choose to create an empty project when prompted:

$ npm create astro@latest

 astro   v1.9.1 Launch sequence initiated.

✔ Where would you like to create your new project? … awesome-website
✔ How would you like to setup your new project? › an empty project
✔ Template copied!
✔ Would you like to install npm dependencies? (recommended) … yes
✔ Packages installed!
✔ Would you like to initialize a new git repository? (optional) … yes
✔ Git repository created!
✔ How would you like to setup TypeScript? › Relaxed
✔ TypeScript settings applied!

  next   Liftoff confirmed. Explore your project!

Next, you can start the website in developer mode by entering the directory of the project you just created and running:
npm run dev and visiting http://localhost:3000.

Pages and routes

The interesting stuff in Astro happens inside the src folder. Checking what's there, we see a single directory called pages with an index.astro file.

Astro pages are a mix of HTML, Javascript, or TypeScript. This is the default index.astro:

---
---

<html lang="en">
    <head>
        <meta charset="utf-8" />
        <link rel="icon" type="image/svg+xml" href="/favicon.svg" />
        <meta name="viewport" content="width=device-width" />
        <meta name="generator" content={Astro.generator} />
        <title>Astro</title>
    </head>
    <body>
        <h1>Astro</h1>
    </body>
</html>

You may have noticed that Astro files start with fences delineated with ---. Whatever code we put inside these fences is executed on the server and prerendered before serving the page.

Below the frontmatter, we find the content for this route, an augmented HTML form that allows for variables. We can, for example, define a variable in the frontmatter and use it in HTML like this:

---
// src/pages/index.astro
const title = "Astro";
---

<html lang="en">
    <head>
        <meta charset="utf-8" />
        <link rel="icon" type="image/svg+xml" href="/favicon.svg" />
        <meta name="viewport" content="width=device-width" />
        <meta name="generator" content={Astro.generator} />
        <title>{title}</title>
    </head>
    <body>
        <h1>{title}</h1>
    </body>
</html>

Astro uses a file-based routing, so every file in the pages folder is mapped to a route in the website. For example, if we create a file called greetings.astro, we should see its contents at http://localhost:3000/greetings.

    ---
    const greeting = "Hello, world!";
    ---

    <html lang="en">
        <head>
            <meta charset="utf-8" />
            <link rel="icon" type="image/svg+xml" href="/favicon.svg" />
            <meta name="viewport" content="width=device-width" />
            <meta name="generator" content={Astro.generator} />
            <title>Astro</title>
        </head>
        <body>
            <h1>{greeting}</h1>
        </body>
    </html>

In addition to .astro files, Astro can parse Markdown, MDX, JSX JavaScript, and TypeScript files. For instance, if we want to write a blog post in Markdown, we create a post.md file under the pages folder. Visiting the route will make Astro convert it into HTML on the fly:

---
title: 'Learning Astro'
pubDate: 2023-01-10
description: 'A post written in Markdown.'
author: 'Tommy'
---

# Learning Astro

This Markdown file should be rendered as HTML when I visit http://localhost:3000/post

Components

Astro components are *.astro files with reusable code and HTML. We can use components to write elements like headers, footers, navigation bars, buttons, and forms — anything that can be expressed as HTML can constitute a component.

Let's create our first component in src/components/Header.astro:

---
// src/components/Header.astro
---
<meta charset="utf-8" />
<link rel="icon" type="image/svg+xml" href="/favicon.svg" />
<meta name="viewport" content="width=device-width" />
<meta name="generator" 
<title>Astro</title>

Once defined, we can import it to any page (or other components) and use it like this:

---
import Header from "../components/Header.astro";
---

<html lang="en">
    <head>
        <Header />
    </head>
    <body>
    </body>
</html>

Astro components are no different from pages. Any code defined between fences is executed on the server. JavaScript is stripped out before sending the content to the browser.

Layouts

Layouts are used to refine reusable UI structures. They are technically components, so the syntax stays the same.

Let's replace the content of index.astro with a layout:

---
// src/pages/index.astro
import SiteLayout from "../layouts/SiteLayout.astro";
---

<SiteLayout></SiteLayout>

As you can see, layouts are stored in the src/layouts folder by convention.

Layouts, like components, can include other components. Here we have extracted the structure in index.astro and added a Footer component:

---
// src/layouts/SiteLayout.astro
import Header from "../components/Header.astro";
import Footer from "../components/Footer.astro";
---

<html lang="en">
    <head>
        <Header />
    </head>
    <body>
        <Footer />
    </body>
</html>

Props and slots

So far, our website has been completely static. In order to pass data
between pages and components, we need to understand how props and
slots work.

Components and layouts can define and accept props(short for properties) via the global Astro.props. Values passed through props are accessible to the component before rendering.

We can read props in our component like this:

---
// src/components/Header.astro
const { title } = Astro.props;
---

<meta charset="utf-8" />
<link rel="icon" type="image/svg+xml" href="/favicon.svg" />
<meta name="viewport" content="width=device-width" />
<meta name="generator" 
<title>{title}</title>

The value of title can be supplied when the component is called, which in the following example happens through the SiteLayout layout.

---
// src/layouts/SiteLayout.astro
import Header from "../components/Header.astro";
import Footer from "../components/Footer.astro";
---
<html lang="en">
    <head>
        <Header title = "Welcome my Astro Blog!" />
    </head>
    <body>
        <Footer />
    </body>
</html>

⚠️ Note that you need spaces surrounding the equal sign, i.e. title="Hello" is NOT correct. Instead, it should be: title = "Hello".

Slot elements create placeholders for content to be injected later. To see how it works, we can add a <slot /> element in src/layouts/SiteLayout.astro:

---
// src/layouts/SiteLayout.astro
import Header from "../components/Header.astro";
import Footer from "../components/Footer.astro";
---

<html lang="en">
    <head>
        <Header title = "Welcome my Astro Blog!" />
    </head>
    <body>
        <slot />
        <Footer />
    </body>
</html>

Now the HTML inside <SiteLayout> is injected into the point where the slot is located.

---
// src/pages/index.astro
import SiteLayout from "../layouts/SiteLayout.astro";
---

<SiteLayout>
    <p>This content is rendered in the slot</p>
</SiteLayout>

Nesting components, layouts, props, and slots give us the flexibility to create reusable UI files across the website.

Islands and hydration

Up to this point, we have not shipped any JavaScript to the user; everything is prerendered and served as pure HTML+CSS. How do we send make Astro send JavaScript to the browser? For that, we need to understand the islands architecture.

The island architecture pattern aims to reduce the amount of JavaScript needed browser-side. Less JavaScript means less data sent and less computational power required on the user's device. An island is an autonomous component that bundles HTML, CSS, and — optionally —
JavaScript. In the islands pattern, a page is made up of several independent islands.

The islands architecture allows combining static HTML, server-rendered content, and interactive client-rendered components on one page without conflicts.

Each island is prerendered, so there is no interactivity right after the page loads. Once the initial page is ready, the islands are converted into interactive content in a process called hydration. Hydration is a technique that converts static content delivered via static hosting or server-side rendering into a dynamic page by attaching event handlers to the HTML elements.

How hydration works. The first contentful paint happens right after all the HTML has been sent to the browser, which typically happens very quickly. The browser then requests and executes all the client-side JavaScript needed to make the page fully interactive.

Using Islands

An example showing how Astro implements islands is in order. Let's first add a UI integration to the project. The following command installs @astrojs/preact and preact.

$ npx astro add preact

Let's create a simple button to test the integration:

// src/components/MyButton.jsx

export default function MyButton() {

const clicked = () => { console.log('Hello!') };

return (
    <div>
      <button style={{ color: 'purple' }} onClick={clicked}>Click me</button>
    </div>
  )
}

As usual, Astro will try to strip away any JavaScript. So nothing would happen if we instantiated the component with <MyButton />. We need to tell Astro that we want this component treated as an island and hydrated accordingly by adding the template directive client:load:

---
import MyButton from "../components/MyButton.jsx";
---

<html lang="en">
    <body>
        <MyButton client:load />
    </body>
</html>

Clicking the button should print "Hello!" in the browser console.

The client directive caused Astro to hydrate the component. There are five levels of hydration with different priorities:

client:load hydrates the component as soon as possible.
client:idle hydrates the component when the page is done loading. Useful for low-priority components that don't need immediate interactivity.
client:visible={string} hydrates the component as soon as it appears in the viewport.
client:media={string} takes a CSS query as an argument and loads the component as soon as it's fulfilled.
client:only skips HTML rendering entirely and renders the component in-browser.

Deploying Astro application with CI/CD

Being a Node application, setting up an Astro build with CI/CD is very straightforward. We only need a couple of jobs.

A possible continuous integration pipeline for Astro.

The first job runs npm ci to populate node_modules. We use sem-version to select a current Node version.

checkout
sem-version node 19.4
npm ci
cache store

The second job runs npm run build and stores the built application (located in the dist folder) as a workflow
artifact.

checkout
sem-version node 19.4
cache restore
npm run build
artifact push workflow dist/

Once the site is built, we can configure continuous
deployment.

Deployment targets

Depending on the nature of the web application, Astro can be deployed as a static site à la Hugo or Gatsby, in which case we only need something as simple as an S3 bucket or GitHub pages, or as a full-fledged server-side rendered (SSR) application, where we need JavaScript- or TypeScript-enabled endpoints.

Astro has built-in support for various popular deployment targets such as Netlify, Firebase, Vercel, and Deno. Some only support either SSR or static hosting, while others can do both.

Once we have chosen our deployment method, we can add a continuous deployment pipeline to automatically deploy the website on every change.

Here's an example deployment pipeline that targets a Netlify static site.

The complete CI/CD workflow with continuous deployment to Netlify.

For reference, the job commands are as follows. This supposes that we have already obtained an API token and saved it as a secret on Semaphore with the env variables NETLIFY_TOKEN and NETLIFY_SITE.

checkout
artifact pull workflow dist
npm install -g netlify-cli
netlify deploy --dir=dist --prod --auth $NETLIFY_TOKEN --site $NETLIFY_SITE

Conclusion

The popularity of this project has been nothing short of astronomical: in the first five months since the Astro 1.0 release, the project has amassed over 25,000 stars on GitHub. The success of Astro is not accidental. The Astro team has made a web framework that offers great ergonomics for developers and fast-loading websites for users, even if they are on low-power devices or slow connections.

Thanks for reading, and happy building!

Ruby Adds Support for WebAssembly: What is WebAssembly and how it benefits Ruby devs?

Tomas Fernandez — Fri, 10 Feb 2023 21:18:38 +0000

Photo by Joshua Fuller on Unsplash

Ruby has joined the ranks of languages capable of targeting WebAssembly with its latest 3.2 release. This seemingly minor update might be the biggest thing that has happened to the language since Rails, as it lets Ruby developers go beyond the backend. By porting their code to WebAssembly, they can run it anywhere: on the frontend, on embedded devices, as serverless functions, in place of containers, or on the edge. WebAssembly has the potential to make Ruby a universal language.

What is WebAssembly?

WebAssembly (commonly shortened as Wasm) is a binary low-level instruction format that runs on a virtual machine. The language was designed as an alternative to JavaScript. Its aim is to run applications on any browser at near-native speeds. Wasm can be targeted from any high-level language like C, Go, Rust, and now also Ruby.

Wasm became a W3C standard in 2019, opening the path to writing high-performing applications for the Web. The standard itself is still evolving, and its ecosystem is growing. Currently, this technology is receiving a lot of focus from the Cloud Native Computing Foundation (CNCF), with several projects under development.

Wasm's design sits on two pillars: portability and security. The Wasm binary can run on any modern browser, even mobile devices. For security, Wasm programs run in a sandboxed, memory-safe VM. As such, they cannot access any system resources: they can’t change the filesystem or access the network or memory.

WebAssembly brings portability to the next level

Let’s say you want to build an application targeting many systems, e.g. Linux, Windows, and macOS. What are your options?

You could use a compiled language like C and build a binary for each target.

Compiler portability creates multiple executable files

Or, if you can rely on having the appropriate runtime installed you could choose an interpreted language like JavaScript or one that compiles to bytecode like Java.

Code is compiled into an intermediate bytecode. This system relies on having a runtime environment, or VM, installed on the client.

What if you have a container runtime in the client? In that case, you could build a Docker image for each platform type.

Code is compiled into platform-dependent images. A container runtime is required for clients, which pulls the correct image automatically.

For Ruby developers historically, the only option was to distribute the code. That meant that users had to install the Ruby interpreter (or developers had to package the interpreter along with the application) to run the application.

Code is shipped directly to users, who must have the interpreter installed in their systems to be able to run it.

All these mechanisms provide portability, but at a cost: you must build, test, and distribute many images. Sometimes, you must also ship a suitable runtime with the release or tell the user to install it independently.

WebAssembly (shortened as Wasm) takes portability to the next level: it allows you to build ONE binary and run it in any modern browser.

WebAssembly compiles into a low-level assembly that every modern browser can execute. As a result, the same Wasm binary can run, unmodified, on every platform (even mobile).

The ability to run code at native speed has allowed developers to build sites like Figma, and Google Earth or even run Vim in the browser.

Ruby adds support for WebAssembly

The latest Ruby release ships with a Wasm port of the interpreter. Therefore, we can run Ruby code directly in the browser without the need for a backend.

As you can see in the example below, all it takes to get started with the Ruby Wasm port is a couple of lines. The script downloads ruby.wasm and instantiates the interpreter in the browser. After that, it takes the text of text/ruby type and feeds it into the WebAssembly program.

<html>
  <script src="https://cdn.jsdelivr.net/npm/ruby-head-wasm-wasi@0.5.0/dist/browser.script.iife.js"></script>
  <script type="text/ruby">
    puts "Hello, world!"
  </script>
</html>

You can confirm that Ruby is running from the browser, i.e. not connecting with a backend, by opening the developers' tools. Here, you'll find once ruby.wasm is downloaded, no further connections are needed.

Traditionally, JavaScript has been touted as the best language to learn because you have it everywhere. With WebAssembly, everyone can learn and experiment with Ruby using a browser. The output is printed in the developer's console.

You can even see the contents of ruby.wasm disassembled into text format in the “Sources” tab:

We can see the downloaded Wasm file in the browsers web developer tools.

You can check out the Wasm port online at the Ruby playground.

Working with the sandbox

As said, Wasm programs run in a sandboxed VM that lacks access to the rest of the system. Therefore, Wasm applications do not have access to the browser, filesystem, memory or the network. We'll need some JavaScript code to send and receive data from the sandbox.

The following example shows how to read the output of a Ruby program and make changes to the page using the ruby-head-wasm-wasi NPM package:

<html>
  <script src="https://cdn.jsdelivr.net/npm/ruby-head-wasm-wasi@latest/dist/browser.umd.js"></script>
  <script>
    const { DefaultRubyVM } = window["ruby-wasm-wasi"];
    const main = async () => {
      const response = await fetch(
        "https://cdn.jsdelivr.net/npm/ruby-head-wasm-wasi@latest/dist/ruby.wasm"
      );
      const buffer = await response.arrayBuffer();
      const module = await WebAssembly.compile(buffer);
      const { vm } = await DefaultRubyVM(module);

      vm.printVersion();
      vm.eval(`
        require "js"
        luckiness = ["Lucky", "Unlucky"].sample
        JS::eval("document.body.innerText = '#{luckiness}'")
      `);
    };

    main();
  </script>
  <body></body>
</html>

The same package can also run Ruby code inside a Node project, allowing you to mix Ruby and JavaScript on the backend. You'll need to install the NPM package ruby-head-wasm-wasi for the example to work:

import fs from "fs/promises";
import { DefaultRubyVM } from "ruby-head-wasm-wasi/dist/node.cjs.js";

const main = async () => {
 const binary = await fs.readFile(
 // Tips: Replace the binary with debug info if you want symbolicated stack trace.
 // (only nightly release for now)
 // "./node_modules/ruby-head-wasm-wasi/dist/ruby.debug+stdlib.wasm"
 "./node_modules/ruby-head-wasm-wasi/dist/ruby.wasm"
 );
 const module = await WebAssembly.compile(binary);
 const { vm } = await DefaultRubyVM(module);

 vm.eval(`
 luckiness = ["Lucky", "Unlucky"].sample
 puts "You are #{luckiness}"
 `);
};

main();

Running ruby.wasm outside the browser

While Wasm's primary design goal is running binary code in the browser, developers quickly realized the potential of a fast, safe, and universally portable binary format for software delivery. Wasm has the potential to become as big a Docker, greatly simplifying application deployment for embedded systems, serverless functions, edge computing, or as a replacement for containers on Kubernetes.

Running a Wasm application outside the browser requires an appropriate runtime that implements the WebAssembly VM and provides interfaces to the underlying system. There are a few competing solutions in this field, the most popular being wasmtime, wasmer, and WAMR.

The Ruby repository provides a complete example for bundling your application code into a custom Ruby image.

Limitations

Let’s remember that this is all cutting-edge tech. The whole Wasm ecosystem is moving fast. Right now, Ruby Wasm has a few limitations which significantly limit its usability in big projects:

No thread support.
Spawning processes does not work.
No network support.
The garbage collector can create memory leaks.
Gems and modules are unavailable unless you build a custom Wasm image.

The future is bright

WebAssembly opens a world of exciting possibilities. It allows Ruby developers to escape the backend. As tooling around WebAssembly improves, Ruby will be able to reach new frontiers: the browser is no longer off-limits, and there will be new opportunities to run Ruby on the edge and as serverless applications.

With the latest release, Ruby developers can begin experimenting with WebAssembly. It's the first step, for sure, and there is much more work to do before we see complex Ruby applications running in with this technology.

Thanks for reading, and happy assembling!