DEV Community: Huseyn

Online Learning Problem

Huseyn — Mon, 29 Dec 2025 07:18:56 +0000

Online Learning Problem (OLP)

What is Online Learning?

Online learning is a learning paradigm where a model learns sequentially while making decisions, without access to the full dataset beforehand. At each step, the learner must act first and learn from feedback afterward.

Unlike offline (batch) learning, online learning operates in real time and continuously updates its strategy.

Standard Online Learning Framework

At each round (t):

The learner selects an action or prediction (a_t)
The environment reveals the outcome (y_t)
The learner incurs a loss (\ell(a_t, y_t)) or receives a reward
The learner updates its decision rule

Future data is never known in advance.

Performance Measure: Regret

Online learning performance is measured using regret, defined as:

[
\text{Regret}(T) = \sum_{t=1}^{T} \ell(a_t) - \min_{a} \sum_{t=1}^{T} \ell(a)
]

Interpretation:

How much worse did the learner perform compared to the best fixed decision chosen in hindsight?

Low regret implies effective learning.

Types of Online Learning Problems

1. Online Prediction with Expert Advice

The learner combines predictions from multiple experts
Goal: perform almost as well as the best expert in hindsight

2. Online Convex Optimization

Loss functions are convex
Common in large-scale machine learning

3. Bandit Feedback

Only the loss or reward of the chosen action is observed
Limited feedback setting

4. Contextual Online Learning

Decisions depend on observed context or features

Bandit Models

What is a Bandit Model?

A (multi-armed) bandit model is a special case of online learning where the learner repeatedly chooses among several actions (arms), each with an unknown reward distribution.

The learner observes feedback only for the selected arm.

Exploration vs Exploitation Trade-off

Exploration: trying uncertain actions to gain information
Exploitation: choosing the action that currently appears best

Balancing these two is the core challenge of bandit problems.

Formal Bandit Setting

A finite set of arms: (A = {1, 2, \dots, K})
Each arm has an unknown reward distribution
At each round:
- Select one arm
- Observe its reward
- Update beliefs

Performance is measured using regret.

Types of Bandit Models

Stochastic Bandits

Rewards drawn from fixed but unknown distributions

Contextual Bandits

Arm choice depends on observed context
Used in recommender systems and ads

Adversarial Bandits

Rewards may be chosen by an adversary
No statistical assumptions

Common Bandit Algorithms

(\varepsilon)-greedy
Upper Confidence Bound (UCB)
Thompson Sampling
EXP3 (adversarial setting)

Relationship Between Online Learning and Bandits

Aspect	Online Learning	Bandit Learning
Feedback	Full loss information	Partial (chosen action only)
Information	More	Less
Scope	General framework	Special case

All bandit problems are online learning problems, but not all online learning problems are bandits.

Applications

Adaptive routing and forwarding
Caching and resource allocation
Recommender systems
Online advertising
Congestion control
Real-time decision systems

Key Takeaway

Online learning focuses on learning while acting, and bandit models address this challenge under limited feedback conditions.

Understanding Domains, DNS, Servers, and Reverse Proxies

Huseyn — Mon, 22 Sep 2025 02:00:23 +0000

A structured guide to demystify domain names, DNS records, servers, and how everything works together with Nginx/Apache.

1. Domain Basics

A domain name (e.g., myapp.com) is just a human-readable label.
When you buy a domain, you’re only buying the name, not the hosting or server.
To make it work, you must link the domain to a server IP (where your app is hosted).

2. DNS Records (Mapping Domain to Server)

DNS records define how your domain/subdomain points to IPs or other domains.

A Record → Maps domain → IPv4 address

Example: myapp.com A 123.45.67.89
AAAA Record → Maps domain → IPv6 address
CNAME Record → Maps domain → another domain

Example: www.myapp.com CNAME myapp.com
Subdomains:
api.myapp.com A 123.45.67.89
blog.myapp.com A 123.45.67.89

→ All point to the same server IP.

3. How Subdomains Work on a Single Server

Your server usually has one public IP.
All subdomains (api.myapp.com, blog.myapp.com) resolve to this same IP.
When the request reaches the server, the web server (Nginx/Apache) checks the Host header to know which site/subdomain was requested.
Based on that, it serves different apps (via Virtual Hosts or server blocks).

4. Role of Web Servers (Nginx / Apache)

Nginx or Apache does:

Listens on ports 80 (HTTP) and 443 (HTTPS).
Reads the Host header (api.myapp.com, blog.myapp.com).
Matches it with configuration (sites-available in Nginx).
Serves the correct project folder or proxies the request to backend apps.

Project Files

Static apps (React, Angular, etc.): Deploy the build folder (HTML, JS, CSS).
Dynamic apps (Node.js, Django, etc.): Run app on internal port (e.g., 5000), then reverse proxy with Nginx.

5. Reverse Proxy Explained

Why do we need it?

Backend apps don’t (and shouldn’t) run directly on port 80/443.
They usually run on local ports like 5000, 8000, etc.
Outsiders don’t see or access those ports.

👉 Reverse Proxy = Gateway

User → https://api.myapp.com (port 80/443).
Nginx forwards it → http://localhost:5000.
App responds → Nginx sends back to user.

6. Why Not Direct Backend Usage?

Security: Don’t expose raw app ports.
Clean URLs: Users see only myapp.com, no :5000.
SSL: Nginx handles HTTPS certificates.
Load Balancing: Can distribute across multiple backend servers.

7. Example Setup with Nginx

Backend

Run a Node.js/Django app on: http://localhost:5000

Nginx Config

server {
  listen 80;
  server_name api.myapp.com;

  location / {
      proxy_pass http://localhost:5000;
      proxy_set_header Host $host;
      proxy_set_header X-Real-IP $remote_addr;
  }
}

Flow

Browser → https://api.myapp.com/login
Request hits Nginx (port 80/443)
Nginx forwards to backend (localhost:5000)
Backend responds → Nginx passes it back → Browser shows result

8. Summary Workflow

Buy domain (get the name).
Set DNS records → Domain → Server IP.
Server receives requests → Nginx/Apache listens.
Nginx decides → Based on domain/Host header.
Serve files (static build) OR proxy to backend (dynamic apps).
User only sees clean domain (no internal ports).

GitHub Actions Cheatsheet and Workflow Analysis

Huseyn — Wed, 16 Jul 2025 17:26:38 +0000

GitHub Actions Cheatsheet

GitHub Actions is a CI/CD platform for automating tasks in GitHub repositories. Workflows are defined in YAML files stored in the .github/workflows/ directory. Below is a cheatsheet of all major components and their roles.

1. Workflow

Definition: A workflow is an automated process defined in a YAML file, containing one or more jobs triggered by events.
Key Attributes:
- name: Workflow name displayed in the GitHub Actions UI.
- on: Specifies events that trigger the workflow (e.g., push, pull_request, schedule).
- jobs: Defines one or more jobs to execute.
- env: Workflow-level environment variables.
- concurrency: Controls concurrent workflow runs to prevent conflicts.
Example:

  name: CI Pipeline
  on:
    push:
      branches: [main]
  env:
    GLOBAL_VAR: value

2. Events (`on`)

Definition: Events trigger workflows. Defined under the on key.
Common Triggers:
- push: On code push to specified branches.
- pull_request: On pull request actions (e.g., opened, synchronized).
- schedule: Cron-based scheduling (e.g., 0 0 * * * for daily runs).
- workflow_dispatch: Manual trigger via GitHub UI or API.
- issue_comment, release, repository_dispatch, etc.
Example:

  on:
    push:
      branches: [main, dev/*]
    pull_request:
      branches: [main]
    schedule:
      - cron: '0 0 * * *'

3. Jobs

Definition: A job is a set of tasks (steps) running on a single runner (virtual machine or container).
Key Attributes:
- runs-on: Specifies the runner (e.g., ubuntu-latest, windows-latest, macos-latest, or self-hosted).
- steps: List of tasks to execute.
- needs: Defines job dependencies for sequential execution.
- env: Job-level environment variables.
- if: Conditional execution (e.g., if: github.event_name == 'push').
- services: Defines Docker containers for external dependencies (e.g., databases).
- strategy: For matrix builds (run job with multiple configurations, e.g., different Go versions).
Example:

  jobs:
    build:
      runs-on: ubuntu-latest
      needs: [test]
      steps: []

4. Steps

Definition: Individual tasks within a job, executed sequentially on the same runner.
Key Attributes:
- name: Step name for UI display.
- uses: References an action (e.g., actions/checkout@v4).
- run: Executes a shell command (e.g., npm install).
- env: Step-level environment variables.
- if: Conditional execution.
- with: Inputs for actions.
- id: Unique identifier for step outputs.
- continue-on-error: Allows subsequent steps to run even if the step fails.
Example:

  steps:
    - name: Checkout
      uses: actions/checkout@v4
    - name: Run script
      run: echo "Hello, World!"
      env:
        MY_VAR: value

5. Actions

Definition: Reusable units of code that perform specific tasks (e.g., checking out code, setting up environments).
Key Attributes:
- Hosted in GitHub Marketplace or custom repositories.
- Used via uses (e.g., uses: actions/setup-go@v5).
- Supports inputs (with) and outputs.
Example:

  - uses: actions/setup-go@v5
    with:
      go-version: '1.21'

6. Services

Definition: Docker containers running alongside a job to provide dependencies (e.g., databases, caches).
Key Attributes:
- Defined under jobs.<job_id>.services.
- Specifies image, ports, env, and options (e.g., health checks).
- Accessible via localhost or service name.
Example:

  services:
    redis:
      image: redis:latest
      ports:
        - 6379:6379

Checkout the PPT . Credit goes to: TechSchool

7. Runners

Definition: Virtual machines or containers that execute jobs.
Types:
- GitHub-hosted: ubuntu-latest, windows-latest, macos-latest.
- Self-hosted: Custom runners on your infrastructure.
Example:

  runs-on: ubuntu-latest

8. Environment Variables

Definition: Variables available to steps, jobs, or workflows.
Levels: Workflow (env), job (env), step (env).
Secrets: Securely stored variables accessed via ${{ secrets.NAME }}.
Example:

  env:
    API_KEY: ${{ secrets.API_KEY }}

9. Matrix Strategy

Definition: Runs a job multiple times with different configurations (e.g., multiple versions of a language).
Key Attributes:
- strategy.matrix: Defines variables (e.g., go-version: ['1.21', '1.22']).
- strategy.fail-fast: Stops matrix on first failure (default: true).
Example:

  strategy:
    matrix:
      go-version: ['1.21', '1.22']

10. Artifacts and Outputs

Artifacts: Files stored after a job (e.g., build outputs).
- Use actions/upload-artifact@v4 and actions/download-artifact@v4.
Outputs: Data passed between steps or jobs via ::set-output.
Example:

  - name: Upload artifact
    uses: actions/upload-artifact@v4
    with:
      name: my-artifact
      path: ./build/

11. Concurrency and Permissions

Concurrency: Prevents multiple workflow runs from conflicting.
- Use concurrency.group and concurrency.cancel-in-progress.
Permissions: Control workflow access to repository resources.
- Use permissions (e.g., contents: read).
Example:

  concurrency:
    group: ${{ github.workflow }}-${{ github.ref }}
    cancel-in-progress: true
  permissions:
    contents: write

12. Common Actions

actions/checkout@v4: Clones the repository.
actions/setup-<language>@vX: Sets up languages (e.g., setup-go, setup-node).
actions/upload-artifact@v4: Uploads files.
actions/download-artifact@v4: Downloads files.
actions/cache@v4: Caches dependencies for faster builds.

13. Expressions and Contexts

Expressions: Use ${{ <expression> }} for dynamic values (e.g., ${{ github.sha }}).
Contexts: Predefined objects like github, env, secrets, matrix.
Example:

  if: ${{ github.event_name == 'push' }}

14. Best Practices

Use specific action versions (e.g., @v4 instead of @main).
Store sensitive data in GitHub Secrets.
Break workflows into modular jobs for clarity.
Use continue-on-error for non-critical steps.
Leverage caching to speed up workflows (actions/cache@v4).

For more details, see the GitHub Actions Documentation.

Analysis of the Provided Go Workflow YAML

The provided YAML file defines a GitHub Actions workflow named unit-test for a Go project with a PostgreSQL database. Below is a detailed breakdown of the workflow, referencing the cheatsheet components.

YAML File

# This workflow will build a golang project
# For more information see: https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-go

name: unit-test

on:
  push:
    branches: ["main", "dev/sajjad/users"]
  pull_request:
    branches: ["main"]

jobs:
  test:
    name: Test
    runs-on: ubuntu-latest

    services:
      postgres:
        image: postgres:latest
        # Provide the password for postgres
        env:
          POSTGRES_USER: root
          POSTGRES_PASSWORD: root
          POSTGRES_DB: simple_bank
        ports:
          - 5432:5432
        options: >-
          --health-cmd pg_isready
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5

    steps:
      - name: Set up Go
        uses: actions/setup-go@v4
        with:
          go-version: "1.24.3"

      - name: Check out code to runner
        uses: actions/checkout@v4

      - name: Install go-migrate
        run: |
          $ curl -L https://github.com/golang-migrate/migrate/releases/download/v4.18.3/migrate.linux-amd64.tar.gz | tar xvz
          sudo mv migrate.linux-amd64 /usr/bin/migrate
          which Migrate

      - name: Run migrations
        run: make migrateup

      - name: Test
        run: make test

Component Breakdown

Workflow:
- Name: unit-test
- Purpose: Automates unit testing for a Go project with a PostgreSQL database.
- File Location: Stored in .github/workflows/ (not specified in the YAML but implied).
Events (on):
- Triggers:
  - push: branches: ["main", "dev/sajjad/users"]: Runs on pushes to the main branch and the dev/sajjad/users branch.
  - pull_request: branches: ["main"]: Runs on pull requests targeting the main branch.
- Explanation: This ensures the workflow runs for code changes to specific branches and pull requests, supporting both development and production workflows.
Jobs:
- Job Name: test
- Runner: runs-on: ubuntu-latest
  - Uses a GitHub-hosted Ubuntu virtual machine (latest version).
- Purpose: Executes the test-related tasks for the Go project.
Services:
- Service Name: postgres
- Configuration:
  - image: postgres:latest: Uses the latest PostgreSQL Docker image.
  - env:
  - POSTGRES_USER: root: Database user.
  - POSTGRES_PASSWORD: root: Database password.
  - POSTGRES_DB: simple_bank: Database name.
  - ports: - 5432:5432: Maps port 5432 for access via localhost:5432.
  - options: Health checks ensure the database is ready:
  - --health-cmd pg_isready: Checks if PostgreSQL is accepting connections.
  - --health-interval 10s, --health-timeout 5s, --health-retries 5: Retries 5 times every 10 seconds with a 5-second timeout.
- Explanation: The PostgreSQL service provides a database for testing and migrations, accessible by the job’s steps.
Steps:
- Step 1: Set up Go
  - uses: actions/setup-go@v4
  - with: go-version: "1.24.3": Installs Go version 1.24.3.
  - Explanation: Sets up the Go environment required for building and testing.
- Step 2: Check out code to runner
  - uses: actions/checkout@v4
  - Explanation: Clones the repository to the runner, making the code available for subsequent steps.
- Step 3: Install go-migrate
  - run: Downloads and installs go-migrate分散
```
   curl -L https://github.com/golang-migrate/migrate/releases/download/v4.18.3/migrate.linux-amd64.tar.gz | tar xvz
   sudo mv migrate.linux-amd64 /usr/bin/migrate
   which Migrate
```

 - **Explanation**: Installs the `go-migrate` CLI (version 4.18.3) to manage database migrations.

Step 4: Run migrations
- run: make migrateup
- Explanation: Runs the migrateup Makefile target to apply database migrations to the PostgreSQL database.
Step 5: Test
- run: make test
- Explanation: Runs the test Makefile target to execute the Go tests, likely using the PostgreSQL database.

How the Workflow Works

Trigger: The workflow runs on:
- Pushes to the main or dev/sajjad/users branches.
- Pull requests targeting the main branch.
Runner Setup: An ubuntu-latest virtual machine is provisioned.
Service Startup: The postgres service starts a PostgreSQL container with the simple_bank database, accessible at localhost:5432.
Step Execution:
- Installs Go 1.24.3.
- Clones the repository.
- Installs the go-migrate CLI.
- Runs database migrations using make migrateup.
- Executes tests using make test.
Completion: The workflow completes, marking success or failure in the GitHub Actions UI. The PostgreSQL container is terminated.

Key Notes

Makefile Integration: The workflow uses make commands (migrateup, test), indicating the project has a Makefile to simplify task execution.
Security: Hardcoded POSTGRES_USER and POSTGRES_PASSWORD (root) are used for simplicity. In production, use GitHub Secrets for sensitive data.
Health Checks: The PostgreSQL service uses health checks to ensure readiness, preventing steps from running before the database is available.
Potential Improvement: The go-migrate installation command has an extra $ before curl, which may cause errors. It should be:

  curl -L https://github.com/golang-migrate/migrate/releases/download/v4.18.3/migrate.linux-amd64.tar.gz | tar xvz

Additional Tips for This Workflow

Add Environment Variables: Explicitly define DATABASE_URL for clarity:

  env:
    DATABASE_URL: postgres://root:root@localhost:5432/simple_bank?sslmode=disable

in the Run migrations and Test steps.

Caching: Use actions/cache@v4 to cache Go modules for faster builds:

  - name: Cache Go modules
    uses: actions/cache@v4
    with:
      path: ~/go/pkg/mod
      key: ${{ runner.os }}-go-${{ hashFiles('**/go.mod') }}

Matrix Testing: Test multiple Go versions:

  strategy:
    matrix:
      go-version: ['1.21', '1.22', '1.24.3']

For further customization, refer to the GitHub Actions Documentation and go-migrate Documentation.

Database Concurrency Phenomena & ISOLATION Label: Read Phenomena and Serialization Anomaly

Huseyn — Tue, 15 Jul 2025 10:16:00 +0000

Introduction

In database systems, concurrent transactions can lead to issues that affect data consistency. These issues, known as read phenomena (dirty reads, non-repeatable reads, phantom reads) and serialization anomaly, occur when multiple transactions read and write data simultaneously. This document defines and explains these phenomena with examples to illustrate their impact in a generic database context.

Concurrency Phenomena

1. Dirty Reads

Definition: A transaction reads data that has been modified by another transaction but not yet committed. If the modifying transaction rolls back, the read data becomes invalid.
Example:
- Transaction 1 updates a customer’s balance to 90 but hasn’t committed.
- Transaction 2 reads the balance as 90.
- Transaction 1 rolls back, restoring the balance to 100.
- Issue: Transaction 2 operates on incorrect data (90 instead of 100).
Impact: Can lead to decisions based on temporary, uncommitted data, causing inconsistencies in applications like financial systems or inventory management.

2. Non-Repeatable Reads

Definition: A transaction reads the same data twice and gets different results because another transaction modified and committed the data between the reads.
Example:
- Transaction 1 reads a product’s stock quantity as 50.
- Transaction 2 updates the stock to 40 and commits.
- Transaction 1 reads the stock again and sees 40.
- Issue: The same query within Transaction 1 yields inconsistent results.
Impact: Inconsistent reads can cause errors in applications requiring stable data, such as order processing or reporting.

3. Phantom Reads

Definition: A transaction re-executes a query to retrieve a set of rows and finds different rows (new or missing) because another transaction inserted or deleted rows and committed.
Example:
- Transaction 1 queries all orders for a customer, finding 3 orders.
- Transaction 2 inserts a new order for the same customer and commits.
- Transaction 1 re-runs the query and finds 4 orders.
- Issue: The result set changes due to “phantom” rows appearing.
Impact: Can affect operations that rely on a consistent set of records, such as generating reports or auditing transactions.

4. Serialization Anomaly

Definition: A serialization anomaly occurs when the outcome of concurrently executing transactions cannot be explained by any serial (one-at-a-time) execution order of those transactions, even though each transaction is individually consistent. This is specific to the SERIALIZABLE isolation level.
Example:
- Initial State: Three records (A: 100, B: 100, C: 100).
- Transaction 1 updates A to 90 and B to 110.
- Transaction 2 updates B to 90 and C to 110.
- Transaction 3 updates C to 90 and A to 110.
- Expected Outcome (serial execution): Balances remain A: 100, B: 100, C: 100 (net zero change).
- Anomaly: Concurrent execution might result in inconsistent balances (e.g., A: 90, B: 100, C: 110) because transactions interleave in a way that doesn’t match any serial order.
- Issue: The final state is inconsistent with expected business logic, despite each transaction being valid.
Impact: Can lead to incorrect data states in applications where the combined effect of transactions must align with a specific logical outcome, such as financial ledgers.

SQL Transaction Isolation Levels

Transaction isolation levels, defined by the SQL standard, control how transactions interact with each other and manage concurrency issues like read phenomena and serialization anomalies. The four standard isolation levels, from least to most strict, are described below, including their impact on the phenomena discussed above.

1. READ UNCOMMITTED

Definition: The least strict isolation level, allowing transactions to read uncommitted changes made by other transactions.
Behavior:
- Transactions can see data modified by other transactions before they commit.
- Offers the highest concurrency but the lowest consistency.
Impact on Phenomena:
- Dirty Reads: Possible, as uncommitted data is visible.
- Non-Repeatable Reads: Possible, as committed changes from other transactions can alter data between reads.
- Phantom Reads: Possible, as new rows inserted by other transactions can appear.
- Serialization Anomaly: Possible, as there’s no guarantee of serial execution.
Example:
- Transaction 1 updates a record but doesn’t commit.
- Transaction 2 reads the uncommitted data, which may be rolled back, leading to a dirty read.
Use Case: Rarely used in practice due to its high risk of inconsistencies; suitable only for applications where data accuracy is not critical (e.g., approximate analytics).

2. READ COMMITTED

Definition: Ensures that transactions only read data that has been committed, preventing dirty reads but allowing other concurrency issues.
Behavior:
- Each query within a transaction sees only committed data at the time of the query.
- Common default isolation level in many databases, including PostgreSQL.
Impact on Phenomena:
- Dirty Reads: Prevented, as only committed data is visible.
- Non-Repeatable Reads: Possible, as data can change between queries within the same transaction.
- Phantom Reads: Possible, as new rows can be inserted by other transactions.
- Serialization Anomaly: Possible, as there’s no guarantee of serial execution.
Example:
- Transaction 1 reads a record’s value as 50.
- Transaction 2 updates the value to 40 and commits.
- Transaction 1 reads the same record again and sees 40 (non-repeatable read).
Use Case: Suitable for applications where dirty reads are unacceptable but some inconsistency (e.g., non-repeatable reads) is tolerable, such as simple reporting systems.

3. REPEATABLE READ

Definition: Ensures that data read by a transaction remains consistent throughout the transaction, preventing non-repeatable reads but not phantom reads.
Behavior:
- Uses a snapshot of committed data at the start of the transaction for all reads, or locks read rows to prevent modifications.
- In PostgreSQL, this is implemented using multiversion concurrency control (MVCC).
Impact on Phenomena:
- Dirty Reads: Prevented, as only committed data is visible.
- Non-Repeatable Reads: Prevented, as the transaction sees a consistent snapshot of data.
- Phantom Reads: Possible, as new rows inserted by other transactions may appear in subsequent queries.
- Serialization Anomaly: Possible, as concurrent transactions can still produce non-serializable outcomes.
Example:
- Transaction 1 reads all records matching a condition (e.g., 3 orders).
- Transaction 2 inserts a new record matching the condition and commits.
- Transaction 1 re-queries and sees the new record (phantom read).
Use Case: Suitable for applications requiring consistent reads within a transaction, such as financial calculations or inventory checks, but where phantom reads are acceptable.

4. SERIALIZABLE

Definition: The strictest isolation level, ensuring that transactions execute as if they were run sequentially, preventing all concurrency anomalies.
Behavior:
- Guarantees that the outcome of concurrent transactions matches some serial execution order.
- In PostgreSQL, uses predicate locking and MVCC to detect conflicts, aborting transactions (error code 40001) if a serialization anomaly is detected.
Impact on Phenomena:
- Dirty Reads: Prevented, as only committed data is visible.
- Non-Repeatable Reads: Prevented, as the transaction sees a consistent snapshot.
- Phantom Reads: Prevented, as conflicting inserts/deletes trigger a serialization failure.
- Serialization Anomaly: Prevented, as the database ensures a serializable outcome.
Example:
- Transaction 1 and Transaction 2 attempt conflicting updates that would lead to a serialization anomaly (e.g., cyclic updates to records).
- The database detects the conflict and aborts one transaction, requiring a retry.
Use Case: Ideal for applications requiring strict data consistency, such as financial systems or critical business logic, though it may reduce concurrency due to potential transaction aborts.

Isolation Levels and Concurrency Phenomena Table

This table illustrates the relationship between SQL transaction isolation levels and the concurrency phenomena they allow or prevent.

Phenomenon	READ UNCOMMITTED	READ COMMITTED	REPEATABLE READ	SERIALIZABLE
Dirty Read	Yes	No	No	No
Non-Repeatable Read	Yes	Yes	No	No
Phantom Read	Yes	Yes	Yes	No
Serialization Anomaly	Yes	Yes	Yes	No

Explanation

Yes: The phenomenon can occur under the specified isolation level.
No: The phenomenon is prevented under the specified isolation level.
Isolation Levels:
- READ UNCOMMITTED: Allows all phenomena due to minimal restrictions.
- READ COMMITTED: Prevents dirty reads but allows others.
- REPEATABLE READ: Prevents dirty and non-repeatable reads but allows phantom reads.
- SERIALIZABLE: Prevents all phenomena, ensuring full consistency.

Summary

Dirty Reads: Reading uncommitted data (possible in READ UNCOMMITTED).
Non-Repeatable Reads: Inconsistent results from repeated reads (possible in READ UNCOMMITTED and READ COMMITTED).
Phantom Reads: Changes in query result sets due to inserts/deletes (possible in READ UNCOMMITTED, READ COMMITTED, and REPEATABLE READ).
Serialization Anomaly: Non-serializable outcomes from concurrent transactions (possible in all but SERIALIZABLE).
Isolation Levels:
- READ UNCOMMITTED: Allows all phenomena; least consistent.
- READ COMMITTED: Prevents dirty reads; moderate consistency.
- REPEATABLE READ: Prevents dirty and non-repeatable reads; higher consistency.
- SERIALIZABLE: Prevents all phenomena; maximum consistency but may impact performance.

Understanding these phenomena and isolation levels is crucial for designing robust database applications, as they guide the choice of isolation level to balance consistency and performance based on application requirements.

Database Transactions: A Comprehensive Guide with Go

Huseyn — Sat, 12 Jul 2025 17:32:30 +0000

What is a Database Transaction?

A database transaction (often called a "TX") is a sequence of one or more database operations (e.g., INSERT, UPDATE, DELETE) executed as a single logical unit. Transactions ensure data integrity and consistency by guaranteeing that either all operations succeed or none are applied, maintaining the database in a consistent state.

Key Characteristics - ACID

Atomicity: All operations in a transaction are completed successfully, or none are applied (all-or-nothing).
Consistency: The database transitions from one valid state to another, adhering to all constraints (e.g., foreign keys, unique constraints).
Isolation: Transactions are executed independently, preventing interference from concurrent transactions.
Durability: Committed transactions are permanently saved, even in the event of a system failure.

Example: Transferring $100 from Account A to Account B involves two operations:

Deduct $100 from Account A.
Add $100 to Account B.

These must be part of a single transaction to avoid inconsistencies (e.g., money deducted but not credited).

Ensuring Transactions

To ensure transactions are reliable, follow these steps:

Begin a Transaction:
- Start with BEGIN or START TRANSACTION in SQL or use the Go database/sql package’s Begin() method.
- This marks the transaction’s start, tracking all operations as a single unit.
Execute Operations:
- Perform SQL queries within the transaction.
- Changes are held in a temporary state until committed or rolled back.
Commit the Transaction:
- Use COMMIT in SQL or Tx.Commit() in Go to make changes permanent.
- Ensures durability, saving changes to disk.
Rollback on Failure:
- Use ROLLBACK in SQL or Tx.Rollback() in Go to undo changes if an error occurs.
- Ensures atomicity, restoring the database to its pre-transaction state.
Set Isolation Levels:
- Use isolation levels (e.g., READ COMMITTED, SERIALIZABLE) to control how transactions interact.
- In Go, set the isolation level when beginning a transaction (e.g., db.BeginTx(context.Background(), &sql.TxOptions{Isolation: sql.LevelSerializable})).
Error Handling:
- Implement robust error handling in Go to catch failures (e.g., constraint violations) and trigger rollbacks.

Database State in Transactions

The database state refers to the data at any given moment. Transactions ensure valid state transitions:

Initial State: The database is consistent, satisfying all constraints.
Temporary State: During a transaction, changes are held in memory or logs (e.g., using write-ahead logging or MVCC) and are not visible to other transactions until committed (depending on isolation level).
Committed State: On COMMIT, changes are written to permanent storage, creating a new consistent state.
Rolled-Back State: On ROLLBACK, changes are discarded, and the database reverts to its initial state.
Concurrency: Isolation levels and mechanisms like MVCC (in PostgreSQL) ensure concurrent transactions see consistent snapshots.

What Happens if a Transaction Fails?

Failures can occur due to errors, crashes, or concurrency issues. The DBMS and application handle these as follows:

Query Errors: If a query fails (e.g., due to a constraint violation), the transaction is rolled back, undoing all changes.
System Crashes: Write-ahead logging ensures committed changes are recovered, and uncommitted changes are discarded on restart.
Deadlocks: The DBMS detects deadlocks and aborts one transaction, requiring the application to retry.
Application Handling: In Go, use error handling to detect failures and call Tx.Rollback().

Implementing Transactions in Go

Go’s database/sql package provides robust support for transactions, compatible with databases like PostgreSQL, MySQL (InnoDB), and SQLite. Below is an example of a transaction in Go using PostgreSQL to transfer money between accounts.

Go Example: Bank Transfer Transaction

This example demonstrates a transaction to transfer $100 from Account A to Account B, with error handling and rollback.

package main

import (
    "context"
    "database/sql"
    "fmt"
    "log"

    _ "github.com/lib/pq"
)

func main() {
    // Connect to PostgreSQL database
    connStr := "user=youruser password=yourpassword dbname=yourdb sslmode=disable"
    db, err := sql.Open("postgres", connStr)
    if err != nil {
        log.Fatal("Failed to connect to database:", err)
    }
    defer db.Close()

    // Begin a transaction
    ctx := context.Background()
    tx, err := db.BeginTx(ctx, &sql.TxOptions{Isolation: sql.LevelReadCommitted})
    if err != nil {
        log.Fatal("Failed to begin transaction:", err)
    }

    // Execute transaction operations
    _, err = tx.ExecContext(ctx, "UPDATE accounts SET balance = balance - 100 WHERE account_id = $1", "A")
    if err != nil {
        tx.Rollback()
        log.Fatal("Failed to deduct from Account A:", err)
    }

    _, err = tx.ExecContext(ctx, "UPDATE accounts SET balance = balance + 100 WHERE account_id = $1", "B")
    if err != nil {
        tx.Rollback()
        log.Fatal("Failed to credit Account B:", err)
    }

    // Check for sufficient balance (example constraint)
    var balance float64
    err = tx.QueryRowContext(ctx, "SELECT balance FROM accounts WHERE account_id = $1", "A").Scan(&balance)
    if err != nil {
        tx.Rollback()
        log.Fatal("Failed to check balance:", err)
    }
    if balance < 0 {
        tx.Rollback()
        log.Fatal("Insufficient balance in Account A")
    }

    // Commit the transaction
    err = tx.Commit()
    if err != nil {
        tx.Rollback()
        log.Fatal("Failed to commit transaction:", err)
    }

    fmt.Println("Transaction completed successfully!")
}

Explanation of Go Code

Database Connection: Uses sql.Open to connect to a PostgreSQL database.
Begin Transaction: db.BeginTx starts a transaction with a specified isolation level (sql.LevelReadCommitted).
Execute Queries: tx.ExecContext runs SQL queries within the transaction.
Error Handling: Checks for errors after each query and calls tx.Rollback() if any occur.
Balance Check: Queries the balance to enforce a business rule (e.g., no negative balance).
Commit or Rollback: Calls tx.Commit() to save changes or tx.Rollback() to undo them.

Prerequisites

Install the PostgreSQL driver: go get github.com/lib/pq
Ensure a PostgreSQL database with an accounts table:

  CREATE TABLE accounts (
      account_id VARCHAR(50) PRIMARY KEY,
      balance DECIMAL(10, 2)
  );
  INSERT INTO accounts (account_id, balance) VALUES ('A', 500.00), ('B', 200.00);

Common Issues and Mitigations

Deadlocks:
- Issue: Multiple transactions lock resources in conflicting orders.
- Mitigation: Access tables in a consistent order, use shorter transactions, and catch deadlock errors (sql.ErrTxDone or database-specific errors) to retry.
Performance:
- Issue: Long transactions hold locks, reducing concurrency.
- Mitigation: Optimize queries, use appropriate indexes, and break large transactions into smaller ones.
Lost Updates:
- Issue: Concurrent transactions overwrite each other’s changes.
- Mitigation: Use stricter isolation levels (e.g., sql.LevelSerializable) or optimistic locking.
Connection Management:
- Issue: Improper connection handling can lead to leaks.
- Mitigation: Always defer db.Close() and ensure tx.Rollback() or tx.Commit() is called.

Best Practices in Go

Use Context: Pass a context.Context to BeginTx and query methods to support cancellation and timeouts.
Explicit Rollbacks: Always call tx.Rollback() in error paths, even if tx.Commit() fails, as some DBMSs require it.
Connection Pooling: Leverage database/sql’s built-in connection pooling by reusing the sql.DB object.
Isolation Levels: Choose the appropriate isolation level for your use case to balance consistency and performance.
Testing: Test transaction failure scenarios (e.g., constraint violations, network issues) to ensure robust error handling.

Conclusion

Database transactions ensure data integrity by adhering to ACID properties. In Go, the database/sql package provides a clean and efficient way to manage transactions, with robust error handling and rollback support. By following best practices and understanding database state transitions, you can build reliable applications that maintain consistency even in the face of failures.

For further details, explore the Go database/sql documentation and your DBMS’s transaction management features (e.g., PostgreSQL’s MV assumptions.

Database Schema Migration Cheatsheet with golang-migrate/migrate

Huseyn — Sat, 12 Jul 2025 10:10:17 +0000

This cheatsheet provides a concise guide to using the golang-migrate/migrate library for database schema migrations. It covers installation, file structure, migration commands (up, down, force down), and best practices.

1. Overview

What is golang-migrate/migrate?
- A CLI tool and Go library for managing database schema migrations.
- Supports incremental, version-controlled, and reversible changes to database schemas.
- Compatible with databases like PostgreSQL, MySQL, SQLite, and more.
- Migrations are written as SQL files (or Go code for advanced cases).
Key Concepts
- Up Migration: Applies changes to evolve the schema (e.g., create tables, add columns).
- Down Migration: Reverts changes to roll back the schema (e.g., drop tables, remove columns).
- Force Down: Resolves issues with a "dirty" database by forcing a specific version.

2. Installation

Install the migrate CLI tool for your operating system.

Prerequisites

Go installed (for Go-based CLI compilation, optional).
A supported database (e.g., PostgreSQL, MySQL) with connection details.

Install via Package Managers

MacOS (Homebrew):

  brew install golang-migrate

Linux (via curl):

  curl -L https://github.com/golang-migrate/migrate/releases/download/v4.17.1/migrate.linux-amd64.tar.gz | tar xvz
  sudo mv migrate /usr/local/bin/

Windows (Scoop):

  scoop install migrate

Verify Installation:

  migrate -version

Install via Go (Optional)

If you prefer to build from source or use as a library:

go get -u github.com/golang-migrate/migrate/v4

3. File Structure

Organize migration files in a dedicated directory for version control and clarity.

Recommended Structure

project/
├── database/
│   ├── migrations/
│   │   ├── 202507120001_create_users_table.up.sql
│   │   ├── 202507120001_create_users_table.down.sql
│   │   ├── 202507120002_add_email_index.up.sql
│   │   ├── 202507120002_add_email_index.down.sql
│   └── schema.sql  (optional: schema dump)
├── main.go
└── Makefile  (optional: for migration shortcuts)

Naming Convention: Files are named <timestamp>_<description>.<up/down>.sql.
- Example: 202507120001_create_users_table.up.sql for creating a table.
- Example: 202507120001_create_users_table.down.sql for dropping it.
Directory Path: Typically database/migrations/ (relative to project root).

Example Migration Files

202507120001_create_users_table.up.sql:

  -- Create users table
  CREATE TABLE users (
      id SERIAL PRIMARY KEY,
      username VARCHAR(50) NOT NULL,
      email VARCHAR(100) NOT NULL,
      created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
  );

202507120001_create_users_table.down.sql:

  -- Drop users table
  DROP TABLE IF EXISTS users;

202507120002_add_email_index.up.sql:

  -- Add index on email column
  CREATE INDEX idx_email ON users(email);

202507120002_add_email_index.down.sql:

  -- Drop index on email column
  DROP INDEX IF EXISTS idx_email;

4. Usage

Run migrations using the migrate CLI with a database connection string and migration path.

Basic Commands

Create a New Migration:

  migrate create -ext sql -dir database/migrations -seq create_table_name

-ext sql: Specifies SQL file extension.
-dir: Path to migration directory.
-seq: Generates a sequential timestamp.
Example output: Creates database/migrations/202507120003_create_table_name.up.sql and .down.sql.
- Apply Migrations (Up):

  migrate -path database/migrations/ -database "postgresql://username:password@localhost:5432/dbname?sslmode=disable" -verbose up

Applies all pending migrations in order.
-path: Directory containing migration files.
-database: Database connection string (replace with your DB details).
-verbose: Shows detailed output.
- Revert Migrations (Down):

  migrate -path database/migrations/ -database "postgresql://username:password@localhost:5432/dbname?sslmode=disable" -verbose down

Reverts the last applied migration.
Use down N to revert the last N migrations (e.g., down 2).
- Force Down (Fix Dirty Database):

  migrate -path database/migrations/ -database "postgresql://username:password@localhost:5432/dbname?sslmode=disable" force VERSION

Use when a migration fails and marks the database as "dirty," preventing further migrations.
Replace VERSION with the desired migration version (e.g., 202507120001).
Caution: Ensure the database state matches the forced version to avoid inconsistencies.
Example: If migration 202507120002 fails, fix the error in the .up.sql file, then run:
```
migrate -path database/migrations/ -database "postgresql://username:password@localhost:5432/dbname?sslmode=disable" force 202507120001
```
This sets the database to version 202507120001, clearing the dirty state.
- Check Migration Status:

  migrate -path database/migrations/ -database "postgresql://username:password@localhost:5432/dbname?sslmode=disable" version

Displays the current schema version.

Using with a Makefile (Optional)

Simplify commands with a Makefile:

# Makefile
DB_URL=postgresql://username:password@localhost:5432/dbname?sslmode=disable

migration_up:
    migrate -path database/migrations/ -database "$(DB_URL)" -verbose up

migration_down:
    migrate -path database/migrations/ -database "$(DB_URL)" -verbose down

migration_force:
    migrate -path database/migrations/ -database "$(DB_URL)" force $(VERSION)

Run: make migration_up, make migration_down, or make migration_force VERSION=202507120001.

5. Best Practices

Version Control: Commit migration files to your repository (e.g., Git) for team collaboration.
Test Migrations: Test migrations in a staging environment before applying to production.
Reversible Migrations: Always define down migrations to allow rollbacks.
Avoid Destructive Changes: Use non-destructive changes (e.g., add nullable columns) to minimize data loss risks.
Backup Database: Always back up your database before running migrations.
Handle Dirty Databases:
- If a migration fails, check the error, fix the .up.sql or .down.sql file, and use force VERSION cautiously.
- Verify the database state matches the forced version.

6. Common Issues and Solutions

Error: Dirty database version X. Fix and force version:
- Cause: A migration failed, leaving the database in a "dirty" state.
- Solution: Fix the problematic migration file, then run migrate force X to set the correct version.
Migration order issues: Ensure migration files are applied sequentially based on timestamps.
Connection errors: Verify the database connection string and ensure the database is running.

7. Example Workflow

Create a migration:

   migrate create -ext sql -dir database/migrations -seq add_posts_table

Edit database/migrations/202507120003_add_posts_table.up.sql:

   -- Create posts table
   CREATE TABLE posts (
       id SERIAL PRIMARY KEY,
       user_id INT REFERENCES users(id),
       title VARCHAR(255) NOT NULL
   );

Edit database/migrations/202507120003_add_posts_table.down.sql:

   -- Drop posts table
   DROP TABLE IF EXISTS posts;

Apply the migration:

   migrate -path database/migrations/ -database "postgresql://username:password@localhost:5432/dbname?sslmode=disable" up

Revert if needed:

   migrate -path database/migrations/ -database "postgresql://username:password@localhost:5432/dbname?sslmode=disable" down

If migration fails and database is dirty:

Fix the error in the migration file.
Force to the last successful version:

 migrate -path database/migrations/ -database "postgresql://username:password@localhost:5432/dbname?sslmode=disable" force 202507120002

8. Additional Resources

Official GitHub: golang-migrate/migrate
Supported Databases: PostgreSQL, MySQL, MariaDB, SQLite, SQL Server, and more.
CLI Documentation: Run migrate -help for full command details.

Complete Guide: Using `sqlc` with Your Go Project

Huseyn — Mon, 07 Jul 2025 09:29:55 +0000

🚀 What is `sqlc`?

sqlc is a code generation tool that converts your SQL queries into type-safe Go functions. It eliminates manual boilerplate code and avoids ORMs.

✅ Benefits

Type-safe queries
No ORM overhead
Native SQL with full control
Auto-generated Go models & methods

📦 Installation

CLI

brew install sqlc  # macOS
# or
go install github.com/sqlc-dev/sqlc/cmd/sqlc@latest

📁 Recommended Project Structure

myapp/
├── db/
│   ├── migrations/
│   ├── queries/
│   │   ├── users.sql
│   └── schema.sql
├── sqlc.yaml
├── go.mod
└── main.go

⚙️ `sqlc.yaml` Configuration

version: "2"
sql:
  - engine: "postgresql"
    queries: "db/queries/"
    schema: "db/migrations/"
    gen:
      go:
        package: "db"
        out: "db/sqlc"
        sql_package: "pgx/v5"
        emit_json_tags: true
        emit_interface: true

✍️ Writing SQL Queries

db/queries/users.sql

-- name: CreateUser :one
INSERT INTO users (username, email)
VALUES ($1, $2)
RETURNING id, username, email;

-- name: GetUser :one
SELECT id, username, email FROM users WHERE id = $1;

-- name: ListUsers :many
SELECT id, username, email FROM users ORDER BY id DESC;

🔖 Query Types in `sqlc`

Tag	Purpose	Go Return Type
`:one`	Returns a single row. Fails if no row or more than one is returned. Use it for `SELECT` with unique constraint or primary key filters.	`(Model, error)`
`:many`	Returns multiple rows as a slice. Use for general `SELECT` queries that return 0 to many rows.	`([]Model, error)`
`:exec`	Executes query with no result rows. Use for `INSERT`, `UPDATE`, or `DELETE` that don't return rows.	`(error)`
`:execrows`	Executes and also returns the number of rows affected. Useful when you care how many rows were impacted by `UPDATE` or `DELETE`.	`(int64, error)`
`:copyfrom`	Generates a method using PostgreSQL's COPY FROM. Use for fast bulk insert operations.	`(int64, error)`

📄 Example Migration (`db/migrations/init.up.sql`)

CREATE TABLE users (
  id SERIAL PRIMARY KEY,
  username TEXT NOT NULL,
  email TEXT UNIQUE NOT NULL
);

🛠️ Generating Go Code

sqlc generate

Creates Go code in db/sqlc with models and query methods.

🧑‍💻 Using in Go Code

import (
  "context"
  "database/sql"
  "fmt"
  _ "github.com/lib/pq"
  "myapp/db/sqlc"
)

dbConn, _ := sql.Open("postgres", "postgresql://root:root@localhost:5432/simple_bank?sslmode=disable")
q := sqlc.New(dbConn)

user, _ := q.CreateUser(context.Background(), "esha", "esha@email.com")
fmt.Println(user)

🧹 Handling Errors

If you see:

error: Dirty database version -1. Fix and force version.

Use:

migrate ... force 0

Then:

migrate ... up

✅ Tips

Use pgx/v5 for performance
Never edit applied migrations
Use .PHONY Makefile targets to run sqlc generate, migrate, etc.

More
These annotations like :one, :many, :exec, etc., are special sqlc query tags that tell sqlc:

What kind of Go function to generate
What kind of return value to expect

They appear in SQL comments just before each query:

-- name: MyFunctionName :<type>
<SQL statement>

Let’s break them down:

🔖 `sqlc` Query Types Explained

Tag	Purpose	Return Type (Go)	Use When...
`:one`	Return exactly one row	`(Type, error)`	SELECT that should return a single record
`:many`	Return multiple rows	`([]Type, error)`	SELECT with multiple rows expected
`:exec`	Execute query, no result rows	`(error)`	INSERT/UPDATE/DELETE without returning
`:execrows`	Like `:exec` but also returns rows affected	`(int64, error)`	UPDATE/DELETE where you want affected row count
`:copyfrom`	Use COPY FROM PostgreSQL bulk import	`(int64, error)`	For high-speed bulk insert

🧠 Examples

✅ `:one`

-- name: GetUserByID :one
SELECT id, username, email FROM users WHERE id = $1;

Generates:

func (q *Queries) GetUserByID(ctx context.Context, id int32) (User, error)

✅ `:many`

-- name: ListUsers :many
SELECT id, username, email FROM users ORDER BY id DESC LIMIT 10;

Generates:

func (q *Queries) ListUsers(ctx context.Context) ([]User, error)

✅ `:exec`

-- name: DeleteUser :exec
DELETE FROM users WHERE id = $1;

Generates:

func (q *Queries) DeleteUser(ctx context.Context, id int32) error

✅ `:execrows`

-- name: UpdateEmail :execrows
UPDATE users SET email = $2 WHERE id = $1;

Generates:

func (q *Queries) UpdateEmail(ctx context.Context, id int32, email string) (int64, error)

✅ `:copyfrom`

-- name: CopyUsers :copyfrom
COPY users (username, email) FROM STDIN;

Generates:

func (q *Queries) CopyUsers(ctx context.Context, r io.Reader) (int64, error)

Used for efficient bulk data import with a CSV reader or similar.

🧠 Summary Cheat Sheet

You want to...	Use this
SELECT 1 row	`:one`
SELECT many rows	`:many`
INSERT/UPDATE/DELETE (no rows)	`:exec`
UPDATE and get row count	`:execrows`
COPY FROM for bulk insert	`:copyfrom`

Docker Cheatsheet

Huseyn — Sun, 06 Jul 2025 17:15:28 +0000

What is Docker?
Docker is a platform for developing, shipping, and running applications inside containers. It allows you to package an application with its dependencies, ensuring consistency across different environments.

Key Concepts

Container : A lightweight, standalone, executable package that includes everything needed to run an application (code, runtime, libraries, and dependencies).
Image : A read-only template used to create containers. Images are built from a series of layers, each representing an instruction in the image’s Dockerfile.

Basic Docker Commands

Managing Containers

docker ps: List all running containers.
Example: docker ps
Use: Check currently active containers.

docker ps -a: List all containers (including stopped ones).
Example: docker ps -a
Use: View all containers, running or stopped.

docker run: Create and start a container from an image.
Example: docker run -d -p 8080:80 nginx
Options:
-d: Run container in detached mode (background).
-p :: Map host port to container port.
--name <name>: Assign a name to the container.
-e <environemnt_variable>: set env variables like db user, pass etc

Use: Start a new container from an image.

docker exec: Run a command inside a running container.
Example: docker exec -it <container_id> /bin/bash
Options:
-i: Interactive mode (keep STDIN open).
-t: Allocate a pseudo-TTY (terminal).

Use: Access a container’s shell or run commands inside it.

docker stop: Stop a running container.
Example: docker stop <container_id>
Use: Gracefully stop a container.

docker rm: Remove a stopped container.
Example: docker rm <container_id>
Use: Delete a container to free up resources.

Managing Images

docker images: List all images on the system.
Example: docker images
Use: View available images locally.

docker pull: Download an image from a registry (e.g., Docker Hub).
Example: docker pull ubuntu:latest
Use: Fetch an image to use locally.

docker rmi: Remove an image.
Example: docker rmi <image_id>
Use: Delete an unused image to free up space.

docker build: Build an image from a Dockerfile.
Example: docker build -t my-app:latest .
Options: -t <name:tag>: Name and tag the image.

Use: Create a custom image from a Dockerfile.

Logs and Monitoring

docker logs: View logs of a container.
Example: docker logs <container_id>
Options: -f: Follow log output (like tail -f).

Use: Inspect container output for debugging or monitoring.

docker inspect : Display detailed information about a container or image.
Example: docker inspect <container_id>
Use: Get metadata, network settings, or configuration details.

System Management

docker info: Display system-wide information about Docker.
Example: docker info
Use: Check Docker version, storage, and running containers/images.

docker system prune: Remove all unused containers, networks, and images.
Example: docker system prune -a
Options:
-a: Remove all unused images, not just dangling ones.

Use: Clean up unused Docker resources.

Networking

docker network ls: List all networks.
Example: docker network ls
Use: View available Docker networks.

docker network create: Create a new network.
Example: docker network create my-network
Use: Set up a custom network for containers to communicate.

Command	Description
`docker build -t my-app .`	Build an image from Dockerfile in current directory.
`docker images`	List all images on local machine.
`docker rmi <image_id>`	Remove an image.
`docker pull <image>`	Pull an image from Docker Hub.
`docker push <image>`	Push your image to a registry.

Command	Description
`docker run -it --name my-container -p 3000:3000 my-app`	Run a container from an image interactively.
`docker ps`	List running containers.
`docker ps -a`	List all containers, running or stopped.
`docker stop <container_id>`	Stop a running container.
`docker rm <container_id>`	Remove a stopped container.
`docker logs <container_id>`	View container logs.
`docker exec -it <container_id> /bin/bash`	Open shell in running container.
`docker container inspect container_name`	Inspect Container

Command	Description
`docker-compose up`	Start all services defined in `docker-compose.yml`.
`docker-compose down`	Stop and remove all services.
`docker-compose build`	Build/rebuild images for services.
`docker-compose logs`	View logs of all services.

Command	Description
`docker network ls`	List all available Docker networks.
`docker network inspect <network_name>`	Show detailed info about a specific network (subnet, containers, driver, etc.).
`docker network create mynet`	Create a new custom network (default: bridge).
`docker network create --driver bridge mynet`	Create a network explicitly using the `bridge` driver.
`docker network create --driver host hostnet`	Create a network using the host driver (shares host network).
`docker network create --driver overlay overnet`	Create an overlay network (multi-host, Docker Swarm).
`docker network rm <network_name>`	Delete a specific network (only if no container is using it).
`docker network prune`	Remove all unused networks.
`docker run -d --name app --network=mynet myimage`	Run a container and attach it to `mynet`.
`docker network connect mynet mycontainer`	Connect an existing container to `mynet`.
`docker network disconnect mynet mycontainer`	Disconnect a container from `mynet`.
`docker inspect <container_id>	grep -A 5 "Networks"`	Show which networks a container is connected to.

Tips

Always specify a tag (e.g., nginx:latest) when pulling or running images to avoid unexpected updates.
Use docker ps -q to get only container IDs (useful for scripting).
Combine docker exec with -itfor interactive commands like shells.
Regularly clean up unused containers and images with docker systemprune to save disk space.

Mind Map: From Powering On Laptop to Loading an Application

Huseyn — Fri, 04 Jul 2025 11:10:54 +0000

1. Powering On the Laptop

HDD (Hard Disk Drive or SSD):
- The HDD stores the operating system (OS), applications, and data in a non-volatile state, meaning they persist even when the laptop is powered off.
- When you power on the laptop, the BIOS/UEFI (firmware on the motherboard) initializes hardware and loads the OS bootloader from the HDD into RAM.
RAM (Random Access Memory):
- RAM is volatile memory, meaning it’s empty when the laptop is off. Upon boot, the bootloader and essential OS files are copied from the HDD to RAM for faster access.
- The CPU can only execute instructions stored in RAM, not directly from the HDD, due to RAM’s much faster access speed.
Flow:
- HDD → Bootloader code is copied to RAM → CPU begins executing the bootloader to load the OS.

2. Operating System Loads

CPU (Central Processing Unit):
- The CPU is the "brain" of the computer, responsible for executing instructions. It consists of several key components: Program Counter (PC), Instruction Register (IR), Register Set, Control Unit (CU), and Arithmetic Logic Unit (ALU).
Program Counter (PC):
- The PC holds the memory address of the next instruction to be executed.
- At boot, the PC is set to the starting address of the bootloader code in RAM, directing the CPU to begin OS loading.
Control Unit (CU):
- The CU acts as the "conductor," coordinating the fetch-decode-execute cycle.
- It reads the address in the PC, fetches the instruction from RAM, and directs other components to process it.
Instruction Register (IR):
- The IR temporarily holds the current instruction fetched from RAM, allowing the CU to decode it.
Register Set:
- Registers are small, ultra-fast storage locations inside the CPU (e.g., general-purpose registers, status registers).
- During OS loading, registers store temporary data like memory addresses, counters, or status flags used by the bootloader and OS initialization code.
Arithmetic Logic Unit (ALU):
- The ALU performs arithmetic (e.g., addition) and logical operations (e.g., comparisons).
- During boot, the ALU may perform calculations for memory allocation or hardware checks as directed by the OS loading process.
RAM:
- As the OS loads, more files (kernel, drivers, system libraries) are copied from the HDD to RAM.
- RAM acts as a workspace where the CPU can quickly access and manipulate data.
Flow:
- PC points to bootloader address in RAM → CU fetches instruction to IR → CU decodes instruction → ALU/registers process data as needed → OS files are loaded from HDD to RAM → OS initializes.

3. Launching an Application

User Action:
- You click on an application (e.g., a web browser or text editor).
- The OS receives this request and initiates a process for the application.
Process and Thread:
- A process is an instance of a program in execution, including its code, data, and state (e.g., memory allocation, open files).
- Each process has at least one thread, which is the smallest unit of execution. A thread contains a PC, register set, and stack, allowing it to execute instructions independently within the process.
- Modern applications may use multiple threads (e.g., one for the UI, another for background tasks) to improve performance.
HDD:
- The application’s executable file (e.g., .exe on Windows) is stored on the HDD.
- When you launch the application, the OS locates the executable on the HDD.
RAM:
- The OS copies the application’s code, libraries, and initial data from the HDD to RAM.
- The process’s memory space (code segment, data segment, stack, heap) is allocated in RAM.
Flow:
- OS identifies application on HDD → Copies executable and dependencies to RAM → Allocates process memory in RAM → Creates initial thread for execution.

4. Executing the Application

Program Counter (PC):
- The PC is set to the starting address of the application’s code in RAM (the entry point of the program).
- For each thread, the PC tracks the next instruction to execute.
Control Unit (CU):
- The CU manages the fetch-decode-execute cycle for the application’s instructions:
  1. Fetch: CU retrieves the instruction at the PC’s address from RAM and stores it in the IR.
  2. Decode: CU interprets the instruction to determine the required actions (e.g., load data, perform a calculation).
  3. Execute: CU directs the ALU, registers, or memory to carry out the instruction.
Instruction Register (IR):
- The IR holds the current instruction during the fetch phase, allowing the CU to decode and execute it.
Register Set:
- Registers store temporary data for the application, such as:
  - Variables or intermediate results (general-purpose registers).
  - Stack pointers for function calls and thread management.
  - Status flags (e.g., zero flag, carry flag) set by the ALU after operations.
- Each thread has its own set of register values to maintain its execution state.
Arithmetic Logic Unit (ALU):
- The ALU performs calculations (e.g., adding numbers in the application) or logical operations (e.g., checking if a condition is true) as specified by the instructions.
- For example, if the application calculates a sum, the ALU adds the values stored in registers.
RAM:
- The application’s code and data reside in RAM during execution.
- The CPU accesses RAM to fetch instructions (via the PC) and read/write data as the application runs.
- If the application opens a file, the file’s data may be copied from the HDD to RAM for processing.
HDD:
- The HDD is accessed if the application needs additional data (e.g., loading a saved file) or if the OS swaps out memory pages to free up RAM (in case of high memory usage).
Flow:
- PC points to application code in RAM → CU fetches instruction to IR → CU decodes instruction → Registers/ALU execute operations (e.g., calculations, data movement) → RAM provides code/data → HDD accessed for additional data if needed → Thread continues executing, updating PC for next instruction.

5. Multitasking and Thread Management

Process and Thread Context:
- The OS manages multiple processes and threads, scheduling them on the CPU using a scheduler.
- Each thread’s state (PC, register values, stack) is saved when it’s paused and restored when it resumes, enabling multitasking.
CPU Components:
- PC: Updated for each thread to point to its next instruction when switched.
- Registers: Save/restore thread-specific data during context switches.
- CU: Coordinates context switches by saving the current thread’s state and loading the next thread’s state.
RAM:
- Stores the memory space for all running processes and threads.
- The OS may use virtual memory, swapping data to the HDD if RAM is full.
HDD:
- Used for virtual memory (swap space/page file) when RAM is insufficient.
Flow:
- OS schedules threads → CU saves/restores PC and registers during context switches → RAM holds process/thread data → HDD handles swap space if needed.

6. Completing the Application’s Task

As the application runs, the CPU repeatedly executes the fetch-decode-execute cycle for each thread:
- PC increments to the next instruction or jumps to a new address (e.g., for loops or function calls).
- CU orchestrates the cycle, directing the ALU and registers.
- IR holds each instruction temporarily.
- ALU performs computations as needed.
- Registers store intermediate results and thread state.
- RAM provides fast access to code and data.
- HDD is accessed for persistent storage (e.g., saving a file).
When the application closes, the OS deallocates its RAM, terminates its process/threads, and updates any saved data on the HDD.

Summary Mind Map (Conceptual Flow)

1. Power On
   └── HDD: Stores OS → Copies bootloader to RAM
2. OS Loads
   ├── RAM: Holds OS code/data
   └── CPU
       ├── PC: Points to bootloader address
       ├── CU: Fetches/decodes/executes boot instructions
       ├── IR: Holds current boot instruction
       ├── Registers: Store temporary boot data
       └── ALU: Performs boot-related calculations
3. Launch Application
   ├── HDD: Stores application executable → Copies to RAM
   ├── RAM: Allocates process memory (code, data, stack)
   └── OS: Creates process and initial thread
4. Execute Application
   ├── CPU
   │   ├── PC: Tracks next instruction address
   │   ├── CU: Manages fetch-decode-execute cycle
   │   ├── IR: Holds current instruction
   │   ├── Registers: Store thread data (variables, stack pointers)
   │   └── ALU: Performs computations
   ├── RAM: Provides code/data for execution
   └── HDD: Accessed for additional data or virtual memory
5. Multitasking
   ├── OS: Schedules threads, manages context switches
   ├── CPU: Saves/restores PC and registers
   ├── RAM: Holds all process/thread data
   └── HDD: Swap space for virtual memory
6. Application Closes
   ├── RAM: Deallocates process memory
   ├── HDD: Saves final data
   └── CPU: Terminates thread execution

Key Roles Recap

HDD: Persistent storage for OS, applications, and data; slow but large capacity.
RAM: Fast, volatile memory for active code and data; serves as CPU’s workspace.
Program Counter (PC): Tracks the address of the next instruction, guiding the CPU’s execution flow.
Instruction Register (IR): Holds the current instruction for decoding and execution.
Register Set: Ultra-fast storage for temporary data and thread state.
Control Unit (CU): Orchestrates the fetch-decode-execute cycle and coordinates CPU components.
ALU: Performs arithmetic and logical operations for computations.
Process/Thread: Process is the program in execution; threads are lightweight execution units within a process, each with its own PC and registers.

This flow illustrates how the components work together seamlessly, from powering on the laptop to running an application, with the CPU’s components, RAM, and HDD playing distinct but interconnected roles. If you’d like a visual diagram or further details on any part (e.g., context switching, virtual memory), let me know!

Concurrency & Goroutine in GO

Huseyn — Wed, 18 Jun 2025 05:35:03 +0000

What Is Concurrency in Go?

Concurrency is when multiple tasks are in progress at the same time — not necessarily running simultaneously, but progressing independently. Go makes concurrency simple and efficient with goroutines and channels.

What Is a Goroutine?

A goroutine is a lightweight thread managed by the Go runtime. You start one with the go keyword:

go doSomething()

Characteristics:

Very lightweight (~2KB initial stack)
Managed by Go runtime, not the OS
Automatically grows/shrinks its stack
Scheduled by Go's internal M:N scheduler (many goroutines on few OS threads)

Real-World Scenario: Web Server Handling Requests

Code Example:

func handler(w http.ResponseWriter, r *http.Request) {
    go logRequest(r) // non-blocking logging
    data := process(r) // main request logic
    w.Write(data)
}

What Happens:

logRequest runs in a separate goroutine → doesn't block response
process(r) runs on main goroutine handling the request
Go's scheduler assigns both to available threads

Under the Hood: What Happens When a Goroutine Runs?

Go's Scheduler (G-M-P Model):

Go uses an internal M:N scheduler with three components:

G: Goroutine (your code)
M: Machine (OS thread)
P: Processor (context for scheduling goroutines)

Lifecycle:

You call go doSomething()
Go runtime:
- Creates a new G (goroutine struct)
- Adds it to a local run queue of a P
- A running M picks it up from P and runs it
If doSomething() blocks (e.g., on IO or time.Sleep), Go parks the G, and the M picks another G

How Goroutines Differ from Threads

Feature	Goroutine	OS Thread
Memory	~2KB stack	~1MB stack
Managed by	Go runtime	OS Kernel
Scheduling	M:N, cooperative	1:1, preemptive
Startup cost	Tiny	Large
Context switch	Fast (no syscall)	Slower (involves kernel)

What Makes Go Concurrency Powerful?

Built-in scheduler ensures fairness and efficiency
Goroutines auto-grow/shrink stack
Go has network poller (epoll/kqueue) to handle blocking IO without blocking the thread
Garbage collector knows about goroutines and stack frames

Important Notes

Blocking a goroutine ≠ blocking a thread: e.g., a goroutine waiting on a channel doesn't block an OS thread
Too many goroutines? Still okay — Go can handle millions if you avoid leaks and deep blocking
Use runtime.NumGoroutine() to inspect count
Use pprof or go tool trace to profile concurrent programs

Conclusion & Best Practices

Use goroutines for anything parallel (IO, background jobs, async tasks)
Don’t forget to sync: use channels, WaitGroups, or mutexes
Avoid goroutine leaks (e.g., waiting forever on a channel)
Monitor memory and goroutine counts in production

Git Commit Convention

Huseyn — Mon, 16 Jun 2025 04:53:53 +0000

Git Commit Convention Rules Every Developer Should Know

Many developers write commit messages like:

"fixed bug"
"updated code"
"changes made"

However, this kind of approach doesn’t work well in professional software development. Instead, you should follow a standardized format for writing meaningful and consistent commit messages.

Commit Message Format:

<type>(<scope>): <subject>

Common Commit Types:

feat: When you add a new feature
fix: When you fix a bug
docs: For documentation changes or updates
style: For formatting, white-space, missing semicolons, etc. (no code logic changes)
refactor: Code refactoring (no new features or bug fixes)
test: Adding or updating tests
chore: Other technical tasks (e.g., dependency updates, build scripts, configs)
perf: Performance improvements
ci: Changes to CI/CD configuration
revert: Reverting a previous commit
update: Updating packages or dependencies

Following this convention makes your commit history clean, understandable, and professional.

Make() in GO

Huseyn — Mon, 16 Jun 2025 04:26:37 +0000

What Does make Do in Go?
The make function is used to create and initialize only three types of built-in data structures in Go:

✅ Slices
✅ Maps
✅ Channels

These are reference types, and unlike arrays or structs, they need some internal setup to work properly. The make function does exactly that — it sets them up so you can use them safely.

Why Not Just Declare Normally?
Because if you just declare a slice/map/channel without using make, it may be nil — and using a nil map or channel will cause a runtime error.

Syntax

make(type, length[, capacity])

For slices, you can give both length and capacity.
For maps, you can give an optional initial capacity.
For channels, you specify the buffer size (for buffered channels).

🔍 Examples

🧺 Slice Example

s := make([]int, 3, 5)
fmt.Println(s)        // [0 0 0]
fmt.Println(len(s))   // 3
fmt.Println(cap(s))   // 5

len = how many items it holds now
cap = how many items it can hold before resizing

You can now safely use s and append more items.

🗺️ Map Example

m := make(map[string]int)
m["apple"] = 10
fmt.Println(m) // map[apple:10]

You must use make for maps before writing to them, or you’ll get a panic.

You can give an initial capacity: make(map[string]int, 100) — this allocates space for ~100 items.

📬 Channel Example

ch := make(chan int, 2)
ch <- 1
ch <- 2
fmt.Println(<-ch) // 1
fmt.Println(<-ch) // 2

This is a buffered channel of size 2.
make is always used to create channels.

❓ What's the Difference Between make and new?

Feature	`make`	`new`
Works with	Only slices, maps, channels	All types
Returns	Initialized value	Pointer to zero value
Ready to use?	✅ Yes	❌ No — you often must set it up manually

🧑‍🍳 Real-World Analogy
Imagine you're setting a table:

new just gives you an empty plate, but you still need to put food on it.
make gives you a fully served plate, ready to eat from.

🧪 In Practice: Shopping Cart

func main() {
    cart := make([]string, 0) // slice
    prices := make(map[string]float64) // map
    orderQueue := make(chan string, 2) // channel

    cart = append(cart, "Book", "Pen")
    prices["Book"] = 9.99

    orderQueue <- "Order #1"
    orderQueue <- "Order #2"

    fmt.Println("Cart:", cart)
    fmt.Println("Price of Book:", prices["Book"])
    fmt.Println("Processing:", <-orderQueue)
}

DEV Community: Huseyn

Online Learning Problem

Online Learning Problem (OLP)

What is Online Learning?

Standard Online Learning Framework

Performance Measure: Regret

Types of Online Learning Problems

1. Online Prediction with Expert Advice

2. Online Convex Optimization

3. Bandit Feedback

4. Contextual Online Learning

Bandit Models

What is a Bandit Model?

Exploration vs Exploitation Trade-off

Formal Bandit Setting

Types of Bandit Models

Stochastic Bandits

Contextual Bandits

Adversarial Bandits

Common Bandit Algorithms

Relationship Between Online Learning and Bandits

Applications

Key Takeaway

Understanding Domains, DNS, Servers, and Reverse Proxies

1. Domain Basics

2. DNS Records (Mapping Domain to Server)

3. How Subdomains Work on a Single Server

4. Role of Web Servers (Nginx / Apache)

Nginx or Apache does:

Project Files

5. Reverse Proxy Explained

6. Why Not Direct Backend Usage?

7. Example Setup with Nginx

Backend

Nginx Config

Flow

8. Summary Workflow

GitHub Actions Cheatsheet and Workflow Analysis

GitHub Actions Cheatsheet

1. Workflow

2. Events (on)

3. Jobs

4. Steps

5. Actions

6. Services

7. Runners

8. Environment Variables

9. Matrix Strategy

10. Artifacts and Outputs

11. Concurrency and Permissions

12. Common Actions

13. Expressions and Contexts

14. Best Practices

Analysis of the Provided Go Workflow YAML

YAML File

Component Breakdown

How the Workflow Works

Key Notes

Additional Tips for This Workflow

Database Concurrency Phenomena & ISOLATION Label: Read Phenomena and Serialization Anomaly

Introduction

Concurrency Phenomena

1. Dirty Reads

2. Non-Repeatable Reads

3. Phantom Reads

4. Serialization Anomaly

SQL Transaction Isolation Levels

1. READ UNCOMMITTED

2. READ COMMITTED

3. REPEATABLE READ

4. SERIALIZABLE

Isolation Levels and Concurrency Phenomena Table

Explanation

Summary

Database Transactions: A Comprehensive Guide with Go

What is a Database Transaction?

Key Characteristics - ACID

Ensuring Transactions

Database State in Transactions

What Happens if a Transaction Fails?

2. Events (`on`)

🚀 What is `sqlc`?

⚙️ `sqlc.yaml` Configuration

🔖 Query Types in `sqlc`

📄 Example Migration (`db/migrations/init.up.sql`)

🔖 `sqlc` Query Types Explained

✅ `:one`

✅ `:many`

✅ `:exec`

✅ `:execrows`

✅ `:copyfrom`