DEV Community: discernible-io

API keys leak. Certificates expire.

discernible-io — Thu, 16 Oct 2025 12:18:39 +0000

Live Event Today: Mutual Trust, No Secrets

RODiT and the Future of API Authentication

📅 When: Today, 3pm UTC

🎯 What: Live Q&A and deep dive into RODiT technology

Join us as we talk with Discernible IO about RODiT (Rich Online Digital Tokens), a new approach to API authentication with verifiable on-chain identities.

About RODiT

Built on @NEARProtocol by Discernible, RODiTs bring mutual blockchain-based trust to API authentication.

Watch Live

NEAR DevHub

https://www.youtube.com/@NEARDevHub

Bring your questions and join the conversation!

Creating Debian Packages: A Quick Guide

discernible-io — Thu, 02 Oct 2025 09:37:54 +0000

Creating your own Debian packages is simpler than you might think. Here's a streamlined approach to get you started quickly.

Basic Setup

First, you'll need the essential tools:

sudo apt install build-essential devscripts debhelper

Package Structure

Create your package directory structure. At minimum, you need:

your-package/
├── debian/
│   ├── control
│   ├── changelog
│   ├── rules
│   └── compat
└── [your source files]

The debian/control file describes your package, changelog tracks versions, rules defines the build process, and compat specifies the debhelper compatibility level.

Building the Package

Once your debian directory is properly configured, building is straightforward:

dpkg-buildpackage -us -uc

The -us -uc flags skip signing the source and changes files, which is perfect for local development and testing. This command will create your .deb file in the parent directory.

Testing

Install your package locally to test:

sudo dpkg -i ../your-package_version_architecture.deb

That's the essence of it. While there are many nuances to packaging, this approach using dpkg-buildpackage -us -uc provides a reliable foundation for creating functional Debian packages.

For a more comprehensive guide with detailed examples, check out this excellent tutorial on creating deb packages.

Internet Trust Chains

discernible-io — Thu, 28 Aug 2025 13:35:01 +0000

Internet trust is built on quite shaky foundations. Moving packets around to the correct computer that controls each IP address involves often no less than 3 large complex pyramids of trust: BGP, DNS and PKI. Any crack in any of these leads to the others crumbling.

Let's start with BGP, it works like this:

The DNS pyramid has the following components:

And the PKI pyramid has these:

We can compare these pyramids as follows:

You can easily see that BGP, the foundation, has enormous gaps. By exploiting BGP it is possible to compromise both DNS and PKI, despite all the current mitigations in place.

In particular, across all levels, there are significant gaps in
Administrative Controls, with fraudulent participants obtaining approval, Real-time Validation and Revocation, which barely works at the PKI level as it can't scale, Transparency Mechanisms, as both IP and domain asset owners can't control who makes security assertions about their assets, Man in the middle attacks due to poor Authentication mechanisms,

All these gaps result in poor security. For example the support for strong mutual authentication is poor, both in server-client and peer to peer applications. Deployments are complex, expensive and brittle.

In the next article we will explain how RODiT technology can help overcoming many of these challenges.

The Chain of Trust in X.500 Digital Certificates: Power, Control, and Real-World Failures

discernible-io — Mon, 07 Jul 2025 10:45:36 +0000

Digital certificates form the backbone of modern internet security, enabling everything from secure web browsing to email encryption. At the heart of this system lies the concept of a "chain of trust" based on X.500 digital certificates. However, while this system appears robust on paper, real-world incidents reveal significant vulnerabilities and power imbalances that affect global internet security.

Understanding X.500 Digital Certificates

X.500 digital certificates are based on the ITU-T X.509 standard, which defines the format for public key certificates. These certificates bind a public key to an identity (such as a person, organization, or device) and are digitally signed by a Certificate Authority to verify their authenticity.

Each X.509 certificate contains:

Subject information: The entity being certified
Public key: The cryptographic key for the certified entity
Issuer information: The CA that signed the certificate
Digital signature: The CA's cryptographic signature
Validity period: When the certificate is valid
Extensions: Additional information like key usage and certificate policies

The Chain of Trust Explained

The chain of trust is a hierarchical model where trust flows from a root Certificate Authority down through intermediate CAs to end-entity certificates. This creates a tree-like structure of trust relationships.

Root Certificate Authorities

At the top of the hierarchy sit Root CAs. These are self-signed certificates that serve as the ultimate trust anchors. Root CAs are pre-installed in operating systems, browsers, and other software applications. When you see a padlock icon in your browser, it's because the certificate chain ultimately traces back to a trusted root CA.

Intermediate Certificate Authorities

Between root CAs and end-entity certificates are Intermediate CAs (also called subordinate CAs). Root CAs typically don't issue certificates directly to end users. Instead, they issue certificates to intermediate CAs, which then issue certificates to other intermediates or directly to end entities.

Chain Validation Process

When a certificate is presented, the validating software:

Checks the end-entity certificate's signature against the issuing intermediate CA
Validates each intermediate CA certificate up the chain
Verifies that the chain terminates at a trusted root CA
Confirms that all certificates in the chain are valid and not revoked

The CA/Browser Forum: Who Holds the Power?

The CA/Browser Forum is the industry consortium that sets standards for Certificate Authorities and web browsers. Understanding its membership structure reveals who truly controls internet certificate policy.

Membership Structure and Power Distribution

Certificate Authority Members: The most influential CAs include:

DigiCert: One of the largest commercial CAs globally
Sectigo (formerly Comodo): Major commercial CA with significant market share
GlobalSign: European-based CA with worldwide operations
Entrust: Enterprise-focused CA with government contracts
Let's Encrypt (ISRG): Non-profit CA that has revolutionized certificate accessibility
GoDaddy: Web hosting company with substantial CA operations

Browser Members: The entities that ultimately enforce certificate policies:

Google (Chrome): Controls the largest browser market share globally
Mozilla (Firefox): Maintains independent root store policies
Microsoft (Edge/IE): Integrates with Windows certificate stores
Apple (Safari): Controls certificate policy for iOS and macOS ecosystems

Power Dynamics and Decision Making

The Forum operates on a consensus model, but practical power is unevenly distributed:

Browser Dominance: Browsers hold ultimate veto power because they control what certificates users actually trust. Google's Chrome team, in particular, has driven major policy changes including:

Mandatory Certificate Transparency logging
Shorter certificate lifespans
Stricter validation requirements

Commercial CA Influence: Large commercial CAs influence standards through:

Technical expertise and implementation experience
Financial resources for standards development
Market relationships with enterprise customers

Voting Structure: The Forum uses a two-thirds majority voting system, but browsers can effectively override decisions by changing their root store policies unilaterally.

Recent Power Struggles

Several incidents illustrate the power dynamics:

Symantec Distrust (2017): Google unilaterally decided to distrust Symantec certificates, forcing the CA to sell its business to DigiCert. This demonstrated browsers' ultimate authority over the certificate ecosystem.

Certificate Transparency Mandate: Browsers, led by Google, mandated CT logging despite resistance from some CAs concerned about operational complexity.

Domain Control Mechanisms: DNS-Based Certificate Authority Authorization (CAA)

Domain holders have limited mechanisms to control which CAs can issue certificates for their domains, that unfortunately are not used much.

DNS CAA Records

Certificate Authority Authorization (CAA) records allow domain owners to specify which CAs are authorized to issue certificates for their domains.

CAA Record Format:

example.com. CAA 0 issue "letsencrypt.org"
example.com. CAA 0 issue "digicert.com"
example.com. CAA 0 iodef "mailto:security@example.com"

CAA Properties:

issue: Authorizes certificate issuance
issuewild: Authorizes wildcard certificate issuance
iodef: Specifies contact for policy violations

DNS-Based Authentication of Named Entities (DANE)

DANE uses DNS to specify which certificates or CAs are valid for a domain:

_443._tcp.example.com. TLSA 3 1 1 [certificate hash]

Certificate Transparency Monitoring

Domain owners can monitor CT logs to detect unauthorized certificate issuance:

Facebook Certificate Transparency Monitoring: Automated detection of certificates issued for Facebook domains
Google Certificate Transparency API: Allows programmatic monitoring of certificate issuance

Real-World Failures: The MyEtherWallet BGP Attack

One of the most significant examples of how certificate vulnerabilities can be exploited for cryptocurrency theft occurred in April 2018 with the MyEtherWallet attack.

The Attack Sequence

Step 1: BGP Hijacking
Attackers compromised Amazon's Route 53 DNS service through BGP hijacking, redirecting DNS queries for MyEtherWallet.com to attacker-controlled servers.

Step 2: Certificate Acquisition
The attackers obtained a valid Let's Encrypt certificate for MyEtherWallet.com by:

Controlling the DNS responses during Let's Encrypt's domain validation
Using the DNS-01 challenge method to prove domain ownership
Receiving a legitimate certificate that browsers trusted

Step 3: Phishing Execution
With a valid certificate, the attackers:

Presented a convincing replica of MyEtherWallet
Displayed the trusted padlock icon in browsers
Harvested private keys from users attempting to access their wallets

Step 4: Cryptocurrency Theft
The attackers used the harvested private keys to steal approximately $152,000 in cryptocurrency.

Technical Analysis

This attack succeeded because:

BGP lacks authentication: Attackers could redirect traffic without cryptographic verification
DNS validation vulnerability: Let's Encrypt's domain validation relied on DNS, which the attackers controlled
User trust in certificates: Users trusted the valid certificate without additional verification
Lack of HSTS preloading: MyEtherWallet wasn't in the HSTS preload list, allowing the initial redirect

Lessons Learned

The MyEtherWallet attack highlighted several critical issues:

Certificate validation isn't sufficient: Valid certificates don't guarantee legitimate websites
DNS security is crucial: BGP and DNS vulnerabilities can undermine certificate security
Need for additional protections: Certificate Transparency, CAA records, and HSTS provide additional layers of security

The MCIP Initiative: Modernizing Certificate Infrastructure

The Multi-Perspective Certificate Issuance and Validation (MCIP) initiative represents the latest effort to address fundamental weaknesses in certificate validation.

Current Validation Problems

Traditional certificate validation suffers from:

Single point of validation: CAs typically validate domain control from a single network perspective
BGP vulnerabilities: Attackers can redirect validation traffic as in the MyEtherWallet case
DNS poisoning: Localized DNS attacks can fool validation processes

MCIP Solution Architecture

Multiple Validation Perspectives: Instead of validating from a single location, CAs must validate domain control from multiple, geographically distributed vantage points.

Validation Requirements:

Minimum of 3 validation perspectives
Perspectives must be in different network locations
Majority consensus required for certificate issuance

Implementation Approaches:

Distributed validation infrastructure: CAs deploy validation servers globally
Third-party validation services: Independent services provide validation perspectives
Cooperative validation: CAs share validation infrastructure

Technical Implementation

DNS Validation Enhancement:

# Traditional validation (vulnerable)
dig @8.8.8.8 _acme-challenge.example.com TXT

# MCIP validation (resilient)
dig @validator1.ca.com _acme-challenge.example.com TXT
dig @validator2.ca.com _acme-challenge.example.com TXT  
dig @validator3.ca.com _acme-challenge.example.com TXT

HTTP Validation Enhancement:
Multiple perspectives attempt to retrieve validation tokens from different network locations, making BGP hijacking attacks much more difficult.

Industry Adoption

Let's Encrypt Implementation: Let's Encrypt has begun implementing multi-perspective validation, using multiple validation points for domain verification.

CA/Browser Forum Requirements: The Forum is developing requirements for multi-perspective validation as part of the Baseline Requirements.

Browser Support: Major browsers are encouraging CA adoption of MCIP through root store policies.

The Reality of PKI Failures

Despite the theoretical benefits of PKI, real-world implementation reveals significant problems that affect users daily.

Certificate Expiration Incidents

The Spotify outage mentioned in recent reports exemplifies a pervasive problem: certificate expiration causing service disruptions.

Common Expiration Scenarios:

Forgotten renewals: Organizations fail to track certificate expiration dates
Complex renewal processes: Multi-step validation requirements delay renewals
Coordination failures: Multiple teams must coordinate for certificate updates
Testing gaps: Renewed certificates aren't properly tested before deployment

Impact Scale: Certificate expiration affects:

Major websites and services
Internal corporate systems
API endpoints and microservices
Mobile applications and IoT devices

Historical CA Failures

DigiNotar (2011): Complete compromise of a Dutch CA led to:

Fraudulent certificates for major websites (Google, Facebook, Yahoo)
Complete removal from all browser trust stores
Bankruptcy of the CA company
Demonstrated that WebTrust certification doesn't guarantee security

Symantec Issues (2017): Improper certificate issuance practices resulted in:

Google's decision to distrust Symantec certificates
Forced sale of Symantec's CA business to DigiCert
Massive certificate replacement efforts across the internet

Comodo Attacks (2011): Attackers obtained fraudulent certificates for:

Gmail, Yahoo, Hotmail
Skype, Mozilla, WordPress
Demonstrated vulnerability of domain validation processes

The Trust Paradox

WebTrust Certification Limitations: As noted in the DigiNotar case, WebTrust certification doesn't guarantee security. This creates a false sense of security where:

Certified CAs can still be compromised
Audit processes may miss critical vulnerabilities
Users have no way to assess actual CA security levels

Validation Inconsistencies: Different CAs use different validation methods:

Domain Validation (DV): Minimal validation, only proves domain control
Organization Validation (OV): Verifies organization identity
Extended Validation (EV): Rigorous identity verification

Users cannot easily determine validation levels, creating confusion about certificate trustworthiness.

Mechanisms for Domain Control

DNS CAA Records in Practice

Implementation Example:

example.com. CAA 0 issue "letsencrypt.org"
example.com. CAA 0 issue "digicert.com"
example.com. CAA 0 issuewild ";"
example.com. CAA 0 iodef "mailto:security@example.com"

Limitations:

Adoption rates: Many domains don't implement CAA records
DNS security: CAA records are only as secure as DNS itself
CA compliance: Not all CAs properly check CAA records

Certificate Transparency as a Control Mechanism

Monitoring Implementation:

import requests
import json

def monitor_certificates(domain):
    ct_api = "https://crt.sh/?q={}&output=json"
    response = requests.get(ct_api.format(domain))
    certificates = json.loads(response.text)

    for cert in certificates:
        print(f"Certificate ID: {cert['id']}")
        print(f"Issuer: {cert['issuer_name']}")
        print(f"Not Before: {cert['not_before']}")
        print(f"Not After: {cert['not_after']}")

Real-World Monitoring: Organizations like Facebook and Google operate sophisticated CT monitoring systems that:

Detect unauthorized certificate issuance within minutes
Automatically alert security teams
Trigger incident response procedures

The Warranty Problem: Who Pays When PKI Fails?

One of the most significant issues with current PKI is the lack of meaningful warranties or liability.

Certificate Authority Liability Limitations

Typical CA Terms:

Liability limited to certificate cost (often $0 for DV certificates)
No warranties beyond technical compliance
Exclusions for consequential damages
Dispute resolution through arbitration

Real-World Impact: When certificate failures cause:

Service outages costing millions in revenue
Security breaches exposing customer data
Cryptocurrency theft (as in MyEtherWallet)
Reputation damage

Affected parties have little recourse against CAs.

The Race to the Bottom

As Ian Grigg noted, the lack of meaningful liability creates a "race to the bottom" where:

CAs compete on price rather than security
Validation processes are minimized to reduce costs
Security investments are seen as unnecessary expenses
Market forces don't reward better security practices

Usability Challenges

User Understanding

Certificate Validation Complexity: Users face challenges in:

Understanding certificate validation levels
Recognizing legitimate vs. fraudulent certificates
Knowing when to be concerned about certificate warnings
Distinguishing between different types of certificate errors

Professional Challenges: Even security professionals struggle with:

Certificate chain validation
Proper certificate deployment
Understanding CA policy differences
Implementing certificate monitoring

Operational Difficulties

Certificate Management: Organizations face:

Complex renewal processes
Coordination across multiple teams
Testing and deployment challenges
Inventory management for thousands of certificates

Automation Limitations: While ACME has improved automation:

Many CAs don't support automated processes
Complex validation requirements prevent automation
Legacy systems can't integrate with modern certificate management
Organizational processes haven't adapted to shorter certificate lifespans

The Future of Certificate Trust

Emerging Solutions

Certificate Transparency Evolution: CT is expanding beyond just logging to:

Real-time monitoring and alerting
Automated response to suspicious certificates
Integration with threat intelligence platforms
Policy enforcement based on CT data

DANE and DNS Security: Improvements in DNS security through:

DNSSEC adoption
DNS over HTTPS (DoH) and DNS over TLS (DoT)
Authenticated DNS responses
Integration with certificate validation

Blockchain and Distributed Trust: Experimental approaches include:

Blockchain-based certificate authorities
Distributed consensus for certificate validation
Cryptocurrency-based incentive models for security
Decentralized identity systems

Regulatory Developments

European Union eIDAS Regulation: New EU regulations may:

Mandate specific CA security requirements
Create liability frameworks for certificate failures
Establish government oversight of CAs
Require specific validation procedures

Industry Standards Evolution: The CA/Browser Forum continues developing:

Shorter certificate lifespans (potentially 90 days)
Stricter validation requirements
Enhanced monitoring and reporting requirements
Improved incident response procedures

Conclusion: The Paradox of PKI

The Public Key Infrastructure represents both the foundation of internet security and one of its greatest vulnerabilities. While PKI enables secure communications at massive scale, it also creates systemic risks and single points of failure that affect global internet security.

The Fundamental Tensions

Security vs. Usability: Stronger security measures often make certificates more difficult to obtain and manage, potentially driving users toward less secure alternatives.

Centralization vs. Decentralization: While centralized CAs provide scalability and consistency, they also create systemic risks and power imbalances.

Market Forces vs. Security: The competitive certificate market often rewards lower prices over better security, creating perverse incentives.

Key Takeaways

Browser Power: Web browsers, particularly Chrome, hold ultimate power over certificate policy, often overriding CA/Browser Forum consensus.

AI Coding Pitfalls and Solutions: A Practical Guide

discernible-io — Mon, 27 Jan 2025 13:18:01 +0000

Introduction

Most developers, if not all, are using AI by now. This is helping both newbies and highly experienced professionals.

Unfortunately, like any other tool, AI coding can be less than helpful. Do you recall stories about people who end up in a lake following a GPS map application blindly? Something similar happens frequently to developers while using AI.

Common Issues

1. Default Pessimism

The AI not only misses the context you don't share with it, it will not ask for more context, assuming you are telling it everything that needs to be told. For example, if you don't say you have access to the production server, it will try to solve the issue assuming there isn't access.

2. Know-it-all Syndrome

The AI will give you a long list of all the things you can check and all the things you can try, like it had the magical bullet for your issue. Good luck working your way through.

3. Just Plaster It!

The AI knows nothing of root causes. Presented with an issue it will try to solve that one issue ONLY, without checking why the issue is there in the first place. It tries to solve symptoms, not problems.

4. Goldberg Machine Patch

The AI tries to give you a solution that will solve this issue and any other possible ever present in any system ever, and check and log everything under the sun. The solution seems to be bulletproof at first sight, but it is so complex it creates issues on its own that may be harder to fix than the original issue.

5. Dude Comparisons

The AI will compare what you need to compare, but will miss essential components, for example overall number of items.

Solutions and Examples

It is not hard to avoid these issues if you remember that you are a driver, not a passenger, in the coding process. Here are the solutions you can use to avoid each issue:

1. Default Pessimism - Solution

Share most of the context, and instruct it to request more context whenever necessary. Provide working similar samples if available.

Example:

Bad Prompt:

How can I check if users are active in my system?

AI's Limited Response:

def check_user_activity(user_id):
    # Assumes only local database access
    with open('user_logs.txt', 'r') as f:
        logs = f.readlines()
    return any(str(user_id) in log for log in logs)

Better Prompt:

I need to check if users are active in my system. We have:
- PostgreSQL database with user_sessions table
- Access to Redis cache
- Cloudwatch logs
- User activity tracked in NewRelic
Please ask if you need more context about any of these systems.

AI's Comprehensive Response:

def check_user_activity(user_id):
    # Check Redis cache first for performance
    activity = redis_client.get(f"user_activity:{user_id}")
    if activity:
        return json.loads(activity)

    # Check database for recent sessions
    with db.connection() as conn:
        result = conn.execute("""
            SELECT last_active_at 
            FROM user_sessions 
            WHERE user_id = %s
            ORDER BY last_active_at DESC 
            LIMIT 1
        """, (user_id,))
        return result.fetchone()

2. Know-it-all Syndrome - Solution

Request the AI to take only one step at a time, checking the result of each step before proceeding with the next.

Example:

Bad Prompt:

My API endpoints are slow. How can I improve performance?

Better Prompt:

My /users endpoint is taking 2.5s to respond. Let's start by profiling the database queries.
Here's the current code:

def get_users():
    users = User.query.all()
    return jsonify([user.to_dict() for user in users])

What's the first step to identify the bottleneck?

3. Just Plaster It! - Solution

Ask the AI to explain how that solution is going to solve the problem.

Example:

Bad Code Issue:

# Original problem
try:
    process_payment(amount)
except Exception:
    return "Payment failed"

Bad Prompt:

How do I fix this payment error?

Better Prompt:

We're seeing payment failures in production. Here's the code:

try:
    process_payment(amount)
except Exception:
    return "Payment failed"

Before implementing a fix, can you help me understand:
1. What types of exceptions might occur here?
2. How would each exception impact our payment flow?
3. What logging would help us identify the root cause?

4. Goldberg Machine Patch - Solution

Ask the AI if this is a standard, best practice way to solve the issue, and if there is a simpler solution that will solve it.

Example:

Problem:

def save_user(user_data):
    db.users.insert(user_data)

Bad Prompt:

How can I make this user save function more robust?

AI's Overcomplicated Response:

def save_user(user_data: Dict[str, Any]) -> Union[str, None]:
    """
    Ultra-robust user saving with extensive validation and logging
    """
    transaction_id = str(uuid.uuid4())
    logger.info(f"Starting user save operation {transaction_id}")

    try:
        # Pre-save validation
        validation_result = validate_user_schema(user_data)
        if not validation_result.is_valid:
            logger.error(f"Schema validation failed: {validation_result.errors}")
            metrics.increment("user.save.validation_error")
            raise ValidationError(validation_result.errors)

        # Backup current state
        existing_user = db.users.find_one({"email": user_data["email"]})
        if existing_user:
            backup_collection.insert_one({
                "type": "user_backup",
                "data": existing_user,
                "timestamp": datetime.utcnow()
            })

        # Complex retry logic
        for attempt in exponential_backoff(max_attempts=5):
            with attempt:
                result = db.users.insert_one(user_data)

        # Post-save validation
        saved_user = db.users.find_one({"_id": result.inserted_id})
        assert saved_user == user_data, "Save verification failed"

        # Extensive logging
        logger.info(f"User save successful {transaction_id}")
        metrics.increment("user.save.success")

        return str(result.inserted_id)

    except Exception as e:
        logger.error(f"User save failed {transaction_id}: {str(e)}")
        metrics.increment("user.save.error")
        notification.alert_team("User save failed", context=locals())
        raise

Better Prompt:

I need to add basic error handling and validation to this user save function.
What's the minimal change that would catch common issues?

AI's Simpler Response:

def save_user(user_data: dict) -> bool:
    """
    Save user with basic validation and error handling
    """
    if not all(k in user_data for k in ['email', 'name']):
        logger.error(f"Missing required fields: {user_data}")
        return False

    try:
        db.users.insert_one(user_data)
        return True
    except DuplicateKeyError:
        logger.error(f"User already exists: {user_data['email']}")
        return False

5. Dude Comparisons - Solution

Ask the AI to perform a bijective comparison of items. This guarantees each and every item is compared with a peer only once.

Example:

Bad Prompt:

Compare these two API response formats

Better Prompt:

Can you do a field-by-field comparison of these two API response formats,
ensuring every field in both formats is addressed:

Format A:
{
    "user_id": "string",
    "name": "string",
    "roles": ["string"]
}

Format B:
{
    "id": "string",
    "full_name": "string",
    "permissions": ["string"]
}

Conclusion

By following these guidelines and remembering that you are the driver in the development process, you can effectively leverage AI as a powerful tool while avoiding its common pitfalls. The key is to be specific, provide context, and guide the AI step-by-step towards the solution you need.

HashiCorp Vault Setup Guide for NEAR Protocol Accounts

discernible-io — Tue, 14 Jan 2025 15:41:20 +0000

HashiCorp Vault Setup Guide for NEAR Protocol Accounts

This guide walks you through setting up a HashiCorp Vault server to securely store NEAR Protocol accounts. Before starting, ensure you have:

A server with Ubuntu/Debian
Domain name configured
SSL certificates ready
Root or sudo access

Initial Setup and Installation

1. Install Vault

First, add the HashiCorp repository and install Vault:

# Add HashiCorp GPG key
wget -O- https://apt.releases.hashicorp.com/gpg | gpg --dearmor | \
    sudo tee /usr/share/keyrings/hashicorp-archive-keyring.gpg

# Add HashiCorp repository
echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] \
    https://apt.releases.hashicorp.com $(lsb_release -cs) main" | \
    sudo tee /etc/apt/sources.list.d/hashicorp.list

# Install Vault
sudo apt update && sudo apt install vault

2. Configure Vault Server

Create the Vault configuration file:

sudo tee /etc/vault.d/vault.hcl << 'EOF'
ui = true
disable_mlock = true

storage "file" {
  path = "/opt/vault/data"
}

listener "tcp" {
  address     = "0.0.0.0:8200"
  tls_disable = false
  tls_cert_file = "/etc/vault.d/vault.crt"
  tls_key_file  = "/etc/vault.d/vault.key"
}

api_addr = "https://your-vault-domain:8200"
cluster_addr = "https://your-vault-domain:8201"

telemetry {
  disable_hostname = true
  prometheus_retention_time = "24h"
}
EOF

3. SSL/TLS Configuration

Place your SSL certificates in /etc/vault.d/:

vault.crt: Your SSL certificate
vault.key: Your private key

Note: If you need to generate certificates, follow our guide on generating Let's Encrypt certificates. Ensure your DNS is properly configured and your server is set up correctly.

4. Set File Permissions

Configure proper ownership and permissions:

# Set ownership
sudo chown vault:vault /etc/vault.d/vault.hcl
sudo chown vault:vault /etc/vault.d/vault.key
sudo chown vault:vault /etc/vault.d/vault.crt

# Set permissions
sudo chmod 640 /etc/vault.d/vault.hcl
sudo chmod 640 /etc/vault.d/vault.key
sudo chmod 640 /etc/vault.d/vault.crt

5. Create Systemd Service

Set up Vault as a system service:

sudo tee /etc/systemd/system/vault.service << 'EOF'
[Unit]
Description="HashiCorp Vault - A tool for managing secrets"
Documentation=https://www.vaultproject.io/docs/
Requires=network-online.target
After=network-online.target
ConditionFileNotEmpty=/etc/vault.d/vault.hcl

[Service]
User=vault
Group=vault
ProtectSystem=full
ProtectHome=read-only
PrivateTmp=yes
PrivateDevices=yes
SecureBits=keep-caps
AmbientCapabilities=CAP_IPC_LOCK
Capabilities=CAP_IPC_LOCK+ep
CapabilityBoundingSet=CAP_IPC_LOCK
NoNewPrivileges=yes
ExecStart=/usr/bin/vault server -config=/etc/vault.d/vault.hcl
ExecReload=/bin/kill --signal HUP $MAINPID
KillMode=process
KillSignal=SIGINT
Restart=on-failure
RestartSec=5
TimeoutStopSec=30
LimitNOFILE=65536
LimitMEMLOCK=infinity

[Install]
WantedBy=multi-user.target
EOF

Vault Initialization and Configuration

6. Initialize and Unseal

Start the Vault service and perform initial setup:

# Start Vault service
sudo systemctl daemon-reload
sudo systemctl enable vault
sudo systemctl start vault

# Configure Vault address
export VAULT_ADDR='https://your-vault-domain:8200'

# Initialize Vault
vault operator init

# Unseal Vault (requires 3 of 5 keys)
vault operator unseal  # First key
vault operator unseal  # Second key
vault operator unseal  # Third key

# Verify status
vault status

7. Configure Access Policies

Set up the following policies for different access levels:

Admin Policy

sudo tee signing-admin-policy.hcl << 'EOF'
path "sys/auth/*" {
  capabilities = ["create", "update", "delete", "sudo"]
}

path "sys/auth" {
  capabilities = ["read"]
}

path "auth/approle/*" {
  capabilities = ["create", "read", "update", "delete", "list"]
}
EOF

Operator Policy

sudo tee signing-operator-policy.hcl << 'EOF'
path "secret/data/signing-keys/*" {
  capabilities = ["create", "read", "update", "delete", "list"]
  allowed_parameters = {
    "data" = []
    "options" = []
  }
}
path "secret/metadata/signing-keys/*" {
  capabilities = ["read", "list"]
}
path "secret/metadata/signing-keys" {
  capabilities = ["read", "list"]
}
path "secret/data/signing-keys" {
  capabilities = ["create", "read", "update", "list"]
}
EOF

General Signing Policy

sudo tee signing-policy.hcl << 'EOF'
# Allow managing auth methods
path "sys/auth/*" {
  capabilities = ["create", "update", "delete", "sudo"]
}

# Allow listing auth methods
path "sys/auth" {
  capabilities = ["read"]
}

# Allow managing roles
path "auth/*" {
  capabilities = ["create", "read", "update", "delete", "list"]
}

# Existing podman-keys permissions
path "secret/data/signing-keys/*" {
  capabilities = ["create", "read", "update", "delete"]
}

# Allow listing secrets
path "secret/metadata/*" {
  capabilities = ["list"]
}

# Allow managing AppRole auth configuration
path "auth/approle/*" {
  capabilities = ["create", "read", "update", "delete", "list"]
}
EOF

Reader Policy

sudo tee signing-reader-policy.hcl << 'EOF'
path "secret/data/signing-keys/*" {
  capabilities = ["read"]
}
path "secret/metadata/signing-keys/*" {
  capabilities = ["read", "list"]
}
EOF

Apply all policies:

vault policy write signing-admin-policy signing-admin-policy.hcl
vault policy write signing-operator-policy signing-operator-policy.hcl
vault policy write signing-policy signing-policy.hcl
vault policy write signing-reader-policy signing-reader-policy.hcl

8. Enable Key-Value Store

Enable the KV secrets engine:

vault secrets enable -path=secret kv-v2

9. Configure AppRole Authentication

Set up authentication for automated access:

# Enable AppRole
vault auth enable approle

# Create role
vault write auth/approle/role/NEAR-MANAGER-ROLE \
    token_policies="near-operator-policy" \
    token_ttl=0 \
    token_max_ttl=0 \
    token_type="service" \
    period="768h"

Retrieve role credentials:

# Get Role ID
vault read -format=json auth/approle/role/NEAR-MANAGER-ROLE/role-id | jq -r '.data.role_id'

# Get Secret ID
vault write -f -format=json auth/approle/role/NEAR-MANAGER-ROLE/secret-id | jq -r '.data.secret_id'

10. Store NEAR Protocol Accounts

On each server that needs to access the Vault:

# Set Vault address
export VAULT_ADDR='VAULT_SERVER_URL'

# Configure credentials
ROLE_ID='your-role-id'
SECRET_ID='your-secret-id'

# Login
vault write auth/approle/login \
    role_id=$ROLE_ID \
    secret_id=$SECRET_ID

# Store NEAR account
vault kv put -mount=secret near-accounts/my-account \
    account_json=@/path/to/near-credentials/mainnet/account.json

Security Considerations

Unsealing Process

The Vault uses a threshold unsealing process:

Requires 3 of 5 keys by default
Vault starts in a sealed state
Cannot decrypt storage until unsealed
Multiple operators must provide keys
Never store unseal keys on the Vault server
Unsealing required after maintenance/restarts

Closing the PKIX Working Group is, apparently, not news

discernible-io — Tue, 14 Jan 2025 12:23:32 +0000

PKI is perfect, isn't it?

The PKIX Working Group was of the digital certificates standards, among them X.509, that enable various applications such as secure email, web security (SSL/TLS), and digital signatures. It officially concluded its operations... more than ten years ago (31 October 2013) .. Coincidentally, more or less at the same time Bitcoin was created.

The closure can only mean that PKI is perfect (many beg to disagree), or that the companies that sponsor work in PKI have zero interest in evolving the technology as their interests are well served with the current one.

PKI, despite years of development and refinement still has challenges of around certificate management, trust hierarchies, and the susceptibility of certificate authorities to breaches and mismanagement.

In order to manage some of the flaws in PKI, the main players created a system called Certificate Transparency (CT), a framework and set of standards aimed at tackling the vulnerabilities in the lifecycle of digital certificates used in public key infrastructures (PKI). CT provides a layer of security by enabling the detection of misissued or malicious certificates through publicly auditable logs. The core components of Certificate Transparency are:

Public Logs: These are append-only logs where certificates are recorded. Each log entry is cryptographically secured to prevent tampering. Logs are operated by various organizations and are designed to be publicly accessible and verifiable.
Monitors: Entities that observe and verify the entries in CT logs. Monitors can detect suspicious certificates and alert domain owners if an unauthorized certificate is issued for their domain.
Auditors: These are entities that check the consistency and integrity of CT logs. Auditors ensure that logs are functioning correctly and have not been tampered with.
Merkle Trees: A data structure used by CT logs to provide a cryptographic guarantee of the log's integrity. Merkle trees allow efficient and secure verification of individual log entries.

CT has several flaws, among them:

The reliance on a limited number of public logs. If a log operator is compromised or goes offline, it could disrupt the CT ecosystem.
Implementing and maintaining CT requires additional effort and resources from certificate authorities, website operators, and auditors. This complexity can be a barrier, particularly for smaller organizations with limited technical capabilities.
As the number of certificates grows, the size and management of CT logs can become challenging. The logs must handle large volumes of data efficiently while maintaining performance and integrity.

CT alone does not solve all issues related to PKI and digital certificates. While it adds a layer of transparency and detection, it does not prevent misissuance or replace the need for robust security practices.

So we are using a blockchain to mitigate the risks that arise form a system that was designed in the late 80s early 90s, with the requirement of being able to operate off-line.

Why not evolve the technology and use self-issued blockchain based digitally signed tokens, and cut the middleman?

Generate your Let's Encrypt Digital Certificates for all your domains using Apache

discernible-io — Mon, 13 Jan 2025 12:59:04 +0000

As someone who's dealt with their fair share of digital certificate headaches, I'm sharing a comprehensive guide to setting up and managing SSL/TLS certificates with Apache. You may actually use a different web server, but Apache seems to be the easiest way to achieve this. Once you are done, you just have to copy the generated certificates to the right directories in your web server of choice.

Initial DNS Setup

First, ensure all your domains point to your server:

# Create A records for each domain
domain1.com -> server_ip
www.domain1.com -> server_ip
domain2.com -> server_ip
www.domain2.com -> server_ip
domain3.com -> server_ip
www.domain3.com -> server_ip

Directory Structure

Create an organized directory structure that scales with multiple domains:

# Create base directories
sudo mkdir -p /var/www/domains
# Create individual domain directories
for domain in domain1.com domain2.com domain3.com; do
    sudo mkdir -p /var/www/domains/$domain/public_html
    sudo mkdir -p /var/www/domains/$domain/logs
    sudo chown -R $USER:$USER /var/www/domains/$domain
    sudo chmod -R 755 /var/www/domains/$domain
done

Virtual Host Configuration

Create separate virtual host files for each domain:

# Create configuration files
for domain in domain1.com domain2.com domain3.com; do
    sudo touch /etc/apache2/sites-available/$domain.conf
done

Template for each domain's virtual host (example for domain1.com):

<VirtualHost *:80>
    ServerAdmin webmaster@domain1.com
    ServerName domain1.com
    ServerAlias www.domain1.com
    DocumentRoot /var/www/domains/domain1.com/public_html
    ErrorLog /var/www/domains/domain1.com/logs/error.log
    CustomLog /var/www/domains/domain1.com/logs/access.log combined

    <Directory /var/www/domains/domain1.com/public_html>
        Options Indexes FollowSymLinks
        AllowOverride All
        Require all granted
    </Directory>
</VirtualHost>

Batch Certificate Management

Here's how to efficiently manage certificates for multiple domains:

# Install Certbot
sudo apt install certbot python3-certbot-apache

# Create a domains list file
echo "domain1.com www.domain1.com" > domains.txt
echo "domain2.com www.domain2.com" >> domains.txt
echo "domain3.com www.domain3.com" >> domains.txt

# Obtain certificates for all domains in one command
sudo certbot --apache $(cat domains.txt | tr '\n' ' ')

Automated Renewal Management

Create a renewal management script:

#!/bin/bash
# /usr/local/bin/cert-renew-manager.sh

# Renew all certificates
certbot renew

# Check renewal status for each domain
for domain in domain1.com domain2.com domain3.com; do
    cert_path="/etc/letsencrypt/live/$domain/fullchain.pem"
    if [ -f "$cert_path" ]; then
        expiry_date=$(openssl x509 -enddate -noout -in "$cert_path" | cut -d= -f2)
        echo "Domain: $domain - Certificate expires: $expiry_date"
    else
        echo "Warning: No certificate found for $domain"
    fi
done

Batch Security Configuration

Apply security headers to all domains:

# Create a common security configuration
sudo tee /etc/apache2/conf-available/security-headers.conf << EOF
Header always set Strict-Transport-Security "max-age=31536000; includeSubDomains"
Header always set X-Frame-Options "SAMEORIGIN"
Header always set X-Content-Type-Options "nosniff"
Header always set Referrer-Policy "strict-origin-when-cross-origin"
EOF

# Enable for all sites
sudo a2enconf security-headers

Monitoring and Maintenance

Create a monitoring script for multiple domains:

#!/bin/bash
# /usr/local/bin/cert-monitor.sh

domains=("domain1.com" "domain2.com" "domain3.com")
notification_email="admin@yourdomain.com"

for domain in "${domains[@]}"; do
    expiry=$(openssl s_client -connect ${domain}:443 -servername ${domain} </dev/null 2>/dev/null | openssl x509 -noout -enddate | cut -d= -f2)
    expiry_epoch=$(date -d "$expiry" +%s)
    current_epoch=$(date +%s)
    days_left=$(( ($expiry_epoch - $current_epoch) / 86400 ))

    if [ $days_left -lt 30 ]; then
        echo "Warning: Certificate for $domain expires in $days_left days" | mail -s "Certificate Expiry Warning" $notification_email
    fi
done

Batch Testing

Create a comprehensive test script:

#!/bin/bash
# /usr/local/bin/cert-test.sh

# Test Apache configuration
sudo apache2ctl configtest

# Test SSL configuration for each domain
for domain in domain1.com domain2.com domain3.com; do
    echo "Testing SSL configuration for $domain"
    curl -sI https://$domain | head -n 1
    openssl s_client -connect ${domain}:443 -servername ${domain} </dev/null 2>/dev/null | openssl x509 -noout -dates
done

Recovery Procedures

Create a backup script for all certificates:

#!/bin/bash
# /usr/local/bin/cert-backup.sh

backup_dir="/root/cert-backups/$(date +%Y%m%d)"
mkdir -p $backup_dir

# Backup all certificates and configurations
sudo cp -r /etc/letsencrypt $backup_dir/
sudo cp -r /etc/apache2/sites-available $backup_dir/

# Archive the backup
tar -czf $backup_dir.tar.gz $backup_dir
rm -rf $backup_dir

Conclusion

Managing multiple domains requires more automation and organization, but with these scripts and structures in place, you can efficiently manage dozens of domains without increasing administrative overhead. Remember to:

Keep domain lists updated
Run regular batch tests
Maintain comprehensive backups
Monitor all domains systematically
Document any domain-specific configurations

This approach scales well whether you're managing a handful of domains or hundreds, while keeping your certificate management process clean and maintainable.

IP Whitelisting, the silent killer

discernible-io — Mon, 13 Jan 2025 09:36:08 +0000

All systems in the same network can communicate with each other freely, from the lowest layers of sending frames and packets, to RPC calls, etc. This implies that we have the same trust in all systems, for example, they are all managed by the same IT team. Sometimes you have systems in the same network that mostly don't trust each other, for example, system interfaces that are Internet facing. So you create rules using firewalls, proxies, VPNs, API calls authenticated with API keys, etc, to enable trusted communication among the systems and applications that your either manage, or you decide to use. IP whitelisting, which should be better called IP allow listing, is very widely used. This is bad news, for several reasons:

It is an inflexible system. When something changes you have to synchronize the changes on both sides of any pair. This makes moving any system to a business continuity mode very unreliable, complicated and slow.
It is obscure. Only administrators with access to privileged operations understand what has been configured, why an how.
It is often poorly documented or undocumented. When there is any kind of incident, it makes troubleshooting complicated.

Probably the two main unnecessary reasons companies face expensive incidents that take several days to fix are expired digital certificates and brittle architectures resulting from using IP whitelising. There are many solutions depending on what you are trying to achieve with IP whitelisting:

An ethernet cable for systems that have free interfaces and are close to each other.
A point to point VPN for systems that you manage but need to cross untrusted networks to communicate.
API keys for authentication of services outside your network.
Among others…

The silver lining is: If you change jobs every couple of year, it will be someone else who will have to fix the mess. Yupee!

You May Prefer to Know Less About PKI Flaws but Now Is Too Late

discernible-io — Mon, 13 Jan 2025 09:16:45 +0000

While you thought it was here to help all along, things like this happen all the time: A Spotify publisher was down Monday night. The culprit? A lapsed security certificate

Public Key Infrastructure is pervasive and some would say it brings great benefits and makes the internet better. You can't disagree that being able to encrypt our connections with the websites and services we use has security and privacy benefits, but while the concept of PKI sounds good on paper, it seems to be a solution that has many issues, and has created some issues of its own.

One: PKIs Behaving Badly

You are supposed to trust PKIs. But can you really trust them? What if a PKI authority fails to verify if someone owns a domain (CertStar 2008), creates intentionally certificates for domains without the knowledge of the owner (ANSSI 2013), or it is plainly hacked and abused (Diginotar 2011). Well, apparently there is a certification for Certificate Authorities that should guarantee that they take appropriate security measures, WebTrust CA. All Certificate Authorities that are certified should be secure, right?

Diginotar, as far as you can check, was WebTrust CA certified! This means that any Certificate Authority being certified does not mean being secure. This either means that WebTrust certificate is useless or our trust in a Certificate Authority is disconnected to it being certified or not.

Two: Who can create a CA?

Anyone! The only gatekeeper for the creation of Certificate Authorities is the pre-installed root list that comes with the most popular operating systems and browsers. If you get included in the list, the user does not need to install your root certificate, which is complicated for many users. Misbehaving CAs are so problematic that a patch was necessary, Certificate Transparency.

Recent developments in EU Law will not make this problem neither better, nor worse.

Three: Who owns the digital certificate, Jekyll or Hyde?

The methods used to validate who owns an ID are different between different Certificate Authorities, an ID can be a URL, an email or any other identifier. So the level of assurance of a certificate issued by different CAs is not only different, but the final non specialist user has no way to determine what level it is.

Four: Warranty aka Responsibility

So anyone with enough money and influence can create a CA, secure it to unknown levels of security, and finally implement inconsistent methods of validation of IDs. So what degree of trust does a user get from this setup? None. No user has ever received compensation for any damage caused by a unreliable or bogus certificate. Certificate Authorities do not provide any warranty on the certificates they issue beyond being compliant with technical standards. As Ian Grigg notes, this creates a race to the bottom in certificate quality between CAs.

Five: Lack of Usability

Users should be able to check if a website is authentic, and current. But how do users, including professionals, check if they are connecting to the right website? Doing a search in a search engine, not checking the digital certificate.

Can you improve your security rotating your key more often? No, it is hardcoded by CA and industry practices. Can you rotate your key less often as your environment is low security and does not need it? Again, no you can't, and many companies suffer incidents related to certificate renewal.

Conclusion

PKI does not deliver hardly any of the benefits it is supposed to. But there is no current alternative so we are stuck with it. Why are most issues identified in PKI Problems Draft RFC version 0 gone in version 5? Are the issues solved, or perhaps there is an interest to keep them muted?

Sometimes one wonders if PKI was a clever way to prevent public key cryptography from being widely deployed to final users…. ever notice how client server side certificates are almost never used due to the many implementation hurdles?

Final note: The PKIX workgroup that publishes Digital Certificates standards has been closed for 10 years now. Are they perfect now?

Sources:

Creating a new Developer Server in Digital Ocean

discernible-io — Fri, 10 Jan 2025 15:05:11 +0000

This guide will walk you through the process step by step, from creating your droplet to adding some essential development tools for Rust and Javascript.

Note: Picture by Christopher Gower's

Prerequisites

A DigitalOcean account
Basic command line knowledge
SSH client installed on your local machine

1. Creating Your Droplet

Initial Setup

Log into your DigitalOcean account and click "Create Droplet"
Choose your region
- Note: All resources in this datacenter will be part of the same VPC network
- They can communicate securely over their Private IP addresses

Configuration

Select Ubuntu (latest version)
Choose NVMe SSD for storage
Select the basic plan:
- 2GB RAM
- 25GB Disk
- 1TB transfer
- Approximately $12/month

SSH Key Setup

Open your terminal and navigate to SSH directory:

   cd ~/.ssh

Generate a new SSH key:

   ssh-keygen

When prompted, save the key as id_NEWSERVERNAME (replace NEWSERVERNAME with your server name)

Skip the passphrase by pressing Enter twice
Verify the key creation:

   ls -l ~/.ssh

Display your public key:

   cat ~/.ssh/id_NEWSERVERNAME.pub

Copy the output and paste it into the "SSH KEY CONTENT" field on DigitalOcean
Fill in the hostname field with your server name
Click "Create Droplet"

2. Initial Server Configuration

First Login and Updates

Log in as root using your SSH key
Update the system:

   sudo apt-get update && sudo apt-get dist-upgrade -y

User Account Setup

Create a new user:

   adduser USERNAMEOFYOURCHOICE

Grant administrative privileges:

   usermod -aG admin USERNAMEOFYOURCHOICE
   usermod -aG sudo USERNAMEOFYOURCHOICE

Swap File Configuration

Check current swap:

   sudo swapon --show

Create and configure swap file:

   sudo fallocate -l 1G /swapfile
   sudo chmod 600 /swapfile
   sudo mkswap /swapfile
   sudo swapon /swapfile

Make swap permanent by adding to fstab:

   echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab

Essential Tools Installation

sudo apt install mc btop ncdu

System Reboot

sudo shutdown -r now

3. Development Environment Setup

Bash Customization

force_color_prompt=yes
PS1='${debian_chroot:+($debian_chroot)}\[\033[0;36m\]\u@\h\[\033[00m\]:\[\033[0;34m\]\w\[\033[00m\]\$ '

Available colors:

Black: \[\033[0;30m\]
Red: \[\033[0;31m\]
Green: \[\033[0;32m\]
Yellow: \[\033[0;33m\]
Blue: \[\033[0;34m\]
Magenta: \[\033[0;35m\]
Cyan: \[\033[0;36m\]
White: \[\033[0;37m\]

Development Tools Installation

Install essential build tools:

   sudo apt install build-essential

Configure Git:

   git config --global user.email "YOUR.EMAIL@EXAMPLE.COM"
   git config --global user.name "YOUR_GITHUB_USERNAME"

Install Rust:

   curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
   source "$HOME/.cargo/env"
   rustc --version

Install additional development dependencies:

   sudo apt install pkg-config xdg-utils httpie libssl-dev net-tools openssh-server \
   debhelper-compat clang-tools libudev-dev

4. Security Configuration

Firewall Setup

Configure UFW:

   sudo ufw allow 22/tcp
   sudo ufw allow 443/tcp
   sudo ufw enable

SSH Configuration

Set up SSH access:
- On your local machine:
```
 cat ~/.ssh/id_NEWSERVERNAME.pub
```

Copy the output and add it to ~/.ssh/authorized_keys on the server

Conclusion

Your development server is now ready for use! You have a secure, well-configured environment with essential development tools installed. The server includes:

A non-root user with sudo privileges
Configured swap space
Essential development tools and utilities
Basic security settings with UFW
Rust programming environment
Various development dependencies

Remember to regularly update your system and monitor resource usage with the installed tools (btop, ncdu).

If you use Visual Code, for example you can now configure your IDE following these instructions: (https://code.visualstudio.com/docs/remote/ssh?WT.mc_id=netbc-meetup-antchu)