DEV Community: Harun R. Rayhan

How I Review Terraform and Lambda PRs with AI Before They Merge

Harun R. Rayhan — Wed, 14 Oct 2026 14:57:46 +0000

If my earlier posts were about how I write infrastructure and how I build AI-assisted workflows, this one is about the gate right before merge.

That distinction matters. Writing code with AI is useful. Reviewing production changes with AI is where the real value shows up for me. The problem is not getting a first draft fast. The problem is making sure the change is still safe when it touches IAM, Terraform state, Lambda runtime settings, or anything that can quietly cost money or break production.

So I started using AI as a first-pass reviewer for Terraform and Lambda pull requests. Not as the decision maker. Not as the person who approves the merge. Just as the reviewer that helps me catch the obvious and semi-obvious issues before I spend my own time on the full manual pass.

That keeps the review fast, but it also keeps the final responsibility where it belongs, with me.

This post is the workflow I trust today.

Short version: AI reviews the diff, I review the risk.

Why This Is a Different Post From My Other AI and AWS Articles

I already write about AI in AWS codebases, Terraform workflows, and agentic DevOps routines. This post is not another version of that same story.

The earlier posts focus on one of two things:

how I use AI to help build infrastructure
how I structure the workflow around the codebase

This one is narrower. It is about reviewing a production change before it merges.

That narrower scope makes it more useful in practice, because the questions are more specific:

Did this IAM policy get broader?
Did someone accidentally make a Lambda slower, hotter, or more expensive?
Did Terraform replace something that should not be replaced?
Did we open a networking path we did not mean to open?
Did the PR forget tests, docs, or rollback notes?

That is the kind of review AI is actually good at helping with.

What I Feed the Reviewer

I do not ask the model to guess from thin air. I give it the actual change surface.

Typically that means some mix of:

the pull request diff
the Terraform plan output
the files touched by the change
the environment or service context
any recent deployment or incident notes that matter

The quality of the review depends on the input. If I give it a sloppy prompt, I get a sloppy review. If I give it the real plan and the real diff, it can usually spot the things I want it to spot.

One caution: redact before you paste. Plan output can include values that are not marked sensitive, so I strip anything I would not put in a public issue.

I want the reviewer to answer a simple question: what would make this change risky to merge?

The Checks I Care About Most

For Terraform and Lambda work, I care about a small set of risk categories more than anything else.

1. IAM and permissions

This is the first thing I look at because permission creep is easy to miss in a diff. A policy can go from narrow to broad with one extra wildcard or one extra resource pattern.

I want AI to flag:

new wildcards in actions or resources
roles that gained access to more services than before
policies that look temporary but are actually permanent
anything that could grant access across environments

2. Lambda runtime and cost impact

Lambda changes can look tiny and still affect cost or reliability.

I want the reviewer to notice things like:

memory jumps that are probably unnecessary
timeouts that are too short or too generous
concurrency changes that might cause throttle or burst issues
extra cold-start risk from dependencies or packaging changes

3. Terraform replacement risk

This is where production changes can become unpleasant. A harmless-looking refactor can trigger a destroy-and-recreate path if someone changes the wrong attribute.

I want the model to call out:

resources marked for replacement
stateful resources with a destructive path
name changes that can force recreation
security groups, targets, or buckets that could disappear briefly

4. Networking and exposure

Anything that changes public access, private access, VPC wiring, or routing deserves extra attention.

I want the reviewer to be loud about:

public exposure that was not there before
new ingress or egress paths
load balancer or API Gateway routing changes
subnet or security group edits with hidden blast radius

5. Missing operational detail

Even if the code is correct, the review is not complete if the PR forgot the operational side.

So I want AI to ask about:

rollback plan
testing evidence
migration order
monitoring and alert impact
what happens if this fails halfway through

What I Still Review Manually

AI helps me move faster, but some things should stay human-led.

My manual pass focuses on:

the actual blast radius
whether the change matches product intent
whether rollback is realistic
what stateful resources are involved
whether the diff makes sense in the larger release plan

I also sanity-check anything that looks too confident. If the model says a change is safe, that is not enough. I want the evidence behind the claim.

The best outcome is not “AI approved it.” The best outcome is “AI surfaced the right questions before I reviewed it.”

My Review Flow

The flow is pretty simple.

Collect the PR diff and Terraform plan.
Ask the AI reviewer to summarize the change in plain English.
Ask for the top risks, sorted by severity.
Ask for anything that looks inconsistent or missing.
Do my own manual pass on the flagged areas.
Approve, request changes, or split the PR if the risk is too broad.

That sounds basic, but it works because it keeps the review structured.

Here is the kind of prompt I use conceptually:

You are reviewing a production AWS change.

Focus on:
- IAM and permissions
- Terraform replacement risk
- Lambda runtime and cost impact
- networking and exposure
- missing rollback or operational detail

Return:
1. plain-English summary of the change
2. top risks ranked by severity
3. specific lines or resources that look suspicious
4. questions I should answer before merge
5. a merge recommendation: safe, caution, or block

I do not need poetry from the reviewer. I need a compact risk report I can act on.

A Good Review Finds the Questions, Not Just the Bugs

Sometimes the most useful thing the AI does is not catch a mistake outright. It points out that something deserves a question.

Examples:

Why did this Lambda need double the memory?
Why is this IAM role suddenly allowed to touch more than one environment?
Why does this Terraform change replace the resource instead of updating it in place?
Why is this public endpoint now reachable from a wider range of traffic?

That kind of review is valuable because it slows me down in the right place. It makes the risky bit visible before I merge.

Why This Complements My Other AI Workflow Posts

If I compare this to my other posts, the shape is different.

Earlier posts are about:

using AI to build faster
breaking work into specialists
keeping agents safe in production codebases

This one is about the final gate before the change goes live.

That is why I think it is a good companion piece instead of a duplicate. It belongs in the same family, but it answers a different question.

One post is about the tools that help me work. This post is about the review layer that keeps me honest.

What I Want the Reviewer To Avoid

There are a few failure modes I do not want from an AI reviewer.

Do not rubber-stamp the diff.
Do not hide uncertainty behind confident language.
Do not recommend merge without explaining why.
Do not ignore infrastructure changes just because the application code looks small.
Do not treat production Terraform like toy code.

If the model cannot explain the risk, it is not helping much.

Final Thought

AI review works best when it makes the important things easier to see, not when it replaces judgment.

That is the real pattern I keep coming back to in AWS work. I want speed, but I want it inside a review process that still respects the cost of being wrong.

So yes, I use AI to review Terraform and Lambda PRs. But the point is not to automate approval. The point is to catch the risky stuff earlier, ask better questions, and make the human review sharper.

That is the version I trust in production.

Originally published at harun.dev. The full interactive version lives there. Follow me on X or read my bio.

Building a Production RAG Pipeline on AWS Lambda and pgvector

Harun R. Rayhan — Tue, 13 Oct 2026 13:36:41 +0000

I was knee-deep in building my SaaS when users started asking questions I couldn't answer from the UI alone. My app had a solid backend, Lambda functions, Fastify, PostgreSQL on RDS. I wrote about that whole architecture in my previous post. But users wanted to search through documentation, find answers buried in help articles, and ask natural-language questions about the product. The UI just wasn't built for that.

So I started looking into RAG. Every tutorial I found showed the same setup: LangChain running locally, Pinecone for vectors, and a Python script tying it all together. Great for demos. Not great for a production SaaS already running on AWS.

Here's the thing. I didn't want to add another service to my stack. I was already paying for PostgreSQL on RDS. I was already managing Lambda functions. Adding Pinecone or Weaviate meant another bill, another SDK, another thing to monitor at 2 AM. Then I found pgvector, and everything clicked.

pgvector is a PostgreSQL extension. It adds vector storage and similarity search right inside the database I was already running. No new infrastructure. No new vendor. Just an extension. I paired it with Lambda for compute, Bedrock for the LLM, and RDS Proxy for connection pooling. The whole pipeline costs me about $54/month at 10K queries per day.

Let me walk you through every piece of how I built it.

Photo by Tara Winstead on Pexels

What Is RAG and Why Does It Work

RAG stands for Retrieval-Augmented Generation. The name sounds academic, but the idea is simple. And once I understood it, I realized it was exactly what my users were asking for without knowing the term.

You have your own data. Documents, help articles, product specs, whatever. You want an LLM to answer questions about that data. But the LLM was trained on the internet, not your private stuff. So you give it context at query time.

The flow looks like this:

User asks a question. "How do I reset my API key?"
Retrieval step. You search your own documents for the most relevant chunks. This is semantic search, not keyword matching. You're comparing the meaning of the question against the meaning of each chunk.
Generation step. You stuff those relevant chunks into a prompt and send it to the LLM. "Based on the following context, answer this question."
LLM responds with an answer grounded in your actual data.

That's it. No fine-tuning. No retraining. No waiting weeks for a model to learn your docs.

And that's exactly why RAG beats fine-tuning for most use cases. Fine-tuning is slow, expensive, and your model becomes stale the moment your data changes. With RAG, you just update the documents in your database. The next query picks up the new content automatically. Your data is always live.

The magic is in the retrieval step. If you retrieve the right chunks, the LLM gives great answers. If you retrieve garbage, the LLM confidently presents garbage. So the vector search layer matters a lot. That's where pgvector comes in. And for my SaaS, it meant I could keep everything inside the PostgreSQL instance I was already running.

Photo by cottonbro studio on Pexels

Setting Up pgvector in PostgreSQL (RDS)

When I first looked at Pinecone, I nearly added it. The docs are slick and the onboarding is smooth. Then I checked the pricing page and realized I was already paying for PostgreSQL. Why add another database when I could just enable an extension?

If you're already running PostgreSQL on RDS, adding vector search is surprisingly easy. You don't need Pinecone. You don't need Weaviate. You just enable an extension.

pgvector adds a vector column type to PostgreSQL. You store embeddings as vectors, then query them with distance operators. Cosine similarity, L2 distance, inner product. All the standard stuff.

Enable the Extension

First, make sure your RDS parameter group allows pgvector. Here's the CloudFormation snippet:

RDSParameterGroup:
  Type: AWS::RDS::DBParameterGroup
  Properties:
    Family: postgres16
    Description: "PostgreSQL 16 with pgvector"
    Parameters:
      shared_preload_libraries: "pg_stat_statements,pgvector"

Then run the SQL to create the extension:

CREATE EXTENSION IF NOT EXISTS vector;

That's it. PostgreSQL now understands vectors.

Create the Embeddings Table

Here's the schema I use. The vector(1536) column size matches OpenAI's text-embedding-3-small model. If you're using Bedrock Titan, switch to vector(1024).

CREATE TABLE hrr_document_chunks (
  id            BIGSERIAL PRIMARY KEY,
  document_id   TEXT NOT NULL,
  chunk_index   INTEGER NOT NULL,
  content       TEXT NOT NULL,
  metadata      JSONB DEFAULT '{}',
  embedding     vector(1536) NOT NULL,
  created_at    TIMESTAMPTZ DEFAULT NOW(),
  updated_at    TIMESTAMPTZ DEFAULT NOW(),

  UNIQUE (document_id, chunk_index)
);

The metadata column is a JSONB field where I store things like the source URL, document title, section heading, and any other context that helps with filtering later. You'll want this when your dataset grows and you need to scope searches to specific document types.

Create an HNSW Index

Without an index, pgvector does a sequential scan. That's fine for a few thousand rows. But once you hit tens of thousands of embeddings, you need an index.

I recommend HNSW over IVFFlat. HNSW is slightly slower to build but gives better recall and works much better for Lambda workloads. More on that in the lessons learned section.

CREATE INDEX idx_chunks_embedding_hnsw
  ON hrr_document_chunks
  USING hnsw (embedding vector_cosine_ops)
  WITH (m = 16, ef_construction = 200);

The m and ef_construction parameters control the index quality. Higher values give better recall but slower inserts. These defaults work well for datasets up to about 500K embeddings. I landed on these numbers after a week of testing different configurations against my own dataset.

Photo by Rahul Pandit on Pexels

Generating and Storing Embeddings

Now you need to get your data into that table. This is the ingestion pipeline: take your documents, break them into chunks, generate embeddings for each chunk, and store everything in pgvector.

For my SaaS, the documents were help articles, product guides, and FAQ pages that users kept asking about. I run ingestion as a separate Lambda function. It gets triggered whenever new documents are added, either through an S3 event or a direct API call.

Chunking

Chunking strategy matters more than most people think. I split documents into chunks of about 500 tokens with a 50-token overlap between consecutive chunks. The overlap prevents information from getting lost at chunk boundaries.

function chunkDocument(text, maxTokens = 500, overlap = 50) {
  const sentences = text.split(/(?<=[.!?])\s+/);
  const chunks = [];
  let currentChunk = [];
  let currentLength = 0;

  for (const sentence of sentences) {
    const sentenceTokens = estimateTokens(sentence);

    if (currentLength + sentenceTokens > maxTokens && currentChunk.length > 0) {
      chunks.push(currentChunk.join(" "));

      // Keep last few sentences for overlap
      const overlapSentences = [];
      let overlapLength = 0;
      for (let i = currentChunk.length - 1; i >= 0; i--) {
        overlapLength += estimateTokens(currentChunk[i]);
        if (overlapLength > overlap) break;
        overlapSentences.unshift(currentChunk[i]);
      }
      currentChunk = overlapSentences;
      currentLength = overlapLength;
    }

    currentChunk.push(sentence);
    currentLength += sentenceTokens;
  }

  if (currentChunk.length > 0) {
    chunks.push(currentChunk.join(" "));
  }

  return chunks;
}

function estimateTokens(text) {
  return Math.ceil(text.split(/\s+/).length * 1.3);
}

Generating Embeddings with Bedrock Titan

Here's how to call Bedrock's Titan Embeddings model. This uses the AWS SDK v3:

import { BedrockRuntimeClient, InvokeModelCommand } from "@aws-sdk/client-bedrock-runtime";

const bedrockClient = new BedrockRuntimeClient({ region: "us-east-1" });

async function generateEmbedding(text) {
  const response = await bedrockClient.send(
    new InvokeModelCommand({
      modelId: "amazon.titan-embed-text-v2:0",
      contentType: "application/json",
      accept: "application/json",
      body: JSON.stringify({
        inputText: text,
        dimensions: 1024,
        normalize: true,
      }),
    })
  );

  const result = JSON.parse(new TextDecoder().decode(response.body));
  return result.embedding;
}

If you prefer OpenAI embeddings instead, the call is straightforward:

async function generateEmbeddingOpenAI(text) {
  const response = await fetch("https://api.openai.com/v1/embeddings", {
    method: "POST",
    headers: {
      "Authorization": `Bearer ${process.env.OPENAI_API_KEY}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({
      model: "text-embedding-3-small",
      input: text,
    }),
  });

  const data = await response.json();
  return data.data[0].embedding;
}

Upserting Chunks

Now bring it all together. Chunk the document, embed each chunk, and upsert into pgvector:

async function ingestDocument(pool, documentId, text, metadata = {}) {
  const chunks = chunkDocument(text);

  for (let i = 0; i < chunks.length; i++) {
    const embedding = await generateEmbedding(chunks[i]);

    await pool.query(
      `INSERT INTO hrr_document_chunks (document_id, chunk_index, content, metadata, embedding)
       VALUES ($1, $2, $3, $4, $5::vector)
       ON CONFLICT (document_id, chunk_index)
       DO UPDATE SET content = $3, metadata = $4, embedding = $5::vector, updated_at = NOW()`,
      [documentId, i, chunks[i], JSON.stringify(metadata), JSON.stringify(embedding)]
    );
  }

  // Clean up old chunks if document got shorter
  await pool.query(
    `DELETE FROM hrr_document_chunks WHERE document_id = $1 AND chunk_index >= $2`,
    [documentId, chunks.length]
  );

  return chunks.length;
}

The ON CONFLICT clause handles re-ingestion cleanly. If a document gets updated, you just run ingest again and it overwrites the old chunks. The cleanup query at the end removes stale chunks if the document got shorter after an edit.

One thing to watch out for: Bedrock Titan has a rate limit. I hit this on day one when I tried to ingest my entire knowledge base at once. If you're ingesting thousands of documents, add a small delay between embedding calls or use batch processing. I throttle to about 10 requests per second and that's been reliable.

Photo by Media Dung on Pexels

Retrieval: Semantic Search on Lambda

This is the part that made it all feel real. A user sends a question, and you need to find the most relevant chunks from your database. Fast.

The first time I ran a semantic search against my own docs and got back exactly the right paragraph, I knew this was going to work. The retrieval Lambda takes a query string, embeds it using the same model you used for ingestion, then runs a cosine similarity search against pgvector. The whole thing takes about 100-200ms on a warm Lambda invocation.

async function retrieveChunks(pool, query, topK = 5) {
  // Embed the user's query with the same model used for ingestion
  const queryEmbedding = await generateEmbedding(query);

  // Cosine similarity search using pgvector's <=> operator
  const result = await pool.query(
    `SELECT
       id,
       document_id,
       content,
       metadata,
       1 - (embedding <=> $1::vector) AS similarity
     FROM hrr_document_chunks
     WHERE 1 - (embedding <=> $1::vector) > 0.3
     ORDER BY embedding <=> $1::vector
     LIMIT $2`,
    [JSON.stringify(queryEmbedding), topK]
  );

  return result.rows;
}

A few things to notice here.

The <=> operator is pgvector's cosine distance. Lower distance means higher similarity. I compute 1 - distance to get a similarity score between 0 and 1. The WHERE clause filters out anything below 0.3 similarity, which cuts out the noise. You don't want the LLM seeing irrelevant chunks.

The topK parameter controls how many chunks you return. I default to 5. Going higher gives the LLM more context but costs more tokens. Going lower risks missing relevant information. Five is a good starting point for most use cases.

Connection Pooling with RDS Proxy

If you read my previous post, you know this pattern. Lambda functions can spin up hundreds of concurrent instances. Each one opens a database connection. Without connection pooling, you'll exhaust PostgreSQL's connection limit in seconds.

RDS Proxy handles this. Your Lambda points to the proxy endpoint, and the proxy multiplexes connections to the actual database. The code is identical to a normal pg.Pool setup:

import pg from "pg";

const pool = new pg.Pool({
  host: process.env.DB_HOST,     // RDS Proxy endpoint
  port: 5432,
  database: process.env.DB_NAME,
  user: process.env.DB_USER,
  password: process.env.DB_PASSWORD,
  max: 5,
  idleTimeoutMillis: 30000,
  connectionTimeoutMillis: 5000,
  ssl: { rejectUnauthorized: true },
});

Set max: 5 on the Lambda side. The proxy handles the actual pooling across all Lambda instances. Keeping the per-function pool small prevents any single Lambda from hogging connections.

Photo by Nataliya Vaitkevich on Pexels

Generation: Calling the LLM with Context

You've retrieved the relevant chunks. Now you need to turn them into an answer. This is where the LLM comes in.

I went back and forth on this part. The idea is simple: build a prompt that includes the retrieved chunks as context, along with the user's question. Then call the LLM and return whatever it generates. But getting the prompt right took more iterations than I expected.

The Prompt Template

Here's the prompt template I use. It's deliberately strict about staying grounded in the context:

function buildPrompt(question, chunks) {
  const context = chunks
    .map((chunk, i) => `[Source ${i + 1}] ${chunk.content}`)
    .join("\n\n");

  return {
    system: `You are a helpful assistant that answers questions based on the provided context.
Rules:
- Only use information from the provided context to answer.
- If the context does not contain enough information, say so honestly.
- Keep answers concise and direct.
- Reference which source(s) you used when possible.`,
    user: `Context:
${context}

Question: ${question}

Answer based only on the context above.`,
  };
}

That system prompt is important. I learned this after my first demo went sideways. Without explicit instructions to stay grounded, the LLM will happily make things up using its training data. One of my test users asked about a feature we didn't have, and the LLM invented a whole tutorial for it. The "say so honestly" part is key. I'd rather get an "I don't have enough information" response than a confident hallucination.

Calling Bedrock Claude

Here's the Bedrock call using Claude Haiku. I use Haiku instead of Sonnet for RAG because it's 10x cheaper and the quality difference is minimal when you're giving it good context:

import { BedrockRuntimeClient, InvokeModelCommand } from "@aws-sdk/client-bedrock-runtime";

const bedrockClient = new BedrockRuntimeClient({ region: "us-east-1" });

async function generateAnswer(question, chunks) {
  const prompt = buildPrompt(question, chunks);

  const response = await bedrockClient.send(
    new InvokeModelCommand({
      modelId: "anthropic.claude-3-haiku-20240307-v1:0",
      contentType: "application/json",
      accept: "application/json",
      body: JSON.stringify({
        anthropic_version: "bedrock-2023-05-31",
        max_tokens: 1024,
        system: prompt.system,
        messages: [{ role: "user", content: prompt.user }],
      }),
    })
  );

  const result = JSON.parse(new TextDecoder().decode(response.body));
  return result.content[0].text;
}

Handling Token Limits

Here's a practical concern. Each chunk might be 500 tokens. If you retrieve 5 chunks, that's 2,500 tokens of context. Add the system prompt, the question, and the response, and you're looking at maybe 4,000 tokens total. That fits easily within Haiku's context window.

But what if your chunks are longer, or you need more of them? I cap the total context at 3,000 tokens and truncate the lowest-similarity chunks first:

function truncateContext(chunks, maxTokens = 3000) {
  let totalTokens = 0;
  const selected = [];

  for (const chunk of chunks) {
    const tokens = estimateTokens(chunk.content);
    if (totalTokens + tokens > maxTokens) break;
    selected.push(chunk);
    totalTokens += tokens;
  }

  return selected;
}

Since the chunks are already sorted by similarity (highest first), this naturally keeps the most relevant ones and drops the least relevant. Simple and effective.

Photo by Matheus Bertelli on Pexels

Wiring It Together: The API Endpoint

Now let's connect everything. This is where my existing SaaS architecture paid off. The API endpoint receives a question, runs retrieval, generates an answer, and returns it. I use Fastify on Lambda (same setup as my previous post), and adding a new route was straightforward. The pattern works with any framework.

async function ragHandler(request, reply) {
  const { question } = request.body;

  if (!question || question.trim().length === 0) {
    return reply.status(400).send({ error: "Question is required" });
  }

  // Step 1: Retrieve relevant chunks
  const chunks = await retrieveChunks(request.server.pg, question, 5);

  if (chunks.length === 0) {
    return reply.send({
      answer: "I couldn't find any relevant information to answer your question.",
      sources: [],
    });
  }

  // Step 2: Truncate context to fit token limits
  const trimmedChunks = truncateContext(chunks);

  // Step 3: Generate answer
  const answer = await generateAnswer(question, trimmedChunks);

  // Step 4: Return answer with source metadata
  return reply.send({
    answer,
    sources: trimmedChunks.map((c) => ({
      documentId: c.document_id,
      similarity: parseFloat(c.similarity.toFixed(3)),
      preview: c.content.substring(0, 150) + "...",
    })),
  });
}

// Register the route
app.post("/rag/ask", {
  schema: {
    body: {
      type: "object",
      required: ["question"],
      properties: {
        question: { type: "string", minLength: 1, maxLength: 1000 },
      },
    },
  },
  preHandler: [app.authenticate],
  handler: ragHandler,
});

The handler is intentionally thin. Retrieval and generation are separate modules. This makes testing easy. You can unit test the chunking, the retrieval query, and the prompt template independently. The handler just orchestrates the flow.

I also return the source metadata alongside the answer. This is useful for two reasons. First, your frontend can show users where the answer came from, which builds trust. Second, it helps you debug retrieval quality. If the answer is wrong, check the sources. Nine times out of ten, the problem is that retrieval returned the wrong chunks, not that the LLM misunderstood them.

The preHandler uses the same authentication middleware from my existing API. This was one of the perks of building RAG into my existing stack instead of bolting on a separate service. Same auth, same rate limiting, same logging. RAG endpoints should always be authenticated. You don't want someone querying your private documents without authorization.

Photo by Brett Sayles on Pexels

Cost Breakdown

Let's talk real numbers. This was a big deal for me because I didn't want RAG to double my AWS bill. I'm going to break down costs for a small AI SaaS handling about 10,000 RAG queries per day.

The Infrastructure

Component	Monthly Cost	Notes
RDS db.t3.micro (PostgreSQL + pgvector)	~$15	2 vCPU, 1GB RAM, 20GB storage
RDS Proxy	~$20	Connection pooling for Lambda
Lambda (retrieval + generation)	~$5	512MB, avg 300ms per invocation
API Gateway (HTTP API)	~$3	$1 per million requests
Bedrock Titan Embeddings	~$3	~$0.10 per 1M tokens, ~30M tokens/month
Bedrock Claude Haiku (generation)	~$8	~$0.25/1M input, ~$1.25/1M output tokens
Total	~$54/month

That's $54/month for a production RAG pipeline handling 10K queries per day. Not bad.

Compared to Hosted Vector Databases

Now let's compare to what you'd pay with a dedicated vector database:

Pinecone Starter: ~$70/month for 1M vectors. And you still need the LLM costs on top of that.
Weaviate Cloud: ~$25/month for their starter tier. Plus compute for the actual search.
Qdrant Cloud: ~$30/month for a small instance.

With pgvector, the vector storage cost is zero because it's part of your existing PostgreSQL instance. You're already paying for RDS. The extension is free. This is exactly why I went this route. I just added vector search to a database I was already running and paying for.

The math is clear: if you already have PostgreSQL on RDS, pgvector is the cheapest vector database alternative out there. You avoid a whole separate service, a separate bill, and separate operational overhead.

Where the Money Actually Goes

The biggest cost in a RAG pipeline is not storage or compute. It's the LLM generation calls. At 10K queries per day with Claude Haiku, you're spending about $8/month on generation alone. Switch to Claude Sonnet and that jumps to ~$80/month. Switch to GPT-4o and you're looking at ~$150/month.

So choose your generation model carefully. For most RAG use cases, Haiku or GPT-4o-mini gives you 90% of the quality at 10% of the cost. Save the big models for cases where answer quality justifies the spend.

Photo by Leeloo The First on Pexels

Lessons Learned

After running this RAG pipeline in production for a few months, here are the things I learned the hard way. Every one of these cost me hours of debugging or unnecessary spending.

1. Chunking Strategy Matters More Than the Embedding Model

I wasted about two weeks on this one. I was obsessing over Titan vs. OpenAI embeddings, running benchmarks, comparing similarity scores. The difference in retrieval quality between them was maybe 5%. Then I changed my chunking from fixed 1000-token chunks to 500-token chunks with overlap, and retrieval quality jumped by 30%. Two weeks of model comparison, undone by a single config change.

The reason is obvious in hindsight. Smaller chunks are more precise. When a user asks a specific question, a 500-token chunk about that exact topic scores much higher than a 1000-token chunk where the relevant part is buried in the middle. The overlap makes sure you don't lose context at chunk boundaries.

If your RAG answers feel "off," look at your chunks before you look at your embedding model. I wish someone had told me that from day one.

2. HNSW Index vs. IVFFlat for Lambda

This one bit me in production. pgvector supports two index types: IVFFlat and HNSW. I started with IVFFlat because the internet says it's faster to build. That was a mistake I didn't catch until users complained about slow first responses.

IVFFlat requires a "warm-up" query to load the index into memory. On Lambda, that warm-up happens on every cold start. It added about 400ms to the first query after a cold start. My users noticed. I got support tickets about it.

HNSW doesn't need warm-up. The first query is about the same speed as subsequent ones. The moment I switched, the cold start complaints stopped. For Lambda workloads where cold starts are a reality, HNSW is the better choice. The trade-off is that HNSW uses more memory and takes longer to build, but you're building the index once and querying it thousands of times. The build cost is irrelevant.

3. Cache Embeddings for Repeated Queries

I noticed this pattern about a month in. Users ask the same questions over and over. "How do I reset my password?" "What are the pricing tiers?" "How do I cancel?" I was paying Bedrock to embed identical strings dozens of times a day.

Embedding the same question every time wastes money and adds latency. I added a simple cache using a hash of the query text:

const embeddingCache = new Map();

async function getEmbedding(text) {
  const cacheKey = createHash("sha256").update(text.toLowerCase().trim()).digest("hex");

  if (embeddingCache.has(cacheKey)) {
    return embeddingCache.get(cacheKey);
  }

  const embedding = await generateEmbedding(text);
  embeddingCache.set(cacheKey, embedding);

  // Keep cache from growing forever
  if (embeddingCache.size > 1000) {
    const firstKey = embeddingCache.keys().next().value;
    embeddingCache.delete(firstKey);
  }

  return embedding;
}

This is an in-memory cache, so it lives as long as the Lambda container. For a warmer setup, you could use DynamoDB or ElastiCache, but the in-memory approach works surprisingly well. In my case, about 40% of queries are cache hits within the same Lambda container.

4. Monitor Retrieval Quality Separately from Generation Quality

I spent an entire weekend debugging a "wrong answer" that a user reported. I was blaming the LLM, tweaking the prompt, trying different temperature settings. Then I finally looked at the retrieved chunks and realized the retrieval step had pulled in completely irrelevant paragraphs. The LLM was doing its best with garbage context. That weekend taught me to always check retrieval first.

Now I log both steps separately:

Retrieval quality: log the query, the returned chunks, and their similarity scores. Review the lowest-scoring returned chunks regularly. If they're irrelevant, your similarity threshold is too low.
Generation quality: log the prompt (with context) and the LLM's response. When users report bad answers, check whether the context contained the right information.

Most of the time, bad answers come from bad retrieval. The LLM is usually fine if you give it the right context. So investing in better chunking, better metadata filtering, and tuning your similarity threshold pays off way more than switching to a fancier LLM.

Photo by Mikhail Nilov on Pexels

Conclusion

My SaaS now has a /rag/ask endpoint that my users actually love. They type a question, get an answer grounded in our own documentation, and see exactly which sources the answer came from. The whole thing runs on the same Lambda and PostgreSQL stack I was already using. No new services. No new vendors. Just pgvector and a few hundred lines of code.

Looking back, the decision to keep everything inside my existing stack was the right call. I avoided a separate vector database bill, a separate SDK to maintain, and a separate service to monitor. pgvector gave me semantic search, HNSW indexing, and cosine similarity queries inside the database I already knew how to operate. The total cost sits around $54/month at 10K queries per day.

If I were starting over, I'd change one thing. I would spend more time on chunking from the very beginning instead of comparing embedding models. Good chunks make everything downstream better. Bad chunks make everything worse, no matter how fancy your LLM is. I learned that lesson the expensive way.

For anyone building an AI SaaS on AWS, this setup hits a sweet spot. Low cost, minimal operational overhead, and your data stays in PostgreSQL where you can query it with regular SQL alongside your vector searches. If you already have the Lambda and RDS stack running, you're closer to production RAG than you think.

Now go build something cool with it.

Hope you enjoyed this walkthrough. If you have questions about any part of the pipeline, or if you've built something similar and want to compare notes, reach out.

Follow me on Twitter/X for more posts about AWS, serverless, and building AI products. I share what actually works in production, not just what looks good in tutorials.

Photo by Nicolas Foster on Pexels

Originally published at harun.dev.

First 3 things I always do after creating a new AWS account

Harun R. Rayhan — Wed, 06 Jul 2022 14:24:50 +0000

As part of my job, I create AWS accounts frequently. I do lots of things depending on account needs before starting to use it. I found 3 common things I do with all of the new AWS accounts. I made a checklist. Nowadays I do these 3 things first and then everything else.

Let's make your account more secure and more protected than ever. 🔐

1. Budget with Alerts

One of the common issues with AWS is bills. People gets surprising bills because of misconfiguration or keep expensive services running unknowingly. Although, budget and alerts can NOT fix these issues but help keep bills tidy. And gives alerts if crosses the budget almost instantly.

Let's set a budget and a couple of alerts.

Go to Budget Console
Click on the "Create Budget" button.
Choose the first option, "Cost budget - Recommended" from the next page.
From the next page:
- A. Give it a name.
- B. Set the Period as Month for this demo, but you can choose any other period.
- C. Set Budget renewal type as Recurring budget
- D. Choose Fixed from the Budgeting method.
- E. Give a real budget amount to your need, mine is a demo account so $10 is fine for me. Let me iterate again, choose a real amount.
- F. From the bottom-right of the page click "Next"
From the next page, click on the "Add an alert threshold. Going to set a couple of alerts.
Let's set an actual budget alert:
- A. Choose 80 from the threshold.
- B. Select Actual from the trigger
- C. Put as many real and active emails as necessary. (⚠️ please add active email to save yourself from disaster).
- D. Let's add another one. Actual budget alert only sends notifications when you spent threshold amount. For example, it will send an email to me after spending $8 (80% of 10$). So, we need something else together with it.
Now set a forecasted budget alert:

Everything is the same as the first alert, only change Trigger to Forecasted. So, it will send you an alert if AWS forecast that you will spend more than your threshold. After configuring it, click the Next button from the bottom of the page.
Click the Next again from the next page. And then click the Create Budget button from the final page.

And you are done with, you probably going to be saved from a disaster by email notifications.

...

2. Add Multi-Factor Authentication (MFA)

You should use MFA for all IAM accounts including the root account. If not possible, at least apply it to root and privileged accounts. Login to the account (root or IAM) and let's protect it.

Go to Security Credentials of IAM Console
Click on the "Assign MFA device"
Choose Virtual MFA device from the popup
You can choose either of these apps: Authy, Duo Mobile, LastPass Authenticator, Microsoft Authenticator, Google Authenticator, Symantec VIP. My personal choice is Authy.
Click on the "Show QR code" from the next popup and scan it from your MFA app. Then enter two codes (one after another). And then click the "Assign MFA" button.

...

(bonus) Set Alias for the Account

Account ID or Alias is required to log in IAM account. Account ID is a long number. Humans are bad with a long number. A meaningful alias can save you from the trouble of remembering the account ID. Let's set an alias for your account:

Go to IAM Console after login into the root account.
You will see a Create button in the right sidebar under the account alias.
Choose a unique and memorable alias for your account and click "Save changes".

Use the alias instead of account ID next time you are logging into an IAM account.

...

3. Create IAM account(s) for Day-to-Day tasks

You need the root account for a few tasks. Other than that you should not use the Root account. Perhaps, not for day-to-day tasks, lunching, and provisioning services.

I always create an IAM account with admin permission. I do all of the tasks from this account except those tasks that require the root account. So, let's create an admin IAM account.

Go to Users page of IAM Console.
Click on the Add users button from top-right corner.
You will be in this page:
- A. Choose an username
- B. Chosse access type. I need both CLI and AWS Console access. (UPDATE: Don’t choose CLI, if not necessary. You can create different IAM users for CLI actions.)
- C. You can enter your custom password or choose auto generated.
- D. It's better to require reset password to choose it's own.
- E. Click on the Next: Permission button from the bottom-right
You are in the permission page:
- A. Choose Attach existing policies directly from the top options
- B. Search for the AdministratorAccess role
- C. Choose the AdministratorAccess role
- D. Click on the Next: Tags button
You are in the tags page. You can add you tag(s) if you want but it's optional. Click in the Next: Review button.
Take a look on everything and check if it's matches to mine. And then click on the Create user button.
Now we are in the final page:
- A. The link to login to the console
- B & C. Username & Password to login to the console
- D & E. Access key & secret to login via CLI or App
- F. You can download the .csv file with credentails. You can save it in safer place for later use.

...

Conclusion

There are more things to do to save bills and tighten security of your account. But these are the first 3 things I do and doesn't require much technical knowledge.

You can subscribe to my newsletter in my blog. Follow me on Twitter, Dev.to, and Hashnode for regular updates. Don't hesitate to show your love by clicking Like 👍 button.

Install Cloud Watch Agent in Amazon Lightsail instance for Monitoring, Logging & Debugging

Harun R. Rayhan — Sat, 04 Jun 2022 04:51:10 +0000

Use the Cloud Watch agent to send logging and debugging to Cloud Watch. You can check logs without logging into instances. It's accessible even after the instance is terminated.

Life is far from perfect, and so does Amazon Lightsail instances. It can be frustrating if there is an error and you don't know how to debug it. The server itself and applications installed in it produces log files. And server stats can be helpful in monitoring, logging, and debugging.

Prerequisites

It would help if you had an existing Amazon Lightsail instance. You can create a new instance by following "Deploy WordPress App to Amazon Lightsail" or "Deploy Laravel Application to Amazon Lightsail".

Let's go 🏃‍♂️

Create an IAM User

Go to IAM Console.
Click on the Users menu from the navigation panel and then click on the Add Users button.
You will be in the Add user screen:
- A. Give it a name. For me, it's LightsailCloudWatchAgent.
- B. Check the Access key - Programmatic access option
- C. Click on the Next: Permissions button.
You will be in the Set permissions page:
- A. Select the Attach existing policies directly option.
- B. Search for CloudWatchAgentServerPolicy.
- C. Select CloudWatchAgentServerPolicy, and
- D. Click the Next: Tags button.
Tags are optional. You can choose your tags. Click on the Next: Review after you are done.
From review page you can check if everything alright and then click the Create user button.
Your account is created and you will be in this screen:
- A. & B. Copy Access Key ID and Secret access key in somewhere safe, or,
- C. Download the csv file. We will need this access later.

Configure AWS Access

Check if AWS CLI is installed by running aws --version. You should see the current version. If you get an error, CLI probably is not installed.
Ignore this if AWS CLI is already installed. Run this on ubuntu:

curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install
sudo ./aws/install --bin-dir /usr/bin --install-dir /usr/local/aws-cli --update

Follow this instruction to install on other servers.

Now, we are going to configure the AWS access generated in the last step. Run this

sudo aws configure --profile AmazonCloudWatchAgent

Enter Access Key and Secret Like this:

Install the Cloudwatch Agent

You need to SSH into your instance. Here is how you can SSH into your Amazon Lightsail Instance.
Now, let's install the agent. I'm using ubuntu, you can just run this command

# Download the agent
wget https://s3.amazonaws.com/amazoncloudwatch-agent/ubuntu/amd64/latest/amazon-cloudwatch-agent.deb
# Install the agent
sudo dpkg -i -E ./amazon-cloudwatch-agent.deb

Now run this command configure the agent.

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-config-wizard

Input your choices as I did. Check the arrow mark. You can choose your own or follow mine.

Give AWS Access to the Agent. Run this:

sudo vim /opt/aws/amazon-cloudwatch-agent/etc/common-config.toml

Paste this in the bottom of the file:

[credentials]
shared_credential_profile = "AmazonCloudWatchAgent"

Now hit ESC and then :wq!.

Start the agent:

sudo amazon-cloudwatch-agent-ctl -c file:/opt/aws/amazon-cloudwatch-agent/bin/config.json -a fetch-config -s

Check the status of the agent by using:

sudo amazon-cloudwatch-agent-ctl -a status

You should see something like this:

Verify metrics on CloudWatch

Go to the Cloudwatch Console
On the left navigation panel, choose Metrics.
Under “Custom Namespaces”, You should see a link for “CWAgent”.
Choose CWAgent.
Choose any ImageId, InstanceId, InstanceType.
Select the checkbox to display metrics on the graph. Here is mine:

Conclusion

Hope you enjoyed this article. This is the last article of the Amazon Lightsail series. You can ask me any Lightsail-related question in the comments or on social media.

You can subscribe to my newsletter to get blog updates every week. Follow me on Twitter, Dev.to, and Hashnode.

Connect Redis/Elasticache to application in Amazon Lightsail instance

Harun R. Rayhan — Sat, 28 May 2022 14:32:18 +0000

You can create Redis/Elasticache even though Amazon Lightsail doesn't offer it. You can create an Elasticache for Redis cluster in AWS and connect using VPC Peering.

Prerequisites

An Amazon Lightsail instance. To create a new instance follow Deploy Laravel Application to Amazon Lightsail or Deploy WordPress App to Amazon Lightsail
Be a little familiar with AWS Console.

Create Elasticache for Redis

Go to Elasticache for Redis console. Change the region that you want to launch it. I selected us-east-1 as my Lightsail Instance(s) are there.
Click the Create Redis cluster button from the center of the page (or wherever you see it)
You will land in this screen 👇
- A. Choose Configure and create a new cluster
- B. Choose cluster mode Disabled
- C. Give it a name
- D. Uncheck Multi-AZ Enable
- E. Uncheck Auto-failover Enable
- F. Select a free-tier node type. Currently it's "cache.t3.micro" and "cache.t2.micro".
- G. Do not need any replica for this demo. Make it "0".
- H. Give subnet groups a name
- I. Select your VPC
- J. And then clieck the Next button from the bottom.
You will be on the second page. Just uncheck Enable automatic backup and click the Next button from the bottom of the page.
Review everything on the 3rd and final page. And then click the Create button from the bottom of the page.
It will take a few minutes to get created. Wait for status changes to Available.
now, click on the Cluster name. And you will be on the details page.

Copy the Primary endpoint in your clipboard or somewhere safe.

Configure Security Group for Redis cluster

Go to the Security groups of VPS Console. Make sure you are in the same region as the Redis cluster. And then click the Create security group from the top-right corner.
You are in create Security Group page
- A. Give it a name
- B. Click on the Add rule button of the Inbound rules section.
- C. Select 6379 as Custom TCP Port
- D. Add your Amazon Lightsail VPC IP range. To generate it, check your instance's Private IP address. Mine is 172.26.3.157. Replace the last two sections with 0 and add /16. So, mine became 172.26.0.0/16.
- E. Optionally, you can add a description
- F. Then hit the Create security group button from the bottom of the page.
Now go to the Network and security tab of your Redis cluster and click the Modify button from the Security groups section.
Click on the Manage button from the next screen
Select the correct security group from the pop and then click on the Choose button.
Click Modify button from the bottom of the page.

It will take a few minutes to complete the modification. Wait until the status changes to Available.

VPC Peering

The important part of this tutorial. We need to peer VPC between Lightsail and Redis cluster's VPC to connect it.

Go to the homepage of the Amazon Lightsail console. Click the Account from the Account menu dropdown.
Click on the Advanced tab from the Account settings page. You will see a list of your VPCs. Connect the one that resides in your Redis cluster. For me, it's *Virginia (us-east-1)
*.

Connect to the Redis cluster from the Lightsail instance

Using CLI

Connect to the instance using SSH. Follow this to connect to SSH.
Install Redis CLI
- A. On Amazon Linux 2

  sudo amazon-linux-extras install epel -y
  sudo yum install gcc jemalloc-devel openssl-devel tcl tcl-devel -y
  sudo wget http://download.redis.io/redis-stable.tar.gz
  sudo tar xvzf redis-stable.tar.gz
  cd redis-stable
  sudo make BUILD_TLS=yes

B. on Ubuntu

  sudo apt install redis-server

Now connect to the Redis cluster. The moment of truth.

redis-cli -h <redis_endpoint> -c -p 6379

Replace <redis_endpoint> with your Primary endpoint of Redis you copied in the clipboard. Remove port (":6379") from the endpoint URL. You will see this screen if successful 👇

You can use the ping, set, and get commands to test it out.

Connect to Application

You will get packages for most of the Applications, languages, and Frameworks. Check docs. For Laravel, you just have to configure it in the .env file.

Conclusion

Hope you enjoyed this short article. You can subscribe to my newsletter, and follow me on Twitter, Dev.to, and Hashnode.

Add Lightsail Load Balancer to Application hosted in Amazon Lightsail instance(s)

Harun R. Rayhan — Sat, 28 May 2022 11:12:16 +0000

Lightsail offers Load Balancer. It's very easy to provision one and attach one or multiple instances to it. Let's create one and attach instances to it.

Prerequisites

You need one or more Amazon Lightsail instances to attach to the load balancer in the same region. You can create it by following Deploy Laravel Application to Amazon Lightsail or Deploy WordPress App to Amazon Lightsail

⚠️ Warning: Amazon Light Load Balancer going to cost you USD$18/month. It has no pre-tier available. Be sure before going to provision one. You can check the whole article before creating.

Provision a Load Balanced

Go to the homepage of the Amazon Lightsail console. Click on the Networking tab and click on the Create load balancer button.
You are in the Create a Load balancer screen like this 👇
- A. Change your region to where your instances are. My instances are in Virginia.
- B. Give it a name, and
- C. Hit the Create load balancer button from the bottom.
Load balancer will be created in a moment and you should land on the Load balancer page.

Attach instances to the Load balance

Go to the homepage again and then click the Networking tab. You should see Loadbalancer(s) including the one just created. Click on the name of the Load balancer.
Our load balancer doesn't have any attached instance(s) yet, we are going to attach one or more. Select your instance in the Target instances section.

I just have one in this region. You should see all of the instances you have in the region where the load balancer is provisioned.
Confirm the attachment of your instance
Do the same process until you attached all of your target instances.
Go to Load balancer URL (DNS name) and you should see your application.

ℹ️ If you don't see the desired page, check your application or other tools (Apache, Nginx, etc.) setting.

Add Domain or Sub-Domain

You can create DNS Zone in Lightsail for your domain and configure it. Follow this to add your domain/sub-domain.
You can add the Load balancer URL (DNS name) as A record in your domain's DNS control panel.

Enable HTTPS

Go to the Inbound traffic tab of your Load balancer setting.
- A. Create an SSL certificate for your domain by clicking the Create certificate + button.
- B. You should see all of the created certificates in the SSL/TLS certificate list and select the one you just created.
- C. Optionally, you can Redirect from HTTP to HTTPS. (After enabling the HTTPS).

Conclusion

Hope you enjoyed this short article. You can subscribe to my newsletter, and follow me on Twitter, Dev.to, and Hashnode.

Add Database to Application hosted in Amazon Lightsail

Harun R. Rayhan — Mon, 23 May 2022 14:04:33 +0000

Amazon Lightsail offers dedicated Databases. You can make use of it instead of an on-instance database. It's easy, managed, secure, and most importantly it's not tied to the instance. So, your database will be safe even if instance crashes.

In "Deploy WordPress App to Amazon Lightsail, the database is installed in the instance. In this article, we are to cover the dedicated databases.

⚠️ Warning: Dedicated Lightsail Databases are going to cost $$$. It has a 3-month free tier at this time of writing. Check your plan before creating a database.

So, what are you waiting for? Let's dive right in 🏃‍♂️

Create a Database Instance

Go to your Amazon Lightsail home page. And then click on the Databases tab.
Now click on the Create database button
Now you are on the database creation page:
- (A) Select AWS Region and Availablity Zone (AZ). for me Virginia and Zone A are perfect.
- (B) Choose the Database engine and version. I selected MySQL 8.
- (C) Click on Specify login credentials, and
- (D) Click on Specify the master database name.
Scroll down a little and configure this:
- (A) Choose Database username
- (B) Password. I'm okay with the generated password, but you can choose yours.
- (C) The first/master database. You can create more later.
- (D & E) Choose your plan type and the actual plan. You can choose the free-tier option like me.
- (F) Give it a name, and
- (G) Hit the Create database button Your database should be created within a few minutes. Please wait for it until it's Running.

Database Credentials

You already know your database username and master database. For everything else, go to your database and click on the Connect tab. You should see your credentials like this:

(A) DB Username
(B) DB Password
(C) DB Endpoint
(D) DB Port

Connect to the database

Instance CLI

You can SSH into your instance.

Install MySQL using the following command:

sudo apt install mysql-server -y
sudo systemctl start mysql.service

And then connect to MySQL:

mysql --host <db_endpoint> --user <db_username> --password

You will be asked for your DB password. If successful, you will see this screen:

SQL Client App over SSH (SSH Tunneling)

Most modern SQL Client applications (ex TablePlus, MySQL Workbench) have the option to connect Database via SSH. You can connect using that option.

Add Database to your application

Laravel and NodeJS

You can update your application's .env file with Database credentials.

WordPress

Update your wp-config.php file with the database credentials.

Any other app

Update your database credential in your application's configuration and/or environment file. Check your application's documentation for that. Or it's custom, you know where to put it.

Conclusion

Thanks for reading the article. Adding dedicated databases to Amazon Lightsail instance is not that hard. You can follow me to get instant updates on my posts. You can also subscribe to my newsletter.

Deploy Laravel Application to Amazon Lightsail

Harun R. Rayhan — Thu, 19 May 2022 15:06:43 +0000

Beginners friendly step-by-step guide to deploy any application to Amazon Lightsail. This article demos a Laravel app but the idea is the same for all kind of applications.

Introduction

Amazon Lightsail is a fixed monthly priced VPS service offered by AWS. Once you install the OS of your choice, you can run almost any application. In this application, I'm showing how to deploy one of my favorite backend frameworks called Laravel. Which most of you might hear of. Let's deploy it.

I don't like/know Laravel

It's totally fine if you don't like or know Laravel. Laravel is just the framework I choose to demo but it can be any app. It can be Node.js, Python, Django, Fast API, Ruby on Rails, or any other app. The idea is the same, but some of the steps in deployment will be different.

Prerequisites

You need a Laravel app to deploy. Don't worry if you don't have one, I have to best Laravel (v9) app you can find in the internet (pun intended).

Let's deploy our Laravel app. 📦👉☁️

Create Amazon Lightsail Instance

Go to Lightsail homepage. Click on the Create instance button under Instances tab.
You will land on the Create instance page. We will use Ubuntu for this demo but you are open to choosing the OS of your choice.

(A) Change the AWS Region and Availablity Zone (AZ) of your choice if not the correct one selected. (B) Select Linux/Unix platform. (C) Select Ubuntu (20.04 LTS) as OS. The blueprint should be OS Only. (D) Choose a plan. (E) Give it a name, and (F) Hit the Create instance button from the bottom of the page.
Your instance is in the Pending state. Will be Running in a few minutes.

Assign a static IP

Read how to assign a Static IP to an Amazon Lightsail Instance.

Connect to the instance using SSH

ℹ️ I described in detail How to connect to an Amazon Lightsail instance using SSH.

Go to the instance page by clicking on the name. And go to the Connect tab. We need these 3 things: (A) Public IP, (B) SSH Username, and (C) Download the SSH Key.
Change permission of Key file:

# Change "path/to/keyfile.pem" with your key file name
chmod 400 path/to/keyfile.pem

Connect to the SSH

#format 
ssh -i "path/to/keyfile.pem" <username>@<ip_address>
# Should look like this with your own value
# ssh -i "LightsailDefaultKey-us-east-1.pem" ubuntu@52.201.59.133

If you are asked to add IP to known hosts, type yes and you should be logged in.

Installing Necessary Scripts

Our server is created and it's empty. You can install any script you want, depending on what you need. In this demo, we are going to use Laravel. You can follow along even if you have a different app/stack.

Update package information

sudo apt-get update

Some packages

sudo apt install curl git unzip -y

Install PHP I love the latest PHP 8.1. It's not officially available in Ubuntu 20.04 yet. We will enable the PHP Repository to install PHP 8.1.

sudo apt install software-properties-common -y
sudo add-apt-repository ppa:ondrej/php -y
sudo apt install php8.1-fpm php8.1-cli php8.1-mysql php8.1-curl php-xml php-mbstring -y

Check php version by using php --version command. You should see something like this:

Install Composer

# Download the bin file
curl -sS https://getcomposer.org/installer | php
# Move it to correct path
sudo mv composer.phar /usr/local/bin/composer
# Make it executeable
sudo chmod +x /usr/local/bin/composer

Install Nginx & configure nginx

sudo apt install nginx -y

Clone your application repo

# go to www directory
cd /var/www/html
# clone the git repo
sudo git clone https://github.com/HarunRay/laravel-to-lightsail.git
# go to application directory
cd laravel-to-lightsail
# copy .env.example file
cp .env.example .env
# Install dependencies
sudo composer install

Create a nginx config file

sudo vim /etc/nginx/sites-available/laravel

Paste this code. Change 52.201.59.133 to your IP address and save.

server {
    listen 80;
    listen [::]:80;
    server_name 52.201.59.133;
    root /var/www/html/laravel-to-lightsail/public;

    add_header X-Frame-Options "SAMEORIGIN";
    add_header X-Content-Type-Options "nosniff";

    index index.php;

    charset utf-8;

    location / {
        try_files $uri $uri/ /index.php?$query_string;
    }

    location = /favicon.ico { access_log off; log_not_found off; }
    location = /robots.txt  { access_log off; log_not_found off; }

    error_page 404 /index.php;

    location ~ \.php$ {
        fastcgi_pass unix:/var/run/php/php8.1-fpm.sock;
        fastcgi_param SCRIPT_FILENAME $realpath_root$fastcgi_script_name;
        include fastcgi_params;
    }

    location ~ /\.(?!well-known).* {
        deny all;
    }
}

Now, run this:

# link nginx config 
sudo ln -s /etc/nginx/sites-available/laravel /etc/nginx/sites-enabled/
sudo systemctl start nginx
sudo systemctl enable nginx

Set Laravel App Permission

# Go to your laravel app directory
cd /var/www/html/laravel-to-lightsail/
# Add files & folder www-data user & group
sudo chown -R www-data:www-data .
# Add your ubuntu to group
sudo usermod -a -G www-data ubuntu
# Set file(s) permission
sudo find . -type f -exec chmod 644 {} \;
# Set folder(s) permission
sudo find . -type d -exec chmod 755 {} \;
# Set cache directory permission
sudo chgrp -R www-data storage bootstrap/cache
sudo chmod -R ug+rwx storage bootstrap/cache

Generate app key

sudo php artisan key:generate

Update the .env file Open file

sudo vim /var/www/html/laravel-to-lightsail/.env

Now, change these values;

APP_ENV=production
# Change IP to yours
APP_URL=http://52.201.59.133

Now go to your public IP address. Mine is http://52.201.59.133. You should see this welcome page.

Congratulations, your Laravel app launched on Amazon Lightsail instance. 🚀

Conclusion

In the next few articles, we are going to configure the database, load balancer & Redis (Elasticache). Follow me and subscribe to my newsletter.

Snapshot and Restore of Lightsail Instance to Lightsail/EC2

Harun R. Rayhan — Thu, 12 May 2022 14:31:04 +0000

Taking regular backup/snapshots of a production application is important. It is one of the main requirements of a disaster recovery(DR) plan. But snapshots are not only for DR, you can use them to move/clone instances to different regions or availability zone(AZ). Maybe move to a higher spec instance. Or maybe you want to move to Amazon EC2 altogether.

Prerequisites

You just need an existing Amazon Lightsail instance. Don't worry if you don't have one, follow how to Deploy WordPress App to Amazon Lightsail.

Take Snapshot (Backup)

Click on the instance name you want to take a snapshot. All of the instances are listed under the Instances tab of the home page.
Go to the Snapshots tab of your instance page.

(B) We are going to create a manual snapshot. This one will take a snapshot instantly.
(C) Optionally, you can set up an automatic snapshot too. It will take daily snapshots.

⚠️ **For production instance, always set up an automatic snapshot. **
Give it a name and click Create.

It will take some time if this is the first snapshot of the instance, size is another factor. Subsequent snapshots are faster, as it takes incremental backups, only changes from the last snapshot.
Upon completion, it will look like this:
You will find all of the snapshots in the Snapshots tab of the homepage. I have just one:

Restore to new Amazon Lightsail instance

Click on these 👇 three dots:
Click on Create new instance from this menu:

ℹ️ You can click on the second item, Copy to another Region to literally copy to another region and restore to a new instance there. Snapshot stored in an S3 bucket in the same region where the instance launched. You can't access the S3 bucket as it's managed by AWS. To restore a snapshot to a new instance in another region, you have to copy it first.
The page looks familiar, right? It's the same create instance page.

(A) You can change the availability zone(AZ) if you want. It's great if you are using multiple instances for horizontal scaling, gives high availability(HA).
(B) Choose an instance the same size or larger than the source. Choose larger for vertical scaling. If you need a larger instance than you originally have, this is the way.
(C) Give it a name, and
(D) Click the Create instance button from the bottom of the page.

Wait a moment, maybe a few minutes. Congratulations 🎉 you just restored your instance.

...

Copy to another Region

Snapshot stored in an S3 bucket in the same region where the instance launched. You can't access the S3 bucket as it's managed by AWS. To restore a snapshot to a new instance in another region, you have to copy it first. Let's copy it to another region.

Go to the Snapshots tab of the homepage. Find your instance and snapshot. Click the three dots. And then click Copy to another Region:
(A) Choose the region you want to copy the snapshot, (B) give it a name, and (C) click the Copy snapshot button from the bottom of the page.
It will take a few minutes to copy the snapshot.

...

Restore/Export to Amazon EC2 instance

Click three dots from the snapshot menu you want to restore to EC2 and click the Export to Amazon EC2 menu item
Click the Yes, continue button in the popup. Click Acknowledge in the next popup.

⚠️ If you are not in the free-tier or cross the free-tier limit, you may get charged for this. If you are not sure, don't do it. Just follow my demo.
You will see a task running like this ↓. Wait for a while to finish. A small snapshot wouldn't take much time.
After the copy task is completed, go to Snapshots of EC2 Management Console. Choose the correct region. You should see your snapshot there.
Select the snapshot and click the Create image from snapshot from the Actions menu
Give the name of the image and click the Create image button from the bottom of the page
Now go to AMIs page of EC2 Management Console. Choose the correct region, N. Virginia (us-east-1) in my case. You should see all of your AMIs.
Select the AMIs you just created and click the Lunch Instance from AMI button in the top right corner.
You will see a traditional EC2 Lunch page:

(A) Give your instance a name. (B) Choose an instance type, for my case, it's free tier eligible t2.micro. (C) Choose existing key pair or create a new one. You need it to log in via SSH. (D) Check the ports you want to open, in my case I selected all of them.

Now, click the Lunch instance button from the right sidebar. Wait for a while to instance state to become Running and Status check 2/2 checks passed.

You will see the EC2 Instances page like this: Select the instance. Copy the public IP address and try it in the browser. You should see the same page as Amazon Lightsail instance.

ℹ️ Manual snapshots will not be deleted even if you delete the instance. So, don't forget to delete unnecessary snapshots to save bills.

...

Conclusion

Hope you enjoyed this article. Ask your questions in the comment. Subscribe to my newsletter to get weekly updates.

Add (Cloudfront) CDN Distribution to WordPress Application hosted on Amazon Lightsail

Harun R. Rayhan — Tue, 10 May 2022 12:15:16 +0000

It's not very hard to add CDN Distribution to a WordPress app hosted on Amazon Lightsail. But if you are new, it can be cumbersome. This article is to demonstrate how to do it easily.

Why Should I use CDN distribution?

You might hear about Content Delivery Network (CDN). If not, here is an article explaining "what is CDN?".

If you are not interested to read a long article, here is TL;DR:
In CDN, Lots of small servers run closer to your customer. Content gets cached there and served from the closer server(s) of your customer. In that way, ease the pressure from the main servers.

What is CDN Distribution?

It is the Amazon Lightsail CDN service. Under the hood, it uses Amazon Cloudfront. It has 300+ points of presence to deliver content closer to your customers. On top of that, it comes with AWS Shield, which protects infrastructure from DDoS attacks.

Are you excited to create your first CDN Distribution? Let's go 🏃‍♂️

Prerequisites

WordPress Application

You need to have an existing WordPress application for this tutorial. Don't worry if you don't have one. I covered How to Deploy WordPress Application to Amazon Lightsail in the first part of this series.

Attach a Static IP to WordPress Instance

Go to the Networking tab and then click on Create static IP
(A) Select the WP App instance. (B) Give it a meaningful name and (C) Hit Create.

Now we are ready to create our first distributions.

Create CDN Distribution

Go to home page of Lightsail. Go to Network tab again. This time hit the Create distribution button.
Select the WordPress Instance as Origin
Hit Yes, apply to apply the predefined WordPress settings
Now, (A) choose a plan. (B) Give it a name and (C) Hit Create Distribution button

Woohoo 🎉 Your first distribution is created. Wait for a while to be available. After getting done, you will see this 👇

(A) Your domain for the distribution. You can access the site using this URL.

Domain

Update the Domain in DB

Login to your WordPress application's database, you may follow this. Update the siteurl and home to your wp_options table to your distribution domain.

Bonus

You can use a custom domain instead of CloudFront's sub-domain. You may follow this tutorial.
You can add Object Storage to your WordPress app with a CDN distribution by following this tutorial.

Conclusion

Hope you enjoyed this simple article and created your first CDN distribution. Ask your questions in the comment. Subscribe to get the next blog post. Thanks 🙏

Deploy WordPress App to Amazon Lightsail

Harun R. Rayhan — Sun, 08 May 2022 11:02:48 +0000

Introduction

AWS Lightsail can be the easiest entry to the AWS Cloud. It has fixed monthly pricing unlike most other services, so you don't have to worry about a huge monthly bill.

Lightsail is really great for small to medium apps. It has out-of-the-box support for WordPress, Magento, Joomla, Prestashop, Drupal, Ghost, and Django apps.

It provides pre-configured stacks like LAMP, NGINX, MEAN, and Node.js. You can also launch pre-configured Plesk, WHM & cPanel in just a few clicks.

But that's not all. You can launch any Linux-based and Windows OS instance. My personal favorites are Ubuntu and Amazon Linux 2.

Prerequisites

Need a valid AWS account to deploy the WP app. If you don't have one, just create one from here. Before starting to deploy your app read about Amazon Lightsail Pricing and Free-Tier. Don't worry, I will try to keep everything within the free tier. At this time of writing, you can run a Lightsail instance for free for 3 months.

Install WordPress App

Are you seeing this? Let's go 🏃‍♂️

Create Instance

Now click on the big Create Instance button. If you don't see one, follow this:
Let's choose Instance location, platform, and blueprint:

A. Click the link to change location and availability zone (AZ) if not what you want. I recommend selecting a location closer to your customers.
B. Choose the Linux/Unix platform.
C. Select the Apps + OS from the blueprint.
D. Now choose WordPress from the app list. Not the multisite one.
Now scroll down to Choose your instance plan section. I chose a free-tier plan and only one instance. Here is my setup:
Finally scroll to the bottom of the page and click the Create instance button. Wait for the status to become Running. Congratulations, your instance has launched 🚀

Assign a Static IP (optional)

AWS Lightsail Instance comes with public (and private) IP. Which changes every time Stop and Start the instance. My Public IP was this when I launched the instance:

After Stop and Start the instance, it got a new Public IP address:

We have to fix that. So, let's create and assign a Static IP:

Click on your application instance name to see the details of your instance.
Click on + Create static IP from here:
Add a meaningful name and click Create: > ⚠️ Warning: You are allowed to assign 5 static IPs for free. You will be charged for static IP that is not attached to a running instance.

Connect to the instance using SSH

You can connect to an instance in two ways:

Using browser

That's the easiest. Go to the Connect tab and click Connect using SSH:

Using SSH Client

Download the SSH key from here:
If you are using PuTTY you can follow this tutorial.
If you are using a terminal in macOS or Linux, run this command on where you downloaded the SSH key.

# Run for the first time to fix the permission
chmod 400 <path/to/your-lightsail-ssh-key>.pem

# Connect to your Lightsail instance
# Username and Public IP can be found in "Connect" tab
ssh -i "<path/to/your-lightsail-key>.pem" <username>@<public_ip>

Get Admin Username and Password

You can get the username from the Connect tab. The username is user for this stack.
To get the password, log in to SSH and run this command

cat bitnami_application_password

Now open a browser tab and go to http://<public_ip>/wp-admin. Enter username and password above. You should see the WP Admin dashboard like me.

Add a Domain

Go to the Home page. Click on the Networking tab and then hit Create DNS zone button
Add your domain name. Scroll to the bottom and click the Create DNS zone button
Add these name server records in your domain control panel (ex. Namecheap, GoDaddy). You might have different records than mine
Now to the good part. Click + Add Record button. Add your instance and click on tick icon to save. > ⚠️ It may take few minutes to days to propagate DNS records.
Now update the domain name in siteurl and home rows of wp_options table.

Import Existing App

Import Database

You may have an existing database. You can import your database to the Lightsail instance. You can use a Database client app like TablePlus, MySQL Workbench, or even just the terminal.

Your database credentials exist in wp-config.php file. Connect to your instance via SSH and run this command:

cat stack/wordpress/wp-config.php

Use DB credentials to connect to DB, and replace localhost with the instance's public IP. Connect to your DB via SSH (recommended) or add the 3306 port here:

ℹ️ Don't forget to change the domain name in siteurl and home rows of wp_options table.

Import existing files

Application folder located in stack/wordpress. You can upload files in two ways:

SFTP Client: You can use SFTP clients like FileZilla or Transmit to upload files. Use the same SSH credential to upload files.
Git: Connect to your instance via SSH. Install git by using this command:

sudo apt-get install git

Now pull your files from git using the git clone/pull command.

Conclusion

This is just the first step toward deploying your WordPress app to Amazon Lightsail instance. There are lot more you can do. You are welcome to ask your questions, give me feedback, or any improvement request.