<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Mohammed Arshad Ansari</title>
    <description>The latest articles on DEV Community by Mohammed Arshad Ansari (@mohammed_arshadansari_f2).</description>
    <link>https://dev.to/mohammed_arshadansari_f2</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F3609964%2Fd0a642d8-0372-4976-99f8-0575aa92a93c.png</url>
      <title>DEV Community: Mohammed Arshad Ansari</title>
      <link>https://dev.to/mohammed_arshadansari_f2</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/mohammed_arshadansari_f2"/>
    <language>en</language>
    <item>
      <title>Self-hosted LLM for a small team: what it takes, and when it pays</title>
      <dc:creator>Mohammed Arshad Ansari</dc:creator>
      <pubDate>Mon, 16 Nov 2026 14:00:00 +0000</pubDate>
      <link>https://dev.to/mohammed_arshadansari_f2/self-hosted-llm-for-a-small-team-what-it-takes-and-when-it-pays-3kg9</link>
      <guid>https://dev.to/mohammed_arshadansari_f2/self-hosted-llm-for-a-small-team-what-it-takes-and-when-it-pays-3kg9</guid>
      <description>&lt;p&gt;A self-hosted LLM is an open-weight language model running on hardware you control, answering requests over your own network. Nothing you send it leaves the building. That is the main reason teams ask about it, and it is a good reason.&lt;/p&gt;

&lt;p&gt;The second reason people give is cost, and that one is often wrong. I ran my own system's thinking on an 8 GB GPU for months, then moved it to a hosted API, and kept one job at home. Which job stayed, and why, is most of what a small team needs to know before buying a card.&lt;/p&gt;

&lt;h2&gt;
  
  
  Is a self-hosted LLM worth it?
&lt;/h2&gt;

&lt;p&gt;It depends on the job, not on the team. Three questions settle most cases:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Must the data stay inside?&lt;/strong&gt; Contracts, health data, client documents under an NDA. If yes, self-hosting may be the only option, and cost is secondary.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Is the work steady or bursty?&lt;/strong&gt; A GPU costs the same idle as busy. Steady, high-volume work uses what you paid for. A few hundred requests a day, arriving in bursts, does not.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Does the job need judgement?&lt;/strong&gt; Embeddings, labelling, short summaries: small models do these well. Extracting the right number from a messy document, or deciding what matters: small models do these badly.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Here is what that looked like in my own system, &lt;a href="https://hikmahtechnologies.com/blog/why-aegis-moved-off-my-local-gpu/?utm_source=devto&amp;amp;utm_medium=referral&amp;amp;utm_campaign=blog" rel="noopener noreferrer"&gt;AEGIS&lt;/a&gt;. On a replay of 36 real financial emails with known answers, a hosted model extracted 32 correctly. A local 4B model got 13. The local model also averaged about 23 seconds a call against 2 seconds hosted. And the hosted bill for the whole agent fleet came to about five cents a day.&lt;/p&gt;

&lt;p&gt;So I moved the judgement work to a hosted API. I did not move the embeddings. The full numbers are in &lt;a href="https://hikmahtechnologies.com/blog/why-aegis-moved-off-my-local-gpu/?utm_source=devto&amp;amp;utm_medium=referral&amp;amp;utm_campaign=blog" rel="noopener noreferrer"&gt;why AEGIS moved off my local GPU&lt;/a&gt;; I won't repeat them here.&lt;/p&gt;

&lt;h2&gt;
  
  
  Hardware: how much VRAM a self-hosted LLM needs
&lt;/h2&gt;

&lt;p&gt;The rule is simple. &lt;strong&gt;The model, plus its context, has to fit in GPU memory.&lt;/strong&gt; If it does not, the server runs part of it on the CPU, and it slows down sharply.&lt;/p&gt;

&lt;p&gt;Model size depends on parameter count and quantisation (how many bits store each weight). From Ollama's &lt;a href="https://ollama.com/library/llama3.1/tags" rel="noopener noreferrer"&gt;Llama 3.1 tags page&lt;/a&gt;, checked on 1 October 2026:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Model&lt;/th&gt;
&lt;th&gt;Build&lt;/th&gt;
&lt;th&gt;Download size&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Llama 3.1 8B&lt;/td&gt;
&lt;td&gt;4-bit (q4_K_M), the default&lt;/td&gt;
&lt;td&gt;4.9 GB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Llama 3.1 8B&lt;/td&gt;
&lt;td&gt;8-bit (q8_0)&lt;/td&gt;
&lt;td&gt;8.5 GB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Llama 3.1 8B&lt;/td&gt;
&lt;td&gt;16-bit (fp16)&lt;/td&gt;
&lt;td&gt;16 GB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Llama 3.1 70B&lt;/td&gt;
&lt;td&gt;4-bit (q4_K_M)&lt;/td&gt;
&lt;td&gt;43 GB&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The context needs memory on top of the weights. Ollama's &lt;a href="https://github.com/ollama/ollama/blob/main/docs/faq.mdx" rel="noopener noreferrer"&gt;FAQ&lt;/a&gt; says required memory scales with the number of parallel requests times the context length. So four users with long prompts need much more than one.&lt;/p&gt;

&lt;p&gt;Ollama also picks a default context from your VRAM. Its &lt;a href="https://github.com/ollama/ollama/blob/main/docs/context-length.mdx" rel="noopener noreferrer"&gt;context length docs&lt;/a&gt; say: under 24 GiB of VRAM, 4k tokens; 24–48 GiB, 32k; 48 GiB or more, 256k. That matters for RAG, below.&lt;/p&gt;

&lt;p&gt;My reading of those numbers, as a guide and not a promise:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;8 GB card:&lt;/strong&gt; embeddings, and small models (up to about 8B at 4-bit) with short context. This is what I have.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;24 GB card:&lt;/strong&gt; 8B-class models with long context, or somewhat larger models at 4-bit.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;70B-class models:&lt;/strong&gt; about 43 GB of weights at 4-bit before any context, so two large cards or a workstation-class one.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Check the real split after you load a model. &lt;code&gt;ollama ps&lt;/code&gt; shows a &lt;code&gt;PROCESSOR&lt;/code&gt; column; you want &lt;code&gt;100% GPU&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;And plan for heat. My GPU runbook fires an alert above 95°C. The first step is to scale the Ollama service to zero and wait until the card is below 70°C before starting it again. A runaway batch of requests can hold a card at 100% long enough to get there.&lt;/p&gt;




&lt;p&gt;This is the first part. The full post — including the rest of the working details — is on my site: &lt;a href="https://hikmahtechnologies.com/blog/self-hosted-llm-for-a-small-team/?utm_source=devto&amp;amp;utm_medium=referral&amp;amp;utm_campaign=blog" rel="noopener noreferrer"&gt;Self-hosted LLM for a small team: what it takes, and when it pays&lt;/a&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>llm</category>
      <category>devops</category>
      <category>opensource</category>
    </item>
    <item>
      <title>Why DMARC fails when SPF and DKIM both pass</title>
      <dc:creator>Mohammed Arshad Ansari</dc:creator>
      <pubDate>Sun, 15 Nov 2026 13:27:00 +0000</pubDate>
      <link>https://dev.to/mohammed_arshadansari_f2/why-dmarc-fails-when-spf-and-dkim-both-pass-806</link>
      <guid>https://dev.to/mohammed_arshadansari_f2/why-dmarc-fails-when-spf-and-dkim-both-pass-806</guid>
      <description>&lt;p&gt;DMARC can fail even when SPF and DKIM both show "pass". The reason is alignment.&lt;/p&gt;

&lt;h2&gt;
  
  
  The short answer
&lt;/h2&gt;

&lt;p&gt;DMARC does not ask only "did SPF or DKIM pass?" It asks: "did one of them pass &lt;strong&gt;for the same domain the reader sees in the From address&lt;/strong&gt;?"&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;SPF alignment:&lt;/strong&gt; the domain in the envelope sender (also called Return-Path or MAIL FROM) must match the From domain.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;DKIM alignment:&lt;/strong&gt; the domain in the &lt;code&gt;d=&lt;/code&gt; tag of the DKIM signature must match the From domain.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;One aligned pass is enough. If both passes are for other domains, DMARC fails.&lt;/p&gt;

&lt;h2&gt;
  
  
  The usual cause
&lt;/h2&gt;

&lt;p&gt;A third-party service sends mail for you. It uses its own bounce domain for SPF and signs with its own DKIM domain. Both checks pass, but for the service's domain, not yours. The From address still shows your domain. Nothing aligns.&lt;/p&gt;

&lt;p&gt;Read the full post: &lt;a href="https://www.domainposture.com/blog/dmarc-fails-spf-dkim-pass" rel="noopener noreferrer"&gt;https://www.domainposture.com/blog/dmarc-fails-spf-dkim-pass&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;a href="https://www.domainposture.com/blog/dmarc-fails-spf-dkim-pass" rel="noopener noreferrer"&gt;https://www.domainposture.com/blog/dmarc-fails-spf-dkim-pass&lt;/a&gt;&lt;/p&gt;

</description>
    </item>
    <item>
      <title>Iceberg catalogs explained: REST, Glue, S3 Tables, Polaris, Nessie</title>
      <dc:creator>Mohammed Arshad Ansari</dc:creator>
      <pubDate>Thu, 12 Nov 2026 14:00:00 +0000</pubDate>
      <link>https://dev.to/mohammed_arshadansari_f2/iceberg-catalogs-explained-rest-glue-s3-tables-polaris-nessie-2ahg</link>
      <guid>https://dev.to/mohammed_arshadansari_f2/iceberg-catalogs-explained-rest-glue-s3-tables-polaris-nessie-2ahg</guid>
      <description>&lt;p&gt;"Iceberg catalog" sounds like a big piece of infrastructure: a metadata service, a governance layer, something with a vendor attached. The core job is much smaller. A catalog keeps &lt;strong&gt;one pointer per table&lt;/strong&gt;, to the table's current &lt;code&gt;metadata.json&lt;/code&gt;, and changes that pointer atomically when someone commits. Everything else about the table lives in files.&lt;/p&gt;

&lt;p&gt;Once you see that pointer, the choice between REST, Glue, S3 Tables, Polaris and Nessie gets easier. They differ in what they add around the pointer, and in which engines can reach them. I have not run Iceberg in production. To show the catalog concretely, I built a table on my laptop with PyIceberg 0.12.0 and DuckDB 1.5.6, using a SQL catalog on SQLite and then on a throwaway Postgres 16. The rows below are from those runs.&lt;/p&gt;

&lt;h2&gt;
  
  
  What is an Iceberg catalog?
&lt;/h2&gt;

&lt;p&gt;An Iceberg table is a folder of Parquet data files plus a tree of metadata. The top of the tree is a &lt;code&gt;metadata.json&lt;/code&gt; holding the schema, the partition spec and the list of snapshots. Every commit writes a &lt;strong&gt;new&lt;/strong&gt; &lt;code&gt;metadata.json&lt;/code&gt;. Nothing is edited in place. So after 50 commits there are 51 of them, and something has to say which one is current.&lt;/p&gt;

&lt;p&gt;That something is the catalog. Here is the entire state of PyIceberg's SQL catalog after I created one table and appended 50 batches to it. The catalog database held two tables:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;iceberg_tables
iceberg_namespace_properties
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;and &lt;code&gt;iceberg_tables&lt;/code&gt; held one row:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;catalog_name                lab
table_namespace             shop
table_name                  events
metadata_location           .../events/metadata/00050-c499c071-....metadata.json
previous_metadata_location  .../events/metadata/00049-bb259239-....metadata.json
iceberg_type                TABLE
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That is it. One row, two file paths. The table's schema, its 50 snapshots and its file lists are all in the 50 KB &lt;code&gt;metadata.json&lt;/code&gt; that row points to. The catalog holds a few hundred bytes.&lt;/p&gt;

&lt;h2&gt;
  
  
  A commit is a compare-and-swap
&lt;/h2&gt;

&lt;p&gt;When a writer commits, it writes new data and metadata files, then asks the catalog to move the pointer. The SQL catalog does it with one statement. Simplified from PyIceberg's source:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;UPDATE&lt;/span&gt; &lt;span class="n"&gt;iceberg_tables&lt;/span&gt;
&lt;span class="k"&gt;SET&lt;/span&gt; &lt;span class="n"&gt;metadata_location&lt;/span&gt;          &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'.../00051-....metadata.json'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;previous_metadata_location&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'.../00050-....metadata.json'&lt;/span&gt;
&lt;span class="k"&gt;WHERE&lt;/span&gt; &lt;span class="n"&gt;table_namespace&lt;/span&gt;   &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'shop'&lt;/span&gt;
  &lt;span class="k"&gt;AND&lt;/span&gt; &lt;span class="n"&gt;table_name&lt;/span&gt;        &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'events'&lt;/span&gt;
  &lt;span class="k"&gt;AND&lt;/span&gt; &lt;span class="n"&gt;metadata_location&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'.../00050-....metadata.json'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;  &lt;span class="c1"&gt;-- the version I started from&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If another writer committed in between, the &lt;code&gt;WHERE&lt;/code&gt; matches no row, and PyIceberg raises &lt;code&gt;CommitFailedException&lt;/code&gt;. I tested it on Postgres: two handles on the same table, both appending. The second append conflicted, PyIceberg retried 108 ms later on top of the new state, and both rows landed.&lt;/p&gt;

&lt;p&gt;That check is why the catalog exists. Object storage has no cheap way for two writers to agree on which file is "current". The catalog gives them one place to take turns.&lt;/p&gt;

&lt;p&gt;After a delete, a schema change, an append, a rewrite and a snapshot expiry, the same row pointed at &lt;code&gt;00055-....metadata.json&lt;/code&gt;, with &lt;code&gt;00054&lt;/code&gt; as the previous one. Every operation moved the pointer once.&lt;/p&gt;




&lt;p&gt;This is the first part. The full post — including the rest of the working details — is on my site: &lt;a href="https://hikmahtechnologies.com/blog/iceberg-catalog-explained/?utm_source=devto&amp;amp;utm_medium=referral&amp;amp;utm_campaign=blog" rel="noopener noreferrer"&gt;Iceberg catalogs explained: REST, Glue, S3 Tables, Polaris, Nessie&lt;/a&gt;&lt;/p&gt;

</description>
      <category>dataengineering</category>
      <category>database</category>
      <category>opensource</category>
      <category>data</category>
    </item>
    <item>
      <title>Snowflake warehouse sizes and what each one costs in credits</title>
      <dc:creator>Mohammed Arshad Ansari</dc:creator>
      <pubDate>Tue, 10 Nov 2026 14:00:01 +0000</pubDate>
      <link>https://dev.to/mohammed_arshadansari_f2/snowflake-warehouse-sizes-and-what-each-one-costs-in-credits-1ach</link>
      <guid>https://dev.to/mohammed_arshadansari_f2/snowflake-warehouse-sizes-and-what-each-one-costs-in-credits-1ach</guid>
      <description>&lt;p&gt;Snowflake warehouse size is the one setting that decides how fast your credits burn. An X-Small warehouse uses 1 credit an hour. Each size up doubles that, all the way to 512 credits an hour for a 6X-Large. Most people learn this from the first bill.&lt;/p&gt;

&lt;p&gt;But size is only half of the sum. The other half is &lt;strong&gt;how many hours the warehouse is awake&lt;/strong&gt;, and that is where most Snowflake bills go wrong. This post gives you the full size table, what a credit costs, worked monthly figures, and a way to check your own account before you change anything.&lt;/p&gt;

&lt;p&gt;A note on sources. I have no live Snowflake account to write this from. Every rate below is from Snowflake's own documentation or its &lt;a href="https://www.snowflake.com/legal-files/CreditConsumptionTable.pdf" rel="noopener noreferrer"&gt;Service Consumption Table&lt;/a&gt;, the version effective &lt;strong&gt;30 September 2026&lt;/strong&gt;, which I read on 1 October 2026. Prices change, and contracts differ. Check the current table and your order form.&lt;/p&gt;

&lt;h2&gt;
  
  
  Snowflake warehouse sizes and credits per hour
&lt;/h2&gt;

&lt;p&gt;From Snowflake's &lt;a href="https://docs.snowflake.com/en/user-guide/warehouses-overview" rel="noopener noreferrer"&gt;warehouse overview&lt;/a&gt;, for a standard Gen1 warehouse:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Size&lt;/th&gt;
&lt;th&gt;Credits / hour&lt;/th&gt;
&lt;th&gt;At $3.00 / credit&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;X-Small&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;td&gt;$3 / hr&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Small&lt;/td&gt;
&lt;td&gt;2&lt;/td&gt;
&lt;td&gt;$6 / hr&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Medium&lt;/td&gt;
&lt;td&gt;4&lt;/td&gt;
&lt;td&gt;$12 / hr&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Large&lt;/td&gt;
&lt;td&gt;8&lt;/td&gt;
&lt;td&gt;$24 / hr&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;X-Large&lt;/td&gt;
&lt;td&gt;16&lt;/td&gt;
&lt;td&gt;$48 / hr&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2X-Large&lt;/td&gt;
&lt;td&gt;32&lt;/td&gt;
&lt;td&gt;$96 / hr&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;3X-Large&lt;/td&gt;
&lt;td&gt;64&lt;/td&gt;
&lt;td&gt;$192 / hr&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;4X-Large&lt;/td&gt;
&lt;td&gt;128&lt;/td&gt;
&lt;td&gt;$384 / hr&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;5X-Large&lt;/td&gt;
&lt;td&gt;256&lt;/td&gt;
&lt;td&gt;$768 / hr&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;6X-Large&lt;/td&gt;
&lt;td&gt;512&lt;/td&gt;
&lt;td&gt;$1,536 / hr&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The $3.00 column is the Enterprise on-demand list price in AWS US East, used here only so the doubling is easy to see.&lt;/p&gt;

&lt;p&gt;Three billing rules sit under that table:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Per second, with a 60-second minimum&lt;/strong&gt; each time the warehouse starts or resumes. A 5-second query that wakes a warehouse bills a full minute.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Suspended warehouses use no credits.&lt;/strong&gt; The meter runs while the warehouse is awake, not while queries run.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;5X-Large and 6X-Large are not in every region.&lt;/strong&gt; The consumption table says they "may not be GA in all Regions".&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Gen2 warehouses cost more per hour
&lt;/h3&gt;

&lt;p&gt;This one catches people out. Snowflake's &lt;a href="https://docs.snowflake.com/en/user-guide/warehouses-gen2" rel="noopener noreferrer"&gt;Gen2 documentation&lt;/a&gt; says that if you create a standard warehouse without a &lt;code&gt;GENERATION&lt;/code&gt; clause, you get Gen2, unless Gen2 is not available in your region. The consumption table prices Gen2 at a higher rate:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Size&lt;/th&gt;
&lt;th&gt;Gen1&lt;/th&gt;
&lt;th&gt;Gen2 on AWS / GCP&lt;/th&gt;
&lt;th&gt;Gen2 on Azure&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;X-Small&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;td&gt;1.35&lt;/td&gt;
&lt;td&gt;1.25&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Medium&lt;/td&gt;
&lt;td&gt;4&lt;/td&gt;
&lt;td&gt;5.4&lt;/td&gt;
&lt;td&gt;5&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Large&lt;/td&gt;
&lt;td&gt;8&lt;/td&gt;
&lt;td&gt;10.8&lt;/td&gt;
&lt;td&gt;10&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Gen2 is meant to finish analytics and data engineering work faster, so per query it may cost less. I can't verify how much faster on your workload, and neither can a table. What I can say: if your bill moved and you did not change a size, check which generation your newer warehouses are on.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="no"&gt;SHOW&lt;/span&gt; &lt;span class="no"&gt;WAREHOUSES&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;   &lt;span class="c1"&gt;-- the size, auto_suspend and generation columns are the ones to read&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  How much is a Snowflake credit in dollars?
&lt;/h2&gt;

&lt;p&gt;It depends on four things: your &lt;strong&gt;edition&lt;/strong&gt;, your &lt;strong&gt;cloud&lt;/strong&gt;, your &lt;strong&gt;region&lt;/strong&gt;, and whether you pay &lt;strong&gt;on demand or by capacity contract&lt;/strong&gt;. Snowflake's &lt;a href="https://www.snowflake.com/en/pricing-options/" rel="noopener noreferrer"&gt;pricing page&lt;/a&gt; does not print a single number for this reason. It points to the consumption table.&lt;/p&gt;

&lt;p&gt;On-demand prices from the table effective 30 September 2026:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Region&lt;/th&gt;
&lt;th&gt;Standard&lt;/th&gt;
&lt;th&gt;Enterprise&lt;/th&gt;
&lt;th&gt;Business Critical&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;AWS US East (N. Virginia)&lt;/td&gt;
&lt;td&gt;$2.00&lt;/td&gt;
&lt;td&gt;$3.00&lt;/td&gt;
&lt;td&gt;$4.00&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;AWS US West (Oregon)&lt;/td&gt;
&lt;td&gt;$2.00&lt;/td&gt;
&lt;td&gt;$3.00&lt;/td&gt;
&lt;td&gt;$4.00&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;AWS EU Dublin&lt;/td&gt;
&lt;td&gt;$2.60&lt;/td&gt;
&lt;td&gt;$3.90&lt;/td&gt;
&lt;td&gt;$5.20&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;AWS EU Frankfurt&lt;/td&gt;
&lt;td&gt;$2.60&lt;/td&gt;
&lt;td&gt;$3.90&lt;/td&gt;
&lt;td&gt;$5.20&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;AWS AP Sydney&lt;/td&gt;
&lt;td&gt;$2.75&lt;/td&gt;
&lt;td&gt;$4.05&lt;/td&gt;
&lt;td&gt;$5.50&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Capacity contracts, paid up front, can bring the price down. The discount is in your order form, not in any public table. If you are on capacity, use your contracted rate in every sum below.&lt;/p&gt;

&lt;p&gt;There is one more line most people never see: &lt;strong&gt;cloud services&lt;/strong&gt;. The table charges it at 4.4 credits an hour, but only for the part of a day's cloud services use that exceeds 10% of that day's warehouse credits. For most accounts it is zero or close to it. If it is not, something unusual is running: very many tiny queries, or heavy metadata work.&lt;/p&gt;




&lt;p&gt;This is the first part. The full post — including the rest of the working details — is on my site: &lt;a href="https://hikmahtechnologies.com/blog/snowflake-warehouse-sizes-and-credit-costs/?utm_source=devto&amp;amp;utm_medium=referral&amp;amp;utm_campaign=blog" rel="noopener noreferrer"&gt;Snowflake warehouse sizes and what each one costs in credits&lt;/a&gt;&lt;/p&gt;

</description>
      <category>dataengineering</category>
      <category>database</category>
      <category>sql</category>
      <category>data</category>
    </item>
    <item>
      <title>Postgres MCP server: which one to run, and how to make it read-only</title>
      <dc:creator>Mohammed Arshad Ansari</dc:creator>
      <pubDate>Fri, 06 Nov 2026 14:00:00 +0000</pubDate>
      <link>https://dev.to/mohammed_arshadansari_f2/postgres-mcp-server-which-one-to-run-and-how-to-make-it-read-only-ifp</link>
      <guid>https://dev.to/mohammed_arshadansari_f2/postgres-mcp-server-which-one-to-run-and-how-to-make-it-read-only-ifp</guid>
      <description>&lt;p&gt;A Postgres MCP server is a small program that lets an AI client such as Claude Code or Cursor list your tables and run SQL against your database. Search for "postgres mcp" and you will find a dozen of them. The one most tutorials still point at is archived, and its read-only mode could be escaped with one word: &lt;code&gt;COMMIT&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;So the useful question is not "which Postgres MCP server?" It is &lt;strong&gt;where does the read-only guarantee actually live?&lt;/strong&gt; My answer, after building an MCP server of my own: in the database role. The server is a convenience on top.&lt;/p&gt;

&lt;h2&gt;
  
  
  Is there an official Postgres MCP server?
&lt;/h2&gt;

&lt;p&gt;There used to be. The Model Context Protocol project shipped a reference Postgres server as &lt;code&gt;@modelcontextprotocol/server-postgres&lt;/code&gt;. It is now in &lt;a href="https://github.com/modelcontextprotocol/servers-archived/tree/main/src/postgres" rel="noopener noreferrer"&gt;modelcontextprotocol/servers-archived&lt;/a&gt;, a repository whose own description is "Reference MCP servers that are no longer maintained". The active &lt;a href="https://github.com/modelcontextprotocol/servers" rel="noopener noreferrer"&gt;modelcontextprotocol/servers&lt;/a&gt; repository keeps seven reference servers today (everything, fetch, filesystem, git, memory, sequentialthinking, time). Postgres is not one of them.&lt;/p&gt;

&lt;p&gt;Its read-only mode worked like this, from its source:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;query&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;BEGIN TRANSACTION READ ONLY&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;result&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;query&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;sql&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;   &lt;span class="c1"&gt;// the model's SQL, as one string&lt;/span&gt;
&lt;span class="c1"&gt;// ...&lt;/span&gt;
&lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;query&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;ROLLBACK&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The model's SQL went in as a single string, and the Node client accepts several statements separated by semicolons. &lt;a href="https://securitylabs.datadoghq.com/articles/mcp-vulnerability-case-study-SQL-injection-in-the-postgresql-mcp-server/" rel="noopener noreferrer"&gt;Datadog Security Labs showed in August 2025&lt;/a&gt; that &lt;code&gt;COMMIT; DROP SCHEMA public CASCADE;&lt;/code&gt; ends the read-only transaction and then runs the drop with whatever rights the connection has. The same write-up says the server was deprecated on 10 July 2025 and archived on GitHub, npm and Docker Hub, and that the npm package still had about 21,000 weekly downloads.&lt;/p&gt;

&lt;p&gt;That is the whole lesson in one bug. A read-only flag in the server is a promise the server makes. A role without write grants is a fact the database enforces.&lt;/p&gt;

&lt;h2&gt;
  
  
  Which Postgres MCP server to run instead
&lt;/h2&gt;

&lt;p&gt;The maintained option I would start with is &lt;a href="https://github.com/crystaldba/postgres-mcp" rel="noopener noreferrer"&gt;Postgres MCP Pro&lt;/a&gt; by Crystal DBA. It is MIT-licensed. I checked the repository on 1 October 2026: the last tagged release is v0.3.0 (May 2025), and the last commit to main was on 16 August 2026.&lt;/p&gt;

&lt;p&gt;It gives the model nine tools, among them &lt;code&gt;list_schemas&lt;/code&gt;, &lt;code&gt;list_objects&lt;/code&gt;, &lt;code&gt;get_object_details&lt;/code&gt;, &lt;code&gt;execute_sql&lt;/code&gt;, &lt;code&gt;explain_query&lt;/code&gt;, &lt;code&gt;get_top_queries&lt;/code&gt; (from &lt;code&gt;pg_stat_statements&lt;/code&gt;) and &lt;code&gt;analyze_db_health&lt;/code&gt;. Two details matter more than the tool list:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;It has two access modes, and the default is &lt;code&gt;unrestricted&lt;/code&gt;&lt;/strong&gt;, which allows writes and schema changes. You have to pass &lt;code&gt;--access-mode=restricted&lt;/code&gt; yourself.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Restricted mode parses each statement&lt;/strong&gt; with &lt;code&gt;pglast&lt;/code&gt; and allows only safe statement types, and wraps execution in a 30-second timeout (in &lt;code&gt;server.py&lt;/code&gt;, &lt;code&gt;SafeSqlDriver(sql_driver=base_driver, timeout=30)&lt;/code&gt;). That timeout runs in the client. I would still set one in Postgres, below, so the database stops the query even if the client goes away.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The README is candid about the limits. It says restricted mode is "conservative" and that "we may add measures in the future to make sure that restricted mode is safe to use with production databases". Read that as: use it with a read-only role, not instead of one.&lt;/p&gt;

&lt;p&gt;Other servers exist, including Supabase's own and several "read-only by default" projects. I have not run them, so I won't rank them. The test I would apply to any of them is the same: does it still hold if the server's own check is bypassed?&lt;/p&gt;




&lt;p&gt;This is the first part. The full post — including the rest of the working details — is on my site: &lt;a href="https://hikmahtechnologies.com/blog/postgres-mcp-server/?utm_source=devto&amp;amp;utm_medium=referral&amp;amp;utm_campaign=blog" rel="noopener noreferrer"&gt;Postgres MCP server: which one to run, and how to make it read-only&lt;/a&gt;&lt;/p&gt;

</description>
      <category>mcp</category>
      <category>ai</category>
      <category>database</category>
      <category>llm</category>
    </item>
    <item>
      <title>Postgres and Iceberg: pg_lake, pg_duckdb and the Postgres lakehouse</title>
      <dc:creator>Mohammed Arshad Ansari</dc:creator>
      <pubDate>Wed, 04 Nov 2026 14:00:00 +0000</pubDate>
      <link>https://dev.to/mohammed_arshadansari_f2/postgres-and-iceberg-pglake-pgduckdb-and-the-postgres-lakehouse-2hcp</link>
      <guid>https://dev.to/mohammed_arshadansari_f2/postgres-and-iceberg-pglake-pgduckdb-and-the-postgres-lakehouse-2hcp</guid>
      <description>&lt;p&gt;"Postgres Iceberg" covers four different jobs. You might want to &lt;strong&gt;query&lt;/strong&gt; Iceberg or Parquet files on S3 from Postgres. You might want Postgres to &lt;strong&gt;write&lt;/strong&gt; Iceberg tables. You might want Postgres to be the &lt;strong&gt;catalog&lt;/strong&gt; that other engines use. Or you might want your Postgres data to &lt;strong&gt;land in Iceberg&lt;/strong&gt; for a warehouse to read. Each job has a different tool, and two of the tools people still recommend are no longer maintained.&lt;/p&gt;

&lt;p&gt;This post sorts them out. I have not run pg_lake or Iceberg in production. What I did run, on my laptop: PyIceberg 0.12.0 with a plain Postgres 16 container as the Iceberg catalog, and the pg_duckdb Docker image (Postgres 18.1) reading the same table. The numbers below come from those runs. Everything else links to its source, checked on 1 October 2026.&lt;/p&gt;

&lt;h2&gt;
  
  
  Postgres and Iceberg: the options at a glance
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Option&lt;/th&gt;
&lt;th&gt;Job&lt;/th&gt;
&lt;th&gt;Licence&lt;/th&gt;
&lt;th&gt;Latest release&lt;/th&gt;
&lt;th&gt;Status&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://github.com/Snowflake-Labs/pg_lake" rel="noopener noreferrer"&gt;pg_lake&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;Write and read Iceberg, Postgres as catalog, COPY to/from S3&lt;/td&gt;
&lt;td&gt;Apache-2.0&lt;/td&gt;
&lt;td&gt;v3.5.3, 2026-09-24&lt;/td&gt;
&lt;td&gt;Active&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://github.com/duckdb/pg_duckdb" rel="noopener noreferrer"&gt;pg_duckdb&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;Read Parquet, CSV, JSON, Iceberg, Delta; fast analytics&lt;/td&gt;
&lt;td&gt;MIT&lt;/td&gt;
&lt;td&gt;v1.1.1, 2025-12-18&lt;/td&gt;
&lt;td&gt;Active, PG 14 to 18&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;PyIceberg / Iceberg JDBC SQL catalog&lt;/td&gt;
&lt;td&gt;Plain Postgres as the catalog only&lt;/td&gt;
&lt;td&gt;Apache-2.0&lt;/td&gt;
&lt;td&gt;PyIceberg 0.12.0&lt;/td&gt;
&lt;td&gt;Active&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Snowflake Postgres&lt;/td&gt;
&lt;td&gt;Managed Postgres with pg_lake&lt;/td&gt;
&lt;td&gt;Commercial&lt;/td&gt;
&lt;td&gt;GA 2026-02-24&lt;/td&gt;
&lt;td&gt;Active&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Supabase Analytics Buckets + Supabase ETL&lt;/td&gt;
&lt;td&gt;CDC from Postgres into Iceberg on S3 Tables&lt;/td&gt;
&lt;td&gt;ETL is Apache-2.0&lt;/td&gt;
&lt;td&gt;—&lt;/td&gt;
&lt;td&gt;Public alpha&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Databricks Lakebase&lt;/td&gt;
&lt;td&gt;Managed Postgres, sync to the lakehouse&lt;/td&gt;
&lt;td&gt;Commercial&lt;/td&gt;
&lt;td&gt;GA on AWS 2026-02-03&lt;/td&gt;
&lt;td&gt;Sync is beta, Delta only&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;pg_mooncake&lt;/td&gt;
&lt;td&gt;"Real-time analytics on Postgres tables"&lt;/td&gt;
&lt;td&gt;MIT&lt;/td&gt;
&lt;td&gt;v0.1.3&lt;/td&gt;
&lt;td&gt;Dormant: team joined Databricks&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;ParadeDB pg_analytics&lt;/td&gt;
&lt;td&gt;DuckDB-backed lake queries&lt;/td&gt;
&lt;td&gt;PostgreSQL&lt;/td&gt;
&lt;td&gt;—&lt;/td&gt;
&lt;td&gt;Archived 2025-03-19&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The short version: &lt;strong&gt;pg_lake if Postgres should own Iceberg tables, pg_duckdb if Postgres only needs to read the lake&lt;/strong&gt;, and a managed product if you want someone else to run the plumbing.&lt;/p&gt;

&lt;h2&gt;
  
  
  pg_lake: Postgres that writes Iceberg
&lt;/h2&gt;

&lt;p&gt;pg_lake began as Crunchy Data Warehouse. Snowflake bought Crunchy Data in June 2025 and open-sourced the code as pg_lake in November 2025. Per its &lt;a href="https://github.com/Snowflake-Labs/pg_lake" rel="noopener noreferrer"&gt;README&lt;/a&gt;, it lets you:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Create and change Iceberg tables from Postgres, inside ordinary transactions.&lt;/li&gt;
&lt;li&gt;Query Parquet, CSV, JSON and Iceberg files in object storage.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;COPY&lt;/code&gt; query results out to S3 as Parquet, CSV or JSON, and back in.
&lt;/li&gt;
&lt;/ul&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;CREATE&lt;/span&gt; &lt;span class="n"&gt;EXTENSION&lt;/span&gt; &lt;span class="n"&gt;pg_lake&lt;/span&gt; &lt;span class="k"&gt;CASCADE&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="k"&gt;CREATE&lt;/span&gt; &lt;span class="k"&gt;TABLE&lt;/span&gt; &lt;span class="n"&gt;events&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;id&lt;/span&gt; &lt;span class="nb"&gt;bigint&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;ts&lt;/span&gt; &lt;span class="n"&gt;timestamptz&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;kind&lt;/span&gt; &lt;span class="nb"&gt;text&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;USING&lt;/span&gt; &lt;span class="n"&gt;iceberg&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="k"&gt;INSERT&lt;/span&gt; &lt;span class="k"&gt;INTO&lt;/span&gt; &lt;span class="n"&gt;events&lt;/span&gt; &lt;span class="k"&gt;VALUES&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;now&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt; &lt;span class="s1"&gt;'view'&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;

&lt;span class="k"&gt;COPY&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="k"&gt;FROM&lt;/span&gt; &lt;span class="n"&gt;events&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;TO&lt;/span&gt; &lt;span class="s1"&gt;'s3://bucket/exports/events.parquet'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The design has two parts. Postgres runs the pg_lake extensions. A separate process, &lt;code&gt;pgduck_server&lt;/code&gt;, runs DuckDB and speaks the Postgres wire protocol locally. pg_lake hands scans and computation to it. You don't query it directly, but you do deploy and size it: it defaults to 80% of system memory, per the README.&lt;/p&gt;

&lt;p&gt;From the &lt;a href="https://github.com/Snowflake-Labs/pg_lake/blob/main/docs/iceberg-tables.md" rel="noopener noreferrer"&gt;Iceberg tables doc&lt;/a&gt;:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Postgres is the catalog.&lt;/strong&gt; Commits update pg_lake's catalog tables in the same transaction as the data change. An &lt;code&gt;iceberg_tables&lt;/code&gt; view lets Spark, PyIceberg and iceberg-rust find the tables through their JDBC or SQL catalogs.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;REST catalogs work too.&lt;/strong&gt; A table can live in an external REST catalog, so other engines see it where they already look.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Compaction is built in.&lt;/strong&gt; &lt;code&gt;VACUUM&lt;/code&gt; merges small files and removes expired ones, and autovacuum runs it every 10 minutes.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Some SQL is not supported&lt;/strong&gt; on Iceberg tables: &lt;code&gt;MERGE&lt;/code&gt;, &lt;code&gt;INSERT ... ON CONFLICT&lt;/code&gt;, &lt;code&gt;SELECT ... FOR UPDATE&lt;/code&gt;. An update or delete locks the table, so only one runs at a time.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Other engines read, Postgres writes.&lt;/strong&gt; External tools cannot yet write to tables pg_lake created. Tables attached from a REST catalog are read-only in pg_lake.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The managed form is &lt;a href="https://docs.snowflake.com/en/user-guide/snowflake-postgres/postgres-pg_lake" rel="noopener noreferrer"&gt;Snowflake Postgres&lt;/a&gt;, generally available since 24 February 2026. Since 14 July 2026 a &lt;a href="https://docs.snowflake.com/en/release-notes/2026/other/2026-07-14-snowflake-postgres-catalog-integration-ga" rel="noopener noreferrer"&gt;catalog integration&lt;/a&gt; lets Snowflake discover those tables and read them as read-only Iceberg tables.&lt;/p&gt;

&lt;p&gt;I did not run pg_lake for this post. Its Docker setup builds the images locally rather than pulling a ready one, and I chose to test the parts I could run honestly in an afternoon.&lt;/p&gt;




&lt;p&gt;This is the first part. The full post — including the rest of the working details — is on my site: &lt;a href="https://hikmahtechnologies.com/blog/postgres-iceberg-pg-lake/?utm_source=devto&amp;amp;utm_medium=referral&amp;amp;utm_campaign=blog" rel="noopener noreferrer"&gt;Postgres and Iceberg: pg_lake, pg_duckdb and the Postgres lakehouse&lt;/a&gt;&lt;/p&gt;

</description>
      <category>dataengineering</category>
      <category>database</category>
      <category>sql</category>
      <category>opensource</category>
    </item>
    <item>
      <title>Iceberg vs Parquet: do you need a table format?</title>
      <dc:creator>Mohammed Arshad Ansari</dc:creator>
      <pubDate>Mon, 02 Nov 2026 14:00:00 +0000</pubDate>
      <link>https://dev.to/mohammed_arshadansari_f2/iceberg-vs-parquet-do-you-need-a-table-format-4pba</link>
      <guid>https://dev.to/mohammed_arshadansari_f2/iceberg-vs-parquet-do-you-need-a-table-format-4pba</guid>
      <description>&lt;p&gt;"Apache Iceberg vs Parquet" sounds like a choice between two file formats. It isn't. An Iceberg table stores its data in Parquet files. Iceberg is a &lt;strong&gt;table format&lt;/strong&gt;: a set of metadata files that say which Parquet files make up the table right now, what the schema is, and what the table looked like at every past commit.&lt;/p&gt;

&lt;p&gt;So the useful question is: &lt;strong&gt;do your Parquet files need a table format on top?&lt;/strong&gt; For many small teams the honest answer is no. A note on where this comes from: I run plain Parquet with DuckDB in production. I have not run Iceberg in production. To get numbers instead of opinions, I built Iceberg tables on my laptop with PyIceberg 0.12.0, PyArrow 25.0.1 and DuckDB 1.5.6, on the local filesystem with a SQLite catalog. Every figure below comes from those runs.&lt;/p&gt;

&lt;h2&gt;
  
  
  Apache Iceberg vs Parquet: a file and a table
&lt;/h2&gt;

&lt;p&gt;A Parquet file is self-contained. It knows its own columns, types and row groups, and nothing about any other file. A "table" made of plain Parquet is a folder and a naming convention. The engine lists the folder and reads what it finds.&lt;/p&gt;

&lt;p&gt;An Iceberg table adds three layers above the same kind of files. After 50 small appends, my test table looked like this on disk:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;warehouse/shop/events/
  data/
    00000-0-061edf85-....parquet          50 Parquet files, 450 KB
  metadata/
    00050-c499c071-....metadata.json      the current table state (50 KB)
    snap-616794116452149308-....avro      manifest list: one per snapshot
    ....avro                              manifests: lists of data files + stats
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Reading goes top down. The catalog points at the current &lt;code&gt;metadata.json&lt;/code&gt;. That file holds the schema, the partition spec and the list of snapshots. Each snapshot points at a manifest list, which points at manifests, which list the data files with per-column statistics. A commit writes new files and then swaps the catalog pointer in one step. That swap is what makes a commit atomic.&lt;/p&gt;

&lt;p&gt;The format is versioned. The &lt;a href="https://github.com/apache/iceberg/blob/main/format/spec.md" rel="noopener noreferrer"&gt;Iceberg spec&lt;/a&gt; has versions 1 to 3 adopted and version 4 in development. Version 3 adds a variant type, nanosecond timestamps, geospatial types, default values, deletion vectors and row lineage. The Java reference implementation reached 1.12.0 on 30 September 2026.&lt;/p&gt;

&lt;h2&gt;
  
  
  What plain Parquet already gives you
&lt;/h2&gt;

&lt;p&gt;My book's position is that plain Parquet plus DuckDB is enough for a lot of real work, and I still hold it. Each file carries its own schema, so DuckDB can read a folder whose files gained a column over time:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="k"&gt;FROM&lt;/span&gt; &lt;span class="n"&gt;read_parquet&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'data/*.parquet'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;union_by_name&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;true&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="c1"&gt;-- files written before the new column return NULL for it&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That handles the most common change, adding a column. The book is just as plain about where it stops. Renames, type changes and dropped columns still break consumers. There is no versioning, no time travel and no atomic commit across many files. Its rule: for single-writer, append-only work, plain Parquet. For multi-writer warehouses, reach for a table format.&lt;/p&gt;




&lt;p&gt;This is the first part. The full post — including the rest of the working details — is on my site: &lt;a href="https://hikmahtechnologies.com/blog/iceberg-vs-parquet/?utm_source=devto&amp;amp;utm_medium=referral&amp;amp;utm_campaign=blog" rel="noopener noreferrer"&gt;Iceberg vs Parquet: do you need a table format?&lt;/a&gt;&lt;/p&gt;

</description>
      <category>dataengineering</category>
      <category>database</category>
      <category>data</category>
      <category>analytics</category>
    </item>
    <item>
      <title>Is Dagster free? Self-hosting Dagster vs Dagster+</title>
      <dc:creator>Mohammed Arshad Ansari</dc:creator>
      <pubDate>Wed, 28 Oct 2026 14:00:00 +0000</pubDate>
      <link>https://dev.to/mohammed_arshadansari_f2/is-dagster-free-self-hosting-dagster-vs-dagster-4i8p</link>
      <guid>https://dev.to/mohammed_arshadansari_f2/is-dagster-free-self-hosting-dagster-vs-dagster-4i8p</guid>
      <description>&lt;p&gt;Yes, Dagster is free. The open-source core is Apache 2.0, and you can run all of it on your own servers without paying Dagster Labs anything. That answers the licence question. It doesn't answer the budget question.&lt;/p&gt;

&lt;p&gt;The budget question is whether you'd rather pay Dagster Labs or pay with your own hours. I self-host a production Dagster instance on a homelab Docker Swarm. It runs 112 assets, 57 schedules and 3 sensors for one person, me. Here is what the paid product costs today, what self-hosting has cost me in time, and a minimal setup you can start from.&lt;/p&gt;

&lt;h2&gt;
  
  
  Is Dagster free? What open source gives you
&lt;/h2&gt;

&lt;p&gt;The open-source packages give you the whole orchestrator: assets, partitions, schedules, sensors, asset checks, the run queue and the web UI. Nothing in my pipeline needed a paid feature. The &lt;a href="https://github.com/dagster-io/dagster" rel="noopener noreferrer"&gt;Dagster repository&lt;/a&gt; is licensed Apache-2.0.&lt;/p&gt;

&lt;p&gt;What you take on is running it. A self-hosted Dagster has four parts:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Postgres&lt;/strong&gt; for run history, event logs and schedule state. SQLite works on a laptop; I use it for local development only.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Code locations&lt;/strong&gt;: containers that load your definitions and serve them over gRPC.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;dagster-webserver&lt;/strong&gt;: the UI and the GraphQL API.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;dagster-daemon&lt;/strong&gt;: schedules, sensors, the run queue, retries and run monitoring.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;None of these is hard to start. Keeping them healthy for a year is the real work.&lt;/p&gt;

&lt;h2&gt;
  
  
  What Dagster+ costs
&lt;/h2&gt;

&lt;p&gt;I checked &lt;a href="https://dagster.io/pricing" rel="noopener noreferrer"&gt;dagster.io/pricing&lt;/a&gt; on &lt;strong&gt;1 October 2026&lt;/strong&gt;:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Plan&lt;/th&gt;
&lt;th&gt;Price&lt;/th&gt;
&lt;th&gt;Includes&lt;/th&gt;
&lt;th&gt;Per credit&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Solo&lt;/td&gt;
&lt;td&gt;$10 / month&lt;/td&gt;
&lt;td&gt;1 user, 1 code location, 1 deployment&lt;/td&gt;
&lt;td&gt;$0.040&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Starter&lt;/td&gt;
&lt;td&gt;$100 / month&lt;/td&gt;
&lt;td&gt;up to 3 users, 5 code locations, 1 deployment&lt;/td&gt;
&lt;td&gt;$0.035&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Pro&lt;/td&gt;
&lt;td&gt;contact sales&lt;/td&gt;
&lt;td&gt;unlimited seats, code locations and deployments&lt;/td&gt;
&lt;td&gt;custom&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Serverless compute was $0.010 a minute on Solo and Starter. Solo and Starter had a 30-day free trial. The page defines a credit as &lt;strong&gt;one asset materialization or one op execution&lt;/strong&gt;, and says a Hybrid deployment, where your own infrastructure runs the code, has no compute charge. There is no free tier on the page, only the trial.&lt;/p&gt;

&lt;p&gt;To see what credits mean for you, count materializations. My pipeline has 112 assets, 36 of them partitioned (34 daily). Every daily partition of every daily asset is a credit, every day, plus every retry. Do that sum for your own graph before you assume a plan fits. I haven't priced my pipeline on Dagster+, so I won't guess my number.&lt;/p&gt;




&lt;p&gt;This is the first part. The full post — including the rest of the working details — is on my site: &lt;a href="https://hikmahtechnologies.com/blog/is-dagster-free-self-hosting/?utm_source=devto&amp;amp;utm_medium=referral&amp;amp;utm_campaign=blog" rel="noopener noreferrer"&gt;Is Dagster free? Self-hosting Dagster vs Dagster+&lt;/a&gt;&lt;/p&gt;

</description>
      <category>dataengineering</category>
      <category>devops</category>
      <category>opensource</category>
      <category>python</category>
    </item>
    <item>
      <title>Querying Parquet on S3 with DuckDB: secrets, speed and the settings that matter</title>
      <dc:creator>Mohammed Arshad Ansari</dc:creator>
      <pubDate>Mon, 26 Oct 2026 14:00:00 +0000</pubDate>
      <link>https://dev.to/mohammed_arshadansari_f2/querying-parquet-on-s3-with-duckdb-secrets-speed-and-the-settings-that-matter-3gij</link>
      <guid>https://dev.to/mohammed_arshadansari_f2/querying-parquet-on-s3-with-duckdb-secrets-speed-and-the-settings-that-matter-3gij</guid>
      <description>&lt;p&gt;DuckDB reads Parquet on S3 in one line, and that part is not the problem. The problem shows up a month later, when a query that took two seconds on your laptop takes forty against the bucket and nobody can say why.&lt;/p&gt;

&lt;p&gt;The answer is almost never bandwidth. &lt;strong&gt;It is the number of requests.&lt;/strong&gt; Object storage charges you a round trip for every file you open and every column chunk you fetch. A layout that costs nothing on local disk can mean tens of thousands of round trips on S3. This post covers credentials first, because they are what people search for, and then the layout and settings that decide speed.&lt;/p&gt;

&lt;p&gt;One honest note. My own production pipeline reads Parquet from an NFS share on a NAS, not from S3. The credential and request-level detail here comes from DuckDB's documentation and from my book. The many-files failure near the end is mine, from that NFS pipeline, and it would have hit the same way on S3.&lt;/p&gt;

&lt;h2&gt;
  
  
  Can DuckDB read from S3?
&lt;/h2&gt;

&lt;p&gt;Yes. The &lt;code&gt;httpfs&lt;/code&gt; extension handles S3, and DuckDB &lt;a href="https://duckdb.org/docs/current/core_extensions/httpfs/overview.html" rel="noopener noreferrer"&gt;autoloads it&lt;/a&gt; the first time you touch an &lt;code&gt;s3://&lt;/code&gt; path. You don't need to install or load anything by hand, though doing so in production scripts makes the dependency explicit.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="k"&gt;count&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;FROM&lt;/span&gt; &lt;span class="n"&gt;read_parquet&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'s3://analytics/events/dt=2026-09-*/*.parquet'&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;For Parquet, DuckDB does not download whole files. Its docs describe &lt;a href="https://duckdb.org/docs/current/core_extensions/httpfs/https.html" rel="noopener noreferrer"&gt;partial reading&lt;/a&gt;: it reads the file's metadata, then uses HTTP range requests to fetch only the parts the query needs. A &lt;code&gt;count(*)&lt;/code&gt; can be answered from metadata alone. That is why Parquet on S3 works at all, and also why the number of files matters so much: every file needs its own metadata read before anything else happens.&lt;/p&gt;

&lt;h2&gt;
  
  
  Setting S3 credentials: CREATE SECRET
&lt;/h2&gt;

&lt;p&gt;DuckDB manages credentials as &lt;strong&gt;secrets&lt;/strong&gt;. The &lt;a href="https://duckdb.org/docs/current/core_extensions/httpfs/s3api.html" rel="noopener noreferrer"&gt;S3 docs&lt;/a&gt; show two providers.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;&lt;code&gt;credential_chain&lt;/code&gt;&lt;/strong&gt; asks the AWS SDK for credentials, the same way the AWS CLI would. That covers environment variables, profiles, SSO, assumed roles and instance or container roles. Use it on servers, so no key ever sits in your code:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;CREATE&lt;/span&gt; &lt;span class="k"&gt;OR&lt;/span&gt; &lt;span class="k"&gt;REPLACE&lt;/span&gt; &lt;span class="n"&gt;SECRET&lt;/span&gt; &lt;span class="n"&gt;aws&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="k"&gt;TYPE&lt;/span&gt; &lt;span class="n"&gt;s3&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;PROVIDER&lt;/span&gt; &lt;span class="n"&gt;credential_chain&lt;/span&gt;
&lt;span class="p"&gt;);&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;&lt;code&gt;config&lt;/code&gt;&lt;/strong&gt; takes the values directly. Use it when the keys come from somewhere else, such as a secrets manager your app already reads. Pass them in from code rather than typing them into SQL:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;os&lt;/span&gt;
&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;duckdb&lt;/span&gt;

&lt;span class="n"&gt;con&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;duckdb&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;connect&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="n"&gt;con&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;CREATE OR REPLACE SECRET aws &lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;(TYPE s3, PROVIDER config, KEY_ID ?, SECRET ?, REGION &lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;eu-west-2&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;)&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;environ&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;AWS_ACCESS_KEY_ID&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;environ&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;AWS_SECRET_ACCESS_KEY&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]],&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Three details that save time:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Scope secrets to buckets.&lt;/strong&gt; Add &lt;code&gt;SCOPE 's3://raw-bucket'&lt;/code&gt; and the secret only applies there. With several matching secrets, the &lt;a href="https://duckdb.org/docs/current/configuration/secrets_manager.html" rel="noopener noreferrer"&gt;secrets manager&lt;/a&gt; picks the longest matching prefix. &lt;code&gt;FROM which_secret('s3://raw-bucket/x.parquet', 's3')&lt;/code&gt; tells you which one a path will use, and &lt;code&gt;FROM duckdb_secrets()&lt;/code&gt; lists them with sensitive values redacted.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Persistent secrets are plain files.&lt;/strong&gt; Secrets live in memory for the session by default. &lt;code&gt;CREATE PERSISTENT SECRET&lt;/code&gt; writes them to &lt;code&gt;~/.duckdb/stored_secrets&lt;/code&gt;, and the docs say they are "stored in unencrypted binary format on the disk." On a shared server, use &lt;code&gt;credential_chain&lt;/code&gt; instead.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;S3-compatible stores need three more fields.&lt;/strong&gt; For MinIO and similar, set &lt;code&gt;ENDPOINT&lt;/code&gt;, &lt;code&gt;URL_STYLE 'path'&lt;/code&gt; and, for a local HTTP endpoint, &lt;code&gt;USE_SSL false&lt;/code&gt;.
&lt;/li&gt;
&lt;/ul&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;CREATE&lt;/span&gt; &lt;span class="k"&gt;OR&lt;/span&gt; &lt;span class="k"&gt;REPLACE&lt;/span&gt; &lt;span class="n"&gt;SECRET&lt;/span&gt; &lt;span class="n"&gt;minio&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="k"&gt;TYPE&lt;/span&gt; &lt;span class="n"&gt;s3&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;PROVIDER&lt;/span&gt; &lt;span class="n"&gt;config&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;KEY_ID&lt;/span&gt; &lt;span class="s1"&gt;''&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;SECRET&lt;/span&gt; &lt;span class="s1"&gt;''&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;ENDPOINT&lt;/span&gt; &lt;span class="s1"&gt;'minio:9000'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;URL_STYLE&lt;/span&gt; &lt;span class="s1"&gt;'path'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;USE_SSL&lt;/span&gt; &lt;span class="k"&gt;false&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="k"&gt;SCOPE&lt;/span&gt; &lt;span class="s1"&gt;'s3://analytics'&lt;/span&gt;
&lt;span class="p"&gt;);&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If you have older code that uses &lt;code&gt;SET s3_access_key_id = ...&lt;/code&gt;, it still runs. Chapter 9 of &lt;em&gt;Local-First Analytics&lt;/em&gt; uses that style for its MinIO example. DuckDB's docs now call it &lt;a href="https://duckdb.org/docs/current/core_extensions/httpfs/s3api_legacy_authentication.html" rel="noopener noreferrer"&gt;a legacy method&lt;/a&gt; and say to avoid it, because settings are easy to print by accident. Move to secrets when you next touch that code.&lt;/p&gt;




&lt;p&gt;This is the first part. The full post — including the rest of the working details — is on my site: &lt;a href="https://hikmahtechnologies.com/blog/duckdb-s3-parquet/?utm_source=devto&amp;amp;utm_medium=referral&amp;amp;utm_campaign=blog" rel="noopener noreferrer"&gt;Querying Parquet on S3 with DuckDB: secrets, speed and the settings that matter&lt;/a&gt;&lt;/p&gt;

</description>
      <category>dataengineering</category>
      <category>database</category>
      <category>sql</category>
      <category>tutorial</category>
    </item>
    <item>
      <title>Dagster and dbt: do you need both?</title>
      <dc:creator>Mohammed Arshad Ansari</dc:creator>
      <pubDate>Fri, 23 Oct 2026 14:00:00 +0000</pubDate>
      <link>https://dev.to/mohammed_arshadansari_f2/dagster-and-dbt-do-you-need-both-27l7</link>
      <guid>https://dev.to/mohammed_arshadansari_f2/dagster-and-dbt-do-you-need-both-27l7</guid>
      <description>&lt;p&gt;"Why use dbt if I have Dagster?" is the number-two Google result for this topic, and it's a Reddit thread. That tells you the question is real. People see two tools that both draw a lineage graph, both run on a schedule, and both run tests, and they wonder if they are paying twice.&lt;/p&gt;

&lt;p&gt;They aren't, quite. dbt is a &lt;strong&gt;SQL transformation framework&lt;/strong&gt;. Dagster is an &lt;strong&gt;orchestrator&lt;/strong&gt;. They overlap in the middle, and the overlap is where the confusion lives. My own production pipeline runs on Dagster with no dbt at all, so I'll also say when skipping dbt is the right call.&lt;/p&gt;

&lt;h2&gt;
  
  
  What each tool actually does
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;dbt&lt;/strong&gt; takes a folder of SELECT statements with Jinja templating. It works out their order from &lt;code&gt;ref()&lt;/code&gt; calls, compiles them for your warehouse, runs them as tables or views, and runs tests (&lt;code&gt;not_null&lt;/code&gt;, &lt;code&gt;unique&lt;/code&gt;, custom SQL) against the results. It does not schedule itself. Something has to call &lt;code&gt;dbt build&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Dagster&lt;/strong&gt; decides when things run, records what each run produced, retries failures, tracks partitions, and runs arbitrary Python. It has its own data-quality checks (asset checks) and its own lineage graph, built from the dependencies you declare.&lt;/p&gt;

&lt;p&gt;The overlap: both have a lineage graph and both have tests. The difference: dbt's graph covers SQL models in one warehouse. Dagster's graph covers everything, including the Python that loads data before dbt runs and the models you train after it.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why use dbt if you have Dagster
&lt;/h2&gt;

&lt;p&gt;You still want dbt when:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Most of your transformations are SQL in one warehouse.&lt;/strong&gt; dbt's templating, &lt;code&gt;ref()&lt;/code&gt;, incremental materializations and the docs site are a decade of work you would otherwise rebuild badly.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Analysts write the models.&lt;/strong&gt; dbt is the tool analysts already know. Asking them to write Dagster assets in Python is a bigger change than it sounds.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;You want the project to be portable.&lt;/strong&gt; A dbt project runs under Dagster, Airflow, Prefect, cron or dbt Cloud. Your SQL is not locked to the orchestrator.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;You can skip dbt when the transformations are mostly Python, or when one engineer writes everything and is happy to keep the SQL inside Python assets. That is my situation, below.&lt;/p&gt;




&lt;p&gt;This is the first part. The full post — including the rest of the working details — is on my site: &lt;a href="https://hikmahtechnologies.com/blog/dagster-with-dbt/?utm_source=devto&amp;amp;utm_medium=referral&amp;amp;utm_campaign=blog" rel="noopener noreferrer"&gt;Dagster and dbt: do you need both?&lt;/a&gt;&lt;/p&gt;

</description>
      <category>dataengineering</category>
      <category>python</category>
      <category>sql</category>
      <category>data</category>
    </item>
    <item>
      <title>AEGIS v1 is archived. Here is what replaces it</title>
      <dc:creator>Mohammed Arshad Ansari</dc:creator>
      <pubDate>Thu, 22 Oct 2026 04:30:51 +0000</pubDate>
      <link>https://dev.to/mohammed_arshadansari_f2/aegis-v1-is-archived-here-is-what-replaces-it-580j</link>
      <guid>https://dev.to/mohammed_arshadansari_f2/aegis-v1-is-archived-here-is-what-replaces-it-580j</guid>
      <description>&lt;p&gt;I've shut down AEGIS v1 and archived &lt;a href="https://github.com/hikmahtech/aegis" rel="noopener noreferrer"&gt;github.com/hikmahtech/aegis&lt;/a&gt;. The code stays public and MIT-licensed. You can still read it, fork it and learn from it. It just won't get fixes, releases or answers to issues any more.&lt;/p&gt;

&lt;p&gt;AEGIS itself isn't going away. It is being rebuilt as version 2, and version 2 is not open source. This post explains why, what v2 looks like, and what is already running on it.&lt;/p&gt;

&lt;h2&gt;
  
  
  What v1 was
&lt;/h2&gt;

&lt;p&gt;AEGIS v1 was one system that did everything for one person: my tasks, email, money, research, and the alerts from the machines in my house. Three services shared one Postgres database. A Temporal worker ran every workflow on one task queue. A handful of named agents each owned a slice of the work, and every time one needed me, it asked through the same approval card.&lt;/p&gt;

&lt;p&gt;I &lt;a href="https://hikmahtechnologies.com/blog/aegis-is-open-source/?utm_source=devto&amp;amp;utm_medium=referral&amp;amp;utm_campaign=blog" rel="noopener noreferrer"&gt;open-sourced it in July&lt;/a&gt;, and the ideas in it held up. One primitive for every interruption. Durable workflows that survive a weekend of waiting for my tap. Local models first. An audit line for everything that matters. None of that is changing.&lt;/p&gt;

&lt;h2&gt;
  
  
  What went wrong with one system
&lt;/h2&gt;

&lt;p&gt;The shape was the problem, not the ideas.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;One deploy for everything.&lt;/strong&gt; A fix to bank-statement parsing redeployed the code that restarts services in my homelab. Every change carried the risk of every other area.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;One database for everything.&lt;/strong&gt; The accounting tables and the alert tables lived side by side. Nothing stopped one area from reading or writing another's rows, so over time some did.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Nothing could leave.&lt;/strong&gt; If someone wanted just the bookkeeping part, there was no way to give it to them. It came with my task manager, my research feeds and my infrastructure agent attached.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The last one matters most. The bookkeeping lane is useful to people who aren't me. To hand it to anyone, it has to stand on its own.&lt;/p&gt;




&lt;p&gt;This is the first part. The full post, including the rest of the working details, is on my site: &lt;a href="https://hikmahtechnologies.com/blog/aegis-v1-is-archived/?utm_source=devto&amp;amp;utm_medium=referral&amp;amp;utm_campaign=blog" rel="noopener noreferrer"&gt;AEGIS v1 is archived. Here is what replaces it&lt;/a&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>llm</category>
      <category>opensource</category>
      <category>architecture</category>
    </item>
    <item>
      <title>DuckDB vs Spark: which workloads actually need a cluster</title>
      <dc:creator>Mohammed Arshad Ansari</dc:creator>
      <pubDate>Wed, 21 Oct 2026 14:00:00 +0000</pubDate>
      <link>https://dev.to/mohammed_arshadansari_f2/duckdb-vs-spark-which-workloads-actually-need-a-cluster-4gbe</link>
      <guid>https://dev.to/mohammed_arshadansari_f2/duckdb-vs-spark-which-workloads-actually-need-a-cluster-4gbe</guid>
      <description>&lt;p&gt;Most "DuckDB vs Spark" questions are really a sizing question. Spark is a distributed engine: it splits a job across many machines and pays to coordinate them. DuckDB is an in-process engine: it runs the whole job on one machine and pays nothing to coordinate. Which one is faster depends almost entirely on whether one machine can hold the work.&lt;/p&gt;

&lt;p&gt;So the useful question is not which engine is better. It is: &lt;strong&gt;how many bytes does your job actually touch, and does that fit on one box?&lt;/strong&gt; Here is the arithmetic, where a cluster earns its cost, and what one node has handled for me. I have not run Spark in production; this is the single-node side of the line, measured, plus what Spark's own documentation says about the other side.&lt;/p&gt;

&lt;h2&gt;
  
  
  DuckDB vs Spark: what each one is built for
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Spark&lt;/strong&gt; spreads work over a cluster. A driver plans the job, executors on many machines each process part of the data, and a shuffle moves rows between machines when a join or group-by needs them together. If a machine dies, Spark rebuilds the lost piece. Its documentation puts it plainly: if any partition of a cached dataset is lost, "it will automatically be recomputed using the transformations that originally created it." That is what you rent a cluster for: data larger than any one machine, and jobs that keep going when one machine fails.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;DuckDB&lt;/strong&gt; is a columnar SQL engine that runs inside your Python process, a CLI or a service. No driver, no executors, no shuffle over a network. It reads Parquet and CSV in place, uses every core, and spills to disk when a join or sort outgrows memory. The &lt;a href="https://duckdb.org/docs/current/guides/performance/how_to_tune_workloads.html" rel="noopener noreferrer"&gt;tuning guide&lt;/a&gt; lists GROUP BY, JOIN, ORDER BY and window functions as operators that can work larger than memory.&lt;/p&gt;

&lt;p&gt;Both read the same Parquet. That matters more than any benchmark: the choice is reversible, because the files don't care which engine opens them.&lt;/p&gt;

&lt;h2&gt;
  
  
  The sizing math that decides it
&lt;/h2&gt;

&lt;p&gt;The number that matters is the &lt;strong&gt;working set&lt;/strong&gt;: the bytes a job has to process after partition pruning and column selection. It is not the size of the lake.&lt;/p&gt;

&lt;p&gt;Three steps get you there.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;1. Start from what the job reads, not what exists.&lt;/strong&gt; A 20 TB events table, partitioned by day, queried for the last 30 days, touching 6 of 60 columns, reads roughly 30/365 × 6/60 of the table. That is about 160 GB on disk before any filter on row values. Parquet compresses, so memory use runs higher than disk size. But most of that work streams through and never has to sit in memory at once.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;2. Budget memory by operation.&lt;/strong&gt; DuckDB's &lt;a href="https://duckdb.org/docs/current/guides/performance/environment.html" rel="noopener noreferrer"&gt;environment guide&lt;/a&gt; gives the rule of thumb: about &lt;strong&gt;1–2 GB of memory per thread for aggregation-heavy work and 3–4 GB per thread for join-heavy work&lt;/strong&gt;. So a 32-thread job needs roughly 32–64 GB for aggregations, or 96–128 GB for joins. That is one ordinary server, not a cluster.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;3. Compare with the biggest box you can rent.&lt;/strong&gt; Single machines are large now. AWS lists an &lt;a href="https://aws.amazon.com/ec2/instance-types/x2i/" rel="noopener noreferrer"&gt;x2iedn.metal&lt;/a&gt; with 128 vCPUs and 4,096 GiB of memory. You don't need that. The point is that the ceiling of "one machine" sits far above where most teams assume it is.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Local-First Analytics&lt;/em&gt; draws the same line. Chapter 5 suggests DuckDB for up to about 1 TB of partitioned Parquet for aggregation-heavy work, and less for join-heavy work. It also warns that many-to-many joins blow up intermediate state whatever the input size. Its rule of thumb for starting local-first is a working set that fits in 32 GB of RAM and fewer than ten people running queries. Both bars are much lower than where a single node actually gives out.&lt;/p&gt;

&lt;p&gt;If your working set after this arithmetic is in the gigabytes or low hundreds of gigabytes, you are running Spark for the cluster's sake, not the data's.&lt;/p&gt;




&lt;p&gt;This is the first part. The full post — including the rest of the working details — is on my site: &lt;a href="https://hikmahtechnologies.com/blog/duckdb-vs-spark/?utm_source=devto&amp;amp;utm_medium=referral&amp;amp;utm_campaign=blog" rel="noopener noreferrer"&gt;DuckDB vs Spark: which workloads actually need a cluster&lt;/a&gt;&lt;/p&gt;

</description>
      <category>dataengineering</category>
      <category>database</category>
      <category>python</category>
      <category>data</category>
    </item>
  </channel>
</rss>
