DEV Community: Philip Hern

back at it

Philip Hern — Thu, 23 Apr 2026 12:27:13 +0000

thesis

this is a small checkpoint post. the heavy lift is not finished, but i am out of the weeds for now, and that is worth naming out loud.

context

the same work i was carrying in stick with it kept growing in weight and surface area. for a while it felt like one endless tangle. i stayed with it anyway, and eventually i approached it the way i should have from the start, as smaller chunks that stack into the much larger change. each piece still had to be real, but the sequencing and scope finally matched how my head and the system can tolerate change.

argument

getting to a stable point did not erase the backlog. i still have more testing to run, more simulations to exercise, and real user acceptance testing ahead. the difference is that the foundation is no longer thrashing. errors and surprises have a place to land without undoing everything at once.

that stability is what gave me room to breathe. i can take a short break on purpose, look at the whole arc with a little distance, and come back to the tuning work with less panic and more optimism. the remaining work is still serious, but it is the kind of serious that fits a calendar instead of the kind that owns every waking hour.

tension or counterpoint

a stable checkpoint is not the same as done. if i confuse relief for completion, i will skip validation i still need. the discipline now is to rest without pretending the job is closed.

closing

so i am back at it in a different posture, not firefighting the whole shape at once, but finishing the test matrix, listening to users, and dialing things in with a clearer mind. sticking with it got me here. the next stretch is about proving it in the world, calmly.

related on this site

stick with it

Philip Hern — Fri, 17 Apr 2026 13:24:41 +0000

thesis

this one is for me as much as anyone reading. the single most important thing i can do on a long, hard project is keep showing up for it. motivation rises and falls, energy comes in waves, and neither of those things matter as much as continuity. if i stay with the work long enough, the payoff arrives, even when progress is invisible for stretches in the middle.

context

i am in the middle of a very heavy lift right now. it started as a change i thought i would finish quickly, and it has turned into something much bigger. the effort, concentration, and validation required are more than i am used to, and the timeline has stretched well past what a typical change would take. the stress is real. i feel it in how i think about the project before bed and how quickly i reach for my laptop in the morning.

i am still in it, though, because the value at the end is worth the cost. when this lands, i will have more stable and explainable historical data, which means my ongoing workload of troubleshooting data validity questions drops. less firefighting later is worth more pressure now, and that tradeoff is the only reason i would keep going through a change this heavy.

argument

continuity beats intensity

motivation is a wave, not a rope. it pulls me forward for a while, then it lets go, then it comes back later with a different shape. if i tie my progress to the wave, i stop whenever the wave stops. if i tie my progress to the habit of showing up, the wave cannot take the project down with it. that is the same pattern i wrote about in little by little, a little becomes a lot, just applied to a single long problem instead of a daily practice.

isolate your changes, even in your own playground

the hardest lesson from this round is about isolation. i have been testing work in an environment i consider my playground, and for a long time that has been fine. this time, my changes broke downstream consumers, and the pressure immediately escalated because other people were suddenly blocked. the takeaway is simple. if my changes can reach downstream consumers, i need to separate my testing from a shared test environment, regardless of how freely i am used to moving in that space. a playground still has neighbors.

do not try to lift several objects at once

i also tried to move multiple pieces of the system at the same time. i thought bundling them would be faster. what actually happened is that each piece depended on the others in a way that made every single one harder to validate, and the total stress grew faster than the total work. smaller, sequential chunks would have finished sooner and felt calmer. one object at a time, even if it feels slower on paper, is almost always faster in practice.

better preparation shrinks the stress

the last lesson is about preparation. i went in expecting a small change and i prepared like it was a small change. when the scope grew, my preparation did not grow with it, and that mismatch is where the break points appeared. better preparation up front, regardless of how small i thought the task was, would have reduced both the stress and the number of places things could go wrong. the cost of preparing for a bigger job than you need is tiny. the cost of not preparing for the job you actually have is not.

tension or counterpoint

persistence is not the same as refusing to reassess. sticking with every hard thing forever is just sunk cost fallacy wearing a motivational t-shirt. the honest check i keep running is whether the value at the end is still real and still mine. if the answer is yes, i keep going. if the answer turns into no, i stop, and that is not quitting, that is discernment.

there is also a stress cost to "push through" language. if the pressure is spilling into health, relationships, or judgment, that is a signal to change the pace, not a signal to try harder. pushing through is a tool, not a strategy, and it only works when i also rest and isolate the work properly. that is part of why i think it helps to get comfortable being uncomfortable without confusing discomfort for permission to keep grinding.

closing

so this is my note to myself. keep going. the work is real, the value is real, and the lessons i am collecting on the way are already paying off for the next change. next time i will isolate my testing better, break the work into one object at a time, and prepare like the task is bigger than i think it is, because it almost always is.

and if the wave of motivation dips again tomorrow, that is fine. waves dip. what matters is that i still show up, finish one more piece, and trust that continuity is the actual engine. stick with it.

related on this site

comfortable being uncomfortable

Philip Hern — Fri, 10 Apr 2026 19:29:32 +0000

thesis

i want to normalize a simple idea that still feels hard in practice. getting outside of your comfort zone is not a side quest. it is the main mechanism by which you stretch, learn, and see the world with more room in it for other people. yes, it is uncomfortable, and that is exactly the point.

context

most of us are trained to seek stability. stability is not bad, but it is also not where the adaptation happens. when everything feels familiar, your brain is mostly rehearsing what it already knows. the moment you step into something new, the cost shows up immediately as awkwardness, uncertainty, or fear of looking foolish. that friction is not a sign you chose wrong. it is often a sign you chose honestly.

argument

change is disruptive by definition. if it did not interrupt your default patterns, it would not be change. i think we should embrace that disruption more often, because it is where new experiences actually enter your life. without that interruption, you mostly get repetition with better packaging.

the growth part is not theoretical. discomfort is where skills get pressure-tested. you learn not only how to do things, but how not to do things, which is just as valuable and often faster feedback. mistakes in public or under stress are expensive emotionally, but they are also unusually clear. they show you boundaries, preferences, and limits in a way that a comfortable afternoon rarely will.

more experiences also broaden your worldview in a practical sense. when you have seen more contexts, constraints, and ways people solve problems, it becomes harder to treat your own habits as universal law. that widening tends to produce more tolerant and compassionate attitudes, not because tolerance is a slogan, but because you have more firsthand evidence that reasonable people can live and work in very different, equally valid ways.

so my encouragement is simple. expose yourself to new experiences on purpose. seek situations where the pressure is on you to perform, because that is where you rise to the occasion and discover how capable you can be. it is also where you might discover that this is not your thing, and you should move on. either outcome is a win, because both give you self-insight you cannot fake. you learn what energizes you, what drains you, and what you are willing to practice until it gets easier.

this connects to how i think about adaptability in general. comfort is a resting state. adaptation requires movement.

tension or counterpoint

there is a real downside to glorifying discomfort without boundaries. not every challenge is worth the cost, and not every "growth opportunity" is ethical or safe. pushing yourself is different from letting yourself be pushed past your values or health. the goal is not suffering for its own sake. the goal is chosen stretch, with recovery and discernment built in.

closing

i am not asking for constant chaos. i am asking for a bias toward the new when you can afford it, and toward the high-stakes try when you are ready. the uncomfortable path is where you find out who you are when the easy defaults are not available, and that knowledge is about as practical as it gets.

related on this site

dbt snapshots: moving from merges to native history

Philip Hern — Fri, 10 Apr 2026 19:29:31 +0000

quick answer

dbt snapshots provide a native way to track slowly changing dimensions over time. by migrating from custom merge statements to native dbt snapshots, you can simplify your codebase, rely on built-in history tracking, and ensure your downstream models always have access to point-in-time records.

who this is for

audience: data engineers and analytics engineers using dbt
prerequisites: basic knowledge of dbt models, sql, and data warehousing concepts
when to use this guide: when you need to track historical changes to mutable source records and want to move away from manual merge logic

why this matters

tracking historical changes is a common requirement in data warehousing. building custom merge logic to handle inserts, updates, and history tracking is error-prone and difficult to maintain. dbt snapshots handle the heavy lifting of history tracking out of the box. this ensures you do not lose historical context when source systems overwrite data.

moving from merge to snapshot

recently, i migrated several historical tables from a custom merge strategy to native dbt snapshots. the previous approach relied on complex merge statements that manually checked for changes and inserted or updated rows to maintain history. this was difficult to read and even harder to debug.

by adopting native dbt snapshots, the logic became declarative. instead of writing the exact update and insert commands, i only needed to define the source query and configure how dbt should detect changes. the downstream consumer views then filter the snapshot output to return the current row or a point-in-time record.

the core shift in thinking

when using snapshots, your snapshot definition should remain source-representative. do not apply business date-window filtering in the snapshot definition itself. instead, capture the raw history and apply your logic for which rows to return in downstream consumer views.

for example, to get the current row in a downstream model, you filter using the sentinel value:

select *
from {{ ref('my_snapshot_st') }}
where dbt_valid_to = '9999-12-31 23:59:59'

to get a freeze record for a specific point in time, you derive a freeze timestamp and filter:

select *
from {{ ref('my_snapshot_st') }}
where dbt_valid_from <= freeze_ts and dbt_valid_to > freeze_ts

basic example

here is a basic example of a dbt snapshot using the check strategy. this snapshot tracks changes to a practice affiliation table.

{% snapshot practice_affiliation_st %}

{{
    config(
        target_schema = 'snapshots',
        strategy = 'check',
        unique_key = ['fmno', 'cycle', 'committee', 'hierarchy'],
        check_cols = 'all',
        hard_deletes = 'invalidate',
        dbt_valid_to_current = "to_timestamp_ntz('9999-12-31 23:59:59')"
    )
}}

select
    fmno,
    cycle,
    committee,
    practice_name,
    type,
    hierarchy
from {{ ref('source_practice_affiliation_v') }}

{% endsnapshot %}

configuration options

dbt snapshots offer several configuration options that control how changes are detected and recorded. you can read more about these in the official dbt snapshot documentation.

here are the key options and what they control:

target_schema: the schema where the snapshot table will be built
strategy: determines how dbt detects changes, with the two main options being timestamp and check
unique_key: the primary key of the record, which can be a single column or a list of columns for a composite key
check_cols: used with the check strategy to specify which columns to monitor for changes, accepting a list of column names or the word all
updated_at: used with the timestamp strategy to specify the column that indicates when the source row was last modified
hard_deletes: controls how dbt handles rows that disappear from the source, such as setting it to invalidate to close the current row when a key is no longer present
dbt_valid_to_current: overrides the default null value for current records, allowing you to set a far-future date to make downstream filtering easier

timestamp vs check strategy

the choice between timestamp and check strategies is critical.

use the timestamp strategy when your source has a reliable updated column that changes whenever the row changes. dbt compares the source timestamp to the snapshot timestamp to decide if a new version is needed.

use the check strategy when you do not have a reliable updated timestamp, or when you want to detect any change in a specific set of columns. dbt compares the actual values of the check columns between the source and the current snapshot row. if any checked column differs, dbt closes the current row and inserts a new version.

in my recent work, i found that the check strategy with all columns checked and a composite unique key was the most robust approach for sources where the updated timestamp was synthetic or not authoritative.

gotchas and lessons learned

migrating to snapshots surfaced a few important lessons:

upstream scope gating: if your upstream source query includes filters that remove keys, and you have hard deletes configured to invalidate, dbt will intentionally close the current rows for those missing keys
composite keys: dbt fully supports composite unique keys, and passing a list of columns ensures that dbt tracks history at the correct grain
duplicate source rows: snapshots expect the source data to be unique at the unique key grain, so if your source contains duplicate keys, the snapshot will fail or bloat
defensive deduplication: in some cases, i had to add a defensive qualify row number guard in the snapshot definition to collapse known duplicate-key source rows before dbt processed them
sentinel values: using a sentinel value for current rows instead of null makes downstream queries much cleaner, allowing you to use an equals operator instead of checking for nulls

deployment and automation

snapshots are not updated automatically when you run your standard dbt build commands. they require a dedicated command: dbt snapshot.

if you do not automate this, your history tracking will be manual and prone to gaps. to ensure continuous history capture, you must schedule the snapshot command to run on a regular cadence.

in a production environment, this usually means setting up a continuous integration workflow or an orchestrator task. for example, you can use automated workflows to run snapshot tags on daily, hourly, or monthly schedules.

a typical automated workflow might look like this:

a scheduled trigger fires the workflow
the workflow checks out the repository and sets up the dbt environment
the workflow executes the snapshot command for specific tags
dbt connects to the warehouse, compares the source data to the existing snapshot tables, and applies any necessary inserts or updates

by decoupling the snapshot schedule from your standard model runs, you can capture history at the exact frequency your business logic requires.

references

little by little, a little becomes a lot

Philip Hern — Tue, 07 Apr 2026 23:13:24 +0000

thesis

the importance of just trying to do a little each day cannot be overstated. we often overestimate what we can accomplish in a single afternoon, but we vastly underestimate what we can build over a year of sustained effort. incremental changes add up, and what seems like a drop in the bucket today becomes a reservoir over time.

context

whether it is work, fitness, or building new routines, habit forming often feels like it takes forever to take hold. we live in a world that expects immediate results, and we naturally get frustrated when the scale does not move or the project does not finish overnight. when that initial burst of motivation inevitably fades, the reality of the daily grind sets in, and that is exactly when most people decide to walk away.

argument

this process is a little easier when you understand you are working toward continuity and not perfection. it is not about executing flawlessly every single day, nor is it about never taking a break. it is about showing up and putting in the reps, even when the effort feels small or uninspired. missing one day is just a bump in the road, as long as you do not let it become two days in a row.

i am starting to see the rewards of that mindset now. little by little, my experience has added up into a foundation i can actually rely on. little by little, i have started sharing via my website, turning scattered thoughts into a structured body of work. little by little, i am starting to have a greater reach and help more people, simply because i chose to publish something small rather than waiting for the perfect masterpiece.

tension or counterpoint

the hardest part is trusting the process when the visible progress is zero. it is incredibly easy to quit when you do not see the immediate payoff of your daily effort. it feels like you are just watering dirt for weeks on end. but the compounding effect of showing up is real, even if it remains completely invisible in the short term.

closing

so right in theme, i will keep this short and keep focusing on the small, daily inputs rather than the distant outputs. the goal is simply to keep the chain going, trusting that a little becomes a lot when you give it enough time.

related on this site

how i automated dev.to and linkedin publishing so visibility stops depending on memory

Philip Hern — Sun, 05 Apr 2026 14:03:19 +0000

after i started writing more consistently, it became obvious that writing is only half the work; distribution is the other half. i wanted a system where i can publish from one canonical source and let automation push the same story to dev.to and linkedin.

quick answer

i set up two publish automations that watch my post changes and sync them to dev.to and linkedin. the first publish creates the post on each platform, and later edits update the same external post instead of creating duplicates. this gives me consistent visibility without adding manual publishing steps after every article.

who this is for

people who publish technical writing and keep forgetting cross-posting
creators who want one canonical source plus repeatable distribution
builders who care about discoverability as much as writing quality

why this matters

if distribution is manual, it eventually slips. then strong posts sit unread because i forgot to copy, paste, format, and re-share them across platforms. automation solves that by making visibility part of the same delivery path as the content itself.

this is the same pattern i described in a practical ai workflow: jira, github, and mcp, define one clear source of truth, then automate the handoff steps so i can spend more time on thinking and less time on clerical work.

step-by-step

1) define the starting point

i chose my site post as the only canonical source. every external platform receives content from that source, not from separate drafts. this keeps language, links, and updates aligned over time.

2) apply the change

i added automation for both targets:

trigger on post updates and support manual runs when i want a full backfill
create posts when no external mapping exists
update existing external posts when a mapping already exists
keep a small state map so each canonical url stays attached to one external post id

the practical result is that i can keep writing in one place and trust the sync layer to handle distribution. this complements the writing habits from my cursor setup, where reusable workflows remove repeated manual work.

3) validate the result

i test in three passes:

dry run to confirm detection and decisions without publishing
publish-all run to verify initial backfill behavior
normal change-trigger run to verify incremental updates on later edits

when all three pass, i know the pipeline is reliable enough for daily use.

faq

what was the biggest setup mistake?

token and redirect mismatches during oauth were the main failure point at first. once i aligned scopes, callback values, and secret placement, the automation became stable.

should i keep manual publishing as a fallback?

yes, especially while you are in early setup. after the workflow proves stable, manual publishing becomes a recovery path instead of a default habit.

references

the future of data engineering workflows with ai

Philip Hern — Fri, 03 Apr 2026 14:11:59 +0000

quick answer

the future of data engineering workflows with ai is about moving from manual coding to intelligent orchestration. ai agents will handle boilerplate code, pipeline generation, and data quality checks, allowing data engineers to focus on architecture, governance, and business value.

who this is for

audience: data engineers, analytics engineers, data architects, and technical leaders.
prerequisites: an understanding of modern data stack concepts and basic ai principles.
when to use this guide: when planning your data strategy and evaluating how to integrate ai into your engineering practices.

why this matters

the volume and complexity of data are growing faster than engineering teams can scale. relying solely on manual workflows leads to bottlenecks, technical debt, and delayed insights. embracing ai is not just about efficiency, it is a strategic imperative to remain competitive.

step-by-step

1) define the starting point

traditionally, data engineering has been a highly manual discipline. engineers spend countless hours writing sql, configuring orchestrators like airflow, and debugging failed pipelines. this approach is brittle and scales poorly as the organization grows.

2) apply the change

the integration of ai changes this paradigm. large language models can now generate complex sql queries, translate between dialects, and even suggest optimal data models based on source schemas. ai agents can monitor pipeline health, automatically retry transient failures, and alert engineers only when human intervention is necessary. this shift transforms the engineer from a coder into a system architect.

3) validate the result

the impact of this transformation is measurable. development cycles shorten, data quality improves through automated testing, and the overall reliability of the platform increases. engineers spend less time firefighting and more time building scalable, resilient architectures that drive business decisions.

faq

what is the most important caveat?

ai is a tool, not a replacement for fundamental engineering principles. you still need a strong understanding of data modeling, governance, and security to build a robust platform.

what should i do first?

start by identifying the most repetitive tasks in your workflow, such as writing documentation or basic transformations. experiment with ai tools to automate these specific areas before attempting to overhaul your entire architecture.

references

the modern data stack

how i use cursor and ai agents to write dbt tests and documentation

Philip Hern — Fri, 03 Apr 2026 14:07:49 +0000

quick answer

writing dbt tests and documentation is often the most neglected part of data engineering. i use cursor and custom ai agents to automate this process by reading my sql models, inferring the business logic, and generating the corresponding yaml files. this ensures high-quality data pipelines without the manual overhead.

who this is for

audience: data engineers, analytics engineers, and developers using dbt
prerequisites: basic knowledge of dbt, sql, and cursor
when to use this guide: when you want to scale your data engineering practices and reduce the time spent on writing boilerplate yaml

why this matters

documentation and testing are critical for data trust, but they are tedious to write manually. when these steps are skipped, data quality suffers and debugging becomes a nightmare. by automating this with ai, you get the benefits of rigorous testing and clear documentation while freeing up your time for higher-value architectural work.

step-by-step

1) define the starting point

most data engineers start with a raw sql model and a blank slate for their schema.yml file. the traditional approach requires manually typing out every column name, description, and test. this is prone to human error and inconsistency, plus almost always falls out of sync with current models with the first change.

2) apply the change

i use cursor to bridge this gap. by creating specific ai rules and skills, i can highlight a dbt model and ask the agent to generate the documentation. the agent reads the sql, understands the joins and transformations, and produces a complete yaml file with standard tests like not_null and unique. it can even infer complex relationships and suggest custom tests based on the data domain.

3) validate the result

once the ai generates the yaml, i review it for accuracy. i then run dbt test and dbt docs generate to ensure everything compiles correctly. the ai rarely makes syntax errors, so the validation step is mostly about confirming the business logic aligns with the documentation.

faq

what is the most important caveat?

you must still review the generated output. ai is excellent at scaffolding and inferring patterns, but it does not possess the full business context that you do.

what should i do first?

start by creating a simple cursor skill that defines your team's standards for dbt documentation. feed it a few examples of your best schema.yml files so it learns your preferred style.

references

dbt documentation

sharing is caring

Philip Hern — Thu, 02 Apr 2026 12:11:05 +0000

the value of early adoption

i have always found a unique kind of energy in being an early adopter. when a new tool emerges, especially something as transformative as cursor and artificial intelligence, diving in headfirst is not just about personal efficiency. it is about understanding the landscape before the map is fully drawn. by spending the hours required to become a high-level user, i build a deep familiarity with the edges of what the technology can do.

this mastery translates directly into value for my colleagues. when you understand the high-level nuance of a complex tool, you naturally become the point person for your team. people have onboarding questions, they hit roadblocks, and they need someone who has already navigated those early frustrations. being that resource is incredibly rewarding. it shifts my role from an individual contributor to a multiplier, helping the entire team elevate their workflow and avoid the pitfalls i have already solved.

the responsibility to share

this dynamic reminds me of a principle i have heard often over the years regarding the importance of using your voice and your platform to share. this is exactly why i started this website. i wanted a dedicated space to share my voice, my knowledge, my opinions, my experience, and the solutions i have discovered along the way.

when you hold onto knowledge, its impact is limited to your own output. when you share it, the impact scales infinitely. writing about these tools, documenting my workflows, and answering the nuanced questions my colleagues ask are all extensions of the same core belief. knowledge is meant to be distributed.

stepping into mentorship

i have reached a point of mastery and experience where the natural next step for me is to mentor others and deliberately increase my visibility and presence. it is no longer enough to simply be good at what i do behind the scenes. the real work now is in lifting others up.

in fact, i am starting to feel the weight of this realization. it feels almost selfish not to share what i have learned. when you spend years honing a craft or mastering a paradigm-shifting tool like ai-assisted development, you accumulate a wealth of invisible context. keeping that context locked away serves no one. stepping into a mentorship role, both directly with my colleagues and publicly through this platform, is how i honor the effort it took to gain that experience in the first place.

looking forward

my goal is to continue exploring the bleeding edge of these tools, but with a renewed focus on how i can translate those discoveries into accessible guidance for others. whether it is through answering a quick onboarding question about cursor, writing a detailed guide on this site, or simply being a sounding board for a colleague, the objective remains the same. i want to use my experience to make the path easier for those who follow.

related on this site

post title

what is art?

Philip Hern — Mon, 30 Mar 2026 23:38:30 +0000

thesis

i keep pondering lately, what are we actually defending when we say "ai art is not real art"?

i do not have a final position yet. i am writing this to think in public, not to close the debate.

context

while driving on a family vacation, i asked my wife to fulfill her duty as the passenger and dj some motown bangers. she searched on spotify and found something that seemed to fit the bill. most of the songs were recognizable, memories from my childhood, riding in the backseat listening to my parents' favorites. however, the first song on the playlist was by an artist called the 19s soulers, which was an artist i did not recognize. this was a user created playlist so not everything might fit perfectly into the motown mold i was asking for, and that was ok. the song started and it was SOOO good. TOO good. i had my suspicions, but the music caught me so hard that i completely forgot. i asked my boys in the back seat to look up the artist and they did not even search and just responded "AI DAD - IT IS AI". i felt so many conflicting emotions, including one of pride that my boys could tell the difference and they have some defense against being fooled.

the conversation around ai-generated images and music feels hotter every week, especially when a new ai music act gets attention or a contract. the reaction is often immediate and predictable outrage, fear, dismissal, and arguments about stolen style.

at the same time, many of us use ai to help write code, review pull requests, or shape architecture notes without the same emotional response. that contrast is interesting to me.

if i call code a craft, and sometimes an art form, then why does ai help feel acceptable there for so many people, but unacceptable when the ai helps write song lyrics? and if code can be expressive, why is the outrage concentrated in painting, illustration, and music.

my code has my fingerprints all over it, just as much as this website and the way i speak and write. it defintely qualifies as expressive. i make stylistic, logic, function, etc. choices that suit my style. how is this different from writing a book? but if you asked which one is acceptable to use ai and which one is not, i could guess your answer 99% of the time.

argument

i see a few possible reasons, and none of them feel complete on their own:

visual art and music are tied to identity in a very direct way
audiences often connect to the "maker story", not only the artifact
creative labor markets in those fields already felt fragile before ai
software teams have normalized tool-assisted output for decades
code is often judged by function first, while art is judged by intention and feeling

still, even with those differences, i cannot shake the inconsistency.

when i use ai in code, i still feel like the author because i set constraints, reject bad output, and own the result. i do not think that is very different from guiding a visual generator, editing outputs, and curating a final piece. maybe the difference is only social permission, not creative mechanics.

this question also links to my concern about ownership in the danger of trusting the ai agent, where speed is useful but responsibility still has to stay human.

tension or counterpoint

there is also a strong counterpoint i take seriously: in code, wrong answers fail in visible ways. tests fail, services break, users complain, and teams can trace accountability. in art, value is less binary, and that makes authorship feel more central and more vulnerable.

another counterpoint is economic, not philosophical. people may not be reacting to "is this art" at all. they may be reacting to "will this replace my livelihood".

both of those points feel real to me.

and i think the later point is one worth exploring because the wide-spread availability of ai has "democratized" creativity, technical endeavors, etc. for people who might have great ideas, but not the musical or technical skill to carry out the plan. well, now they do. and that instant competition that was not present before can certainly feel intimidating and encroaching.

i am currently mostly pro-ai, but with caution. we should have caution regarding how the models are being trained (and on what data) and regulated. we should exercise caution surrounding who is doing the regulating, as well. ai is a powerful assistant, and as we all know from spiderman, with great power comes great responsibility.

closing

i am left with questions, not conclusions.

maybe we value human touch most where we believe the human story is the product. maybe we accept ai more where we believe the product is utility. maybe those boundaries are changing and we are all reacting in real time.

for now, i am trying to keep the question open.....when ai is part of the process, what still makes something mine, yours, or ours?

related on this site

dbt tests

Philip Hern — Mon, 30 Mar 2026 23:34:36 +0000

we all know testing is valuable, but almost all dbt projects still underinvest in it. i am guilty of this, shipping the model, promising i will add tests later, then moving on to the next urgent request.

that pattern feels fast in the moment, but it is expensive over time. dbt tests are one of the easiest ways to protect trust in your data, and the setup cost is usually smaller than people expect.

quick answer

dbt tests are assertions about your data that run inside your transformation workflow. they verify assumptions like uniqueness, non-null keys, valid categorical values, and referential integrity. when a test fails, dbt surfaces the exact failing records so you can debug quickly. if you are new to the feature, start with the official dbt data tests documentation and add a few high-signal tests to your most consumed models first.

who this is for

analytics engineers who already build dbt models but still rely on manual spot checks
data teams that have recurring data quality incidents in dashboards or reports
anyone who wants a practical starting point instead of a perfect testing framework

why this matters

the cost of bad data is usually delayed, not immediate. a broken metric can sit in production for days before someone notices, and by then that number has already been used in a deck, decision, or leadership update.

the frustrating part is that most of these issues are predictable. duplicate primary keys, null foreign keys, unexpected status values, and invalid date ranges are all common failures. dbt tests can catch these early, near the model that introduced the issue.

i also think testing helps teams move faster, not slower. when tests are in place, i can refactor a model with more confidence because i have a safety net. without tests, every change feels risky and review cycles become slower because everyone is relying on intuition.

step-by-step

1) define the starting point

pick one model that is heavily consumed, for example an order fact table or a customer dimension. identify three assumptions that must always be true:

the model key is unique
important keys are never null
status fields only contain known values

then encode those assumptions directly in your yml.

2) apply the change

start with generic tests in your schema file.

version: 2

models:
  - name: fct_orders
    columns:
      - name: order_id
        tests:
          - not_null
          - unique
      - name: customer_id
        tests:
          - not_null
          - relationships:
              to: ref('dim_customers')
              field: customer_id
      - name: order_status
        tests:
          - accepted_values:
              values: ["placed", "shipped", "cancelled", "returned"]

this single block gives you strong baseline coverage:

not_null protects required fields
unique protects grain
relationships protects joins and referential integrity
accepted_values protects enum-like business states

next, add one singular test for a business rule that generic tests cannot express cleanly.

-- tests/orders_non_negative_amount.sql
SELECT
    order_id,
    order_amount
FROM {{ ref('fct_orders') }}
WHERE order_amount < 0

this test fails only when the query returns rows. singular tests are ideal for custom rules like range checks, cross-column logic, and impossible combinations.

3) validate the result

run tests in a tight loop while you are developing:

dbt test --select fct_orders

for a broader gate in CI or before merging, run:

dbt build --select fct_orders+

dbt build runs models and tests together, which is useful when you want to validate both transformation logic and data quality in one pass. the + after the model name tells dbt to also run any downstream models according to the dag to make sure dependencies also pass.

a practical prioritization rule

when time is limited, i prioritize tests in this order:

key integrity on high-consumption models (unique plus not_null)
foreign key integrity (relationships) on joins that power dashboards
controlled fields (accepted_values) where business logic depends on a finite set of values
one custom singular test for the highest-risk metric or business rule

this sequence catches a large share of real incidents with minimal setup.

faq

what should i test first in a mature project with little coverage?

start where breakage is most expensive, not where modeling is most elegant. choose one or two heavily consumed models and add key integrity plus relationships first. then add one singular test for the business rule that has caused the most historical pain.

do dbt tests slow down delivery?

they add some upfront work, but they usually reduce cycle time later. test failures are cheaper during development than after release, and tests make refactors safer because you can verify assumptions continuously instead of rediscovering issues in production.

references

dbt docs

Philip Hern — Mon, 30 Mar 2026 11:39:08 +0000

most data engineers i know will spend hours getting a model right, then skip the one step that makes it discoverable to everyone else. dbt docs are that step, and they are worth the effort.

quick answer

dbt docs is a built-in feature that generates a browsable website from your dbt project. it pulls descriptions from your yml files, renders a searchable model catalog, and draws a lineage graph showing how every model connects. running dbt docs generate followed by dbt docs serve gives you a local site instantly. the real payoff is that teammates who never open your sql files can still understand what each model does, what columns it exposes, and where the data comes from. this is especially useful to downstream consumers of your objects because they can see the exact format of the object, including column names and data types, etc.

who this is for

data engineers who use dbt but skip writing descriptions
analysts, product managers, or business users who need to understand available data without reading sql
team leads looking for a low-effort way to make data structures discoverable

why this matters

when you build a dbt project, the models represent real business concepts. customers, products, sales, inventory, whatever the domain is. the people who consume those models in dashboards or reports often have no involvement in building them and no reason to read raw sql.

without documentation, those consumers rely on tribal knowledge, slack messages, and guesswork. that does not scale. dbt docs solve this by turning the metadata you already maintain (yml files, project config, source definitions) into a navigable reference that anyone on the team can use. the effort to write a good description is small, and the compound value to the rest of the organization grows with every model you add.

i think of it like this: if i write a model and do not document it, the only person who truly understands it is me, and even that fades after a few months. if i write a two-sentence description and add column-level context, that knowledge lives in the project permanently and serves everyone who touches the data.

what dbt docs generates

when you run dbt docs generate, dbt produces two main artifacts in your target/ directory:

manifest.json contains the full project graph, including every model, source, seed, snapshot, and macro, along with their descriptions, tags, and configuration
catalog.json contains the schema-level metadata pulled from your warehouse, including column names, data types, and row counts

together with a bundled index.html, these files power a static site that you can open locally with dbt docs serve or host anywhere that serves static files.

the site includes

a searchable list of every model and data source in your project
model-level and column-level descriptions pulled from your yml files
the full sql compiled for each model (in each environment)
a lineage graph (dag - directed acyclic graph) that shows upstream sources, intermediate models, and downstream consumers for any selected node

the lineage graph is especially useful when someone asks "where does this column come from" or "what breaks if i change this source table". instead of tracing through sql files manually, the graph answers it visually.

how to document your models

dbt reads documentation from yml files that live alongside your models. you are probably already using these for configuration and source definitions, so adding descriptions is a natural extension.

model and column descriptions in yml

the most common approach is adding description fields directly in your yml files. here is what that looks like for a view in an information delivery layer:

version: 2

models:
  - name: ORDER_SUMMARY_V
    description: >
      aggregated view of customer orders with totals and status
      breakdowns, consumed by the reporting dashboard and the
      customer details page in the application.
    columns:
      - name: CUSTOMER_ID
        description: "unique identifier for the customer"
      - name: TOTAL_ORDERS
        description: "count of all orders placed by this customer"
      - name: TOTAL_SPEND
        description: "sum of order amounts across all completed orders"
      - name: LAST_ORDER_DATE
        description: "most recent order date for this customer"

every model and every column can have a description. the more specific you are, the more useful the generated docs become. "id" as a column description does not help anyone. "unique identifier for the customer, sourced from the application database" does.

source descriptions

sources benefit from the same treatment. when your project ingests raw data from an external system, describing those sources in your shared sources file makes the lineage graph meaningful from the very first node:

version: 2

sources:
  - name: RAW_ORDERS
    description: "raw order data ingested from the transactional database via kafka"
    database: ANALYTICS_DEV_DB
    schema: RAW_DATA
    tables:
      - name: ORDERS
        description: "one row per order, includes order status and timestamps"
      - name: ORDER_ITEMS
        description: "line items for each order, one row per product per order"

doc blocks for longer descriptions

when a model needs more than a sentence or two of context, dbt supports doc blocks. these are markdown files (.md) that live in your project and can be referenced from yml descriptions:

{% docs order_summary_description %}

this view surfaces the aggregated order history for each customer.
it joins the hub, satellite, and link tables from the refined data
layer to produce a single wide row per customer.

**grain:** one row per customer.

**consumers:** reporting dashboard, customer details api endpoint.

{% enddocs %}

then in your yml:

models:
  - name: ORDER_SUMMARY_V
    description: '{{ doc("order_summary_description") }}'

doc blocks are useful when the context is long enough that embedding it inline in yaml becomes awkward. they also let you reuse the same description across multiple references if needed.

a few useful options in dbt docs

persist_docs

by default, dbt docs only live in the generated static site. if you want the descriptions to also appear in your warehouse catalog (so someone querying snowflake information_schema can see them), you can enable persist_docs:

models:
  my_data_product:
    +persist_docs:
      relation: true
      columns: true

with this enabled, dbt run pushes your yml descriptions into the COMMENT property on the table or view and on each column in the warehouse. this is valuable because it means the documentation is available even outside the dbt docs site, directly in the database catalog that tools like snowflake and bi platforms already read.

the lineage graph

the generated site includes an interactive dag that visualizes every model, source, and their connections. you can click on any node to see its upstream dependencies and downstream consumers. this is one of the most powerful features in dbt docs because it makes the data flow tangible for people who do not read sql (or maybe they just do not have access to your workspace, but still have a need to understand it).

when you have a project with dozens or hundreds of models organized into layers (raw, refined, business, information delivery), the lineage graph shows how a raw source table flows through transformations into the final views that analysts query. it replaces the need for manually maintained architecture diagrams that go stale the moment someone adds a new model.

exposures

exposures let you document where your dbt models are consumed outside of dbt. dashboards, applications, api endpoints, anything downstream. defining them makes the lineage graph extend beyond the dbt project boundary:

exposures:
  - name: customer_dashboard
    type: dashboard
    description: "executive dashboard showing customer order trends and retention"
    depends_on:
      - ref('ORDER_SUMMARY_V')
      - ref('CUSTOMER_RETENTION_V')
    owner:
      name: analytics team
      email: analytics@example.com

exposures show up in the lineage graph as leaf nodes, making it clear which models are actively consumed and by what. this is helpful when you are deciding whether it is safe to refactor or deprecate a model.

hosting dbt docs on github pages

running dbt docs serve is great for local browsing, but the real value comes from hosting the site where the whole team can access it without installing dbt or cloning the repo. github pages is a straightforward, free option for this.

how it works

after dbt docs generate runs, the target/ directory contains everything needed to serve the site: index.html, manifest.json, and catalog.json. you copy those files to a branch or directory that github pages serves, and the docs are live.

a basic github actions workflow

here is a minimal workflow that generates the docs on every push to main and deploys them to github pages:

name: deploy dbt docs

on:
  push:
    branches: [main]

permissions:
  contents: read
  pages: write
  id-token: write

jobs:
  deploy:
    runs-on: ubuntu-latest
    environment:
      name: github-pages
      url: ${{ steps.deployment.outputs.page_url }}
    steps:
      - uses: actions/checkout@v4

      - name: set up python
        uses: actions/setup-python@v5
        with:
          python-version: "3.11"

      - name: install dbt
        run: pip install dbt-snowflake

      - name: generate docs
        run: dbt docs generate --profiles-dir .
        working-directory: my_dbt_project
        env:
          DBT_SNOWFLAKE_ACCOUNT: ${{ secrets.SNOWFLAKE_ACCOUNT }}
          DBT_SNOWFLAKE_USER: ${{ secrets.SNOWFLAKE_USER }}
          DBT_SNOWFLAKE_PASSWORD: ${{ secrets.SNOWFLAKE_PASSWORD }}

      - name: prepare pages artifact
        run: |
          mkdir -p pages
          cp my_dbt_project/target/index.html pages/
          cp my_dbt_project/target/manifest.json pages/
          cp my_dbt_project/target/catalog.json pages/

      - name: upload pages artifact
        uses: actions/upload-pages-artifact@v3
        with:
          path: pages

      - name: deploy to github pages
        id: deployment
        uses: actions/deploy-pages@v4

once this runs, your dbt docs are available at https://<org>.github.io/<repo>/ and automatically update every time someone merges to main. no one needs to install anything or run any commands to browse the documentation.

keep the credentials out of the repo

the workflow above uses github secrets for warehouse credentials. never commit profiles with real credentials. use environment variables or a ci-specific profiles.yml that references secrets, and make sure your .gitignore excludes any local profiles that contain actual passwords or tokens.

faq

do i need to write descriptions for every column?

you do not need to, but the columns that matter most to consumers deserve it. at minimum, describe the primary key, any business key, and any column whose meaning is not obvious from the name alone. over time, filling in the rest pays off as the team grows.

can i generate docs without connecting to the warehouse?

dbt docs generate pulls catalog metadata from the warehouse, so it does need a connection. however, if you already have a catalog.json from a previous run, you can serve the site locally with just those files.

how is this different from a wiki or confluence page?

dbt docs stay in sync with your code automatically. a wiki page about your data model goes stale the moment someone adds a column or renames a table. dbt docs regenerate from the source of truth (your yml files and your warehouse) every time you run the command, so the documentation and the code never drift apart.

DEV Community: Philip Hern

back at it

thesis

context

argument

tension or counterpoint

closing

further reading

related on this site

stick with it

thesis

context

argument

continuity beats intensity

isolate your changes, even in your own playground

do not try to lift several objects at once

better preparation shrinks the stress

tension or counterpoint

closing

further reading

related on this site

comfortable being uncomfortable

thesis

context

argument

tension or counterpoint

closing

further reading

related on this site

dbt snapshots: moving from merges to native history

quick answer

who this is for

why this matters

moving from merge to snapshot

the core shift in thinking

basic example

configuration options

timestamp vs check strategy

gotchas and lessons learned

deployment and automation

references

related reading

little by little, a little becomes a lot

thesis

context

argument

tension or counterpoint

closing

further reading

related on this site

how i automated dev.to and linkedin publishing so visibility stops depending on memory

quick answer

who this is for

why this matters

step-by-step

1) define the starting point

2) apply the change

3) validate the result

faq

what was the biggest setup mistake?

should i keep manual publishing as a fallback?

references

related reading

the future of data engineering workflows with ai

quick answer

who this is for

why this matters

step-by-step

1) define the starting point

2) apply the change

3) validate the result

faq

what is the most important caveat?

what should i do first?

references

related reading

how i use cursor and ai agents to write dbt tests and documentation

quick answer

who this is for

why this matters