<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: TECHWORKSLAB</title>
    <description>The latest articles on DEV Community by TECHWORKSLAB (@techworkslab).</description>
    <link>https://dev.to/techworkslab</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4217889%2F0d240f0b-06ed-4674-8778-3f8bf127058d.png</url>
      <title>DEV Community: TECHWORKSLAB</title>
      <link>https://dev.to/techworkslab</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/techworkslab"/>
    <language>en</language>
    <item>
      <title>Your First Hour With a Dataset You Have Never Seen</title>
      <dc:creator>TECHWORKSLAB</dc:creator>
      <pubDate>Wed, 11 Nov 2026 20:12:23 +0000</pubDate>
      <link>https://dev.to/techworkslab/your-first-hour-with-a-dataset-you-have-never-seen-21f8</link>
      <guid>https://dev.to/techworkslab/your-first-hour-with-a-dataset-you-have-never-seen-21f8</guid>
      <description>&lt;p&gt;Most analysis that goes wrong goes wrong in the first ten minutes, before anyone writes a single chart. A date column was read as text. A survey used minus one to mean no answer and the average quietly dropped. Two rows exist for every customer because of a join nobody checked. None of these are hard problems. They are easy to catch and almost impossible to notice later, once a dashboard is built on top of them and the number already looks reasonable.&lt;/p&gt;

&lt;p&gt;The habit that separates analysts who get trusted from those who get corrected is simple. Before building anything, they spend the first hour getting to know the data on its own terms. Not exploring for insights yet, just establishing what they are holding and where it might lie to them. This article is that first hour, as a routine you can run on any dataset, in any tool, every single time.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step one: get the shape
&lt;/h2&gt;

&lt;p&gt;How many rows, how many columns, and what is one row. That last question is the most important one in analytics and the one beginners skip. Is a row a customer, an order, a customer-month, or a line item on an order? Everything you compute later depends on the answer, and getting it wrong is how a total ends up counted five times.&lt;/p&gt;

&lt;p&gt;Start by printing the dimensions and reading the first and last handful of rows. The head shows you the columns and a sample of values. The tail often reveals a surprise the head hides: a stray total row at the bottom, a block of blank rows, a change in format partway down a file that was stitched together from several exports. Then say out loud, in one sentence, what a single row represents. If you cannot, that is the first thing to resolve before anything else.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step two: check the types
&lt;/h2&gt;

&lt;p&gt;Look at what type each column actually is, not what you assume it is. The classic trap is a date stored as text, which sorts alphabetically so that the tenth of a month lands before the second, and which cannot be used for any time calculation until it is converted. Just as common is a number stored as text because one value somewhere contains a comma, a currency symbol, or the word "null", which forces the whole column to be read as strings.&lt;/p&gt;

&lt;p&gt;In pandas, &lt;code&gt;df.dtypes&lt;/code&gt; and &lt;code&gt;df.info()&lt;/code&gt; tell you this in seconds. A column you expect to be numeric that shows up as &lt;code&gt;object&lt;/code&gt; is a flag: something non-numeric is hiding in it. Find that value before you convert, because it usually tells you something real about how the data was collected.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step three: find what is missing
&lt;/h2&gt;

&lt;p&gt;Count the blanks in every column. A column that is ten percent empty is a different problem from one that is ninety percent empty, and both are different from a column where the missing values are not blank at all but coded. Watch for the sentinel values that pretend to be data: zero where zero is impossible, minus one, 9999, a date like 1900-01-01, or the literal text "N/A". These slip past a blank-count and poison an average silently, because to the tool they look like ordinary numbers.&lt;/p&gt;

&lt;p&gt;The question is not only how much is missing but whether the missingness has a pattern. If every blank row comes from one region or one month, that is a data collection story, not random noise, and it will bias any conclusion you draw. The counting comes first, and it comes from you.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step four: sanity-check the numbers
&lt;/h2&gt;

&lt;p&gt;For every numeric column, look at the minimum, the maximum, and a few points in between. You are not looking for insight yet. You are asking whether the values are physically possible. An age of 200, a negative quantity sold, a percentage above one hundred, a price of zero on a product that clearly sold: each of these is the data telling you how it was really recorded, and each one would quietly distort a chart you built without looking.&lt;/p&gt;

&lt;p&gt;A quick summary of each column, the kind &lt;code&gt;df.describe()&lt;/code&gt; produces, puts the extremes in front of you. When a maximum looks absurd, chase down the row it came from. Sometimes it is a genuine outlier worth keeping. Often it is a data entry error, a unit mismatch, or a placeholder, and knowing which changes everything you do next.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step five: look for duplicates
&lt;/h2&gt;

&lt;p&gt;Check whether rows repeat when they should not. This connects straight back to step one: if one row is supposed to be one customer, then two rows with the same customer id is a problem, and it is usually the fingerprint of a join that matched more rows than you expected. Duplicate rows inflate counts and totals in a way that is invisible on the surface, because each duplicated value looks perfectly legitimate on its own.&lt;/p&gt;

&lt;p&gt;Count the distinct values of whatever is supposed to be unique and compare that to the row count. If they do not match, you have either duplicates to remove or a wrong idea of what one row means. Resolve that now, because every sum and every average downstream depends on it.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step six: write down what you found
&lt;/h2&gt;

&lt;p&gt;The last step costs five minutes and saves a week. Write a short note of what you learned: what one row is, which columns you had to convert, where the missing values are and how they are coded, which outliers you decided to keep or drop and why. This is not bureaucracy. It is the difference between an analysis someone can trust and one you cannot even reconstruct yourself in a month.&lt;/p&gt;

&lt;p&gt;These notes are also what make your work reviewable. When a stakeholder asks why a number looks the way it does, the answer is in your notes, not in your memory. Analysts who keep this habit build a quiet reputation for being right, because their results come with a paper trail that holds up when someone pushes on them.&lt;/p&gt;

&lt;h2&gt;
  
  
  The short version
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Get the shape and say in one sentence what a single row is&lt;/li&gt;
&lt;li&gt;Check every column's type, especially dates and numbers read as text&lt;/li&gt;
&lt;li&gt;Count missing values and catch the ones coded as fake numbers&lt;/li&gt;
&lt;li&gt;Sanity-check ranges for values that are not physically possible&lt;/li&gt;
&lt;li&gt;Look for duplicates against whatever should be unique&lt;/li&gt;
&lt;li&gt;Write down what you found so the work is trustworthy and reviewable&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;In the first hour you are not asking what this data says. You are asking whether you can trust what it says. Those are different jobs, and doing the second one first is what makes the first one safe.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Originally published on the &lt;a href="https://edusharktraining.com/blog-first-hour-with-a-new-dataset" rel="noopener noreferrer"&gt;EDUSHARK TRAINING blog&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>datascience</category>
      <category>python</category>
      <category>beginners</category>
      <category>analytics</category>
    </item>
    <item>
      <title>Writing a Pinnacle 21 style check in R, and why the check is not the evidence</title>
      <dc:creator>TECHWORKSLAB</dc:creator>
      <pubDate>Sat, 31 Oct 2026 01:33:06 +0000</pubDate>
      <link>https://dev.to/techworkslab/writing-a-pinnacle-21-style-check-in-r-and-why-the-check-is-not-the-evidence-43ka</link>
      <guid>https://dev.to/techworkslab/writing-a-pinnacle-21-style-check-in-r-and-why-the-check-is-not-the-evidence-43ka</guid>
      <description>&lt;p&gt;Conformance checking feels like the end of the job. You run the validator, the report comes back clean or close to it, and the dataset is pronounced ready. We want to make a narrower and more useful point: the check is trivial to write, the running of it is where all the value is, and a passing check on its own is worth almost nothing as evidence. To make the first part concrete, here is a small conformance check in R. To make the second part concrete, here is what it is missing.&lt;/p&gt;

&lt;h2&gt;
  
  
  A check is just a function that returns findings
&lt;/h2&gt;

&lt;p&gt;A conformance rule is a question asked of a dataset that returns the rows where the answer is no. That is it. Here are three rules that cover a surprising amount of real world trouble: a required variable is missing, a key is not unique, and a date is not valid ISO 8601. Each returns a tidy table of findings, and the empty table is the pass.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight r"&gt;&lt;code&gt;&lt;span class="n"&gt;library&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;dplyr&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="n"&gt;library&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;tibble&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;&lt;span class="c1"&gt;# Rule 1: every required variable for the domain is present&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="n"&gt;check_required_vars&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;lt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;function&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;domain&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;required&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="n"&gt;missing&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;lt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;setdiff&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;required&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nf"&gt;names&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="n"&gt;tibble&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="n"&gt;domain&lt;/span&gt;&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;domain&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="n"&gt;rule&lt;/span&gt;&lt;span class="w"&gt;     &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"required-variable-present"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="n"&gt;severity&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"error"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="n"&gt;detail&lt;/span&gt;&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;missing&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;&lt;span class="c1"&gt;# Rule 2: the key variables are unique (one row per key)&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="n"&gt;check_unique_key&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;lt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;function&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;domain&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;key&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="n"&gt;dup&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;lt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;%&amp;gt;%&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;count&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;across&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;all_of&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;key&lt;/span&gt;&lt;span class="p"&gt;)))&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;%&amp;gt;%&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;filter&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="n"&gt;tibble&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="n"&gt;domain&lt;/span&gt;&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;domain&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="n"&gt;rule&lt;/span&gt;&lt;span class="w"&gt;     &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"unique-key"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="n"&gt;severity&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"error"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="n"&gt;detail&lt;/span&gt;&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;nrow&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;dup&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;==&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;character&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="m"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
               &lt;/span&gt;&lt;span class="k"&gt;else&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;paste&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;nrow&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;dup&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"duplicated key value(s)"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;&lt;span class="c1"&gt;# Rule 3: --DTC values are ISO 8601 date or datetime (allowing partials)&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="n"&gt;check_iso8601&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;lt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;function&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;domain&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;dtc_var&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="n"&gt;iso&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;lt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"^\\d{4}(-\\d{2}(-\\d{2}(T\\d{2}:\\d{2}(:\\d{2})?)?)?)?$"&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="n"&gt;vals&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;lt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="p"&gt;[[&lt;/span&gt;&lt;span class="n"&gt;dtc_var&lt;/span&gt;&lt;span class="p"&gt;]]&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="n"&gt;bad&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="o"&gt;&amp;lt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;vals&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="nf"&gt;is.na&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;vals&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nf"&gt;nzchar&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;vals&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="n"&gt;grepl&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;iso&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;vals&lt;/span&gt;&lt;span class="p"&gt;)]&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="n"&gt;tibble&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="n"&gt;domain&lt;/span&gt;&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;domain&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="n"&gt;rule&lt;/span&gt;&lt;span class="w"&gt;     &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;paste0&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"iso8601-"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;dtc_var&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="n"&gt;severity&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"error"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="n"&gt;detail&lt;/span&gt;&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;unique&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;bad&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;em&gt;Three rules, each a plain function returning a findings tibble. The pattern scales to as many rules as you care to write.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Running them is a matter of binding the findings together. A real validator does this across every domain and hundreds of rules, but the shape is the same.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight r"&gt;&lt;code&gt;&lt;span class="c1"&gt;# A tiny AE-like frame to check&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="n"&gt;ae&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;lt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;tibble&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="n"&gt;STUDYID&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"PILOT01"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="n"&gt;USUBJID&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nf"&gt;c&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"01-001"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"01-001"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"01-002"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="n"&gt;AESEQ&lt;/span&gt;&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nf"&gt;c&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;&lt;span class="w"&gt;                 &lt;/span&gt;&lt;span class="c1"&gt;# note: duplicate key on subject 01-001&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="n"&gt;AESTDTC&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nf"&gt;c&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"2026-03-01"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"2026-14-01"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"2026-03-05"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="c1"&gt;# note: month 14&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;&lt;span class="n"&gt;findings&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;lt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;bind_rows&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="n"&gt;check_required_vars&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ae&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"AE"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nf"&gt;c&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"STUDYID"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"USUBJID"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"AESEQ"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"AETERM"&lt;/span&gt;&lt;span class="p"&gt;)),&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="n"&gt;check_unique_key&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ae&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"AE"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nf"&gt;c&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"USUBJID"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"AESEQ"&lt;/span&gt;&lt;span class="p"&gt;)),&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="n"&gt;check_iso8601&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ae&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"AE"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"AESTDTC"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;%&amp;gt;%&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;filter&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;nzchar&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;detail&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;lengths&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;detail&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;&lt;span class="n"&gt;findings&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="c1"&gt;#&amp;gt; # A tibble: 3 x 4&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="c1"&gt;#&amp;gt;   domain rule                      severity detail&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="c1"&gt;#&amp;gt;   &amp;lt;chr&amp;gt;  &amp;lt;chr&amp;gt;                     &amp;lt;chr&amp;gt;    &amp;lt;chr&amp;gt;&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="c1"&gt;#&amp;gt; 1 AE     required-variable-present error    AETERM&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="c1"&gt;#&amp;gt; 2 AE     unique-key                error    1 duplicated key value(s)&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="c1"&gt;#&amp;gt; 3 AE     iso8601-AESTDTC           error    2026-14-01&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;em&gt;The findings table is the whole output. Writing more rules is more of the same, which is exactly why the rules are not the hard part.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;You could spend a week turning this into something that reads real SDTM, loads a rule pack, and formats a report, and it would be useful work. But notice what you would have built: a program that, given a dataset, tells you which rows it does not like today. That is necessary. It is nowhere near sufficient.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;The findings table is an output. The evidence is everything that lets someone reproduce it.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Why the check is not the evidence
&lt;/h2&gt;

&lt;p&gt;Picture the clean report, printed, in a binder, a year later. It says zero errors. Now ask the questions an auditor asks. Which version of the dataset did it run on. Which rule set, at which version, with which thresholds. Who ran it, and on what machine. Could someone re-run it today and get the same report. A bare report answers none of these. It is a screenshot of a moment, and it can be a screenshot of a moment that no longer exists.&lt;/p&gt;

&lt;p&gt;The evidence is not the green tick. The evidence is the recorded run: the exact dataset it saw, identified by a hash, not a filename; the exact rule pack and its version; the person who ran it, established through authenticated sign on; the environment it ran in; and a timestamp that cannot be backdated. Capture those and the report becomes reproducible and therefore defensible. Omit them and the report is an assertion, and we have &lt;a href="https://techworkslab.com/insights-oversight-must-be-evidence" rel="noopener noreferrer"&gt;written before&lt;/a&gt; about why an assertion of oversight is not oversight.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Anyone can produce a clean conformance report. The question that matters is whether someone else can reproduce yours, exactly, and show who stood behind it.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  What we actually build around the check
&lt;/h2&gt;

&lt;p&gt;So the check is the small part, and the record around it is the system. In our platform the run is the unit that is captured, not the result. Each conformance run records the dataset it read by content hash, the rule pack and version it applied, the identity of the operator through single sign on, and the environment it executed in. The whole sequence lands in a tamper evident, hash chained audit trail, so the order of events and who caused them cannot be quietly rewritten afterwards. The findings table from the check is one field in that record, not the record itself.&lt;/p&gt;

&lt;p&gt;This is also why we keep saying you should &lt;a href="https://techworkslab.com/insights-validate-the-artifact-not-the-author" rel="noopener noreferrer"&gt;validate the artifact, not the author&lt;/a&gt;. It does not matter whether a person, a script, or an assistant produced the dataset. What you validate is the dataset, and what you keep is the evidence of the validation, bound to the exact thing that was validated. The rule in R took ten lines. The discipline around running it is the part that holds up under inspection.&lt;/p&gt;

&lt;p&gt;If you are building conformance into a pipeline and want it to produce evidence rather than just reports, we are happy to talk it through. You can &lt;a href="https://techworkslab.com/contact" rel="noopener noreferrer"&gt;reach us here&lt;/a&gt;.&lt;/p&gt;

</description>
      <category>rstats</category>
      <category>r</category>
      <category>datascience</category>
    </item>
    <item>
      <title>Pinning an R environment with renv so a result still rebuilds in three years</title>
      <dc:creator>TECHWORKSLAB</dc:creator>
      <pubDate>Sat, 31 Oct 2026 01:32:54 +0000</pubDate>
      <link>https://dev.to/techworkslab/pinning-an-r-environment-with-renv-so-a-result-still-rebuilds-in-three-years-2l2d</link>
      <guid>https://dev.to/techworkslab/pinning-an-r-environment-with-renv-so-a-result-still-rebuilds-in-three-years-2l2d</guid>
      <description>&lt;p&gt;A regulator can ask you to reproduce a result years after you filed it. Not a similar result. The same one, from the same code, to the same digits. That is a high bar for R, because an R script does not carry its dependencies with it. Run it on a machine where one package has moved a minor version and an output can change in ways that are easy to miss and hard to defend.&lt;/p&gt;

&lt;p&gt;The &lt;code&gt;renv&lt;/code&gt; package exists for exactly this. It gives a project its own private library and writes down, in a lockfile, the exact version and hash of every package the project uses. We treat that lockfile as part of the submission record, not as developer convenience. Here is how we set it up, what it actually guarantees, and the gap you have to close yourself.&lt;/p&gt;

&lt;h2&gt;
  
  
  Turning a project into a pinned project
&lt;/h2&gt;

&lt;p&gt;One call bootstraps it. &lt;code&gt;renv::init()&lt;/code&gt; creates a project local library, scans the code for the packages it loads, installs them into that private library, and writes the first &lt;code&gt;renv.lock&lt;/code&gt;.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight r"&gt;&lt;code&gt;&lt;span class="n"&gt;install.packages&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"renv"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;&lt;span class="c1"&gt;# From the project root, once:&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="n"&gt;renv&lt;/span&gt;&lt;span class="o"&gt;::&lt;/span&gt;&lt;span class="n"&gt;init&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;&lt;span class="c1"&gt;# From then on, after adding or updating a package:&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="n"&gt;renv&lt;/span&gt;&lt;span class="o"&gt;::&lt;/span&gt;&lt;span class="n"&gt;snapshot&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="c1"&gt;# record the current state into renv.lock&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;em&gt;After init, the project installs into its own library. The global library on the machine no longer matters.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;The lockfile is the whole point, so it is worth looking at. It is plain JSON, it is meant to be read, and it belongs in version control next to the code.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight r"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="s2"&gt;"R"&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="s2"&gt;"Version"&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"4.4.1"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="s2"&gt;"Repositories"&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="w"&gt;
      &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"Name"&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"CRAN"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"URL"&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"https://packagemanager.posit.co/cran/2026-09-01"&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="s2"&gt;"Packages"&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="s2"&gt;"admiral"&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
      &lt;/span&gt;&lt;span class="s2"&gt;"Package"&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"admiral"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
      &lt;/span&gt;&lt;span class="s2"&gt;"Version"&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"1.1.1"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
      &lt;/span&gt;&lt;span class="s2"&gt;"Source"&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"Repository"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
      &lt;/span&gt;&lt;span class="s2"&gt;"Repository"&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"CRAN"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
      &lt;/span&gt;&lt;span class="s2"&gt;"Hash"&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"a1b2c3d4e5f6a7b8c9d0e1f2a3b4c5d6"&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;em&gt;Illustrative lockfile excerpt. The version and hash together pin the exact package, not just its name.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Three years later, on a clean machine, one call rebuilds the library the script expects. &lt;code&gt;renv::restore()&lt;/code&gt; reads the lockfile and installs each package at the recorded version, verifying the hash as it goes.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight r"&gt;&lt;code&gt;&lt;span class="c1"&gt;# On the machine that has to reproduce the result:&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="n"&gt;renv&lt;/span&gt;&lt;span class="o"&gt;::&lt;/span&gt;&lt;span class="n"&gt;restore&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="c1"&gt;# installs every package at the version and hash in renv.lock&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;&lt;span class="c1"&gt;# Confirm the environment matches the lockfile before you trust an output&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="n"&gt;renv&lt;/span&gt;&lt;span class="o"&gt;::&lt;/span&gt;&lt;span class="n"&gt;status&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;em&gt;renv locks the top of the stack. The bottom of it is your job.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Two gaps renv does not close on its own
&lt;/h2&gt;

&lt;p&gt;renv is excellent at what it does, and it is important to be precise about what it does not do, because the gaps are where false confidence lives.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;It does not install R itself.&lt;/strong&gt; The lockfile records that the project was built under, say, R 4.4.1, but &lt;code&gt;renv::restore()&lt;/code&gt; will not put R 4.4.1 on the machine for you. If you restore under a different R version, a package may build differently or refuse to install. So the R version has to be pinned by whatever stands up the machine, not by renv.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;By default it does not pin where packages come from in time.&lt;/strong&gt; If your repository is the live CRAN, then a package that was yanked or superseded can make a restore fail or drift. The fix is to point the project at a dated snapshot repository, so that the restore always sees CRAN as it looked on one fixed day.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight r"&gt;&lt;code&gt;&lt;span class="c1"&gt;# Pin the repository to a fixed date, so a restore sees a frozen CRAN&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="n"&gt;options&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;repos&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nf"&gt;c&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="n"&gt;CRAN&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"https://packagemanager.posit.co/cran/2026-09-01"&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="n"&gt;renv&lt;/span&gt;&lt;span class="o"&gt;::&lt;/span&gt;&lt;span class="n"&gt;snapshot&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="c1"&gt;# the dated URL is now recorded in renv.lock&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;blockquote&gt;
&lt;p&gt;renv pins the packages. A dated repository pins time. A container pins everything below them. You need all three before you can say a result rebuilds.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  Closing the last gap with a container
&lt;/h2&gt;

&lt;p&gt;The way we actually guarantee a rebuild is to combine renv with a container built on a fixed base image, for example a versioned Rocker image that carries a specific R version and operating system. The container fixes R and the system libraries; renv fixes the packages; the dated repository fixes where those packages came from. The image is built once, tagged, and kept. Reproducing the result three years on means pulling the tagged image and running &lt;code&gt;renv::restore()&lt;/code&gt; inside it, and the answer comes back the same because nothing underneath it was free to move.&lt;/p&gt;

&lt;p&gt;This is the practical core of what we wrote about more generally in &lt;a href="https://techworkslab.com/insights-reproducing-the-environment" rel="noopener noreferrer"&gt;reproducing the environment, not the answer&lt;/a&gt; and in &lt;a href="https://techworkslab.com/insights-validated-sce" rel="noopener noreferrer"&gt;standing up a validated computing environment&lt;/a&gt;. For deterministic R code, reproducing the environment is reproducing the answer. renv is the piece of that stack you can adopt this afternoon, and it earns its place long before anyone asks you to prove it.&lt;/p&gt;

&lt;p&gt;If you are setting up a reproducible R environment for regulated work and want a second pair of eyes on where the gaps are, we are happy to help. You can &lt;a href="https://techworkslab.com/contact" rel="noopener noreferrer"&gt;reach us here&lt;/a&gt;.&lt;/p&gt;

</description>
      <category>rstats</category>
      <category>r</category>
      <category>datascience</category>
    </item>
    <item>
      <title>Rebuilding the CDISC pilot ADSL with admiral, and reading it like a reviewer</title>
      <dc:creator>TECHWORKSLAB</dc:creator>
      <pubDate>Sat, 31 Oct 2026 01:32:45 +0000</pubDate>
      <link>https://dev.to/techworkslab/rebuilding-the-cdisc-pilot-adsl-with-admiral-and-reading-it-like-a-reviewer-2i6g</link>
      <guid>https://dev.to/techworkslab/rebuilding-the-cdisc-pilot-adsl-with-admiral-and-reading-it-like-a-reviewer-2i6g</guid>
      <description>&lt;p&gt;There is a good public dataset for practising this. The CDISC SDTM/ADaM pilot project shipped a small but complete set of SDTM domains, and the R package &lt;code&gt;pharmaversesdtm&lt;/code&gt; makes them available as data frames. It is synthetic, it is open, and it is the same data the &lt;a href="https://techworkslab.com/cdisc" rel="noopener noreferrer"&gt;admiral&lt;/a&gt; documentation builds its own examples on. So we can walk through deriving a subject level analysis dataset, ADSL, from it, and nothing here depends on a client study or a number we are not allowed to show.&lt;/p&gt;

&lt;p&gt;We want to be careful about what this is. It is a method walkthrough, written the way we approach a build, not a benchmark. We are not going to claim a match percentage against a reference ADSL, because a match percentage without the full provenance of both sides is exactly the kind of figure we argue against. What we will do is build ADSL from the pilot data, and then read the result the way a reviewer reads it.&lt;/p&gt;

&lt;h2&gt;
  
  
  Setup
&lt;/h2&gt;

&lt;p&gt;Three packages. &lt;code&gt;admiral&lt;/code&gt; for the derivations, &lt;code&gt;dplyr&lt;/code&gt; for the data manipulation grammar admiral is built on, and &lt;code&gt;pharmaversesdtm&lt;/code&gt; for the pilot SDTM data.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight r"&gt;&lt;code&gt;&lt;span class="n"&gt;install.packages&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;c&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"admiral"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"dplyr"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"pharmaversesdtm"&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;&lt;span class="n"&gt;library&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;admiral&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="n"&gt;library&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;dplyr&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="n"&gt;library&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;pharmaversesdtm&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;&lt;span class="c1"&gt;# The public CDISC pilot SDTM domains, as shipped in pharmaversesdtm&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"dm"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="c1"&gt;# demographics&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"ds"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="c1"&gt;# disposition&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"ex"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="c1"&gt;# exposure&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;&lt;span class="c1"&gt;# One row per subject in DM; this is the spine ADSL is built on&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="n"&gt;dm&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;%&amp;gt;%&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;count&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="c1"&gt;# number of subjects in the pilot DM&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;em&gt;The pilot SDTM domains load as plain data frames, so the whole walkthrough runs on a laptop.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  The spine: start from DM
&lt;/h2&gt;

&lt;p&gt;ADSL has one row per subject, and DM already has one row per subject, so DM is the spine. The planned and actual treatment variables come straight across from the SDTM arm variables. This is a predecessor origin, and it is worth saying so out loud now, because origin is the first thing a reviewer will check later.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight r"&gt;&lt;code&gt;&lt;span class="n"&gt;adsl&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;lt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;dm&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;%&amp;gt;%&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="n"&gt;select&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;DOMAIN&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;%&amp;gt;%&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="n"&gt;mutate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="n"&gt;TRT01P&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;ARM&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="c1"&gt;# planned treatment, predecessor of DM.ARM&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="n"&gt;TRT01A&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;ACTARM&lt;/span&gt;&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="c1"&gt;# actual treatment, predecessor of DM.ACTARM&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Treatment start and end, derived from exposure
&lt;/h2&gt;

&lt;p&gt;Treatment start date is not collected, it is derived, and the derivation is the first place real care is needed. We take the first dosing record where the subject actually received drug, impute a missing day to the first of the month so partial dates do not drop out, and carry the datetime onto ADSL. admiral has a dedicated function for the merge so we do not hand roll a join and a de-duplication.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight r"&gt;&lt;code&gt;&lt;span class="c1"&gt;# Impute partial exposure start dates to the month level, then build a datetime&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="n"&gt;ex_ext&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;lt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;ex&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;%&amp;gt;%&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="n"&gt;derive_vars_dtm&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="n"&gt;dtc&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;EXSTDTC&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="n"&gt;new_vars_prefix&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"EXST"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="n"&gt;highest_imputation&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"M"&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;&lt;span class="n"&gt;adsl&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;lt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;adsl&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;%&amp;gt;%&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="n"&gt;derive_vars_merged&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="n"&gt;dataset_add&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;ex_ext&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="n"&gt;filter_add&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;EXDOSE&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;|&lt;/span&gt;&lt;span class="w"&gt;
      &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;EXDOSE&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;==&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;grepl&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"PLACEBO"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;EXTRT&lt;/span&gt;&lt;span class="p"&gt;)))&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="nf"&gt;is.na&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;EXSTDTM&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="n"&gt;new_vars&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;exprs&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;TRTSDTM&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;EXSTDTM&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="n"&gt;order&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;exprs&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;EXSTDTM&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;EXSEQ&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="n"&gt;mode&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"first"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="n"&gt;by_vars&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;exprs&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;STUDYID&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;USUBJID&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;%&amp;gt;%&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="n"&gt;derive_vars_dtm_to_dt&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;source_vars&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;exprs&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;TRTSDTM&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="c1"&gt;# TRTSDT, a date from the datetime&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;em&gt;The filter is the derivation. It encodes a clinical rule: a zero dose still counts as treatment start for placebo.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;The same pattern, with &lt;code&gt;mode = "last"&lt;/code&gt; and the exposure end date, gives treatment end. Once start and end exist, treatment duration is a single admiral call rather than a date arithmetic expression a reviewer has to verify by hand.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight r"&gt;&lt;code&gt;&lt;span class="n"&gt;adsl&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;lt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;adsl&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;%&amp;gt;%&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="n"&gt;derive_vars_duration&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="n"&gt;new_var&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;TRTDURD&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="n"&gt;start_date&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;TRTSDT&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="n"&gt;end_date&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;TRTEDT&lt;/span&gt;&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="c1"&gt;# derived with the same merge pattern, mode = "last"&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Groupings the tables will need
&lt;/h2&gt;

&lt;p&gt;Age group and a few population flags are not interesting statistically, but they are where silent errors hide, because an off by one in a cut point moves subjects between rows of a demographics table without erroring. Writing the cut explicitly, and keeping it next to the data it classifies, is the honest way to do it.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight r"&gt;&lt;code&gt;&lt;span class="n"&gt;adsl&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;lt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;adsl&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;%&amp;gt;%&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="n"&gt;mutate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="n"&gt;AGEGR1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;case_when&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="w"&gt;
      &lt;/span&gt;&lt;span class="n"&gt;AGE&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;65&lt;/span&gt;&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="o"&gt;~&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"&amp;lt;65"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
      &lt;/span&gt;&lt;span class="n"&gt;AGE&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;65&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;AGE&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;lt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;80&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;~&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"65-80"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
      &lt;/span&gt;&lt;span class="n"&gt;AGE&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;80&lt;/span&gt;&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="o"&gt;~&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"&amp;gt;80"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
      &lt;/span&gt;&lt;span class="kc"&gt;TRUE&lt;/span&gt;&lt;span class="w"&gt;                &lt;/span&gt;&lt;span class="o"&gt;~&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kc"&gt;NA_character_&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="n"&gt;SAFFL&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;if_else&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="nf"&gt;is.na&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;TRTSDT&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"Y"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"N"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="c1"&gt;# safety population: took at least one dose&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;em&gt;Illustrative. The pilot ADSL assembled from DM and EX through named admiral derivations.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Now read it like a reviewer
&lt;/h2&gt;

&lt;p&gt;Building the dataset is the easy half. The half that matters is whether the result survives someone asking hard questions of it. When we open an ADSL we did not write, these are the first checks we run, and they are the same ones we run on our own.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;One row per subject, no duplicates.&lt;/strong&gt; &lt;code&gt;adsl %&amp;gt;% count(USUBJID) %&amp;gt;% filter(n &amp;gt; 1)&lt;/code&gt; should return zero rows. If it does not, a merge somewhere fanned out and the whole dataset is suspect.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Treatment start is not after treatment end.&lt;/strong&gt; A single &lt;code&gt;filter(TRTSDT &amp;gt; TRTEDT)&lt;/code&gt; finds date logic that reversed on a partial date imputation. It is a two line check that catches a class of error no table would reveal.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Population flags are internally consistent.&lt;/strong&gt; A subject with &lt;code&gt;SAFFL == "Y"&lt;/code&gt; and a missing &lt;code&gt;TRTSDT&lt;/code&gt; is a contradiction, because the flag was defined from that very date.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Every derived variable can name its method.&lt;/strong&gt; For each column, we should be able to point to the admiral call that produced it or the SDTM variable it came from. Where we cannot, the variable is not ready, however plausible its values look.&lt;/li&gt;
&lt;/ul&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight r"&gt;&lt;code&gt;&lt;span class="c1"&gt;# The duplicate-key check every ADSL has to pass&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="n"&gt;dups&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;lt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;adsl&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;%&amp;gt;%&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;count&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;USUBJID&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;%&amp;gt;%&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;filter&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="n"&gt;stopifnot&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;nrow&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;dups&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;==&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;&lt;span class="c1"&gt;# The date-order check&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="n"&gt;bad_dates&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;lt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;adsl&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;%&amp;gt;%&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;filter&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="nf"&gt;is.na&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;TRTSDT&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="nf"&gt;is.na&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;TRTEDT&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;TRTSDT&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;TRTEDT&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="n"&gt;stopifnot&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;nrow&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;bad_dates&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;==&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;em&gt;These are not conformance checks in the Pinnacle 21 sense. They are the sanity checks that should pass before conformance checking is even worth running.&lt;/em&gt;&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;The question is never whether the numbers look reasonable. It is whether you can say, for each one, exactly how it was produced and show the step that produced it.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  Why we work this way
&lt;/h2&gt;

&lt;p&gt;A monolithic ADSL program can produce the same columns. What it cannot do easily is answer the reviewer questions above, because the logic for any one variable is tangled with the logic for the rest. Building from named derivations, as we described in &lt;a href="https://techworkslab.com/insights-adam-admiral" rel="noopener noreferrer"&gt;building ADaM datasets with the admiral package&lt;/a&gt;, means traceability is a property of the structure rather than something reconstructed afterwards. On the public pilot data that is a convenience. On a live submission it is the difference between a smooth review and a long one.&lt;/p&gt;

&lt;p&gt;If you want to pressure test your own ADSL build against this kind of reading, or move a monolithic program onto a derivation library, we are happy to help. You can &lt;a href="https://techworkslab.com/contact" rel="noopener noreferrer"&gt;reach us here&lt;/a&gt;.&lt;/p&gt;

</description>
      <category>rstats</category>
      <category>r</category>
      <category>datascience</category>
      <category>clinicaltrials</category>
    </item>
    <item>
      <title>The rules we build regulated analytics tooling under</title>
      <dc:creator>TECHWORKSLAB</dc:creator>
      <pubDate>Sat, 31 Oct 2026 01:31:10 +0000</pubDate>
      <link>https://dev.to/techworkslab/the-rules-we-build-regulated-analytics-tooling-under-1doc</link>
      <guid>https://dev.to/techworkslab/the-rules-we-build-regulated-analytics-tooling-under-1doc</guid>
      <description>&lt;p&gt;Most writing about AI in regulated work argues about capability. Can a model map a dataset, write a derivation, draft a define file. We find the capability question less interesting than the constraint question, because the constraints are what decide whether any of it is usable in a filing. So here are the rules we actually build &lt;a href="https://techworkslab.com/norva" rel="noopener noreferrer"&gt;Norva&lt;/a&gt; under. They are not aspirations. They are the lines we do not cross, and every feature we ship has to live inside them.&lt;/p&gt;

&lt;h2&gt;
  
  
  Rule one: the AI never sees raw data
&lt;/h2&gt;

&lt;p&gt;A model does not need a patient record to help build a derivation. It needs the structure of the data, the variable names, the types, the codelists, the specification. So that is all it gets. The assistive layer works against metadata and specifications, not against the clinical values themselves. The values are processed by validated, deterministic code that runs in an isolated environment with the data mounted read only.&lt;/p&gt;

&lt;p&gt;This is partly a privacy position and partly an engineering one. If the model never touches a subject value, then a whole category of risk simply does not exist: there is no raw data in a prompt, no raw data in a log, no raw data in a place it was not meant to be. We wrote separately about why &lt;a href="https://techworkslab.com/insights-isolate-the-agent-at-infrastructure" rel="noopener noreferrer"&gt;telling an agent not to open the folder is not a control&lt;/a&gt;. The short version is that isolation has to be enforced by the platform, not requested in an instruction.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;The model works on the shape of the data. The deterministic code works on the data.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Rule two: every derivation ships with its validation and a risk score
&lt;/h2&gt;

&lt;p&gt;A derivation that arrives without evidence is not finished, no matter who or what produced it. In our model, the unit of work is not a line of code. It is the derivation plus the test that exercises it, plus a record of what it was checked against, plus a risk score that says how much scrutiny it still needs. The three travel together. You cannot promote the code and leave the evidence behind.&lt;/p&gt;

&lt;p&gt;The risk score is the part people ask about most, so to be plain about it: it is an indicator, not a verdict. It combines things we can measure, such as whether the derivation is a standard pattern or bespoke study logic, whether it touches a variable that feeds a primary endpoint, and whether its inputs changed since the last run. A high score does not block anything. It directs attention. It tells a reviewer which twenty derivations in a study of hundreds are the ones to read first. We covered the broader idea in &lt;a href="https://techworkslab.com/insights-fmea-risk-scoring-submissions" rel="noopener noreferrer"&gt;FMEA risk scoring for a submission package&lt;/a&gt;; the rule here is simply that the score is attached at the point of creation, not bolted on at the end.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Evidence that is generated with the work is cheap. Evidence reconstructed after the fact is expensive, and it is the first thing an auditor distrusts.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  Rule three: a qualified human is the last gate
&lt;/h2&gt;

&lt;p&gt;Nothing becomes a controlled artifact because a model produced it. It becomes a controlled artifact because a qualified person reviewed it and attested to it, and that attestation is recorded. We call the moment this happens the &lt;a href="https://techworkslab.com/insights-ai-freeze-point" rel="noopener noreferrer"&gt;freeze point&lt;/a&gt;. Upstream of it, the assistive layer can draft, suggest, and iterate freely. Downstream of it, the record is fixed and change is controlled.&lt;/p&gt;

&lt;p&gt;The attestation is not a checkbox. It is a scoped electronic signature under 21 CFR Part 11, tied to the specific artifact and the specific version, with the reviewer identified through single sign on and separation of duties enforced so that the person who built a thing is not the only person who approves it. The signature, the identity, and the state change all land in a tamper evident, hash chained audit trail, so the sequence of who did what, and in what order, cannot be quietly rewritten later.&lt;/p&gt;

&lt;h2&gt;
  
  
  What these rules cost, and why we keep them
&lt;/h2&gt;

&lt;p&gt;They cost speed, in the narrow sense. A system that let a model touch raw data, skip the attached evidence, and self approve its output would demo faster and feel more magical. It would also be unusable in a filing, because none of the questions a reviewer asks would have answers. Where did this value come from. Who checked it. How do we know the thing that ran is the thing that was validated. The rules exist so that those questions always have an answer that was captured at the time, not assembled under pressure during an inspection.&lt;/p&gt;

&lt;p&gt;We think this is the honest shape of AI in regulated analytics. The model is assistive and human in the loop. It makes a skilled person faster at producing work that still has to meet the same standard it always did. The rules are what keep the speed from becoming a liability.&lt;/p&gt;

&lt;p&gt;If you are drawing your own version of these lines for a validated environment, we are happy to compare notes. You can &lt;a href="https://techworkslab.com/contact" rel="noopener noreferrer"&gt;reach us here&lt;/a&gt;.&lt;/p&gt;

</description>
      <category>datascience</category>
      <category>ai</category>
      <category>clinicaltrials</category>
    </item>
  </channel>
</rss>
