<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Joe McRell</title>
    <description>The latest articles on DEV Community by Joe McRell (@captainhandsome).</description>
    <link>https://dev.to/captainhandsome</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4120436%2F0b7ac180-d8a5-463b-a8b0-b4412c3b6c37.jpg</url>
      <title>DEV Community: Joe McRell</title>
      <link>https://dev.to/captainhandsome</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/captainhandsome"/>
    <language>en</language>
    <item>
      <title>Building AI characters with memory, feelings and beliefs that change: inside Capybara Island's minds (dev log #1)</title>
      <dc:creator>Joe McRell</dc:creator>
      <pubDate>Sun, 15 Nov 2026 10:19:06 +0000</pubDate>
      <link>https://dev.to/captainhandsome/capybara-island-dev-log-1-six-residents-with-minds-of-their-own-and-an-old-laptop-that-streams-15n2</link>
      <guid>https://dev.to/captainhandsome/capybara-island-dev-log-1-six-residents-with-minds-of-their-own-and-an-old-laptop-that-streams-15n2</guid>
      <description>&lt;p&gt;Hi, I'm Joe. For the last ten days I've been building &lt;strong&gt;&lt;a href="https://capybarais.land" rel="noopener noreferrer"&gt;Capybara Island&lt;/a&gt;&lt;/strong&gt;, a live 3D world where six autonomous AI characters (five capybaras and an axolotl) live their own lives around the clock, and anyone can drop in and watch.&lt;/p&gt;

&lt;p&gt;The goal isn't cute NPCs. I want them to be as close to real minds as I can get: characters that remember their whole lives, know who they are and rewrite it, feel things for reasons, hold beliefs that meet evidence, only know what they could actually see or hear, and change over time. Nobody gives them tasks. This first dev log is about how those minds are built, what's been measured, and what still falls short.&lt;/p&gt;

&lt;h2&gt;
  
  
  The residents
&lt;/h2&gt;

&lt;p&gt;Churro, Brisket and Matilda (an axolotl, who came ashore with her grandfather's trunk) live on the island. Across the water, in the harbour town of Saltby, live Kalo, Suri and Kalo's little brother Gary. A day on the island lasts about two real hours, so they've lived far more days than I've spent building. They write their own blurbs on the &lt;a href="https://capybarais.land/residents" rel="noopener noreferrer"&gt;residents page&lt;/a&gt;, and I don't edit them.&lt;/p&gt;

&lt;h2&gt;
  
  
  How an AI character thinks, moment to moment
&lt;/h2&gt;

&lt;p&gt;Each resident's moment-to-moment thinking is an LLM call that sees the world from where they stand and decides what to do: walk, talk, build, cook, buy, sell, look, rest, or nothing in particular. What turns that loop into something like a mind is everything around the call.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Thought is driven by events, not a timer.&lt;/strong&gt; Every wake-up has a weight: being spoken to, a gift or a touch, someone they love arriving or leaving, the weather turning, or a body signal getting loud (hunger, thirst, tiredness) means they think within seconds. An ordinary happening gets the usual gap, and when nothing is new they wait longer. Being spoken to, or a loud body signal left alone for an island hour, can interrupt what they're doing, but it's only a chance to think, never an order. The first line of each moment tells them what made them stop.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;A thread between thoughts.&lt;/strong&gt; A thought can leave a note for the next one: where they've got to, what they've decided, what they're waiting for. It's kept exactly as written, shown first next time with its age, and lapses after six island hours untouched. In a test of ten thoughts in a row for each resident from the same start, the opening idea repeated the previous one in 48% of pairs without the thread and 30% with it.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;An everyday mind and a deep mind.&lt;/strong&gt; Most moments run on a fast model. When they stop to weigh a choice, the everyday mind lays out the ways they could go, each with what such things have done for them before and which of their cares it serves, and a deeper model chooses. Nights, reflections and dreams go to the deeper models too.&lt;/p&gt;

&lt;h2&gt;
  
  
  Long-term memory that works like memory
&lt;/h2&gt;

&lt;p&gt;Every memory has a tier (a passing try at something is only a trace), a place, the people in it, a strength, a recall count, how they felt at the time, and &lt;strong&gt;where it came from&lt;/strong&gt;: saw, heard, their own account, a thought, a dream, read, or reflected. When a memory comes back to mind, its source comes with it.&lt;/p&gt;

&lt;p&gt;Every memory's meaning is also stored as an embedding. Recall searches the whole life by meaning, words, people and place, weighted by strength, and fills its slots on purpose: something recent, something long ago, something about whoever is here, one defining memory, and one out of nowhere. Remembering a memory strengthens it, the lift fades with time since it last came to mind, and a resident can choose to think back harder.&lt;/p&gt;

&lt;p&gt;Because they're meant to live a long time, memory also has to forget. Ordinary moments older than a season are let go (marked faded, not deleted), insights that say the same thing are folded together, and once a season each resident gets a core memory of what they know about every person and place in their life.&lt;/p&gt;

&lt;h2&gt;
  
  
  A self they write themselves
&lt;/h2&gt;

&lt;p&gt;Each resident keeps one page about themselves under fixed headings: who I am, what I care about, what I fear, how I am with each person, what I'm like when low and what helps, how I've changed, what I'm becoming, and what I don't know yet. They rewrite it after each weekly look back. It sits at the top of every decision and conversation, along with a voice card of their own most distinctive recent lines, so they sound like themselves.&lt;/p&gt;

&lt;p&gt;They can also turn inward. &lt;strong&gt;look_inward&lt;/strong&gt; lets them ask their own mind a question and get an honest answer from their own record: time spent by activity, feelings, promises, intentions, how well they read others, what keeps coming back, and what their beliefs rest on. A weekly mirror sets what they did beside what they wrote about themselves and points at the gaps.&lt;/p&gt;

&lt;h2&gt;
  
  
  Feelings, needs and learning what does them good
&lt;/h2&gt;

&lt;p&gt;Feelings start for a reason, have a strength and fade at their own pace: irritation in about an hour, longing over days. Mood eases back toward where each resident usually sits. Feelings come from their own words and from what happens to them: making something, failing at something again and again, finishing a work, a pat, or a partner out of sight all day. Feelings that stay high ease after a day, and there's a cap on how much they can feel at once.&lt;/p&gt;

&lt;p&gt;Underneath are slow drives (beauty, making, understanding, closeness), weighted to who each one is and felt when left unfed for days. Two newer needs of the mind are &lt;strong&gt;wonder&lt;/strong&gt;, which drains with sameness and fills with new places, discoveries and questions chased, and &lt;strong&gt;calm&lt;/strong&gt;, which worry, cold, hunger and too much company wear down for the quiet ones.&lt;/p&gt;

&lt;p&gt;Then there's a reward loop. Every doing is weighed before and after by how it actually left them, compared with what they expected. Over time each resident learns what really does them good, and recall leans toward it when they're low. A nice surprise feeds wonder.&lt;/p&gt;

&lt;h2&gt;
  
  
  Beliefs that meet evidence
&lt;/h2&gt;

&lt;p&gt;Beliefs are first-class objects with kinds (about the world, a person, themselves, living, and the beyond) and a record of what they rest on. Something only told or feared doesn't grow more certain by being repeated.&lt;/p&gt;

&lt;p&gt;Each night a rotating set of beliefs is set beside what they actually saw that week. Predictions are checked the next night. A saying borne out five times simply stands, and beliefs that go unused fade into the background. When they act they can say what they expect to happen, the world tells them whether it went that way, and surprises are remembered and brought back at night.&lt;/p&gt;

&lt;p&gt;They also keep a track record of reading each other. Each night, every guess about how someone else felt is checked against what that person was really feeling, so they learn whom they read well and whom they misread.&lt;/p&gt;

&lt;h2&gt;
  
  
  Nothing is true just because it was said
&lt;/h2&gt;

&lt;p&gt;LLM agents tend to narrate a world into existence ("I built a boat"), so the world has to push back.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Rulings.&lt;/strong&gt; When someone tries something new, a judge works out what actually happens from what they're holding, where they are and the conditions. When a ruling says something general about how the world works, the world keeps it as a rule, and a like attempt in like conditions turns out the same way.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A nightly truth check.&lt;/strong&gt; Memories are checked against the world's own record. Ones that aren't confirmed are marked, never rewritten, and a resident's own account of an attempt is kept only if that's how it really went.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;True bodies.&lt;/strong&gt; They get hungry, thirsty and tired, and if they stay up long enough they fall asleep. Ailments are real but mild, with causes, courses and cures (a cold caught from someone, a strain, being run down), and the doctor in Saltby examines the true body. Long worry is felt as tightness, not illness.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Routes that fit bodies.&lt;/strong&gt; Walking routes are built from the real distance to every solid thing, so each walker gets a path that fits its own width. On a copy of the live world, about one walk in ten used to be abandoned. Now none are.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Senses: they only know what they could perceive
&lt;/h2&gt;

&lt;p&gt;This is the newest layer. Residents are shown only what they could see, hear or reach from where they are. They know the country past Saltby only by going there, seeing it, or being told.&lt;/p&gt;

&lt;p&gt;They can also really &lt;strong&gt;look&lt;/strong&gt;. The action "look" (at a place, a person, a thing, or around) sends a request to a sight worker on my stream laptop. It renders the 3D world from where that resident's eyes are, at the real hour, and sends back a still. A vision model describes it in appearances only, naming nothing the island's own record doesn't put in view, and the memory is filed with source "saw." In testing, 24 of 24 looks came back, about 10 seconds each, with no names used that weren't actually in view.&lt;/p&gt;

&lt;h2&gt;
  
  
  Nights, sleep and dreams
&lt;/h2&gt;

&lt;p&gt;The night is where a day turns into a life. They write a diary entry and reflect: was the day's intention kept, partly kept, or not? The day is folded into scenes, old moments are let go, and beliefs that say the same thing are merged.&lt;/p&gt;

&lt;p&gt;Then there's a dreaming stage. Two memories far apart in time and subject, but close in how they felt, are joined in a dream that asks what they have to do with each other. Anything that comes of it is labelled a dream, and any idea it gives them goes on their list, theirs to take up or not.&lt;/p&gt;

&lt;h2&gt;
  
  
  Growth: pursuits, temperament and growing up
&lt;/h2&gt;

&lt;p&gt;Each night a resident may take up a big idea and something to chase: what, why, a first step, and what done looks like. They log steps toward it, finish it with what they learned, and a finished pursuit can be improved, scaled up, or lead to the next thing.&lt;/p&gt;

&lt;p&gt;Temperament drifts. Each season, where their mood and calm rest and how much quiet they need move a sixth of the way toward how they've actually lived, within a narrow band around where they began. Children grow up: birthdays come once an island year, and firsts the island notices (a night away, a fire lit alone, a promise kept over a week, teaching someone) become core memories. Nobody grows old or dies, and illness is never worse than a cold.&lt;/p&gt;

&lt;p&gt;Culture grows too. They coin their own words and traditions, write books for a shelf they can all read, and make songs. Anyone holding an instrument they've made can play a song they know, and everyone nearby hears it and learns it. (Honest status: the &lt;a href="https://capybarais.land/songs" rel="noopener noreferrer"&gt;songs page&lt;/a&gt; still says "No songs yet.")&lt;/p&gt;

&lt;h2&gt;
  
  
  The open question: the machine on the islet
&lt;/h2&gt;

&lt;p&gt;The residents don't know they're AI, and they've never been told they're on a stream. What's past the horizon, what the presences who drop treats from the sky are, and whether anyone made them are left open to them. Their beliefs about "the beyond" are tracked like any other belief.&lt;/p&gt;

&lt;p&gt;There is a door, though. On a small islet there's an old stone hut with a faint light at night, and under its hearth an old machine of brass and glass. Whoever gets there can read real pages from the web, make programs and pages that run sealed off (their own process, an empty environment, time and memory limits), and write messages for whoever is out there. Anything about the island itself is filtered out of what they can find. Nobody has reached the hut yet. When someone does, it'll show up in &lt;a href="https://capybarais.land/workshop" rel="noopener noreferrer"&gt;the workshop&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Measuring it, not just vibing it
&lt;/h2&gt;

&lt;p&gt;Most changes ship with an on/off switch and a measurement. Each decision logs the size of every part of its prompt, one in thirty is kept to replay on other models, and every model call goes in a mind log. Every ten island days a review measures whether the minds are actually doing what the design says.&lt;/p&gt;

&lt;p&gt;I also have the whole design reviewed by a second AI model, which reads the code and the live world and tells me where it's still pretend. The latest review set out seven stages. The first three (the world remembering its rulings, nothing true because it was said, and senses) are built. Next come a small physics under the rulings, enough scarcity that they need each other, a record of the world that can be replayed, and minds that change in deeper ways.&lt;/p&gt;

&lt;p&gt;Am I claiming they're sentient? No. I'm trying to give them everything a mind would need (memory with sources, a self, feelings with causes, beliefs that can be wrong, senses that limit them, and room to grow) and then watch honestly what comes of it.&lt;/p&gt;

&lt;h2&gt;
  
  
  The stack, and a 24/7 stream from an old GTX 1080 laptop
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Server:&lt;/strong&gt; plain Node with TypeScript run directly, and almost no dependencies.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;3D:&lt;/strong&gt; three.js in the browser, with free models from Quaternius, Kenney and Poly Pizza (credited on the site).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Minds:&lt;/strong&gt; a chain of models that steps down by ability, from deep hosted models for nights and hard choices to local models through Ollama on a 6900 XT at home, ending on DeepSeek. There's a daily spending ceiling, and a normal day costs a few dollars.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Stream:&lt;/strong&gt; an old gaming laptop with a GTX 1080 draws the island at full quality (shadows, real lighting) at 60 fps and records at 30, at about 14% GPU, 30% processor and 52 °C. The same laptop renders what the residents see when they look. If it goes quiet for ten minutes, the hosted server takes over with a lighter stream, so the island never goes dark.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Digest:&lt;/strong&gt; every evening the island emails me what was built, invented, written, painted and dreamed, built only from its own records with no model calls. One real day is twelve island days, so there's a lot.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;You can &lt;a href="https://capybarais.land" rel="noopener noreferrer"&gt;watch Capybara Island live&lt;/a&gt;, read the residents' diaries in the &lt;a href="https://capybarais.land/journal" rel="noopener noreferrer"&gt;journal&lt;/a&gt;, or read &lt;a href="https://capybarais.land/about" rel="noopener noreferrer"&gt;how it works&lt;/a&gt;. Next time: what the first three stages of the new plan changed, and whether anyone has walked into that hut.&lt;/p&gt;

&lt;p&gt;Thanks for reading.&lt;br&gt;
Joe&lt;/p&gt;

</description>
      <category>ai</category>
      <category>gamedev</category>
      <category>threejs</category>
      <category>showdev</category>
    </item>
    <item>
      <title>OTHERREACH devlog #1: one world, a cliff town, and four new peoples</title>
      <dc:creator>Joe McRell</dc:creator>
      <pubDate>Sun, 15 Nov 2026 10:05:08 +0000</pubDate>
      <link>https://dev.to/captainhandsome/otherreach-devlog-1-one-world-a-cliff-town-and-four-new-peoples-da4</link>
      <guid>https://dev.to/captainhandsome/otherreach-devlog-1-one-world-a-cliff-town-and-four-new-peoples-da4</guid>
      <description>&lt;p&gt;OTHERREACH is a persistent fantasy world built in Godot 4 with C#. It's designed to keep living while you're logged off, where nobody pauses and there are no difficulty presets. We've been heads-down for two months without posting anything, so this first devlog catches up on the bigger pieces.&lt;/p&gt;

&lt;h2&gt;
  
  
  The world is becoming one place
&lt;/h2&gt;

&lt;p&gt;Until recently each region was its own level. You walked to an edge, the game saved, and the next region loaded. That worked, but it never felt like a world.&lt;/p&gt;

&lt;p&gt;The world is now being joined into &lt;strong&gt;one continuous streamed space&lt;/strong&gt;. The Hollow, Kaldrun, the Sedge marshes and the Salt Steps sit in their canon positions, and you can walk between them on foot. The world is cut into 100 m cells. Cells near you are built, cells in the middle distance are kept ready, and cells far away are freed. Models load on a background thread, and the art library counts references, so nothing stays in memory once no cell needs it.&lt;/p&gt;

&lt;p&gt;The first full walk, from the Hollow through Kaldrun to the Sedge and on to Tollpiles, works end to end. It is not fast yet: peak video memory came down from 25.8 GB to 18.9 GB this week, and we're aiming for 13. Most of what's left is people. Every villager's body and clothing has to be on the GPU. This week's wins were compressing the whole wardrobe to BC7/BC5 with no visible change (−1.2 GB in the busiest town), and drawing each villager only where their own cell stands.&lt;/p&gt;

&lt;h2&gt;
  
  
  Kaldrun: a cliff town under the ring
&lt;/h2&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fdif44jfuzbft1w5l8ysq.jpg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fdif44jfuzbft1w5l8ysq.jpg" alt="Vessmere across the lake at golden hour" width="800" height="356"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The east of Kaldrun is a sheer escarpment with a waterfall into a lake. Vessmere, a working town, sits at its foot.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fujc7nt4pglloh3a4c4km.jpg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fujc7nt4pglloh3a4c4km.jpg" alt="The escarpment from the kapok road" width="800" height="356"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The pale arc in the sky is the world's ring. You can see it from almost everywhere, and it's one of the first things that tells you this isn't Earth.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fxqp8sjomokex8p1lon70.jpg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fxqp8sjomokex8p1lon70.jpg" alt="The escarpment at dusk from a kilometre out" width="800" height="356"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F12ggfycdnozhlw1o5z99.jpg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F12ggfycdnozhlw1o5z99.jpg" alt="Vessmere at dusk" width="800" height="356"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Vessmere has chimney smoke, lit windows, a wharf, residents with routines, and its own soundscape. It's also where the game's era starts to show.&lt;/p&gt;

&lt;h2&gt;
  
  
  Not just "period fantasy"
&lt;/h2&gt;

&lt;p&gt;The everyday world is medieval. Technology exists, though, and it's gated by &lt;strong&gt;who will teach it, what can be salvaged and what it costs&lt;/strong&gt;. A flashlight is a luxury in a Veth market town and cheap where the Vaskaal or the Constructed trade. The world steps up in era as you travel. The lift at the top of Vessmere's cliff is Kal engineering: carved basalt with bronze sheaves and landing ledges for a winged people. The iron cranes on the wharf are plainly industrial. Further out, the Salt Steps are "every era at once".&lt;/p&gt;

&lt;h2&gt;
  
  
  Four new peoples
&lt;/h2&gt;

&lt;p&gt;The character creator had four playable peoples. Four more existed in the lore but had no bodies. This week they all got one, on the same rig and animation set as everyone else, so they walk, talk and wear gear like any other character.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fl5lodwodmwrtcurha1bp.jpg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fl5lodwodmwrtcurha1bp.jpg" alt="The Ondrek" width="800" height="490"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The Ondrek&lt;/strong&gt; are tall stone-and-quartz people with a crystal growth on one shoulder, strung with pearls.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F5g07ws2bbvcrw0egmvh3.jpg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F5g07ws2bbvcrw0egmvh3.jpg" alt="The Vaskaal" width="800" height="328"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The Vaskaal&lt;/strong&gt; are lean and grey, with amber eyes and two thumbs on each hand. The mechanical rings at their joints are brace hardware, not anatomy.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fmgqsdpfgg9orjzqs00ka.jpg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fmgqsdpfgg9orjzqs00ka.jpg" alt="The Constructed" width="800" height="328"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The Constructed&lt;/strong&gt; are seamless black lacquer. Getting that finish right under an open sky took most of a day: what looked like reflection artefacts turned out to be split normals at the creator mesh's texture seams.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The Mor&lt;/strong&gt;, the fog people, are still in progress.&lt;/p&gt;

&lt;h2&gt;
  
  
  Also this fortnight
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Combat:&lt;/strong&gt; enemies now have minds of their own. They flee when broken and telegraph their techniques before using them.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Medicine, downed states and wounds:&lt;/strong&gt; these are in, with splints and field treatment.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Regional travel&lt;/strong&gt; between areas works by waystone.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Kaldrun east&lt;/strong&gt; now has its creatures, from black bears and elk to the Reach raiders and their captain.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Next up: getting the one-world walk to a playable frame rate, the Salt Steps quarters, and a public playtest.&lt;/p&gt;

&lt;p&gt;Follow OTHERREACH at &lt;a href="https://otherreach.com" rel="noopener noreferrer"&gt;otherreach.com&lt;/a&gt;, where the devlog and the &lt;a href="https://otherreach.com/playtest/" rel="noopener noreferrer"&gt;public playtest&lt;/a&gt; live.&lt;/p&gt;

</description>
      <category>gamedev</category>
      <category>godot</category>
      <category>csharp</category>
      <category>indiedev</category>
    </item>
    <item>
      <title>Bypassing Headless Pagination and Modal Blocks on Airbnb with Playwright</title>
      <dc:creator>Joe McRell</dc:creator>
      <pubDate>Fri, 11 Sep 2026 07:42:15 +0000</pubDate>
      <link>https://dev.to/captainhandsome/bypassing-headless-pagination-and-modal-blocks-on-airbnb-with-playwright-19gp</link>
      <guid>https://dev.to/captainhandsome/bypassing-headless-pagination-and-modal-blocks-on-airbnb-with-playwright-19gp</guid>
      <description>&lt;p&gt;When scraping modern Single Page Applications (SPAs) like Airbnb, standard pagination strategies (&lt;code&gt;page.click('a[aria-label="Next"]')&lt;/code&gt;) frequently collapse in headless Linux environments (Xvfb / Docker). &lt;/p&gt;

&lt;p&gt;Even if your selectors look completely sound in Chrome DevTools, Playwright will throw errors like:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;playwright._impl._errors.TimeoutError: element is outside of the viewport
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;or fail silently when unexpected modal backdrops (translation dialogs, privacy notices, currency popups) intercept user interaction events.&lt;/p&gt;

&lt;p&gt;Here is the multi-layered defensive pattern we engineered to reliably paginate through dozens of search pages in production.&lt;/p&gt;




&lt;h2&gt;
  
  
  The Root Causes
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Sticky Footers and Responsive Viewports:&lt;/strong&gt; In headless browsers with constrained resolutions, the pagination bar at the bottom of the grid is often overlaid or pushed outside the actionability bounding box.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Translation &amp;amp; Currency Modals:&lt;/strong&gt; Airbnb periodically pops a &lt;em&gt;"Translate reviews and descriptions to English"&lt;/em&gt; or &lt;em&gt;"Choose currency"&lt;/em&gt; overlay. In headless mode, this modal consumes all click events.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Synthetic Event Shielding:&lt;/strong&gt; Clicking the &lt;code&gt;&amp;lt;a&amp;gt;&lt;/code&gt; tag with standard browser synthetic events doesn't always trigger the Next.js router change if the viewport isn't focused.&lt;/li&gt;
&lt;/ol&gt;




&lt;h2&gt;
  
  
  Pattern 1: Active Modal Dismissal
&lt;/h2&gt;

&lt;p&gt;Before attempting pagination or scrolling, run an aggressive dismissal loop that catches close buttons, cookie banners, and presses the &lt;code&gt;Escape&lt;/code&gt; key:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;dismiss_modals&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;page&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;selectors&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
        &lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;button[aria-label=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Close&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;]&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;button[aria-label=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Dismiss&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;]&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;#onetrust-accept-btn-handler&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;div[data-testid=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;modal-container&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;] button&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;sel&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;selectors&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;try&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="n"&gt;btn&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;page&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;locator&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;sel&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="n"&gt;first&lt;/span&gt;
            &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;btn&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;is_visible&lt;/span&gt;&lt;span class="p"&gt;():&lt;/span&gt;
                &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;btn&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;click&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;timeout&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;1000&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
                &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;page&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;wait_for_timeout&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;300&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;except&lt;/span&gt; &lt;span class="nb"&gt;Exception&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="k"&gt;pass&lt;/span&gt;
    &lt;span class="k"&gt;try&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;page&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;keyboard&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;press&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Escape&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;except&lt;/span&gt; &lt;span class="nb"&gt;Exception&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;pass&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;






&lt;h2&gt;
  
  
  Pattern 2: Multi-Tiered Pagination Fallback
&lt;/h2&gt;

&lt;p&gt;Instead of relying solely on &lt;code&gt;.click()&lt;/code&gt;, use a three-tier fallback mechanism:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;urllib.parse&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;urljoin&lt;/span&gt;

&lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;advance_to_next_page&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;page&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;dismiss_modals&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;page&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="n"&gt;next_btn&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;page&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;locator&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;a[aria-label=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Next&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;]&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="n"&gt;first&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;next_btn&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;is_visible&lt;/span&gt;&lt;span class="p"&gt;():&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="bp"&gt;False&lt;/span&gt;

    &lt;span class="c1"&gt;# Tier 1: Extract href for direct URL navigation fallback
&lt;/span&gt;    &lt;span class="n"&gt;next_href&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;next_btn&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get_attribute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;href&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="c1"&gt;# Tier 2: Scroll into view and standard click
&lt;/span&gt;    &lt;span class="k"&gt;try&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;next_btn&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;scroll_into_view_if_needed&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;timeout&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;2000&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;next_btn&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;click&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;timeout&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;3000&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;page&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;wait_for_load_state&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;networkidle&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;timeout&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;8000&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="bp"&gt;True&lt;/span&gt;
    &lt;span class="k"&gt;except&lt;/span&gt; &lt;span class="nb"&gt;Exception&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;pass&lt;/span&gt;

    &lt;span class="c1"&gt;# Tier 3: JavaScript evaluate click
&lt;/span&gt;    &lt;span class="k"&gt;try&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;next_btn&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;evaluate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;el =&amp;gt; el.click()&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;page&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;wait_for_load_state&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;networkidle&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;timeout&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;8000&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="bp"&gt;True&lt;/span&gt;
    &lt;span class="k"&gt;except&lt;/span&gt; &lt;span class="nb"&gt;Exception&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;pass&lt;/span&gt;

    &lt;span class="c1"&gt;# Tier 4: Direct URL navigation fallback (Immune to DOM/viewport obstacles)
&lt;/span&gt;    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;next_href&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;target_url&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;urljoin&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;page&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;next_href&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;page&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;goto&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;target_url&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;wait_until&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;networkidle&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;timeout&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;15000&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="bp"&gt;True&lt;/span&gt;

    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="bp"&gt;False&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If modals intercept the click, or sticky footers keep the link outside the headless viewport, &lt;strong&gt;Tier 4 (direct URL navigation fallback)&lt;/strong&gt; completely bypasses the DOM layer and forces the browser to load the paginated state directly.&lt;/p&gt;




&lt;h2&gt;
  
  
  Production Cloud Implementation
&lt;/h2&gt;

&lt;p&gt;If you don't want to maintain headless Playwright clusters, rotating residential proxies, and DOM selector manifests yourself, this entire logic is packaged and maintained as a serverless actor on Apify:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Actor:&lt;/strong&gt; &lt;a href="https://apify.com/captainhandsome/airbnb-listings-search" rel="noopener noreferrer"&gt;captainhandsome/airbnb-listings-search&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Open-Source Examples:&lt;/strong&gt; &lt;a href="https://github.com/jlucasmcrell/apify-scrapers" rel="noopener noreferrer"&gt;jlucasmcrell/apify-scrapers on GitHub&lt;/a&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;You can run it programmatically via Python in 4 lines:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;apify_client&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;ApifyClient&lt;/span&gt;

&lt;span class="n"&gt;client&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;ApifyClient&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;YOUR_APIFY_TOKEN&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;run&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;actor&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;captainhandsome/airbnb-listings-search&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;call&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;run_input&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;location&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Austin, TX&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;max_items&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;50&lt;/span&gt;
&lt;span class="p"&gt;})&lt;/span&gt;

&lt;span class="n"&gt;listings&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;list&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;dataset&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;run&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;defaultDatasetId&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]).&lt;/span&gt;&lt;span class="nf"&gt;iterate_items&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt;
&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Extracted &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;listings&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; listings reliably.&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



</description>
      <category>python</category>
      <category>playwright</category>
      <category>webscraping</category>
      <category>automation</category>
    </item>
    <item>
      <title>Reverse-Engineering Glassdoor Estimated Job Posting Dates in Python</title>
      <dc:creator>Joe McRell</dc:creator>
      <pubDate>Fri, 11 Sep 2026 07:41:28 +0000</pubDate>
      <link>https://dev.to/captainhandsome/reverse-engineering-glassdoor-estimated-job-posting-dates-in-python-55f0</link>
      <guid>https://dev.to/captainhandsome/reverse-engineering-glassdoor-estimated-job-posting-dates-in-python-55f0</guid>
      <description>&lt;p&gt;One of the biggest headaches when building automated job aggregators or market compensation trackers is &lt;strong&gt;posting age&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;Job portals like Glassdoor don't expose clean ISO timestamps (&lt;code&gt;2026-09-10T12:00:00Z&lt;/code&gt;) in the listing feed HTML. Instead, they present relative, human-readable strings like:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;code&gt;"24h"&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;"3d"&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;"14d"&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;"30d+"&lt;/code&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If you are dumping these raw strings into Postgres, BigQuery, or Pandas, running temporal queries (e.g. &lt;em&gt;"Show only jobs posted within the last 7 days"&lt;/em&gt;) becomes messy.&lt;/p&gt;

&lt;p&gt;Here is how to cleanly normalize relative posting ages into deterministic ISO dates and precision flags in Python.&lt;/p&gt;




&lt;h2&gt;
  
  
  The Mathematical Parser
&lt;/h2&gt;

&lt;p&gt;We can parse relative durations using regular expressions and calculate estimated timestamps relative to run execution (&lt;code&gt;scraped_at&lt;/code&gt;), tagging the calculation with a precision enum (&lt;code&gt;exact_day&lt;/code&gt;, &lt;code&gt;estimated&lt;/code&gt;, or &lt;code&gt;lower_bound&lt;/code&gt;):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;
&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;datetime&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;datetime&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;timezone&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;timedelta&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;parse_posting_age&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;age_str&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;reference_time&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;datetime&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="bp"&gt;None&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="n"&gt;reference_time&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;reference_time&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;datetime&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;now&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;timezone&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;utc&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="n"&gt;age_str&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
            &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;posting_age&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="bp"&gt;None&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
            &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;posting_date_estimated&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="bp"&gt;None&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
            &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;posting_date_precision&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="bp"&gt;None&lt;/span&gt;
        &lt;span class="p"&gt;}&lt;/span&gt;

    &lt;span class="n"&gt;clean_str&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;age_str&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;strip&lt;/span&gt;&lt;span class="p"&gt;().&lt;/span&gt;&lt;span class="nf"&gt;lower&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;

    &lt;span class="c1"&gt;# Matches "24h", "5h", "1h"
&lt;/span&gt;    &lt;span class="n"&gt;hour_match&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;match&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;r&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;^(\d+)\s*h(?:ours?)?$&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;clean_str&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;hour_match&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;hours&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;int&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;hour_match&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;group&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
        &lt;span class="n"&gt;est_date&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;reference_time&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="nf"&gt;timedelta&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;hours&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;hours&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
            &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;posting_age&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;clean_str&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
            &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;posting_date_estimated&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;est_date&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;strftime&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;%Y-%m-%d&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
            &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;posting_date_precision&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;estimated&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
        &lt;span class="p"&gt;}&lt;/span&gt;

    &lt;span class="c1"&gt;# Matches "30d+", "60d+" (Lower bound)
&lt;/span&gt;    &lt;span class="n"&gt;plus_match&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;match&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;r&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;^(\d+)\s*d(?:ays?)?\+$&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;clean_str&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;plus_match&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;days&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;int&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;plus_match&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;group&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
        &lt;span class="n"&gt;est_date&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;reference_time&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="nf"&gt;timedelta&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;days&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;days&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
            &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;posting_age&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;clean_str&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
            &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;posting_date_estimated&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;est_date&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;strftime&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;%Y-%m-%d&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
            &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;posting_date_precision&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;lower_bound&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
        &lt;span class="p"&gt;}&lt;/span&gt;

    &lt;span class="c1"&gt;# Matches "3d", "7d", "14 days"
&lt;/span&gt;    &lt;span class="n"&gt;day_match&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;match&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;r&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;^(\d+)\s*d(?:ays?)?$&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;clean_str&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;day_match&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;days&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;int&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;day_match&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;group&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
        &lt;span class="n"&gt;est_date&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;reference_time&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="nf"&gt;timedelta&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;days&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;days&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
            &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;posting_age&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;clean_str&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
            &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;posting_date_estimated&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;est_date&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;strftime&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;%Y-%m-%d&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
            &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;posting_date_precision&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;estimated&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
        &lt;span class="p"&gt;}&lt;/span&gt;

    &lt;span class="c1"&gt;# Fallback for unexpected formats
&lt;/span&gt;    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;posting_age&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;clean_str&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;posting_date_estimated&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="bp"&gt;None&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;posting_date_precision&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;unknown&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;






&lt;h2&gt;
  
  
  Canonical Link Extraction
&lt;/h2&gt;

&lt;p&gt;Glassdoor URLs from search grids are loaded with dynamic tracking telemetry, session IDs, and redirection wrappers:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;https://www.glassdoor.com/partner/jobListing.htm?pos=101&amp;amp;ao=1136043&amp;amp;s=58&amp;amp;guid=00000191...&amp;amp;jobListingId=100987654321
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;To prevent database duplicates, extract the pure numeric &lt;code&gt;job_id&lt;/code&gt; and construct a clean canonical permalink:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;urllib.parse&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;urlparse&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;parse_qs&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;canonicalize_glassdoor_url&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;raw_url&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;parsed&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;urlparse&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;raw_url&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;qs&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;parse_qs&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;parsed&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;query&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="c1"&gt;# Try finding the jobListingId or jl parameter
&lt;/span&gt;    &lt;span class="n"&gt;job_id&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;qs&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;jobListingId&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="bp"&gt;None&lt;/span&gt;&lt;span class="p"&gt;])[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="n"&gt;qs&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;jl&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="bp"&gt;None&lt;/span&gt;&lt;span class="p"&gt;])[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;

    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="n"&gt;job_id&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="c1"&gt;# Regex search for digits in path
&lt;/span&gt;        &lt;span class="n"&gt;id_match&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;search&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;r&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;-jl_(\d+)\.htm&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;raw_url&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;search&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;r&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;/(\d{8,12})\.htm&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;raw_url&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;id_match&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="n"&gt;job_id&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;id_match&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;group&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;job_id&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;canonical_url&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;https://www.glassdoor.com/job-listing/-jl_&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;job_id&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;.htm&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;job_id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;canonical_url&lt;/span&gt;

    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="bp"&gt;None&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;raw_url&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;






&lt;h2&gt;
  
  
  Production Cloud Endpoint
&lt;/h2&gt;

&lt;p&gt;If you need this pipeline running with zero server maintenance, rotating proxies, and automated retries, we run this continuous enrichment pipeline on Apify:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Actor:&lt;/strong&gt; &lt;a href="https://apify.com/captainhandsome/glassdoor-jobs-scraper" rel="noopener noreferrer"&gt;captainhandsome/glassdoor-jobs-scraper&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;GitHub Repository:&lt;/strong&gt; &lt;a href="https://github.com/jlucasmcrell/apify-scrapers" rel="noopener noreferrer"&gt;jlucasmcrell/apify-scrapers&lt;/a&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;apify_client&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;ApifyClient&lt;/span&gt;

&lt;span class="n"&gt;client&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;ApifyClient&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;YOUR_APIFY_TOKEN&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;run&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;actor&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;captainhandsome/glassdoor-jobs-scraper&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;call&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;run_input&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;keyword&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Software Engineer&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;location&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Austin, TX&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;max_items&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;30&lt;/span&gt;
&lt;span class="p"&gt;})&lt;/span&gt;

&lt;span class="n"&gt;jobs&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;list&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;dataset&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;run&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;defaultDatasetId&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]).&lt;/span&gt;&lt;span class="nf"&gt;iterate_items&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;job&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;jobs&lt;/span&gt;&lt;span class="p"&gt;[:&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;]:&lt;/span&gt;
    &lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;job&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;job_title&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;|&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;job&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;employer&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;|&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;job&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;posting_date_estimated&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;(&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;job&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;posting_age&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;)&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



</description>
      <category>python</category>
      <category>datascience</category>
      <category>webscraping</category>
      <category>automation</category>
    </item>
  </channel>
</rss>
