DEV Community

The Bike Shed

297: We’re Making A Pixar Movie!

Chris gives the deets on that new new – (he joined a startup!) and laments about the back button being so complicated. Steph talks about extracting an untrustworthy service and likens the scenario to making a Pixar movie. You don't wanna miss this hero's journey!

Transcript:

STEPH: Yes, I was getting text messages from you where you were like, “Go on without me.”

CHRIS: [laughs] Leave me behind!

STEPH: [laughs] No developer left behind!!

CHRIS: Hello and welcome to another episode of The Bike Shed, a weekly podcast from your friends at thoughtbot about developing great software. I'm Chris Toomey.

STEPH: And I’m Steph Viccari.

CHRIS: And together, we're here to share a bit of what we've learned along the way. So, Steph, how's your week going?

STEPH: Hey, Chris. It's been a very busy week. There's been a lot going on. But the most delightful part of my week has been that Eric Bailey, another thoughtboter and also a former guest on this show, has a tiny, little baby bunny living in his backyard, and so he has been sharing updates about this little baby bunny. In fact, he's been sharing some pictures on Twitter as well. So I'll include a link in the show notes so other people can experience the joy. Also, the name of the bunny gets me every time. But they have named the bunny “Corndog.”

CHRIS: Checks out. It seems like a very obvious name for a small bunny.

STEPH: It gets me because that's such a big name. I don't know why it's a big name, but it feels like a big name for a little bunny.

CHRIS: I can say yeah, it's a cornball. Yeah, that's a large name. And so a tiny bunny is a...it's like Little John from Robin Hood. It's perfect.

STEPH: [chuckles] I kept referring to him as Corn Nugget, I guess, because of size. But yes, it’s not corn nugget; it’s Corndog. [chuckles] So watching Eric's little bunny has been delightful and a wonderful addition to the week. How about you? How has your week been?

CHRIS: My week has been great. I was off on vacation last week (so you had a guest on), which was fun to just take a week off and reset the system. But actually, this week has been interesting. It was my first full-time week with a new startup that I have joined. I think, yeah, that seems to be the truth in the world. So a bit of a shift from what I've been doing for the last year and a half, almost something like that. The reason there's hesitance in my voice is because I've actually been working with this organization for six months-ish, depending on how you count it. I've been having conversations, and then it’s slowly grown over time where it was just conversations, and then it was an afternoon a week, and then one day a week, and then two, and three. And finally, we decided we think we've got an idea. We've got a thing that we want to build. And so I am the developer on this team, but we are an early-stage startup trying to build something. I'm now full-time on the project. I rotated down the other projects that I was working on from a freelance consulting perspective, and now I'm trying something new. So it's a very different vibe. Even though I'd been working with the organization for a long time, this week just felt so much more real. And there was so much more space, so much more room for activities, having a full week to actually work on things. So yeah, it's very exciting, it’s very new, it's very early stage, so all of those things are true. But there are a lot of great aspects to that, and I'm super excited about it.

STEPH: That is some big news. That's a big change too. Well, I guess with consulting, there are the stresses that go into consulting and then changing projects and managing the projects that you're taking on. But then to joining a team and such an early startup team too...anytime, someone says startup life, I'm always like, well, tell me more. How calm is the startup life, or how uncalm is this startup life?

CHRIS: It's somewhere between calm and uncalm, I would say, but in a, I would say purposeful and intentional way. I was looking for...this has largely been true over the entire time that I was freelancing, but freelancing was a way for me to keep the lights on, and stay engaged with tech, and continue working, and frankly, have more conversations and meet more organizations. But I was looking for something that I could engage with a bit more. I was looking for, largely, something like this. So it definitely is occupying a different space in my head than, say, any individual consulting client where with consulting, I was pretty rigid, you know, these are the hours that I'm working. When I'm off the clock, I'm really not thinking about it too much. I'm responsive if I see an incident or something like that or if the database falls over; I’m going to look at that on the weekend but otherwise, largely not doing anything. Whereas with this project, I'm somewhat purposefully allowing it to have a little bit more space in my head off-hours, that sort of thing. And I'm more invested in the work. It's not just a thing that I'm doing, but it's a project that I believe in. It's something that I want to exist in the world. And so, I'm engaged with it in a different way in that manner. I'm also engineer number one, so I'm choosing all of the technologies and setting the standards. Thankfully, there's a lot of good thoughtbot material out there that I can link to, which is great. But yeah, so it's mostly within the context of what I think startups can be. The expectations and the way that the team is working is very reasonable. And I think it's more for my own self. I'm allowing it to occupy a little bit more of my space, but in a fun way so far.

STEPH: Well, along that line, in terms of choosing the tech stack and starting greenfield, I am curious to hear more about the type of project that you're going to be working on. But I'm also recognizing y'all may be in stealth mode. Is that where you're at, or can you talk a bit more about the type of work you'll be doing?

CHRIS: We're stealth-ish right now, I would say, partly because we're likely in the process of rebranding, and renaming, and things like that. So partly it's just like, oh, I probably shouldn't say that. But at some point, this will become public, and so at that point, I can probably be a little bit more open about it. But at the end of the day, we're building a financial product, FinTech sort of thing. And the tech stack is relatively straightforward. I'm actually using my preferred tech stack is...I got to choose, so it's Rails, Inertia, and Svelte with some TypeScript because why not? And I love it, and it's fantastic. I continue to believe deeply in that tech stack. So, yeah, that's most of what I think is good to say now. But I think over the coming weeks, I'll be able to say more and share more. And I certainly will be able to talk about the details of building and growing a team and things like that.

STEPH: Awesome. Yeah, you answered my other question too. I was going to ask what tech stack you chose.

CHRIS: I chose the tech stack, the one with the acronym, which I don't even know...the STIR stack I think we went with or something.

STEPH: I was about to say I don't remember the acronym. [laughs]

CHRIS: I think I never committed to an acronym previously, and then that was the one that got thrown around on the internet. I think I just was like, in the next episode of The Bike Shed, I'll choose an acronym so STIR, why not?

STEPH: I like it, causing a stir.

CHRIS: But yeah, so it's a pretty sizable shift in my life. But frankly, I don't even know exactly the shape that the coming weeks will have. So it will be interesting to report back as things evolve and as new concerns and considerations come up. But, yeah, we'll save that for future weeks. For now, what else is up in your world?

STEPH: Yeah, it's been an interesting week. There have been really two things on my mind, so one of them has been focused on writing a task that's going to process a sizable CSV. And then it's going to essentially enqueue a bunch of jobs and send off a bunch of data to other third-party systems. So that's been a big focus of the week. The other topic is what I'm going to call extracting an untrustworthy service into its own service. And I know that’s a bit vague, but I've got both of those topics. So which one would you want to hear about first?

CHRIS: I definitely want to hear about both. But because you veiled it in mystery and said, “An untrustworthy,” that one's just going to call to me a little bit more. So yeah, what about this extracting and untrustworthy service? What more can you say there?

STEPH: Good question. I'm glad that you picked the mysterious one and started there. That feels right. So this is a part of our codebase, and it's very related to also the task that I'm writing. So to provide a bit of context, this particular portion of the codebase manages a big part of where we are sending data from our application over to third-party systems. And it's a very important feature of our application. And it's also probably one of the gnarliest sections of our application in terms of there are tons of conditionals based on which type of service we're sending to or the discreet customer that we're sending it to, and any particular preferences that they need and how we're sending that data. And then there's also just a lot of room for ambiguity and errors. And when we are sending that data, was it actually successful? And what if it was successful, but we still got back error messages? What does that mean? Is that successful with warning? And so there are just a lot of unknowns.

It's also one of the less tested areas of the codebase. So even though it's important, we really don't feel confident making changes at this point until we've added some more test coverage. And testing it can be a beast because right now, we really just want to add some security around that section of the codebase. So we're often going for high-level tests, which are then our slower tests, but then also means it's hard to test the more granular aspects of that code. This is that untrustworthy section of the code in terms that we're a bit skittish to make changes, but yet it's a very active part of the codebase, so not the best place to be. But we also recognize that this part of the codebase would be really well-fitted to live outside of the application. It really doesn't need to live with the rest of the application. And there are other services that need to be able to talk to the service as well. So instead of having it grouped together, which -- It's funny. I see your eyebrows go up when I talk about -- For people who can't see, Chris raised his eyebrows when I talked about extracting this to another service. [chuckles]

CHRIS: That doesn't sound like me at all. I don't ever…

STEPH: [chuckles] And since we do have other services that need to be able to pull data or to talk to this particular portion of the codebase, we are looking to then move it out into its own application, so that way, it can stand alone. It can focus on this one task, and then other services can benefit from it as well. And there's been an interesting discussion around, well, we need to make changes to this codebase. And we also have some recognition that we need to make improvements. Do we go ahead and go heads down for a bit and improve the section of the codebase, add more test coverage, get to understand more of what this code does, where the risks are? Or do we go ahead and extract it in its current form to the new greenfield space and just essentially port it, and then we work on it from that space? And so, there's been a conversation around which one do we do first? And I'll tell you my thoughts, and then I'd love to hear yours.

As one of the primary individuals that's been working in this codebase, my stance has been let's leave it in place for now because I want to build some confidence around what this does. So I really want to have some confident understanding about the requirements, about when we extract this, what is that going to look like? But also, I feel like I'm in a place where I'm starting to understand the beast enough that I want to continue that progress and add some testing around it before then we just move it to this new location. And I can't decide if that's one of those decisions where like, I just feel too close to it, and extracting it feels risky to me. So I feel like we're adding on this extra level of complexity. Like, this is already code that's hard to understand. And then we're going to add this network connection on top of that where then we have to talk to it in a different way. And in my mind, that's adding another level of risk and another level of having to debug this service. So my current approach is let's leave it in place. Let's try to identify some low-hanging fruit. Let's go ahead and add some more tests. And I feel pretty good about that decision. I'm curious, what are your thoughts?

CHRIS: I have a bunch of them. The first is that the story that you're telling here feels like the hero's journey of software development. Like, all right, we got this gnarly bit of the code. It's super important. It's super complicated. It doesn't really have any test coverage for historical reasons that are complicated, but here we are. What do we do? That story feels so true. It feels like there are nine Pixar movies about it if Pixar made movies about writing code, and they would be great movies.

STEPH: That's amazing. [laughs] I would watch those movies.

CHRIS: I think of it like Katrina Owen’s therapeutic refactoring, which I feel like is probably my most referenced...It's one of my two most referenced talks that I bring up on the show all the time, but it is almost exactly about that sort of thing. We've got this gnarly piece of code. It's super important, but nobody really knows how it works. But we know it does work, which is an interesting bit. And so to the question of would you extract as is or would you try and shore it up before you extract it? I am 100% on the side that you are on, which is let's shore this thing up before we move it over. Because moving it over, like you said, that's going to add the additional complexity and failure modes of network latency, network timeouts, async disconnects, whatever, any of those complexities. That's another set of failure modes that you'll be introducing or just complexity and things that you have to think about. So that feels complicated. Also, there's probably a poor analogy that I have in my head. But imagine that you're moving, and your bedroom is just a complete mess, and you're like, oh, there are some old to-go food containers over there. And I haven't done my laundry in a couple of weeks. I'm just going to throw it all on a blanket and take it to the new house, and I'll figure out what I want to keep on the other side. It's like, that doesn't feel like the right move. I would definitely throw some things out before I move to a new house. So I definitely lean in to let's clean this up and understand it so that when it's in the new place, we have a slightly more contained, understood, manageable version of the software to try and extract to a service.

STEPH: I feel very judged for my moving style.

CHRIS: [laughs] I mean, obviously, with software, you're doing the one thing. But did I just describe exactly how you move house?

STEPH: [laughs]

CHRIS: To each their own now, you know, whatever works for you.

STEPH: No, I'm with you. I'm definitely the person that's going to clean up first before I put stuff in boxes. I'm going to try to give away as much stuff as possible.

CHRIS: It's a great time to just figure out what's true in your life or what's true in your software. I am intrigued. So yes, I did raise my eyebrows when you mentioned extracting a service and other services talking to each other. In particular, the way you described this piece of the system, I would be surprised if there weren't data requirements and/or transactional consistency things that you wanted to uphold. And that's one of the main things that causes me concern when we're extracting services is if this thing still needs to know about a bunch of different pieces of data and if it's going to make multiple updates to different records where if one succeeds and the other doesn't, we should roll back the whole thing. You lose all of that by moving to a service. And so that's where my broad…like, I'm always going to question if we're going to surface this. So I'm intrigued. Is this thing a very functional piece of your system where some data comes in, some stuff happens, and you get data out at the end of the day? Or is it more operating on related data within your system and potentially updating records after the fact?

STEPH: Yeah, that's a great point. For this area of the codebase, it does feel more functional in terms that we have data, and we essentially want to notify other people that we have this data, and then we want to share it with them. So there is still that coupling of where we need access to those values. So if we're sending it over to the new system, either that new system needs to be able to read from the same database, or we have to send all of those details over to the new system. So then it can build up the message and then send it over to the other third-party systems. So it feels more functional, but there are still some of those requirements that we need to think about.

CHRIS: Okay. That definitely clarifies things. And I wouldn't say that I have a unified theory of services. But what you're describing feels like the type of service that I'm more open to. It sounds almost like a SendGrid where I want to deal with all of my application data. And then I send a bunch of structured data over to SendGrid, and their job is to send an email and retry as necessary or send a text message or even do a voice call if it's Twilio or something like that. And so they're really good at those weird things and the failure modes that exist in those communication channels. But that's not logic that I need to live in my app. And so what you're describing there definitely makes sense as something that could comfortably be extracted to a service and not have more complexity be introduced by that. You did mention something about services talking to services and other things. So is the idea that this would be extracted, then other parts of the system would also use it to communicate out messages or something like that.

STEPH: Yeah, one of the motivations for extracting this is because we have another application that also wants to perform similar behavior. So now we have two applications that need to do similar work, and they feel more in that line of functional work where it would be great if we could share this. But it doesn't fit in the space that we want to extract it in regards to extract it to a gem and make it shareable. It feels more appropriate for it to be its own service and then also capture. Because the other nice thing that we want to include that we're doing now as well is we want to capture feedback from whenever we are sending that data over to other systems. We want to know, hey, how did it go? Did you give us back that successfully, but maybe with some warnings or some errors? Maybe you accepted the data, but then you also gave us a response about something else.

I think one really important question to consider is when is it trustworthy enough to extract? Because we know we're headed down this path. So at what point are we ready to then go ahead and extract this over to its own service? And that was the more interesting conversation because I think those who were in favor of extracting it now had the concern that we can't add test coverage in its current form. So my first response was if I need to make changes and I can't add test coverage, I will sound the alarm, and we will reconsider. But my goal right now is to turn this untrustworthy service into a little more trust. Just dial up the trust a little bit further, and then we can port this over. So then, as we do add some network complexities on top of this, we will at least have more faith and understanding the underlying behavior of the system. But then we still want to understand that it's not going to be perfect. And we're not going to wait until it's perfect before we do extract it. But that's the tale or the mysterious extracting an untrustworthy service. So I think it will be an interesting journey. And it was a very interesting conversation that I was excited to have your thoughts because I know you and I often lean so far away from extracting stuff to a service that it was an interesting conversation to have around; well, this code is a bit of a mess. When do we start to tackle that mess?

CHRIS: I like that you didn't even frame it necessarily in terms of that, but I still definitely got there. I was like, wait, wait, wait, but let's actually talk about whether or not. But this is definitely the sort of thing that I think makes sense to consider as a service extraction. I think the question that you're asking around when do we feel good enough in its current state to do the extraction? That's right on the line of art in the software world as opposed to the science of this is how we connect HTTP. So I'm very interested to see where you get to both with that question and how you actually make that decision and then how the extraction goes. And I imagine this will be the sort of thing that goes on for a bit of time. So it feels like we could make a mini-story arc that'll span a couple of episodes, and you can follow the characters on their journey. This is the Pixar movie. We're making a Pixar movie.

STEPH: We're making a Pixar movie. They're missing an entire genre for their Pixar movies. If they just appeal to developers, that'd be wonderful. I’m so in for that. We should write Pixar.

CHRIS: There are more developers every day, so think Hackers meets Up. That's what we're going for. We're just going to fuse those two together. It's going to pull at your heartstrings, but it's also going to talk about hacking the Gibson. It's going to be great.

STEPH: Oh man, you reached for the most heartfelt one going for Up. That one has the toughest beginning. [laughs]

CHRIS: That's what I'm going for here.

STEPH: For anyone that hasn't seen Up, you can go watch the beginning of it. Just be prepared.

CHRIS: And if anyone hasn't seen Hackers, also be prepared. [laughs]

STEPH: Which is me. I haven't seen Hackers.

CHRIS: All right. You still haven’t. All right, that's a thing we need to work on.

STEPH: [chuckles]

CHRIS: But cool. Okay. So we're going to work on the Pixar movie. You're going to update us because we need to actually gather the information. But yeah, we'll come back to that in future episodes. But shifting gears just a little bit, actually, I have a couple of things, two small things, and then one more sizable thing that is more just like, I'm confused. So yeah, we're going to go in that order. Thing number one is, we are, again, it's a very early-stage startup that I'm working with. And part of what we're doing that I really like is that we are talking to potential customers, potential end-users of the application doing lots of user interviews, which is a thing that I have more from a distance seen often. But now, because we're actually working as a distributed team, we're remote because that's the nature of the world right now. We'll probably meet each other in person at some point, but that's down the road. But all of these conversations are happening over Zoom calls, all of these user interviews. And so I made the suggestion that we use a tool to actually manage those. And so we're using a tool called EnjoyHQ, I think is the name of it. There's another similar tool called Aurelius. We can put the links in the show notes for both of those. But what it does is it basically makes the video available after the fact. I think it automatically transcribes it, and then it allows you to annotate and add notes and things like that, which is great for aggregating this body of information that we're collecting over time as we do all of these user interviews and start to tag common themes that we're seeing. And bringing them together will also allow us to revisit them. But for me as the developer, I've been to a few of them, but not as many as the rest of the team. And what's great is I've now taken to...as I'm doing more mundane…cleaning up email or whatever sort of tasks, I will just put on one of these videos in the background at 2X. And what's great is I can now just hear literally the voice of the users of the application. What are the words that they're choosing? How are they talking about it? What matters to them? What doesn't matter to them? What do they get really passionate about? And it's been just such a wonderful thing to have available. It's almost like a podcast of our app that we're building, and it's like, that's awesome.

STEPH: I love that. Yeah, I would love to be able to hear from people that are using the application. And like you mentioned, just turn it on in the background so that way I can process what they're saying. But then, I don't know, depending on what they're saying, maybe it needs full attention or otherwise, maybe you're able to just absorb little bits and pieces while you're hacking away on something else. And now I've got the word hacking stuck in my mind. [laughs]

CHRIS: It's the best word to describe what we do. Yeah, there's definitely a version of someone should be reviewing...someone's actually doing the interview, so they're going to be very close to it. And then there maybe is a secondary someone's watching it closely and trying to glean, and categorize, and all of that. And I could potentially be any one of those, but I really like this version of this is just a background soundtrack that I'm exposing myself to so that I'm all the more immersed in the problem space that we are working on. And it's one of the things that I fundamentally believe about software development is developers shouldn't be hidden in the corner just writing code. We should always care about what the end-user wants, and what better way to get there than to actually hear their voice and hear the words that they're using. So this is a magical little trick that I have now found that I'm like, oh my God, this is amazing.

STEPH: Funny enough, I had a similar experience this past week where I realized I was feeling very disconnected from the people that are using the application and also the people that are setting priorities for the work that our team is doing. And that is something that I'm very accustomed to with thoughtbot that we always want to be part of the team. We're not necessarily just we can churn through a backlog. But we also really want to be in touch with product decisions, and share opinions, and then also be in touch with users too. So I had some similar revelations this week where I realized I was feeling very disconnected where I was picking up tickets, and I was like, I don't really understand why this is great or how this is helpful. And so, I shared that with the team, and someone encouraged me to attend a specific meeting. And that was wonderful because then I got to hear from the people who were creating those tickets and then giving them a high priority because something was urgent and why it was urgent. And having that insight was huge to me. And I realized that it was incredibly motivational as well. Because then I'm like, yes, okay, I understand how this is going to impact someone. And I'm now very encouraged to get this done.

CHRIS: I think that idea, that ethos of wanting to get into the user persona and understand that better is a very strong thoughtbot ideal. So it’s unsurprising both of us share that. But yeah, that was a really great thing and particularly a tool that facilitated that in a really straightforward way, which I appreciated. Another thing that I used this week, which I've talked about at length in a previous episode, so we can link to that episode, but it's a project called dry-monad. So there's dry-rb is the collection of, I think, a set of gems, but dry-monad is one that allows for defining sequential tasks, so tasks that you have to do a bunch of steps in order and the outcome of a previous step will be the input of the next part of the process. So it can fail in a bunch of ways like, okay, fetch this thing from the network and then look up a user based on that. And then get the user's profile, which may or may not exist, and then assuming that all of that's gone well, actually persist to this new record, to the database. And they're really finicky to write that sort of sequential processing. And so I actually had written that thing manually. And part of it was I'd wrapped the whole thing in a database transaction, but I was trying to make it so that if something went wrong, I would manually roll back the transaction. And then I wanted to return an object to the caller that indicated that things had failed and an error message or something like that. And that was actually really hard to do because of the way transactions work.

The mechanism that I was using was apparently deprecated in Rails. And so the whole thing was just kind of confusing, and it was a bit messy in the code. And I knew in the back of my mind that dry-monad exists. I've used it before. I've really enjoyed it. But I was trying to minimize the amount of new technologies that I'm bringing in this early on in the project. It's like, yeah, I'll bring that in when I need it. But finally, I was like, you know what? I think I've reached that point. I grabbed it, brought it in, and I haven't worked with it in a while, but I was very quickly able to refactor my class to use dry-monad. It cleaned it up immensely. The tests remained identical, which was really interesting. I didn't have to change anything on the test side. And one of my tests was failing before and then passed because of the introduction of dry-monad. And yeah, it was just like a win-win-win, and also the fact that I was able to revisit dry-monad as a library and just get running with it again was really interesting to me because it is a bit complicated and interesting in how it works. But again, I was able to just sort of pick it up and run with it. So that was wonderful. And I will now all the more staunchly suggest that folks reach for that when they have more complex, procedural type code that they need to write.

STEPH: I remember you highlighting dry-monad before in previous episodes and talking about the pain of writing that sort of procedural code, but then we also want to return something helpful. And I looked at it briefly, but I haven't used it. But now that you are reminding me of it, I'm very interested in it because I agree that process is difficult to write, at least in Ruby it's difficult to write. I understand the hesitancy that you have around bringing something in that's new. But then if you recognize that it's going to be a theme in your application around this is something that we're going to do a fair amount, and we want to do it in a clean, efficient way, then it starts to feel more reasonable to say, “Okay, I'm bringing in something new, but it is representative of how we want to handle this step or this type of process in our application.” So it's not just bringing in a gem to handle one small area of the code, something that we could have written, but it is elevating our process and our system.

CHRIS: Yup. Indeed. In this case, these are command objects within the system. That's actually the name that I got from the creator of the project. That was his suggestion on Twitter as to what to call these objects. And it's a pattern that I do want to encode and has become the standard within the application for any of these more complex processing tasks. So, again, we'll link to the previous episode. I talked about it in more depth and the ideas behind it. Railway Oriented Programming is a phrase that's used, which talks about how to sequence failures or successes and whatnot together. And there's some good material behind it, more general, but yeah, wonderful, little library.

STEPH: What is Railway Oriented Programming? I'm not familiar with that term.

CHRIS: That refers to the sequential processing that I was describing. So imagine that you have a bunch of different steps where first you fetch from the network to get this record, then using what you got back, you look up a user in your database, then you fetch that user's profile. Then you do something else. Each of those steps along the way could fail. And so the railway metaphor is the track is going forward, but if at any point you branch off the track because of a failure, then you're in the failure track, and that's a different thing. And so it's a very...the dry-monad or other similar Railway Oriented Programming or monads generally I think is the actual...it's the words in there. And I wish it wasn't in there because it's such a complicated word. But that idea is the fundamental, underlying thing that's going on there. And it is conceptually somewhat complicated, but if you don't try and think about the category theory behind it, and you're just like, well, I want to do a bunch of stuff, and it may fail at any point, and I want to return either a success message with everything having gone well or an error message at the point that it failed and stopped processing, then that's what this thing does, and it's fantastic at it.

STEPH: Okay, cool. Thank you.

CHRIS: You are welcome. And I think there's a bit more in the previous episode as well. So if that sounded interesting to anyone, I think I rambled more in a previous episode about it and probably better because I feel like I was more prepared that time than this time.

STEPH: Well, along those lines of running a process and then being able to fail at any moment, I'm going to circle back to that other topic that I highlighted where most of my week has been focused on writing a task that is processing a CSV, something probably a number of us have done at some point in our career but processing a number of rows, and then sending and queuing jobs to then send data to a third party system. And it was really interesting less so because of the processing of the CSV and then enqueuing jobs. But it was more of the reporting that went behind it and the process that went into writing this task. So Joël and I were pairing on this task. Joël being another thoughtboter and also a former guest on this show. And we had an interesting process of where we started with one, let's do the simplest thing. Let's get it done. Let's also check through the CSV because you're often going to find stuff that doesn't align with what you expect it to when it's a CSV that's provided from an external source. One of the risks that we highlighted right away was how are we going to get the CSV on the server? Because we just have this one CSV that we need to run. We don't want to add it to the repo, and we can't generate it ourselves. So how are we actually going to get the CSV in a place that we can run this in a production mode? I learned that I could pass a CSV as standard input into the Rake task. So then I could actually run it locally because we're using AWS. So I could inform AWS to run this task, but then I could actually stream the CSV into the task that way. And that was really nice because then we no longer had the question of how are we going to get this file on the server?

CHRIS: That's interesting. I didn't know...Yeah, the streaming of it from local to remote is an interesting one. On Heroku, I will typically open up a bash prompt, so Heroku run bash. And then, I will curl the file down onto the server and then run it locally. But that’s an ephemeral dyno. It may die at any point. There are various things that could go wrong there. So that's always interesting. I imagine a similar thing could be done, but I don't know, actually, if you can directly stream into a Heroku dyno like that, which is an even more straightforward one because I end up having to bounce a file off of a random. Like, I'll often put it in a Gist or a Pastebin or something like that. And then I'll curl it down to the server, and yeah, this is interesting.

STEPH: Yeah, I'm also not sure the specifics of how it would work with Heroku. But it was a really nice process for us to be able to use versus having to then read the file from, like you mentioned, curl it from somewhere else and then be able to parse it that way. Two other things that were top of mind for working on this task is one, item potency. You're going to rerun it, friends. At some point, your task is either going to bomb, and it's going to err. And then you're going to have to triage and run it again. Or whoever requested that you run this task and they said, “Oh, it's just temporary. We're just going to run the once,” that's not true. You're going to run it again. So keep in mind how to make that safe, that you can rerun it. And then that won't be its own scenario that then you have to triage and figure out.

CHRIS: Item potency is one of those critical ideas, and I just wish the word were different. I feel ridiculous every time I say it. And I feel like I have to push my glasses up on my nose, and I'm like, well, have we considered item potency for this? But it's such a good idea. And it's the sort of thing that...you're totally right. Every time you're doing this sort of thing, it is something that you should consider. And we use GET requests, and they have rules about it. And it's such a good idea and such an important idea. And I just wish the word were different so that I felt more comfortable using it in polite conversation.

STEPH: [laughs] I don't know why… and this may be sharing too much of myself. But the song Under Pressure by Queen and David Bowie the Under Pressure song has been in my head. But I've been replacing the under pressure part with item potent. So it's [singing] under pressure, and I've been [singing] item potent. [laughter] And that has just been my song for the week.

CHRIS: You've normalized it enough for me now that I'll just hear you singing it every time, and I'll be like, this is a nonsense word. We're fine. We can just go – [laughs]

STEPH: That's what I'm here for, to turn technical terms into nonsense. [laughs]

CHRIS: It's really what this show is about at the end of the day. So you are our hero.

STEPH: I just have to work on more lyrics for the song. I really just have that one line, that one hook. [laughs]

CHRIS: Now I just want to scrap the rest of the episode and just come up with lyrics to item potent. [chuckles] But maybe we don't do that.

STEPH: [laughs]

CHRIS: Maybe that'll be after the credits B-roll, something like that.

STEPH: The other way I do phrase that question is I'm like, what happens when it fails? And that always feels like a safe way. Because if I ask someone like, “Hey, is the item potent?” It feels more natural for people to be like, “Oh, it's fine. It doesn't need to be.” But if you say, “What happens when it fails?” It's harder for someone to say, “Oh, it's never going to fail. [laughs] There is nothing that could go wrong.” So it feels like a more intentional question in regards to how are we going to handle this when we need to rerun it? The other part that really came in handy was the fact that we spent more time on the reporting as well. So we really wanted to know what happened when we are processing all of these rows. So were there any invalid rows? And if we do encounter an invalid row, do we want to just stop processing and stop right there, or do we want to keep moving? Do we have any rows that didn't match, a row in our database, and how do we capture those? And because it's item potent, maybe we want to capture skipped rows so then that way when we rerun it, we can see okay, well, we skipped, you know, a thousand rows because we'd already run them before. And all of that reporting has been so handy because we're also using this to triage. Like, hey, we're sending a bunch of messages to this third-party system. We reach out to that third-party group, and we say, “Hey, we sent you all of this. This is how it went. Let us know how it went on your end.” And then, we can have a more collaborative discussion around what happened on their end versus what happened on our end, and then we can make tweaks to each system.

So overall, it felt more of that run-of-the-mill task where you're going to write a Rake task, you're going to parse a CSV. But something about the reporting really resonated with me because in the past, when I've written Rake tasks, I've leaned more on the this is temporary, so it's okay if it's not great. But the reporting has been so crucial that from now on, I always want reporting from any Rake tasks that I run, and I want to know what happened. And then I also want to be able to rerun it. And I'm very wary of any time that someone says, “Oh, this is temporary,” because then I also think that leads to interesting discussions around testing. Because initially, when we started this, we were under some pressure. Hey, that goes back to my song. We were under some pressure for writing this particular task. And then the question came up: do we want to test it? And to be frank, testing a Rake task isn't great; it’s not fun, which is one of the reasons we get out of a Rake task as quickly as possible and put it into a class. So that was also one of the drivers or one of the conversations that went against, like, oh, this is temporary. So it's okay if it's not the best code. It's okay if it's not tested. And then I was more of an advocate for, like, well, I don't feel good about this. And I'm rerunning the Rake task every time I want to confirm that the changes that I've made are correct. And so once I hit that manual labor point where I'm like, okay, I'm testing this. I just don't have automated tests for this, that then I actually started adding test coverage around it.

CHRIS: I'm so excited that we have transcripts, particularly for that last minute that you were just talking about, because I feel like that was a mini master class in software development. And more generally, there's been almost like a poetic something to the two different topics that you’ve brought up today are the sort of mundane, very real things that actual software development is made almost entirely of. It's not often that we're just starting with a greenfield app and building a new thing. I happen to be doing that this week, but it's rare. It's going to be over very soon. And then I'm going to be in the world of oh; we have to backfill a bunch of data. How are we going to do that? Or we have this portion of the code that, frankly, we should have been testing more, but we didn't. How do we deal with that? And these murky, gray areas where there isn't a clear answer and you have to go with intuition, and you have to...a bunch of the things that you just listed as these good heuristics that you have around how you think about software. I'm just really excited for the transcript to that because that was awesome.

STEPH: I'm so glad you enjoyed it because I think it's not until right now where I'm processing this and talking about it with you that it is...I was trying to think earlier, like, why is this so interesting? Why am I so excited to bring this here to this conversation? And I think it is for the reasons exactly that you said, that it does feel like one of these...this is a mundane task. We write a task; we process some things; we send some data. We do that all the time. But then it's all the other bits around it, and the other ways that we've been bitten, and how we avoid those scenarios, and then how we identify a risk like when someone says, “Oh, it's temporary. It's fine.” That part, I think, is always the very interesting aspect of writing software.

CHRIS: Do you consider this sort of stuff the distraction from the work or actually the work? And in my experience, this makes up a lot of the work. And treating it like what you were saying about testing like, “Yeah, that thing where you're telling me that it's going to be temporary and we probably don't need to test it, I've been told that before,” and I just want to spot-check that real quick. Or what you said of the when I was manually testing, and I crossed a threshold where I'd done that enough, that now adding a test harness around that totally makes sense. It's worth the investment at that point. Those little heuristics that we build up over time are the things that are hard to get. And so, yeah, I love that conversation.

STEPH: I really like how you also asked and then responded to that question around is this distraction, or is this the work? And I am wholeheartedly with you that this is the work. This is the part of the work that I do find interesting, and knowing when to make those trade-offs, and when you've hit a decision point, and which direction you're going to go, and being able to recognize something that otherwise could have been a fire. It could have been much worse in terms of if we'd built a task that wasn't robust. Because of course, then the second time that I ran it, you know, emphasis on the second time that I ran it because we needed to do it again to process more data. It erred halfway through, and I panicked in the moment. But then I was like, oh yeah, this is fine. We planned for this. This was in the game plan. So it goes back to that we want the calm execution. We want to plan so we are back in that calm state. We want calm software. And this feels very in line with how do we make this more calm?

CHRIS: I love that theme that you're bringing up there, which I think is a theme that we've touched on a bunch of times. I think we actually have an episode called Seeking Calm. And I think that little title there, as much as I love the nonsense titles that we have for most of the episodes, that one I think really captures the theme that a lot of what we talk about is in orbit around yeah, we want it to be calm. We don't want things to be on fire every day. And what does it look like to build software with that in mind?

STEPH: Yeah. I also love that theme. And I like that it's something that we have surfaced and then really stuck to because it resonates deeply with me. But that's pretty much all I have for my Rake task adventure. What else is new in your world?

CHRIS: Well, I have one more hopefully quick thing. I'm going to try and boil this down to its essence, but I ran into, let's call it, a subtlety. It's not an issue. It wasn't a bug per se. But looping back to the previous episode that you and I recorded together, we talked a bunch about multi-step forms, which was a great conversation in and of itself. But I eventually completed the feature that I was working on, put it up for acceptance. And the product manager who was reviewing it highlighted a couple of different things. They recorded a video which, as an aside, I love that as a way to do acceptance and show what's going on and talk through it. There were a couple of smaller issues, which I was able to resolve very quickly, but there was one more interesting one that I've extracted this as future work because it was too complex to try and solve in the moment. But basically, what's happening is imagine that we have a two-step form. So there's the first page of the form. The first form that you see is for your name. So it's just an input that says, “Name,” and you fill in your name and then you hit continue. That posts to the server. We save off that data. And then, we redirect you to the next page on the form, which is, say, phone number. So two steps. We start with name; we go to phone number. What happens if you type in your name, you hit continue, everything processes correctly. You end up on the phone number page, but you hit back. What do you think happens?

STEPH: I would expect to go back to the name field and probably expect my name to be populated but would also be fine if it's not.

CHRIS: I like that you would be fine with the fact that it's not, if it's not, because it's not is the answer. And what's unfortunate is so if someone goes back, they will see the unpopulated form, so not filled out. But if they reload at that point, we will serialize down the value and pre-fill the input with their saved data. And so that inconsistency is not great. It's all the more unfortunate because as I tried to resolve it, I'm like, oh, okay, this feels like a solvable thing. I just need to tell Chrome, “Hey, if someone hits the back button, do a better thing than what you're doing.” I needed a way to instruct Chrome or whichever browser because this should be a standards-level thing. And there are things in the HTTP spec about this. So there's the Cache-Control Header is one of the headers that you can send down with a response. And there's a bunch of different values that can be in there, no-cache, no-store. There's also the…I want to say it's the max-age, or I think it's Expires. That's a different header. But you can set it to have an expiry that's just already expired. There's also a Pragma, which you can say no-cache. Some of these are standard. Some of these are not standard. Chrome ignores all of them. Chrome’s just like, “Nevermind.” So the idea is that those headers are intended to inform intermediate proxies.

Say you have a caching layer, so you're using Fastly or CloudFront or something like that. When that service fetches the page from your backend, from your actual, say, Rails app, then it will look at that header and say, “Should I hold onto this for a little while or not? Is it public, or is it private? What should I do as an intermediate caching proxy?” Ideally, Chrome would also look at those and say, like…there should be a version of me being able to tell Chrome, “Listen, if someone hits the back button, please go to the server and ask for it.” Like, I'll take the second of latency that that introduces in navigating back because I always want to show them the correct data. Unfortunately, I have not found a way to do that. There's a bunch of things on Stack Overflow and other places of JavaScript solutions where I can listen to the window.popstate event and then force location.reload. But that feels like a pile of hacks that I don't want to get into. It feels like it will be very inconsistent between browsers. So I am still searching for a solution. But I would like to figure something out here.

As a more pointed version of this to try and explain an example where this could happen, imagine that you've got the header of your application, and in it, you have a sign-out button. And so that sign out is going to delete to the session's endpoint. So you're deleting your session. And after that, you get redirected to the login form. If you then hit back, you will be taken back to the browser’s cached version of the previous logged-in state page that you were at. This is probably fine in a lot of cases. If you reload, you can't do any nefarious action at that point because you are logged out. But you are seeing potentially sensitive information. So imagine that you log in in a cafe, you go through Gmail, or whatever, or your bank, then you log out, you walk away. If you leave that page up and someone hits back, they can now see what was on the page. And part of that particular version, I read a bunch of backstory about that on the Inertia repo because someone posted this as an issue against Inertia as a framework. And the Inertia team...and I really love how they handle these sorts of things. So they were very kind, very welcoming to the issue but also said, “Actually, we're doing...like, this isn't us. But let's talk about it,” and gave a ton of detail and went through the HTTP spec. And it's a fantastic issue as a read. It's like a fun bedtime reading sort of thing to learn about how the internet works. But the Inertia crew really, really cares about being spec-compliant and doing the right thing. So, unfortunately, this is outside of their purview as well. But yeah, I don't have a solution, and it makes me sad.

STEPH: I liked that second example that you provided because I feel like I see that one more commonly when I'm on an application, and I don't know why. But I hit back, and then it shows that I'm signed in, and I'm like, that's a lie, I'm not signed in. I also really appreciate how Inertia is responding so kindly and welcoming to folks and then providing such thoughtful responses. That sounds immensely helpful. I don't know, yeah, I am also interested in this. It's something that I haven't worked with in a while, so I don't have any grand ideas at the moment. So I'm also curious if other people have run into this and how they've approached it.

CHRIS: Yeah. If we're being honest, partly I wanted to share this with you, but also I wanted to say this into a microphone, and then hopefully someone out there on the internet knows an answer. I've tried, I think, all of the normal things, all of the different variations of headers. I haven't actually poked at the JavaScript things yet, but that's probably the direction I'm going. But if anyone out there has an idea, I would absolutely love it. I think in my mind, the ideal version of this is if I'm making GET requests and I'm clicking around on a page, it's perfectly fine for Chrome to use its cache version of the previous page because, sure, that's fine. It may actually be stale just based on it's been a few seconds, and something's changed on the server, but I'm willing to accept that. But if I've posted, or patched, or deleted, or done any action that by definition should be changing data on the server, then I would love for a way to invalidate Chrome's back cache, so its version of the pages that it's restoring when I'm hitting back. I'd love that as the heuristic to get to. I don't know if I can get there. My sense says chrome’s like, “No, I want to go fast. That's all I care about.” [chuckles] I’m like, all right. Well, I get that vibe but --

STEPH: Yeah, that’s a nice, succinct way to say that if I've changed data, then I want to invalidate that browser cache, so then we don't show them a fresh page and we actually show them the name that they entered on the form.

CHRIS: As we know, though, cache invalidation is one of the very easy things to do in software development. So I'm sure my naive, quick idea is very easy to implement and will have no edge cases of its own.

STEPH: Well, this will be our parallel Pixar movie. We have one that we highlighted earlier, and this will be the other one, The Cache Buster. I'm not great with titles. [laughs] This will be our other Pixar movie.

CHRIS: Buster the Lonely Cache. Yep.

STEPH: All right. Well, in parallel, we'll work on Buster the Lonely Cache. Is that the name of this?

CHRIS: Yep.

STEPH: Cool. We'll work on that script. And in the meantime, I'll also think about it if I encounter this or come up with some ideas and share them with you. And then also if other people have any ideas, that'd be fantastic.

CHRIS: That would be fantastic.

STEPH: Yes. Please write in to help Buster with the lonely cache, which, wait; I don't get it. Why is it the lonely cache?

CHRIS: Because the cache has been busted and evicted. So he's got no friends. There's nothing...There's no data left. I don't know.

STEPH: [laughs]

CHRIS: I came up with it real quick. I don't stand by it. It's not a great idea, but we'll workshop it. It'll be fine.

STEPH: That's true. Yeah, we'll go through it. I'm asking too many questions for a very quick creative. We're in the creative space, not the critical space. But please write in to help Buster figure out [laughs] the lonely cache or how to bust the cache. Oh, goodness. I'm done with my jokes for today. I'll try to stop.

CHRIS: I believe that's a perfect note. Shall we wrap up?

STEPH: Let's wrap up. Show notes for this episode can be found at bikeshed.fm.

CHRIS: This show is produced and edited by Mandy Moore.

STEPH: If you enjoyed listening, one really easy way to support the show is to leave us a quick rating or a review in iTunes as it helps other people find the show.

CHRIS: If you have any feedback for this or any of our other episodes, you can reach us @_bikeshed on Twitter. And I'm @christoomey.

STEPH: And I’m @SViccari.

CHRIS: Or you can email us at hosts@bikeshed.fm.

STEPH: Thanks so much for listening to The Bike Shed, and we'll see you next week.

All: Bye.

Announcer: This podcast was brought to you by thoughtbot. thoughtbot is your expert design and development partner. Let's make your product and team a success.

Support The Bike Shed

Episode source