Summary
A few weeks of user testing has revealed specific problems with the syntax of Fault. Marianne ponders various approaches to solving those problems and talks to James Houghton about the intersection between programming and system dynamic modeling.
Links
Rosetta Code
JamesPHoughton.com
PySD a cross compiler and simulation engine that translates models created in Vensim to python
Bonus Content
Become a patron to support this project.
Transcript
MB: Oh boy…….
MB: I’m back.
MB: It took me longer than I thought it would to finish user testing the pre-pre-pre alpha version of Fault. Partly that was scheduling. Partly that was wrestling with design issues highlighted by the user testing itself.
MB: But that’s good because that was the whole point.
MB: I conducted two different types of user testing. The first was passive. I logged every time someone used the repl-- just logging the spec before parsing it. About 50% of people just ran an example model without any changes, just to see what the code looked like and what the output was.
MB: 20% then ran the same model again with a minor tweak— usually changes a single value and seeing what the difference was.
MB: Only a few people attempted to build their own models. One person decided to lint an example model, keeping the significant values the same but removing whitespace in certain areas.
MB: This data was useful and will continue to be useful in its own way.
MB: The second type of testing was pairing with people who volunteered and either writing a model or going through the documentation with them and asking them questions about how they expected pieces of code to behave. This was really useful but sometimes frustrating because I often wanted to put different options in front of them and say “well how about this?” but for a lot of the issues they raised I did not actually have a good alternative available.
MB: I got what I needed out of the conversations but I couldn’t help feeling like I could have been better prepared.
MB: The first problem that—truth be told bugged me a little going into the tests and continued to bug me more and more every time it was mentioned is the required parameters of the stock/flow objects.
MB: If you recall: you must define a name and value with a stock and a function with a flow. But in the same key-value block you can also define any normal of custom properties. There are different rules for these special properties with their reserved keywords and the user defined properties. And sometimes special behaviors. Functions can be initialized with a starting value so that you can reference historical outputs of the function— one person used this to write a Fibonacci sequence in Fault, but you shouldn’t do that with a flow’s default function.
MB: A precedent— a special, non-required property of flows— must have a int value. A name property of a stock must return a string somehow.
MB: I told myself going into testing that all languages have special reserved keywords and builtin functions with specific parameters. But the more I worked with people interacting with the language for the first time …. the more it felt clumsy and brittle.
MB: In the case of flows and their default functions this created a weirdly redundant syntax. Okay we want a reserved keyword to represent the flow’s function …. “fn” for function seems logical. But oh…. since functions themselves are declared with the word ‘func’ followed by a set of curly braces… that means…..
User1: When I saw FN and then function, I was like, oh, that's function and that means function.
MB: Yeah.
User1: So I don't know, is this sort of like a key value pair or is this I mean, this is the function and this is the property?
MB: Yeah, and this is this is like touching on like one of the things I've been exploring, kind of like meditating about.
User2: The input flow then is taking as a target the queue and its function again function seem special here. That’s another one of my notes. If flows always have a function like is this a keyword really? Or are there other function names or what's the … thought here?
MB: Yeah…. for sure
MB: I need to figure out how to simplify things. Its true all languages have these little quirks, but typically they develop those quirks overtime as they change direction and attempt to maintain backwards compatibility. It’s seems stupid to start off with so many quirks from the beginning.
MB: Then another problem I picked up while preparing to run test exercise themselves. If you opted to write a model the task I would suggest is modeling a message queue. There are a couple of ways you could approach thinking about a message queue as stocks and flows. You could think about the number of consumers and the number of publishers as stocks. Or you could think about the messages themselves as a stock and the movement of messages from a queue to either sent or a retry queue as a flow.
MB: Except…. how do you change which stock the flow is sending data to? There’s certainly conditional logic in Fault, that part is easy, but the flow has one source and one target stock maximum. With no side effects.
MB: So you either change the flow target stock dynamically or you have to build a way more complicated model that doesn’t really map well to the behavior you’re trying to represent. You end up painting yourself into a corner.
User2: So then producers have value, this is another thing I noted here to talk about… value seems to be special. It's definitely special within the repl, but is it special overall is is my question. It also seems to be special overall because of the way target and source work, because they implicitly change that dot value value.
MB: Yeah…. So this is something I've been going back and forth on rather regularly, basically the idea of can you assign— how do you do assignment as the output of a function? Right. So effectively, the function of a flow assigns values to the value property of the stock it's connected to, but doesn't have to be that way. Right?
MB: And there are various places where, as I'm trying to work through models for my own tests, I think like, oh God, it would be so easier if the flow could send this value over here and this value over here. So I could definitely like with some fairly minimal retooling, develop the syntax and the logic to sort of say, like this value gets sent over here and it's now assigned to this part of the stock and this value gets sent over here and assigned to this part of the stock.
MB: The thing that keeps me from doing that is kind of not wanting to move away from these very tight, pure functions that only really do one thing, or change one thing. Right? So that's the part where I'm like trying to figure out how much utility do I get from giving that level of flexibility where it's like it's not like it's not necessarily value that's changing.
MB: You can have the function of flow, change other things, and therefore build more complex and lighter rate models versus adding a level of like. State change chaos to the overall model, right, because you are— It is a bounded model, but it is still potentially, as the model grows more and more complicated, it gets more and more computationally complex.
MB: So I’m still struggling with that. I think what I will probably end up doing is testing out that kind of delegation and seeing how I like it and then maybe running some more tests with people and seeing what kind of chaos it opens up if we try to build a really big model. But yeah, so noted for that. So that's really why Value is this special thing that doesn't behave quite the way the other properties behave
MB: Then on top of that Fault tries to draw conclusions about what you want it to do with the value produced by the flow that in retrospect might have been confusing.
User2: The source is the queue size that is now that consumers.
Yeah, and that the.
User2: Yeah, and that the….They're going nowhere, so I want to create a new value which goes from zero, this is kind of related.
User2: Yeah, I guess the new value goes from zero, even though it's got no target. Is that the right way to think about it or?
MB: Yeah….
User2: It's always a constant zero?
MB: if there is a source value and not a target value, it will essentially treat the source as if it is the target value. It's just at this point, it's just syntax to make it clear for people's heads rather than like what the code is doing. Right? If there's only one stock it's connected to, it will do the same thing. No matter if it's targeted at the source, it only changes its behavior when there are two stocks on either side. I see.
User2: I see. I didn't didn't realize that because it feels like sources being subtracted from the targets being pulled.
MB: That is, of course, the idea of it. But if you were to connect the flow to a stock as a source, but the output of the function was a greater number than the the source like incremented the source, it would just increment the source. It wouldn't go, hey, wait a minute, this is supposed to subtraction. This is flowing out. I just did not seem worthwhile to do that, picking you in about it. But it did seem worthwhile to sort of help people, people with. So to think about it in terms of this is the source and this is a flow out of the source and to be able to allow them to define it that way if they so wished.
MB: Okay…. what to do about this. Well first, if I allow the function to explicitly assign values to stocks then having special keywords for source target and value is unnecessary. You can declare a flow with stocks assigned any key name you like and change whatever property of that stock you need to. That one change adds a tons of expressiveness and eliminates three out of three out of our five reserved keyword.
MB: The stock name itself is unnecessary because the statement where you declare it already gives it an identifier. 4 down….
MB: That leaves the function-function problem. It took me a loooooooooooooooooong time to figure this one out. Sure I could make the reserved word something like “rate” and eliminate the ugliness of the repetition, but that isn’t really satisfying. It’s still a special property with slightly different behavior than identical looking properties elsewhere.
MB: I spent a lot of time pondering different ways of arranging blocks of code to separate out this one special thing in such a way that its different behavior could be anticipated. I spent a lot of time on Rosetta Code looking at different syntax from different languages hoping for inspiration.
MB: (If you’re unfamiliar with Rosetta Code it’s a great site displays versions of the same simple programs in different languages side by side. An amazing resource if you want to think deeply about design patterns in programming language grammars.)
MB: Then suddenly, while doing something else the answer hit me. And it was one of those situaitons where once I thought of it I was shocked it took me so long to think of it.
MB: The reason why the function-function block seemed necessary is because Fault had no syntax for call expressions. When the model runs it needs to know what functions it’s executing. I was using a special keyword to determine that, and then adding additional special optional keywords like delay and precedent to configure what the model does as it runs through a single step. All that logic is abstracted away from the user and the special keywords expose those configuration options.
MB: …..Why on earth was I doing that? These are programmers right? Why don’t I just let them program what the model does in a single step?
MB: …………Yeah. It’s obvious right?
MB: So in addition to changing flows to allow them to assign values to any number of different stocks. I added something I’m calling a run block.
MB: At the end of every model is an optional configuration statement that reads for x run all; I say optional because if you remove that line or never add it in the first place it executes all your flows in a five step model by default. But if you change the statement you can make Fault run part of the model or run as many steps as you like.
MB: Now instead of run all, the proper syntax will be run curly braces followed by a definition of what is happening in each step. Now if you want to write a delay you just write a delay as a conditional in that block. If you want to change the order flow execute, you change lines of code in that block. And you can call the function in the flow object whatever you like because you call the function from that run block.
MB: Of course the trade off is that you can produce something in Fault that is not a true stock/flow model if you want… but maybe that’s not as damning I would have assumed.
MB: In my user tests I spent a lot of time talking to programmers of various skill levels, but there was one other group I wanted to chat with….. people who write system dynamic models.
MB: A few weeks ago I sat down with James How-tan a researcher at the University of Pennsylvanian who teaches college students how to build models and also has created a suite of tools to run those models in python.
MB: So I always thought of ASD is like being almost niche to the stock flow abstractions. Yeah, but it sounds like for you it is broadly modeling these kind of social dynamics as a whole.
JH: Yeah. You know, you have human actors with behavioral realistic decision roles that are interacting to create some some phenomena that doesn't exist in the individuals themselves.
MB: So that being said, how do you pick, you know, your model constructs like how do you decide this fits best with an agent base, Stock/Flow based, This fits best with some other construct.
JH: Yeah, there's two sort of tools that you can use for thinking about that.
JH: One is the level of aggregation of the model or its scope, like what do I want to stick in the model and what are the boundaries and the other. So that's sort of like capping a top line on on the scope of the model. And the other thing you want to think about is the granularity of the that you want to represent. And so in this case, I needed to get the granularity all the way down to the level of not just the individuals, but the individual beliefs within the individuals. And so for me, that made it a an agent based model. But I have a very good friend who is modeling the current epidemic and his models are compartmental because the questions that he's asking are more at the top level about the society as a whole. And so in that case, you can create these aggregations, the stocks of individuals and represent these groups of people as a single number.
MB: This is so fun because this is weirdly intersecting with something else I'm doing in my work environment around programmers intent, because I think the the core issue with sophistication on the technical side is that you are never going to write a specification that is 100 percent accurate to every every conceivable factor in a system. Right. So at some point, you have to make a decision of the level of detail. Yeah, but that always drives computer people crazy because they want absolute truth. And they like this idea that I can write the spec and I can prove that it's correct and that's objective, not subjective. And so there's this question of like, how do you reliably set those boundaries of like what level of detail you need? So connecting it to. Well, what research questions are you asking them to model?
JH: Yeah.
MB: It makes a lot of sense. But it's also not an answer that my community, people like because they want to share models and reuse them and build on top of them. And it's like your research questions are not in line with somebody else's research questions. You can't reuse their work, right?
JH: Yeah, well, I think the most important thing in modeling and probably in academia as a whole is to really know what question you're asking.
MB: Yeah.
JH: And when I read an academic paper that doesn't really hang together and I can't understand what it's saying often the reason is that the question isn't clearly articulated and possibly isn't clearly understood by the authors. And the same is true when you look at a model and if it has a whole bunch of pieces and you're not really sure why they're there, maybe the reason is that it was unclear why this model is being built.
MB: Yeah.
JH: And so the best models are the ones where you can really clearly articulate exactly what the research question is.
MB: Sometimes I think there’s a certain magic to having specific conversations at specific times. I don’t always work on this project as diligently as I could (another secret reason why it took me a bit longer to finish user testing than I thought it would) I feel like I have to sit with things sometimes … waiting … not exactly for inspiration but for …. something.
MB: As it happened I procrastinated on reaching out to James for oh … about a month. I don’t know why, I just kept putting it off. And as luck would have it by the time I did reach out I was in the middle of exploring approaches to capturing programmer’s intent for a completely different reason. Right now at work I’m working with a team to build new technology, but of course I’m coming from a career of slaying dragons of tech debt on existing systems. Naturally I can’t build a new system without thinking about how functionality will drift over time, how documentation will fall out of date, how more and more lines of code will make it harder and harder to onboard new engineers. There’s some interesting research work around formalizing programmer’s intent into the code and some theoretical work around compilers that can identify when a function violates the programmer’s stated intent.
MB: Had I not waited, these comments in this conversation probably would not have stood out to me the way they did, and that’s a stroke of luck because I think they reflect something critical about Fault’s design that I hadn’t put much thought into: a model’s scope and assumptions are defined by it’s research question. Ergo the thing that a spec has to communicate most clearly is not how the thigh bone connects to the leg bone but what the intention of the model is.
MB: Assertion have been a part of Fault’s grammar from the beginning, but I was mainly thinking about them in terms of simple boolean statements like x > 5. Yet just as it is my habit to read the unit tests first when trying to figure out a new code base, these assertions will broadcast information about the programmer’s intent. It’s going to be important to invest some time and energy into figuring out how to emphasize their significance.
MB: Something that came up in our chat that genuinely surprised me is how from James perspective the characteristics of visual programming compiling to code wasn’t just a crutch for silly non-technical people. His tools don’t convert to python to remove the proprietary software used to build the models. He wants to keep that drag and drop stock and flow diagramming stage.
MB: This caught me off guard because I find being given a visual abstraction and maps to autogenerated code frustrating. I’d much rather open up the hood and write the code directly.
JH: The thing is, the stock flu models, the diagrams themselves are actually… think about it like a technology in a way, in that this is a tool that we can use to think about the structure of systems. So everything that's represented in that stock flow diagram is just something you could represent with math if you wanted to. But by structuring it with the stock and flow diagrams in the feedback loops, you see the structure of the system in ways that you don't see when you just have lines of code on the screen. So mathematically they're equivalent. But in terms of your understanding of the system, you do a lot better when you have the visuals.
MB: Is the value of these tools actually the visual? I'm sort of approaching it probably too much as a programmer and seeing the value of the tools being the interface of code, basically giving you shortcuts, right? Where it's like, OK, this understands what a stock is, understands what the flow is and like makes it easier. You have less code to write because there are some built in like syntactic sugar that makes it easier to get into that place and then when I actually look at the code. I'm like, no, this is just… like you could do this in any (programming) language.
JH: Absolutely.
MB: Is it really like the visualization and like maybe some graphing is the why these tools are popular?
JH: It is, exactly.
JH: And and and these diagrams are a tool for thinking about the structures of complex systems. Once you have them built, you can translate them to any language you want and the math will work just the same.
MB: The programmer in me stubbornly insists that I can probably understand a system better from looking at code…. and yet … reading code is so much harder than writing it. Perhaps visualizations do make it easier to communicate general information about a system.
JH: The the writing down of the equations is the easy part. Honestly, the the understanding, the structure of the system well enough that you can write down equations that accurately represent it is really difficult.
MB: Yeah.
JH: And what's dangerous is that it doesn't look difficult.
MB: Yeah!
JH: When it's done well, it becomes really clean and and and easy to understand. And that's when you know that a real master has been doing their craft, there.
MB: So okay…. I have a few more design questions that I need to develop answers to before this journey is over.
- I need to figure out how best to present assertions so that they are treated both as tests but also as a documentation of the programmer’s intent.
- I need to figure out how to incorporate visualizations into Fault’s output. The prototype would generate a dotviz flow chart before running your model and I found that to be really helpful. I still don’t like programming via visual methods, but when I’m building Alloy models the ability to check what Alloy thinks my system looks like is often the best debugging tool.
- In general I need to think about how a person interacts with and understands a spec someone else has written and how the code arrangement makes that easier too harder.
Then there was another key challenge that this conversation highlighted:
MB: What are the common mistakes people make when they're building models?
JH: There's a lot of things.
JH: I’d say the biggest ones are not really being specific about what they mean by their their stocks and flows, so that that's in some ways it's a conceptual sloppiness where you haven't gotten all the way down to what is it you really, really, really mean here.
MB: Yeah.
JH: And then there are just some mathematical mistakes that are easy to do. Or, you know, an obvious one is that if you have a stock with an outflow, and the stock is something that, like water in a bathtub, can't really go negative. You need to go to check that it hasn't gone negative. And so what you need is a feedback between the level of the stock and the outflow. And so we'll call it a first order control on the outflow so that you you don't go negative when you shouldn't. And if you forget that, it'll work most of the time until you get to an edge case where your stock has gone negative and you didn't notice. And then all of a sudden you get really, really strange behavior.
MB: So, like, I almost like instinctually want to think about this as a type system issue. Right?
MB: Like, if you have a particular stock that cannot go negative, that's something that can be reasoned about through type systems. You could sort of say this is a this is this is a characteristic of this type that it never is negative and then possibly reason about it on the compiler or the runtime with that thinking in mind. Like, this is the problem with mixing multiple interests, they collide in strange ways sometimes.
JH: This is the power of mixing multiple interests!
MB: laughs….. perhaps, we'll see.
Latest comments (0)