DEV Community: Thomas Jebson

Pycraft progress update!

Thomas Jebson — Tue, 23 May 2023 18:56:29 +0000

Hello there everyone! We have had a very busy week of progress on Pycraft and have a lot to share, including some interesting changes to the rendering pipeline for Pycraft's 2D engine and some detailed statistics on the ongoing optimisations of the settings menu!

But before we get started with that, let's talk about rendering and introduce a few new key terms that we will mention in this article and in future discussions. In Pycraft we have a game loop for every individual menu for Pycraft, whether that's the theme selection menu, the credits menu or even the 3D game engine. In every game loop, we have a rendering pipeline, this controls when, what and how we render content to the display. Before Pycraft v9.5.0dev8 every menu in Pycraft used a synchronous rendering technique. In this approach, the rendering pipeline gets used every time the game loop gets run.

This is very helpful for situations where all the content on-screen is continually updating, like for example the 3D game engine, and this is where this discussion splits to focus on the 2D engine. The 3D engine cannot take advantage of asynchronous rendering. In this approach, the rendering pipeline only gets called occasionally in the game loop, either when an event triggers the graphics to update, or after a certain number of frames elapse. This essentially means that if the content onscreen isn't updated then the rendering pipeline won't get called.

However there is more to a game loop than just rendering, and we can therefore split the game loop into three main components:

The first component is comparatively small in length and contains the scheduling and logic required to control the rest of the game loop, like for example controlling when to update the graphics and compute stages.

The second component is the compute component. This contains all the logic for widget interactions, events much more.

The third and final component is the rendering component. This contains all the computing and controls for rendering content to the screen (by adding the compute for the render stage here as well, we don't inefficiently compute for something that won't get rendered, like a graph).

The concept of asynchronous rendering is very new to Pycraft's 2D engine, and will not be rolled out to Pycraft's 3D engine due to it not being effective. For those menus that can take advantage of asynchronous rendering though, there is currently no inherent support built into Pycraft. Typically, when adding new features to Pycraft we try and gradually add them to existing mechanics over a series of updates and then link them together when we are ready to deploy them, however in the case of asynchronous rendering we are starting almost completely from scratch.

Fortunately, it's a relatively simple technique to roll out, instead of calling the compute and render components for every game loop, you call the render component only where needed. In Pycraft we are currently also requiring scheduled frames, so after a certain number of frames the graphics will be updated. This is because we have not yet finished prototyping an events-only system, and this will also be much more complex to benchmark and test.

A quick fun fact for you, asynchronous rendering, if not referred to by that term, is very commonly used in other applications, so much so we have been using it in Pycraft's Installer and that significantly improves the efficiency of that engine.

Now, there is plenty more to share on this topic, but we are still working on introducing this feature to Pycraft. We have finished work on the settings menu, and have also done a series of optimisations to improve the performance of this menu, as it was very inefficient.

In the graph below, the blue line represents the baseline performance of Pycraft's settings menu, taken before we started any changes and optimisations to this menu. The orange line shows the performance of the settings menu after we have spent some time optimising it, however, it is clear there is more work to do. All other lines that drop to this orange line are render frames. This means that the yellow and grey lines show asynchronous rendering, where for every 4 runs of the game loop, one will be a graphics as well as a compute frame, instead of just a compute frame. The example below ignores forced frames due to events!

Pycraft progress update!

Thomas Jebson — Tue, 23 May 2023 18:51:59 +0000

Hello there everyone, it's time for our weekly progress update, and over the last week, we have made the finishing touches to numerous features that have been adjusted and improved in Pycraft, and have made major changes to the widgets system for Pycraft!

Let's start with some of the features that have been changed and improved over the last few weeks, and the one that immediately comes to mind is the new performance monitoring window for Pycraft. The one we have replaced it with takes advantage of the new metrics system (more on that in a bit) and is considerably shorter and more efficient, taking up just 53 lines compared to the 125 lines it took before. And that is a good way of looking at the progress we have been working on and aiming to achieve as we work on Pycraft currently; doing more in fewer lines.

Now, let's discuss the new metrics system for Pycraft. Whilst this is different to the performance monitoring/graphs, they were added in the same update, and for a significant period - close to 5 years - their only purpose was to collect information to display in the graphs. Whilst the way we do this has remained the same in that time, where we do that has changed significantly. In addition to this, the desire to collect information that is relevant to the performance impact of Pycraft on your system instead of the performance impact on your system from Pycraft.

This means we have now made a new utility module that handles all of the metrics collection. We have also removed the relationship between the collection metrics and the performance graph, so instead of only collecting metrics on demand, it runs passively in the background. This is in preparation for future scenarios where we add performance logging or better error reports but that's very much something we are looking into in future.

Note though that the metrics system is disabled automatically in-game if Pycraft determines it to be causing poor in-game performance. Also, this is something we are very strong on, any performance metrics are only ever stored in memory on your system, and are never stored locally or on a network location. Additionally, the ability to disable performance monitoring is available in the settings menu under the "developer options" tab.

Now, let's discuss widgets in Pycraft. Over the last few weeks, we have been going through every widget in Pycraft and have been converting them to an object-oriented programming paradigm, instead of a procedural programming paradigm. There are two main reasons for this. Firstly by doing this, we can make the widgets much easier to deploy outside of the settings menu, which was where they were first introduced. Secondly, this allows us to adapt the new widgets in such a way that we can optimise their rendering and computing functions.

The reason behind decoupling the rendering and computing of the widgets for Pycraft is that we are working on a new - better optimised - 2D rendering pipeline that should significantly improve the performance of Pycraft. This will work by using two dedicated clocks, one that runs at the specified frame rate, and the other that runs at intervals (this will be adjustable in settings under "video"). So this means if I set my game to run at 60 Hz (or 60 frames per second) the computer will run to target that frame rate. if I then set an interval of 0, then for every compute cycle the graphics will update. This is going to be identical in performance to what we are currently doing in Pycraft. However, if I set the interval to 4, then for every 4 compute cycles, the graphics will update, leading to a compute frame rate of 60 Hz, and a graphics frame rate of 15 Hz.

Naturally, it will be very noticeable if every frame's graphics were only updated for example 15 times per second, therefore the game will automatically "force" a frame at most with a delay of 1/(the current frame rate of the compute stage).

I hope you have enjoyed this weekly progress summary! Unfortunately I cannot seem to upload images to Dev currently: "Unknown end of json input" so apologies about that, you can find the image that was supposed to be here on Hashnode here: https://hashnode.com/edit/clhi4pvga000109jn90457pyd or on Instagram here: https://www.instagram.com/p/CsLeianICCK/?utm_source=ig_web_copy_link&igshid=MzRlODBiNWFlZA==

Pycraft progress update!

Thomas Jebson — Thu, 20 Apr 2023 18:08:34 +0000

Hello there everyone, it's time for our weekly progress summary for Pycraft, and I think that we have just had one of the most productive weeks in the entire 5 years or so we have been working on Pycraft! Needless to say, there is plenty to talk about today - starting with the announcement that Pycraft's installer is now "finished"!

Although unfortunately by "finished" I mean that every individual component of the installer that we have been working on over the last few weeks is now finished, there are a few final 'housekeeping' jobs to do before we can announce that the installer is finished.

But this time last week we were not able to discuss the installer even as being in this "finished" state, so what have we been working on over the last 7 days?

Well, there are 3 main areas to Pycraft's installer, there is the installer, which we had already finished working on last week, there is the uninstaller and the updater, and it is those last two components that we have been working on and have finished this week.

Let's start with the uninstaller. Of any area in Pycraft's installer, the install component was probably the worst written, and the uninstaller was the least optimised, close to 40% of the uninstaller was repeated code, so naturally the first task of this week was to fix this, as we quickly realised we didn't need 3 different methods of removing the same file, so decided that the "action stage" where we would go through and uninstall Pycraft should be completely re-written.

Completely re-writing an entire section like this was something we had hoped to avoid, due to how much additional time that would take, however it wasn't something we would avoid doing if necessary, and in the uninstaller, we decided to go ahead with it as we are rewriting a whole section, but the rewritten section is comfortably 75% smaller than the original. By doing this we were able to optimise the file removal process as much as possible, and significantly reduce the complexity of Pycraft's uninstaller.

From there, now that the uninstaller was a lot more concise, we were able to conclude that the rest of the uninstaller was functionally fine, so decided to tidy up the whole program and move on to the update component, well ahead of schedule.

Right, let's talk about the update component of Pycraft's installer - by first talking about the installer and uninstaller. In Pycraft's installer, you can consider the installer and uninstaller to be fundamental components, and the updater as a composite combination of the two fundamental components of Pycraft's installer. Additionally, a lot of the components of the updater had been migrated into the main installer, for example before you could click on "update" and check for updates and only then if there were any available would you be able to continue, now only if there are any updates available will you be able to access the update program.

What that means is that the updater has been significantly simplified and as such only took around 2 days to work on and finish. Additionally, whilst the actual code itself is simpler, so is the user experience, as we have significantly improved the UI and taken on user feedback to improve its overall user-friendliness.

Now, with all the major components of Pycraft finished (and the repair component coming in a not-too-far-away update) this means that we are nearly finished with Pycraft's installer - with only a bit of code cleaning left to do. Fortunately, this is not the end of work on Pycraft v9.5.0dev8 as we still have numerous improvements to make in Pycraft itself, like for example improving controller compatibility.

We hope you enjoyed this week's progress update!

Pycraft Progress Report!

Thomas Jebson — Tue, 11 Apr 2023 14:27:59 +0000

Hello there everyone, It's time for our weekly Pycraft progress report! If you were wondering why there isn't one for the previous week, that is because we were on holiday, and we were planning to combine this week's progress update into next week, however in the last 4 days we have made a lot of progress, and we are super excited to share that today!

Over the last few days then we have been focusing on the installer for Pycraft, and more specifically on its install component, but before we focus on that, let's talk about the history behind the installer and what we have done with the installer in general.

Since we started working on the installer, there have been 3 major versions, and we have only recently added this into the installer, for reasons we will get onto later. We have listed the major versions below, and what differentiates them, but it's important to note that minor, patch and developer versions of that major install version may change its feature set.

Firstly we have installer v1.x - This installer was never made public and was more of a proof of concept that we could have an installer for Pycraft. This installer worked for approximately 1 year and approximately 4 major updates to Pycraft.
Then after a gap of 3 years, we have installer v2.x - This was our first installer to be made public, it was written to only support Windows, as that was the only operating system we actively tested Pycraft on at the time. This installer 'worked' for approximately 2 years, and 9 major releases of Pycraft - because of this, if you have ever used the installer for Pycraft before today, chances are it was this version.
Then now we have installer v3.x - This is not as different in terms of code as the jump from v1.x and v2.x, however, this is the first installer to use GitHub as the main download site for Pycraft instead of PyPi (although PyPi is still used for Pycraft's dependencies). This installer has been reworked (but not rewritten) to include at fundamental level support for a range of operating systems but has official and hard-coded support for Windows and Linux users. This has worked for approximately 0.003 years and 0 major releases of Pycraft - it's currently been working for one day!

Now that we have discussed the different revisions of the installer, and in general, what sets them apart, let's talk about the changes we have made over the last few days to Pycraft's installer (v3.x).

Initially, we started by updating the fundamental structure of the installer to match the structure we have for Pycraft, making switching between the development of either system much easier. The biggest of the structural changes involved adding a registry to Pycraft's Installer - however, we later experimented with the idea of having one central registry and then dedicated registries for each major program that makes up the installer, where we felt it was necessary. This wasn't something we had experimented with before, and whilst we have no plans to roll out this structural change globally, it does open up some interesting possibilities in the future.

Following that we did a fair amount of cleaning up of both the code and the UI design. This improved the installer's use of PEP-8 and also involved the removal of a large number of global variables, which is an underlying job we have throughout Pycraft and its companion programs, and we are now at a point where the majority of the global variables we have are for communicating with code running in parallel and are not easily removed.

From this point onwards through this blog, all of the changes we make are to the main body of the installer, and the install section and its components.

The final thing we did in the cleaning-up process was to remove the strange off-white background to some widgets, something we had previously thought not possible. This was much harder than we were expecting, but makes the entire installer look a lot less haphazard and rushed.

Finally, this leaves us with two final tasks for the install section of the installer before we can move on to the other components - the uninstaller and the update processes:

We needed to update the process of actually installing Pycraft so that it uses GitHub releases, and can navigate the new directory structure we have for Pycraft, which we don't plan on changing again. The process of installing Pycraft's dependencies relies on us being able to navigate Pycraft's directory structure to find the 'requirements.txt' file and then use PyPi to install the additional packages as required from there.
Then we needed to improve Linux functionality - we had already improved general operating system compatibility through the use of Pathlib for file paths - So now we needed to add functions to add desktop and start menu shortcuts when using a Linux-based operating system. Unfortunately this isn't something we can do on a general level like we could use file paths, and as such only supported operating systems (Windows and Linux) can have desktop and start menu shortcuts made.

Now, with all those changes made, we can share the progress we have made, there are a few small things we have to change still, but the installer is functionally complete and working!

Pycraft progress update!

Thomas Jebson — Wed, 29 Mar 2023 17:15:56 +0000

Hello there everyone! Its time for our weekly progress summary and this week was the first big test of the major changes we made to the way we work on updates, so in this article, we will be reflecting on how that transition went, and what we have been doing in this week of development!

For those unfamiliar, a week ago we made a major change to the way we worked on updates for Pycraft by sharing our progress to a dev branch on GitHub instead of only sharing code upon release. By doing this we can get much faster turnarounds on identified bugs and makes collaboration with other developers much easier. We have also made the entire GitHub repository more accessible by following a more commonly used structure and simplifying the repository considerably.

Now having recapped the work we did last week, this week we put that to the test and after a few days spent working out how to optimise our workflow to make the most of the adjustments, and to our surprise, Pycraft ran pretty much without fault through this transition, needing very little work to get to run, which is good news as it shows the added flexibility we introduced through the development on Pycraft v9.5.0 is working.

So all this meant we could get started on making improvements to Pycraft v9.5.0dev8 very quickly, so despite having allocated this week to get everything working, we were able to start making improvements early, and have already cleaned up many different areas of Pycraft, fixing 2 major bugs and making numerous behind the scenes improvements, like this change to the splash screen that we have also added, which we think makes Pycraft's startup look much cleaner:

Now, the main focus of Pycraft v9.5.0dev8 is to get the installer working and up to date, and so I think it would be wrong in talking about all the progress we have made without talking about the installer, which we have spent a large proportion of our week working on. Currently, we are updating the structure and design of the installer, and then from there, we will expand to support more features, like for example other operating systems. Expect the installer to become the focus of our progress over the next few weeks.

One final thing we would like to announce is that we are away next week, and subsequently, there will be no progress update next week - as we won't know what progress has been made - and we are currently unsure if the week after that we will have enough work done to discuss, so we might combine that into another two-week summary. Apologies for that!

Now, as we have been working on the installer this week, we are proud to share our progress here visually:

Pycraft progress update!

Thomas Jebson — Thu, 23 Mar 2023 20:19:16 +0000

Hello there everyone, its been a bit of a busy two weeks since we last spoke, and whilst there is plenty now to talk about, we didn't feel that in the first week, we had much to talk about, so, in addition to the focus on getting Pycraft v9.5.7 (or Pycraft v9.5.0dev7) released on GitHub, we decided to combine the two.

So, to start: within the first week, we were predominantly finishing up with work on Pycraft, as we were nearly done changing the way data gets transferred through Pycraft, and making some pretty significant under-the-hood changes to Pycraft. This change was designed to improve readability, expandability and flexibility within Pycraft's source code, which we have already talked about a fair bit.

During this first week, we also made a series of plans on what we wanted to do, not on the source code for Pycraft as such, but around Pycraft to make things much simpler and hopefully more accessible. Therefore before we continue into the rest of this progress update, we must make these two big announcements.

We are changing the way versions work in Pycraft, non-developer releases are backwards compatible and unaffected by this change. For developer versions of Pycraft, the developer code we would assign each release of Pycraft has moved to allow for the re-introduction of patches as a release option. Developer updates now include the phrase 'dev' after the main version code and a number to represent the iteration in the cycle towards the release of the next version of Pycraft. The first and last digits of the main version code are cumulative, and as such, they should never decrease as we release future versions of Pycraft. All this means that Pycraft v9.5.0 remains the same before and after the change, and developer versions like Pycraft v9.5.7 now become Pycraft v9.5.0dev7. This change has already taken effect at the time of writing on GitHub and we will use the two version codes when referencing new versions of Pycraft for this article only.

In addition to all this, the structure of the GitHub repository has changed. Originally we had branches for every currently in-development version of Pycraft, and for the most recent stable release of Pycraft. This has changed slightly now, the most recent stable release of Pycraft is now known as the 'master' branch, and the most recent developer release is known as 'main', in order of decreasing priority. From there the 'dev' branch is where the currently in-development version of Pycraft lives. When the code in the 'dev' branch gets completed, it gets its own (pre) release on GitHub, and replaces the code in 'main'. The code is now what changes in the repository, rather than the branches. Changes to the documentation and to adding content around Pycraft and the installer are affected by this change.

These two major changes around Pycraft are the main reason we have decided to release Pycraft v9.5.0dev7, as by splitting what was already going to be a pretty large developer update into two do make these changes benefits us and you, as it allows us to start development with Pycraft v9.5.0dev8 (Pycraft v9.5.8) with the changes applied, and benefits you as this is a much more standard and clearer way of development.

As we progressed into week two, which is ironically when those changes we just discussed take effect, our focus moved away from making progress on Pycraft, and more towards getting Pycraft ready for release. This was a very abrupt change of plan, and didn't allow us a lot of time to make the changes we wanted to, which makes us say that Pycraft v9.5.0dev7 works, and runs at a low level, but lots of the features we added more recently and changes we are currently working on have yet to be re-added. This includes things like the new built in pop-up mechanism, the new saves mechanic (which we plan on revising the backend for) and in general things like fullscreening the window or controller support have yet to be added.

All this makes us say that Pycraft v9.5.0dev7 is very much a 'two-parter' as the code is in an operational but not nessasarily finished state as yet.

We hope you enjoyed this two week progress update, we plan to get back to our normal schedule again for next time, and have also got plenty to share from the operational version of Pycraft, although at this time we cant upload them here, feel free to check them out on Hashnode, Instagram or Twitter at PycraftDev!

Pycraft progress update!

Thomas Jebson — Wed, 08 Mar 2023 20:13:28 +0000

Hello there everyone! It's been another week, and what a successful one it has been! This is for the week commencing the 27th of February, and this week we began to shift focus from the 2D game engine to the 3D engine.

For that unfamiliar, Pycraft is comprised of two fundamental graphics engines, each one is built around the same general structure for Pycraft, but expands upon that in different ways. For instance, the 2D engine is designed to be user-friendly, look good and be adaptable to change, without perhaps being the most efficient in resources. In the 3D game engine, efficiency counts as we add another dimension of complexity by bringing hardware-accelerated rendering to Pycraft. At the time of writing, there is one menu that takes advantage of the 3D engine and 9 menus that take advantage of the 2D engine, and this isn't going to change much, with at present, the final menu composition for Pycraft being 3 that are using the 3D engine, and 8 that use the 2D engine.

Whilst built off the same architecture, the different graphics engines follow very different structures for rendering and handling events, with the 3D engine being much more complex. This knowledge, therefore, means we can split the development on improving the way data is shared through Pycraft into 3 fundamental categories, the start-up, 2D engine and 3D engine. The installer adds the 4th category to complete the list. The first two categories we have managed to complete, and we are now moving into the 3rd category. This was inherently going to lead to a few days of reduced progress as we adapt our workflow to be more optimised for the change, and also to refamiliarise ourselves with the code in question. Even before we started working on this stage of development, we were already however a good way through the changes we had yet to make as we had gone through the whole of Pycraft and removed at the start all of the interactions we had planned to replace.

Unfortunately, despite this head-start, we are still at a disadvantage as, if I draw your mind back to earlier, we currently only have 1 3D engine-based menu, where we have 9 2D menus, meaning that essentially we were able to go through the 2D engine much faster, as the same interactions are often repeated (but to a minimum where possible) for each menu. Additionally, the 3D engine may only have one menu, but that menu is the longest file in Pycraft, and is considerably more complex. It also shares two additional menus with the 2D engine.

Sharing data and transitioning between the 2D and 3D engines has always been an inherent problem for us, and is one we have gotten much better at addressing recently, however one way we are planning to improve this further is to bring the two attached 2D engines to the main 3D engine into the 3D engine. Currently, this means that the installer and map, the two 2D menus, linked to Pycraft's 3D engine, will eventually actually become a part of the 3D engine, which is why they haven't been updated much, as they are going to be needing a complete rewrite for that to be effective.

This means we have got lots of work to do, and plenty of which by the end of the week we have now done, which is excellent news, as we managed to get the loading screen, map and inventory sections each operational after only a day of work, and we are now getting close to finishing the full loading of the 3D engine. This is huge news as the 3D engine, by being the most complex menu, is where the capabilities of the latest structural changes will be most challenged and pushed to its limits, and currently, we have seen the 3D engine adapting well to the slight structural change, which is excellent news, and even if we don't expect the changes we are making now to be noticeable in any way to the end user, it has revolutionised the transfer of data throughout Pycraft, not just the 3D game engine, and that will allow future expansion to be much easier!

We hope that you have enjoyed this article, and we are planning to update this within 24 hours with some images to share for our progress this week, although sadly don't have time to do that today! Thanks for reading.

Pycraft progress update!

Thomas Jebson — Tue, 28 Feb 2023 16:13:22 +0000

It's time for our weekly progress summary for the week beginning the 20th of February 2023 - and what a week it has been! This week we have continued with the ongoing work on Pycraft v9.5.7, with a particular focus on improving the way data is transferred through different areas of Pycraft's source code.

So, last week was extremely successful and we managed to finish phase one of this process of changing the way data is transferred through Pycraft, which consists of removing all the parameters that made use of 'self'. 'Self' was used as a reference to an instance of Pycraft's registry that we would use as a central place to store variables often used by Pycraft. Initially, just this reference was passed through every individual section of source code, which made the transfer and modification of the registry easy, but at the cost of reliability. Then, in Pycraft v9.5.5 we made many major changes to Pycraft, one of which was only allowing 'self' to be used by main menus, with utility programs taking only the information that they needed. This worked very well and significantly improved the readability of Pycraft. However, this came at the cost of Pycraft becoming much longer - reducing the advantage gained by the changes we made.

Now, we are working through Pycraft's source code and undoing these changes, but don't worry, this doesn't mean we wasted our time in Pycraft v9.5.5, many of those changes we made then are still used today. Our solution is for each utility program to have its own instance of Pycraft's registry, but make that registry shared. This way we can transfer all the variables used in the registry behind the scenes, vastly simplifying Pycraft, shortening it and improving reliability and editability.

Since last week, which we can consider almost as the demolition phase, we began phase two of the process of changing the way data is transferred through Pycraft, which we can consider as the construction phase. In this process, we begin to actually start adding new code to allow different sections of Pycraft to take advantage of this new principle. We had feared that this would be a very slow process, however, the utility programs have significantly increased the speed of this, as it allows a single change to make multiple other areas of Pycraft work.

This meant in almost no time at all, we had Pycraft at a state where we would run it, and get to the home screen. This is where our real genius comes in though. Every single 2D menu, including the map, inventory and load screen for Pycraft follows what is essentially the same design, and so by working on the home screen and getting that operational, already makes a significant chunk of the other GUIs work, and as such, our progress was accelerated and we have been smashing through our plan again, and now the achievements, character customiser, credits, home screen, settings and startup menus work.

Now, this progress is in and of itself amazing, and we can now start to see visually the progress we are making - which we expect to be fast as we are not changing the source code itself, just the way different sections interact!

This now means we are about ready to start work on other areas of Pycraft, and having now finished the 2D engine, we can move on to the 3D engine, including the main game engine, the load screen, the inventory and map sections too (although they are technically 2D, they may not end up remaining as such). And we have more good news, the first stage - the destruction phase - we rolled out to the whole of Pycraft, meaning we are now comfortably well over 50% of the way through the process of changing the way data is transferred in Pycraft and the installer...

...The installer? We haven't talked about that much yet, but don't worry, we have plans to do the same thing we are doing to Pycraft now, to the installer, although we haven't started that yet, but will be doing that as well in Pycraft v9.5.7, but will also incorporate the changes we had originally planned into Pycraft v9.5.7 too. The installer is also based on an older architecture than Pycraft is currently, however by the end, we plan for them to both have the same architecture where possible, with a centralised registry that each file of the source code can instantiate!

Now, we have lots to show for our work this week, so it has been difficult choosing the best ones, but here we go - as always we hope you enjoy!

Pycraft progress update!

Thomas Jebson — Tue, 21 Feb 2023 18:56:47 +0000

Hello there everyone, its time for our weekly programming update for Pycraft, and this week has been a very interesting week as we managed to condense 3 weeks of work into a single week and have significantly pulled forward our unreleased planned completion date for this slight side project to the development of Pycraft v9.5.7!

For those unfamiliar, last week we started, what we are calling a 'side project' to the planned development of Pycraft v9.5.7, meaning that we have diverted slightly from what we were originally planning to do in Pycraft v9.5.7 - which is currently meant to be working on improving the installer - to roll out some changes we had been planning for a while, but hadn't decided yet when to implement them. This all came in light of a discovery we made recently in Pycraft that allows us to significantly improve the way data is shared through Pycraft. We talked more about this last week, so for more information, we would recommend checking that out as we will be extending the work we started in that progress summary for a few weeks still.

However, those few weeks we still plan to be working on this are now much shorter than we were expecting due to the progress we made this week. Towards the start of the week we were continuing with the work on converting the main menu programs for Pycraft so that they followed the new structure, and now having done that we were able to move on to the utility programs for Pycraft. For those unfamiliar, the main menu programs are almost exclusively focused on the front-end design and mechanics of Pycraft, so the things you see and interact with onscreen, and the utility programs are what add functionality to those onscreen elements and are the driving force behind Pycraft.

Due to the nature of these utility programs, which we added in the 2nd large-scale restructuring of Pycraft, they are called often repeatedly by different areas of Pycraft, sometimes up to hundreds of times. This is excellent as it shows they are doing what we had planned, they shorten the length of Pycraft by taking sections of larger programs and putting them into their program so that they can be called repeatedly. Whilst this was added years ago now, we are still familiarising ourselves with the advantages this gives us, and in Pycraft v9.5.5 we made many major changes to Pycraft's overall structure, and changing the way utility programs interact was one of them.

In Pycraft v9.5.5 we split the parameters taken by the utility programs out from 'self' which was the variable containing an instance of the registry that was needed by every section of code in Pycraft. This significantly improves readability, at the cost however of line length. But the utility programs are also where we will be able to make large-scale changes very quickly to Pycraft, something that we proved this week.

Because the utility programs for Pycraft are called so regularly, changes to one utility program can affect multiple times it is referenced. Last week and at the beginning of this week, we were changing the way that the main body of Pycraft called the utility programs, and now we are changing the utility programs to correspond to this. Pycraft's source code will not run unless we make these changes.

We have been very focused on making these changes over the past few days, and this has led to some significant results, as over a few days we went from Pycraft barely booting to getting as far as the startup animation for Pycraft, an immense achievement given that so many of the utility programs needed to get that far are re-used, so whilst Pycraft isn't finished still, it is much further than we had planned to be, which is excellent. Now, this progress is also possible because where possible we are avoiding changing the actual body of the source code, we are only looking to change the way it interacts, meaning that this process is much faster than the development in Pycraft v9.5.5.

Unfortunately, we are expecting progress to slow, simply because whilst lots of utility programs get reused, there is a lot that doesn't, so changing these will result in a much less significant - but equally important - improvement in progress.

We hope that you have enjoyed this next week of progress on Pycraft, and whilst our progress to show for this week may seem strange, here are some images taken from Pycraft for this week:

Pycraft progress update!

Thomas Jebson — Wed, 15 Feb 2023 17:56:42 +0000

Hello there everyone, it's time for our weekly progress summary for the week commencing the 6th of February. This week has been a very interesting week, as the work we started this week will go on to set the trend for everything we do in Pycraft going forward. But before we get too into that, let's talk about what else we were up to this week!

We started this week then, as we started last week, by doing a lot of reading and research behind the scenes, as we had decided to begin the rollout of some changes we have been planning for a while behind the scenes. We are of course talking about the switch in some of Pycraft's utility programs to take advantage of the object-oriented approach to solving problems (often abbreviated to OOP, Object-oriented paradigm). This we have already seen to be very useful in theory and had revolutionised the way some utility programs for Pycraft operate.

It's important to note though that not all of Pycraft's source code will be switched over to this approach to programming, this is currently strictly focused on the utility programs for Pycraft, and even then, not all utility programs will need this switch. OOP takes advantage of the concept of objects, which are things that the program uses to interact with to solve problems. Therefore, these changes will be especially advantageous in sections like 'display_utils' which handles the way Pycraft interacts with Pygame's window, and provides additional functionality to it. So this week we have been working on these changes, and we expect this to continue for several weeks still.

Now, for a while, we have been developing Pycraft with the plan to switch from a procedural approach to OOP, and that is why we have included classes in Pycraft for some time. Classes in Python are commonly associated with OOP, although we regularly chose not to use them for their intended purpose. However, on Thursday we discovered Python and the way that classes interact, which would go on to change everything about the way we make Pycraft going forward.

So then, what was this discovery? Well to answer that, we need to talk about how Pycraft handles the transfer of data between programs. Previously this was done through copious use of the 'self' command, which stored the contents of a program we call the 'registry', this program stores the default values of variables used commonly across Pycraft. This worked well, except for two problems: readability and sharing data between programs running in parallel.

However, we thought we would address this in Pycraft v9.5.5 by completely changing the way data gets transferred between programs in Pycraft. To do this, we make a change that would force all utility programs to have parameters for the variables they needed, not just generic 'self', then they would need to return new data that would be transferred back into 'self' by the main program, where 'self' is the contents of the registry. This solved the two problems we faced: readability and the sharing of data between programs. However, this caused problems with scalability. Now, to change the parameters of a section of code, you would need to change all, sometimes hundreds, of references to that section of code. This is a problem.

The solution then is to backtrack on some of the changes we made in Pycraft v9.5.5 (we are by no means undoing everything we did in that update, we are however amending some features we introduced then) so now we are using classes properly, instantiate the registry, but using the updated values, and then adding parameters for everything we can't pass through the registry. This improves the scalability of new features and changes and improves the way data is transferred between processes running in parallel. This also improves readability, as we are significantly shortening the number of lines in Pycraft. Strictly speaking, this is a process from next week (a.k.a the future) but we manage to trim 2,000 lines off the source code for Pycraft in a single day, without changing any of the functionality of the source code we changed!

We hope that you have enjoyed this article, attached below is a preview of one of the sections of code we have applied these changes to, 'achievements.py', the right image shows the original, and the left one shows after we made the changes!

Pycraft Progress Report

Thomas Jebson — Tue, 07 Feb 2023 18:21:09 +0000

Hello there everyone, its time for our usual weekly progress summary! This is for the week beginning the 30th of January. This was a more relaxed week then the weeks immediately prior, as now that we have gotten the release of Pycraft v9.5.6 out of the way, we can focus work on Pycraft v9.5.7!

This week we have been busy working on a range of different areas of Pycraft, mainly improving features already in the source code for Pycraft, but also continuing with our work on the documentation. This week can be considered more of an intermediary week between two fairly significant events, the release of Pycraft v9.5.6, and the major changes to the installer which we start work on next week!

We started off the week with the rollout of the new pop-ups feature. Previously to this week, it was used almost exclusively in the new change-log screen that opens after updating or running Pycraft for the first time on your system. Now however it has replaced most of the pop-up requests that we used to implement through Tkinter’s messagebox functionality, mainly these are permission requests. We will just emphasise that Pycraft’s pop-up messages to display errors will not be seeing any changes due to the nature of this feature. In addition to the wider rollout of this feature, we have been extending its functionality, as prior to this change, there was no way of interacting with the window, other than to dismiss it. Now however you are able to interact with widgets in that pop-up, which makes things like permission requests possible, and is something that we have shown in the image at the end of this post!
The settings menu widgets where always made with modularity in mind, and so including them in the pop-up menu was really easy. However, we have had to expand the functionality of the text based programming language we are making to support Pycraft. For those unfamiliar, there are certain syntaxes that can be used in strings that get rendered to change the look of that particular bit of text, and now add in new elements. This isn’t something that we have written up in the documentation yet, but at the time of writing, this is one of the next things on our to-do list.

Aside from these changes, we have also been working on the documentation for Pycraft, our aim here was to (ironically) update the Update Timeline and Sound Preview sections of Pycraft’s repository and to transfer them over to the documentation too. For those unfamiliar the Update Timeline was yet to be updated with the most recent reviews to the plan we have for Pycraft going forward, so much so that it still uses the old version formatting system! This has been now done at the time of writing. What hasn’t yet been done yet at the time of writing is transferring the Sound Preview for Pycraft from Pycraft’s GitHub repository to Pycraft’s documentation, however since we last talked about this, we have found a solution to adding the sound previews, and have plans to extend the previews into extended forms, and perhaps even their final form, but this is going to take a considerable amount of time to do, so don’t expect much change to that for a bit!

Now, as we continue work on Pycraft v9.5.7, it’s important to make sure that when we add new features, we don’t neglect or break old ones, so we have spent this last weekend going through Pycraft and its online sites and making sure that everything is all up to scratch. Mostly issues have been fixed, especially any issues we noticed in the source code for Pycraft, however there are still some things that need fixing, and some slightly larger tasks to do behind the scenes to bring some sites up to date. But don’t worry, even though next week we start work on the changes to the installer, these important changes wont get overlooked, and we will allocate considerably more time after the completion of work on the installer to fix the things we have identified before they become a problem, and also plan to use that time to go through and test the installer, and also to run the same checks on the installer, which we didn’t include as we have plans to make some major changes there soon. We are not sure yet how long this process may take, but we are talking more than a week! After this, Pycraft v9.5.7 will be released, and then work on the new benchmark system for Pycraft can begin, and ideally sometime in the summer probably we will completely finish work on the 2D engine (in terms of adding new features, or improving the implementation of old ones) meaning that we plan then to spend a significant amount of time (on the order of years) working on the game engine!

Now, the changes we made this week where comparatively minor compares to the ones made last week, but the effect that has had is much larger, as we have now added in settings widgets to the pop-ups for Pycraft, which makes a big difference! Here are the most recent previews of the progress we have been making in Pycraft:

Pycraft Progress Report

Thomas Jebson — Tue, 31 Jan 2023 18:21:00 +0000

Hello there everyone, its time for our weekly progress update, and this one is a bit different, as in light of the recent release of Pycraft v9.5.6 we have had to combine the one from the week beginning the 16th of January, and the one the week beginning the 29th of January as we had to push the release of the first article back so that we could prioritise getting the latest version of Pycraft out!

Now we start off this article by discussing the testing of Pycraft v9.5.6, and we have decided that we are going to make a slight change to the way we test and make improvements to Pycraft. In Pycraft v9.5.5 we spent a very long time thoroughly testing all of Pycraft and the installer, this was because Pycraft v9.5.5 was a much larger release than we would expect for a developer version of Pycraft (if we were to rate our usual update size as a 5 out of 10, then Pycraft v9.5.5 was comfortably a 7 or 8) and as such, because so much had changed, almost everything needed testing.
Moving on then from that we are going to be changing our testing strategy so that we test the features that we have changed in a developer version of Pycraft, and the systems that use that feature. In addition to this, we will be fixing any bugs we find as we go, which is something we do as we make changes to Pycraft anyway. This has been put in place as Pycraft is now getting rather large, and sitting mid-way through the development of Pycraft v9.5.7, we are currently at a total of 22, 343 lines for the source code for Pycraft, and testing every single line would be incredibly time consuming. For comparison, we spent longer testing Pycraft v9.5.5 than we did on the majority of the work on Pycraft v9.5.6!
Naturally, the final major/minor (but not developer) releases of Pycraft will still have extensive tests and performance analysis carried out, so that we can make optimisations for slow areas of Pycraft, and fix all the bugs we can that we have identified.

Now we have been also putting in a lot of time on the documentation for Pycraft, as we continue with the development of Pycraft, this is because we have a goal that for every release (major, minor or developer) of Pycraft we must update the documentation with the changes to any areas of code that we have worked on, and an additional 5% of other sections of Pycraft that we haven’t yet written documentation for. This applies exclusively to the Module Breakdown section of the documentation.

However we have also been very busy filling up other areas of the documentation with useful information and content, and managed to finish the Introduction and Formatting guide before the release of Pycraft v9.5.6, and have also transferred over the about section from Pycraft’s GitHub repository here: https://github.com/PycraftDeveloper/Pycraft and have plans to add in the other areas too, like the update timeline and sound preview, but those will be added in Pycraft v9.5.7 because they need work done on them before they are release ready.

Now, we have been also making some pretty significant changes to the installer for Pycraft v9.5.6, as we have equally larger changes planned for Pycraft v9.5.7, which will be when the installer for Pycraft will become completely separate from Pycraft, and therefore have no reliance on any code from Pycraft its self, making updates easier to do in future for Pycraft through the installer. In light of this major planned change, we have made the decision to change some of the mechanics of the installer behind the scenes so that this change in Pycraft v9.5.7 will require less work to the core structure of the installer, although we have plans to make some pretty large changes to the entire installer based off of the latest changes we have introduced to Pycraft. This therefore meant that we had lots to do in the installer, but because this was prior to the launch of Pycraft v9.5.6, we were limited in what we could do, and what we wanted to do given that there would be so many major changes coming in Pycraft v9.5.7!

Immediately prior to the release of Pycraft v9.5.6, we shared a lot of interesting game mechanics and ideas we have for Pycraft when we start work on the game engine. Now some of these features are yes, very far way still, but some are much closer than you might think, and keep an eye out for the latest game mechanic previews on our twitter account or on Pycraft’s repository when we have finished compiling our list!

Now, moving onto the second week of work, we had just released Pycraft v9.5.6, and immediately started work on introducing the previously prototyped features into Pycraft. For those unfamiliar with how we work on new releases of Pycraft, as a group we spend around half our time working on features that will be coming in the next version of Pycraft, sometimes these features have been worked on for a considerably longer time than that, for example collisions is something that we first looked at adding to Pycraft nearly two years ago, and since then we have considerably refined the implementation, and plan to add it into Pycraft v10! By doing this we can significantly increase the quality of the features we add to Pycraft, as we have had time to better prefect them, and remove any ideas that may not be in our best interests for that release, all whilst significantly cutting down on the time taken to work on that version of Pycraft!

Now, on the subject of work on Pycraft v9.5.7, we don’t only add or revise features in Pycraft, sometimes we remove them too, and the benchmark system is no different. In the testing of Pycraft v9.5.6 we discovered a bug in the benchmark menu that prevented the process from completing successfully, with this in mind we made the decision to lock the user out of gaining access to the benchmark menu in Pycraft v9.5.6, and then in Pycraft v9.5.7 we have started going about the process of removing this flawed section of Pycraft, which is actually as planned as in Pycraft v9.5.8 we plan to completely redesign the entire benchmark section for Pycraft, and we need to clear up the previous attempt at this in Pycraft v9.5.7!

Now, with the benchmark section for Pycraft now completely removed from Pycraft v9.5.7, we have started working on adding the first new feature to Pycraft v9.5.7, and this is of course the new pop-up design for Pycraft. These pop-ups are going to replace in most situations the Tkinter message box windows that we have used previously, and because they are built into Pycraft, they should be better themed and more manageable than the alternative. Now, we cannot use this new pop-up mechanic in the event that Pycraft was to crash or develop an error, simply because we do not know what caused the problem, and therefore keeping our method of detecting and reporting crashes to the end user as simple as possible is crucial in order to make sure that everything works as intended in the event something does go wrong.
Now these new pop-ups will occasionally include widgets that we first introduced in the new settings menu, and will be used to: Ask the user for permissions, display minor errors and to display additional content to the user when a new menu is not required. This will not replace the messages that appear at the bottom of the home screen, and will instead build off that concept. These pop-ups can appear anywhere in Pycraft’s 2D engine, however they will never appear in the game engine, or anywhere in the 3D engine to avoid interrupting something important.

Now as Pycraft grows in complexity, it can be easy to overlook some really basic expectations for any application really; help.
Lots of applications and programs will come with some form of help guide that will help you to fix problems that cannot be necessarily fixed through code, for example if there is a problem with how Pycraft interacts with your graphics driver, or with actually installing Pycraft. There have been times when we have faced problems like this with Pycraft, and want to use our combined knowledge of Pycraft, and other common problems when running code to help user’s fix more common problems. This does not mean that we won’t help out if we need to, just let us know if your having problems, and if we think we have overlooked something, then the help section for Pycraft’s documentation (in versions 9.5.7 and newer) will be updated regularly with the latest information!

Now, in two weeks, lots has happened, and admittingly a large amount of it has been admin and documentation related, but we have got some new content to share as we begin work on Pycraft v9.5.7, and expect more as we continue to add new features!