DEV Community: jreighley

Graphing RPG source code with Neo4j

jreighley — Tue, 09 Oct 2018 05:17:25 +0000

As I learned Clojure, I had a lot of crazy, over-ambitious thoughts. I find that it is really good to try hard things. Software documentation and training materials seem mighty dry and cryptic until you have built "Buckets" to put the knowledge into. But once you have tried and failed, or tried and succeeded inefficiently, when you get training, you think, "If I had known that, I would have been able to write my program that way." It is much easier to retain knowledge once you know how and why it is useful. You have hooks to hang it on.

One such experiment stemmed from my attempt to "mass document" our software. Legacy RPG code is highly structured. It contains 'F specs' which define what files are used for input and output. I had been working on writing documentation, and thought "That would be useful to have" So I wrote a Clojure program to read through all 8000+ files we have and built maps out of the f specs contained within each of them.

As I scanned the code I also searched for the most common patterns our programmers used to have one program call another program and added those into the map. Many programs are controlled by constants questions, so I searched the code for references to those as well.

It was a fun project, but eventually I just had a big map (2.9 MB!). I was unsure how to use that map effectively. It seemed to unwieldy to use directly. It got filed away as a cool learning experiment.

A year or so passed, and I started to become interested in Graph databases. I downloaded an installed Neo4j. I tinkered with it enough to suspect it could do the job. Once I had "Buckets built" (questions that I needed to solve) I enrolled in a 1 day Neo4j class.

I quickly wrote another clojure program to load my map into a Neo4j DB. I created nodes for all the programs, files, and relationships between them. Once I got the data into Neo4j, I was able to start to write queries. I added labels indicating which programs users can call from the menus.

So now I can write a Cypher query that gets all of the possible paths that branch off of a menu item. A customer may call and say "I ran this program, and this report is wrong" -- And I can quickly identify the actual program that wrote the file in question (sometimes the third or forth in a sequence of events) This gives me a more certain starting point for troubleshooting. In the map above for example, the customer calls GRJRNL_C -> program GRGLUP -> file GLDET -> program GRGLPR -> file GLDT. It is much easier to look at the bubble graph than it is to read through 3 programs worth of source code to figure that out.

It is still far from perfect. Programmers have changed a lot over the last 35 years, and I have identified most of their patterns, but not all. The CL code can also override the files to redirect input or output to use another file. I suspect my automated documentation about 80-85% accurate, but if you take it with that grain of salt in mind, it is still pretty hand as a starting point.

And I learned a lot about RPG, about Clojure, and about Neo4j -- so even if it comes to nothing, I've got that!

Dynamic SQL cheatsheet

jreighley — Sat, 06 Oct 2018 07:39:45 +0000

I have been writing a lot of my own IBMi diagnostic tools lately using Clojure and ClojureScript.

One of the problems I often encounter is needing to track down some data without having a great understanding of the tables and data I am querying.. So I built this tool to help me build queries.

When I select a machine, it selects a default schema and gives me a list of all possible tables. I select a table and press 'Get Sample' and it fetches me the 'Select * FROM table Limit 5' and displays the results. This is very helpful in that I can see the field names as well the format used for the data in each field. Sometimes my 5 samples don't give me enough information. Perhaps all five samples have the same value in a field or no value at all. I added a sample button in each row to help me with this case. When it is pressed, it fetches me the first five 5 distinct values from that field. Does this customer use this field at all? Is it what I think it is? Answers are often a click away.

At the bottom of the sample, I have a textbox in which I can type a SQL query. It's much easier now that I have a dynamically generated cheat sheet that gives me hints for any table.

It is already very helpful. Questions like "what date format" are no longer guesswork.

I plan to enhance it by having it suggest joins for me. I also plan to have it save queries for future use, and have it suggest those queries when a particular table is selected. I may add full interactive SQL capabilities to it. Time will tell.

Open Source: Many of the benefits of socialism without the baggage

jreighley — Fri, 17 Aug 2018 16:10:14 +0000

I wonder how much Open Source has contributed to the massive decline in global poverty.

I suspect it's role is not at all trivial. The ethic of "let solve problems because they are worth solving, with or without profit" and "let's share those powerful tools to the masses" Changed the game in exponential ways.

How many business decisions are made in the world today without the use of Linux, apache, mysql, php, or another free variant of such? Nearly none. We have cheap smartphones in the pockets of the poor and developing world powered by linux or a variant.

When I was a kid, It cost dollars to call a state away. I recently watched my kids play 'Magic the Gathering' with over video chat with their friends in Australia for several hours. If technology can be used this trivially, How big of an impact is it having where it generates goods and services? If you can afford to negotiate for hours over a game, you can afford to negotiate for seed, tools, medical missionaries, well drilling missionaries, and all kinds of things that will make life better, freeing people from doing subsistence tasks and being able to move to more exponential endeavors of building tools and infrastructure.

There is a libertarian socialism that has emerged without the need for government revolution or force. It is handing the means of production to the masses. Those tools being distributed allow communities much better ability to solve problems locally and to communicate and coordinate with others to obtain resources that are needed.

You get many benefits of socialism without the the need for government involvement at all. We still have a ways to go. As Open Source ethos spreads outside of the software realm, and into hardware, we can expect it to further accelerate it's compounding interest of making things cheaper and better. Gradually we will let go of the old way of doing things, and migrate to the new. It only makes sense -- Even the big oligopoly software companies are joining the movement. The old expensive systems will be circumvented by the path of least resistance.

This thought was inspired by Mike Overby's post

Clojure on the IBM i

jreighley — Fri, 10 Aug 2018 20:02:13 +0000

I have been using Clojure to make API's that talk to IBMi servers.

Clojure is nice because it interops nicely with Java, so it can access the IBMi via the JT400 toolkit.

I do a lot of support for various clients running IBMi servers. There are many troubleshooting exercises that I used to have to do manually, serially and asynchronously. Connect to a client, run a report, scan the results, check a constant here, check a constant there - verify certain jobs are running and have no messages etc.

I can now point my browser at an URL, and my Clojure program will run the whole troubleshooting tree, and present me a webpage showing me all of the results. Often I can even include a "Fix it button" next to the problems.

If you are interested, here is a pretty bare bones starter gist: