DEV Community: PNS11

Fresh PHP development noIDE adventures

PNS11 — Fri, 24 May 2019 16:19:55 +0000

Instant interaction response and common interfaces across different development tasks help my productivity immensely. Having started a new job in PHP web development where the host OS on the everyday workhorse is Windows 10 this turned out to be a bit of a challenge. I'll be doing legacy code maintenance in parallel with fresh development of the next, well thought out version of the same application, mainly on a PHP/MySQL/Linux stack.

As it happens, targeting a Linux server environment from a Windows development box isn't exactly straightforward. If I'm not mistaken it is common in this situation to pay for and run a big IDE and possibly order around a fleet of handcrafted containers and wrestling with undebuggable Services rather than good old daemons.

I immediately tried out the Windows Subsystem Linux, but it turned out to be somewhat underwhelming. Somehow I had expected more than the lack of GUI and workarounds with MobaXterm being terribly wobbly at best (just don't go there) so I guess I'd been away from Microsoft fairyland long enough to forget the gritty struggles one must endure here.

However, I'm accustomed to SSH:ing into remote servers and fine with doing work in the terminal, which is basically the experience one gets with the WSL. Quite far from the full unixlike splendour but it is good enough.

There I mainly do work on the new application, with tmux, vim+plugins and ripgrep as the primary tooling, all of which install and run nicely under WSL. Once upon a time I used tag files but ripgrep chipped away at that and this time I haven't bothered with them at all. It's just too nice to

:!rg 'function weirdMethod' -A 5 -B 5

from within vim (or command line equivalent in a fresh tmux pane) and instantly get a view of how weirdMethod() is defined and commented and where it is. Especially when working with huge files that mix two browser layout languages, one browser scripting language and two backend languages freely and sometimes erratically it helps a lot to have a consistent, fast and regex capable interface to them.

Filesharing with the WSL is fairly convenient, instead of using SSH/SFTP for transfer one can reach the host filesystem through a mount point. Other than that it is the regular headless VPS experience for the most part.

Since I needed a setup with particular versioning of PHP, web server and so on, I looked into Hyper-V and running a Linux VM as well. I'm just not a container type of monkey. This turned out to be easy at first, a quick reboot to turn on the hardware hypervisor support in BIOS and it was good to go with a well supported Linux image.

VM:s are usually a bit sluggish but I soon found out that Hyper-V is aimed solely at mid to high end servers running headless guests. The UI was lagging intensely and constantly littered with graphical noise, for example, switching between workspaces in the window manager meant that I also needed to do a quick hop between tmux windows afterwards to render and clear up the screen to see my code.

So I went looking for a remedy. At first I only came across forum threads patiently describing how this wasn't a technology for virtual desktop computing, but through some search I managed to find some Linux VM tools from Microsoft and instructions for enabling XRDP connection to the VM.

Happily I chugged through it, rebooted, chugged some more, rebooted again, and got the XRDP VNC login greeter form. Great success, or so I thought.

There was one pretty obvious problem, logging in dumped me into the fairly hideous Ubuntu Unity desktop environment instead of my prefered and tailored i3 setup.

Panicking and revulsed I fumbled out a terminal and ripped out the line containing 'unity' from the .xsession and promptly entered 'i3wm' instead.

Quite the mistake.

What happened after reboot and entering my credentials was a sudden death of the XRDP connection. Over and over. It took me a little bit of patiently retrying while thinking about the issue to realise my mistake.

The i3 binary isn't named i3wm, hence the .xsession script crashed and burned on login, killing the XRDP connection.

After some thinking and reading up on help Set-VM in Powershell I figured out how to turn off XRDP on the Windows end, dropping me into the LightDM greeter instead, and also realised that I could just SSH into the machine to edit the .xsession file. So I did the fixing thing, and a few reboots later it had also been rudely pruned of all the rest of the preinstalled Unity-related calls the .xsession file had been cursed with.

Compared to the basic Hyper-V VM-connection XRDP is really snappy and almost entirely free from weird graphical artifacts littering the screen every now and then. I had to change the terminal font to make it render better and be more easily read but other than that i3 just worked and pretty good too.

If you are in a similar situation but haven't found that Github repo yet, go there and try it out. Just make sure your VM boots up to sshd being started (and reachable over your network or virtual switches) so you can PuTTY-rollback any mistakes while axing the defaults those scripts install for you, just in case you manage to break the LightDM config and get locked out from that one as well.

The commands for switching connection modes on the Windows end are these, to turn XRDP on first, off second:

PS C:\Windows\system32> Set-VM -VMName your-vm-name-here -EnhancedSessionTransportType HvSocket
PS C:\Windows\system32> Set-VM -VMName your-vm-name-here -EnhancedSessionTransportType VMBus

Powershell probably needs admin privileges for it to take effect. It should always have, to keep the chance open of bricking one's Windows system and getting an upgrade to something unixlike.

Other than this we also run some source control and keep remote repos, but that's rarely something one can decide as freely over as whether to suffer under the PHPStorm colossus or integrate oneself neatly and elegantly into a guest operating system with tmux-vim-ripgrep in WSL or Hyper-V.

Oh, and as a royal grande finale, here's the regedit you need to run to switch Caps and left Ctrl:

REGEDIT4

          [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Keyboard Layout]
          "Scancode Map"=hex:00,00,00,00,00,00,00,00,03,00,00,00,1d,00,3a,00,3a,00,1d,00,00,00,00,00

Figuring that out was severely cumbersome and boring and nasty (and if this hex happens to be wrong, comment and I'll share the other I stored which then is sure to be the correct one). Compared to throwing in a xkbdmap or whatever into one's startup scripts RegEdit is a nightmare. And yes, I know there are applications one can soil the system with that supposedly kind of solve it, but I don't like how they dress so they aren't welcome.

Notes on my first PHP job

PNS11 — Thu, 18 Oct 2018 17:14:13 +0000

This is kind of long and somewhat repetitive. That's sort of the idea, to show where a few bad decisions or structures will pop up everywhere, over and over.

Background

As a kid I dabbled in BASIC and Comal80 on a Commodore 64. The next computer was a 23 kilo 286 which managed Win 3.x over DOS and taught me batch scripting, then came others and I picked up some C, HTML and Linux before high school. There I was taught some Java, C++ and more web development, including standard SQL and JavaScript. The impression was that the schooling was abysmally bad and that adults knew nothing about computers, especially since my 'thesis', a project consisting of a small OpenGL engine at the end of high school, was graded solely on the documentation because the teacher didn't manage to get a C++ compiler and run it.

So I vowed to never make a living as a programmer and only use my interest in computers for evil. Instead I played around with a little cracking and a lot of malicious input into web apps over the next years, in a kind of grey hat not-very-legal manner.

Fast forward some eighteen years and I have changed my mind. At uni I studied some theology and law but never took a degree, in my spare time I passed through Python, Common Lisp, Clojure and then to the more obscure post-LISP language picolisp. This led to an offer I couldn't refuse, emigrate to the european south and work with modernising legacy PHP in an addiction business.

Having no ethical issues with profiting from semicompulsory behaviour, I wouldn't turn down Twitter either, and wanting the experience I jumped on the offer and relocated. At the time I had also found that my home country was turning weird and dangerous to leftists so it was attractive to leave and watch from a distance. It turned out to be great fun, absolutely terrifying and a very valuable experience.

Organisation

When I arrived I found myself in a group of roughly twenty developers including two bosses in one room where managers from elsewhere in the company considered it a virtue to take part in micromanagement of development. The result was an interesting, fragmented development process where specs and UX analysis seemed taboo, more or less replaced by vague instructions like 'port this' and 'add this feature' and a lot of silence following questions and suggestions.

After watching one boss integrating with a third party API for a couple of days I started getting assignments and worked away on them. Weeks passed without constructive feedback and it turned out that code review was not a part of the regular development work, and also that devs were explicitly forbidden review among peers.

Instead a kind of code review was implemented after one of the bosses moved to another country and started working remotely. However, it was done by skimming through commit diff snippets and commenting on string concatenation or recommending hands down bad coding practices. Sometimes these practices were implemented instead of the present, more adequate solution as to avoid friction or conflict.

Since every developer chugged along alone with little contact inbetween bosses also had to repeat themselves quite a bit, telling the same thing to different people, something which appeared to be inconvenient and wasteful to me.

The lack of structured planning meant deadlines didn't exist, it was rather a culture of quick fix driven development and incremental design. No cohesion in the team parallelled with little cohesion in code output, also this an interesting choice of managerial doctrine.

A result of this was that the code bases were quite irregular. One person liked warts on variables very much and no one else did. Another thought PHP lacked web templating and invented some in PHP, arguably the worlds most widely adopted web templating language. One had failed at creating REST API:s for years and adopted GraphQL instead, and thus coerced others to do it as well. Some liked Eloquent while others prefered passing in strings to Fluent QB for raw execution, yet others wrote a custom layer on a little of both.

When there was quitting and laying off ahead it was obvious but not spoken of, it seemed like the bosses thought the rest of us didn't notice anything. It meant quite weird moods among some colleagues and a lot of loose rumour when someone no longer worked there. Perhaps this was the idea, I found that hard to tell.

It took several months before I was first given a private talk about issues and performance, where I brought up concerns regarding this and some SQLi:s I'd found. I was told something something company policy and that we tend to patch such vulnerabilities when they were found.

Leadership by fear, anxiety and alienation can be profitable but only in sectors where quality is a non-issue, where machines are the producers and humans are used for pulling levers or something similar. It seems it doesn't translate very well to the art of programming.

Lessons learned

If your boss is more impressed by LoC output or obedience than code quality and ability to reason about it, start looking for another job because when this becomes an issue you're going to get fired rather than promoted
If every branch in a repo has a style of its own you'll learn a lot, but it will also slow down bughunting a lot unless it was you who wrote the code
If there is no constructive feedback on your work you'll have trouble performing well
A manager has the power of the payslip while you have the power of trust, hence you'll be more likely to be fired if your colleagues start trusting you more than their managers
Don't work unpaid overtime, and if you did anyway, make sure to tell your superiors you have so they know that they owe you more than they thought
When confronted with incoherent code bases consistently write in your own style to the best of your ability and try to make sure the result is good, rather than ask superiors or colleagues about it
Listen closely to what your colleagues are saying about past employees and what resulted in them no longer working there, this can help avoid sudden changes in your employment

Code

What I thought would be modernising legacy code eventually turned out to be conservation and wrapping of it in more and more projects. The core application was an elderly publishing platform rewritten in the details over the years but never refactored. A term that happened to be used as a synonym to 'changing existing code' rather than the common technical meaning at that workplace. At first this was very confusing.

This core was written in a procedural style with some OOP on top, ruminated for many years and having a high degree of interconnectivity between files and functions. The resulting complexity turned out as big red circles in PHPmetrics and quite challenging to anyone, regardless of previous experience. What had been described as microservice architecture was rather wrapping of bulky frameworks around new or old instances of this.

At some point when developing new software one needs to make some strategic decisions about what kind of programmers one wants to write code for. If one is the only one at a company who appreciates higher-order functions and currying as a way to break down problems one should avoid it and make a series of classes and interfaces instead.

On this issue I found no leadership and struggled quite a bit while figuring out that this was my issue and that I had to try and predict who might read my code later on. Not having a common idea of what kind of software shop the place should become seems to be bad for productivity. Instead the environment was one where old-timey for i < count(input) i++ was a viable pattern even though foreach was introduced in PHP 4 and a code review comment could consist of a demeaning comment about education and a link to the PHP manual on strings. Because apparently the problem with pasting unvalidated input into database queries are whether it is done with single or double quotes.

While the absence of strategic discussion and peer based code review together with quick fix driven development makes it imperative for a junior to quickly learn new tools and code patterns it also seems like it has some cost in the long run, since victorious debugging becomes reserved for the ruthless one cannot just assign anyone to a ticket without risking that they'll spend a week in circles.

Increasingly I got better at quickly judging the code and found more and more serious issues to add to my private bug database. For some reason there was no common one. These findings alarmed me more and more, since I thought it too risky to have some of them in production. When sounding the alarm I was however met with an indifference I had not anticipated, and later I learned that some of these had been known for years and that those who made a fuss consistently had been laid off.

Lessons learned

When confronted with incoherent code, relax and follow the execution over and over in your mind until you understand it and why the output is as it is
Thinking in incoherent code is demanding, never skip lunch because you'll get nowhere the last hours of the day
If you aren't getting good criticism of your code you need to invent it yourself
The worse the code, the more power is needed in your tooling as well as a solid understanding of command line scripting
Coding is an art so insensitive comments about it are sure to hurt, use this might wisely and with consideration
Don't give up bughunting until you have spent a fair bit of time breaking the application in creative ways

Conclusions

I strongly recommend anyone willing to do PHP but lacking work experience to take a job at a place that will hire you on just a few lines crappy code and some talk about knowledge in areas that have nothing to do with the business they're in.

You'll learn fast, learn a lot and see things that you otherwise would never understand how much you need to avoid until you've done the same mistake yourself and messed up bad. And this is very valuable, impossible to learn on your own (arguably you could pick some up in OSS dev) and will be with you for the rest of your life as a programmer.

Next I'll be more useful and show the toolchain I got comfy with.

Let's build a picolisp guestbook

PNS11 — Mon, 23 Apr 2018 11:07:11 +0000

Introduction

This is a short write-up on a guestbook web app I wrote for practice some time ago.

It renders a form for entering a message and posting it to a guestbook object.

To run it you install picolisp, save the source code below as 'gub.l', then you do

$ pil gub.l +
: (new! '(+Gbk) 'nm "Your guestbook name here")
-> {2}
:(go)(wait)

or replace the last line with

:(bye)
$ pil gub.l -go -wait

and now you can point a web browser to localhost:8080.

How-to

A good place to start developing a web app is to design your data model. For a guestbook just about anything from loose message objects to a complex schema with users, roles and sub-guestbooks could be thought up in a short time.

For this example I chose to go about halfway between the simplest possible, loose messages, and a model with an object holding messages in a list in one of its fields, naming them and making it easy to replace a guestbook without deleting all of the messages at once.

(class +Msg +Entity)
(rel hdr (+String))
(rel bdy (+String))
(rel sdr (+String))
(rel dor (+String))
(rel book (+Joint) msgs (+Gbk))

(class +Gbk +Entity)
(rel nm (+Ref +String))
(rel msgs (+List +Joint) book (+Msg))

With a model like this one we can expose a certain '+Gbk object in the GUI by adjusting this line:

*Gbk (car (collect 'nm '+Gbk))

# for example, (db 'nm '+Gbk "Certain Name")

A more production ready way to do it would be to replace this with a function for browsing and selecting among available guestbooks.

Fetching data from the GUI form is done by holding values in three global variables and then passing these as method parameters to the object held in '*Gbk.

The method looks like this:

(dm addM> (Hdr Bdy Sdr) 
    (let  
        D (new! '(+Msg) 
            'hdr Hdr # header
            'bdy Bdy # message body
            'sdr Sdr # sender
            'dor (stamp)
            'book This )
        (put!> This 'msgs D)
    (commit 'upd) ))

It is possible to nest 'put!> and 'new! but it would be harder to read and extend, for example, if the newly created object should be added to more places in the database it is usually more convenient to have a 'let scoped symbol for doing so instead of relying on a nest of implicit symbol passing.

Typically one doesn't use 'ht:Prin directly, it is a fairly low level function used by GUI components, but it is quite useful for experimentation and learning (and, of course, construction of new components). It is more idiomatic and a better practice to use '+Chart with a pilog expression, but it also requires a little more and harder to read code.

            (for X (get *Gbk 'msgs) # one can use 'get 
                (with X # as well as 'with and the shortcuts 
                    (<h3> NIL (ht:Prin (: hdr)) )
                    (<br>)
                    (<p> NIL (ht:Prin (: bdy)) )
                    (<br>)
                    (<p> NIL (ht:Prin (pack (: sdr) " - " (: dor))) )

For the next steps in refining this application it could be a good idea to replace this presentation code with 'url> or chart constructs, or perhaps prepend the 'form with another one that lets the user choose among available guestbooks, perhaps giving the choice to create a new one.

The entire source code

(load "@lib/http.l" "@lib/xhtml.l" "@lib/form.l")

# in a real application you would use 'allowed to restrict access to 
# certain directories and functions. 
# you would also put httpGate, nginx or somesuch between the Internet 
# and the port where your program is listening.

(class +Msg +Entity)
(rel hdr (+String))
(rel bdy (+String))
(rel sdr (+String))
(rel dor (+String))
(rel book (+Joint) msgs (+Gbk))

# here a TODO could be to write a method that outputs message contents

(class +Gbk +Entity)
(rel nm (+Ref +String))
(rel msgs (+List +Joint) book (+Msg))

(dm addM> (Hdr Bdy Sdr) 
    (let D (new! '(+Msg) 
        'hdr Hdr 
        'bdy Bdy
        'sdr Sdr 
        'dor (stamp)
        'book This )
        (put!> This 'msgs D)
    (commit 'upd) ))

(pool "gb.db")

(setq 
    *Gbk (car (collect 'nm '+Gbk)) # decide which guestbook we'll serve
     )

(de work ()
    (app) # sets up session handling
    (action # says there will be action in some web forms
        (html 0 "Guestbook" "@lib.css" NIL 
            (<h2> NIL "Picolisp guestbook")
            (<hr>)
            (form NIL 
        (<p> NIL "Message header:")
        (gui 'hdr '(+Var +TextField) '*MsgHdr 30)
                (do 2 (<br>))
        (<p> NIL "Message text:")
                (gui 'bdy '(+Var +TextField) '*MsgBdy 30 5)
                (do 2 (<br>))
        (<p> NIL "Your name:")
                (gui 'sdr '(+Var +TextField) '*MsgSdr 30)
                (do 2 (<br>))
                (gui '(+Button) "Send" 
                    '(ask "Post this message?" 
                        (addM> *Gbk *MsgHdr *MsgBdy *MsgSdr) )) )
            (<br>)
        (<hr>)
            (for X (get *Gbk 'msgs) # instead of '+Chart, '+QueryChart or 'url>
                (with X
                    (<h3> NIL (ht:Prin (: hdr)) )
                    (<br>)
                    (<p> NIL (ht:Prin (: bdy)) )
                    (<br>)
                    (<p> NIL (ht:Prin (pack (: sdr) " - " (: dor))) )
            (<br>) )) )))

(de go ()
    (server 8080 "!work") )

The end

If you thought this was worthwhile reading, consider donating a coffee or two, Stagling@Patreon.

Procrastinating with 'from

PNS11 — Sun, 08 Apr 2018 12:51:25 +0000

Introduction

Since I have trouble finding the time to finish that guestbook article I'll keep you bored with some more interactive parsing, this time we'll wring out and save a copy of a subset of a remote database we can access through a browser or similar web client.

Earlier I've written a little about how one can use the picolisp function 'match for exploring XML or similarly structured data. This is a very powerful pattern matching tool that can be utilised to extract just about any data from any set as long as you have an idea of how its environment looks like.

Sometimes you want something else and don't know enough for an efficient 'match expression, though. Perhaps you want to exfiltrate every occurrence of a pattern in a repetitive data set but isn't yet sure what the pattern ought to look like. If this is the case one usually applies the pair functions 'from and 'till instead, and in a slightly different manner.

I have used this technique for all sorts of interactive searching and pattern matching, including real work applications like chugging through gigabytes of web sites hunting for contact information.

The model

The other day I was in a pickle, I had some files containing markup and information about swedish metal bands from metal-archives.com and wanted to extract the information and make it searchable. Basically create a subset of their database locally and query it for specifics that I was interested in.

As you can see in the link it was about nine pages, and if you look at the DOM after the JavaScript has assembled it from the Ajax request and some HTML you'll see that the search result is a one line repetitive tbody-tr-td-construct. This makes it hard to parse out a pattern line by line, for which 'match would have been a perfect fit.

Instead we get to figure out what we want in sequential pattern fragments instead.

How to save a copy of a JS-generated, dynamic DOM is a small science in itself, which I will conveniently assume you have already doctored in. As for my setup, I did it with a Chromium instance and quickly got nine HTML files to examine and write a small parser for.

What we want to do is have the picolisp reader look over that search result line in the HTML files and give us some data structure back that contains band name, location, status, genre and URL. So we start it and tell it what our data model should look like and in what file to keep it persistent.

$ pil +
: (pool "mase.db") 
-> T
: (class +Band +Entity)
-> +Band
: (rel nm (+Ref +String))
-> +Band
: (rel loc (+Ref +String))
-> +Band
: (rel status (+Ref +String))
-> +Band
: (rel genre (+Ref +String))
-> +Band
: (rel url (+Ref +String))
-> +Band
:

Or as a snippet in an er.l:

(pool "mase.db") 

(class +Band +Entity)
(rel nm (+Ref +String))
(rel loc (+Ref +String))
(rel status (+Ref +String))
(rel genre (+Ref +String))
(rel url (+Ref +String))

This can then be loaded with '(load "er.l").

The 'from

Now that you have your model, HTML files and a pooled database it is time to start working on our parser. When doing this with a new set of data it is likely one does some examining from the REPL by reading a file into a variable as a list of transient symbols, or a list of strings, or a list of lines. One could also use a text editor or 'View source' in a browser, whatever works for you when you want to skim through and quickly get an idea about where the valuable information is and how the data is structured there.

Having looked at 1.html and found the line with ~500 bands and some data on them we can start figuring out how to get it out of there. 'from is like a hook while 'till is a release, so you tell 'from how it looks where the extraction should begin and 'till when to stop.

: (make # we want this to return a list
    (in "1.html" # from this file
        (until (eof) # and keep going until the end
            (link # linking lists into a list of lists
                (make 
                    (from "tr class") # from this pattern, look for
                        (from "href=\"") # this pattern, and then 
                            (link (pack (till "\""))) ] # grab it.

If you run this at your REPL and have the HTML files in order you'll get a list of URL:s, something like this but about 500 elements long '(("https://www.metal-archives.com/bands/Name/ID")("https://www.metal-archives.com/bands/Another_Name/AnotherID")).

This is built by the 'make-'link constructs, where the innermost one gets the URL as a transient symbol from 'pack. If we didn't use 'pack we would instead get the URL:s as lists of characters.

Just getting these links aren't satisfactory, though. We also want the rest of the information, so we'll extend the above like this.

: (make 
    (in "1.html" 
        (until (eof) 
            (link 
                (make 
                    (from "tr class")
                        (from "href=\"")
                            (link (pack (till "\""))) 
                                (from ">") # name is between >
                                    (link (pack (till "<"))) # and <
                                        (from "><")(from "><") # after two >< 
                                            (from ">") # then a >
                                                (link (pack (till "<"))) ]

As you can see, we can be quite ignorant of the details in the data and extract what we want anyway, it is enough to know that there will be two '><' until there is '>Data<'. This is slightly different from working with 'match, here we are extending our exfiltration incrementally rather than expressing ourselves with a dense list as pattern.

To get all the interesting information out of this data one would continue adding 'from expressions, the next one would look for a single '><' before extracting 'from '>' 'till '<', and the last extract would probably be 'from 'span', 'from '>' and 'till '<'.

Putting it in objects

Once all of this is in place we get a list where each element is a list '(Url Name Location Genre Status), making it almost trivial to save it as database objects. Assuming you ran it and got the result in the REPL you would probably do something like this, where @ in the regular REPL context is short for 'last on the result stack' and @@ and @@@ dig deeper.

: (for X @ 
    (new! '(+Band)
        'nm (cadr X)
        'loc (caddr X)
        'status (last X)
        'genre (cadddr X)
        'url (car X) ]

Once 'new! has run successfully your data is persistent, browseable and searchable. To be really, really sure it is in your database and will be available next time you open it, run '(commit 'upd) an extra time.

From here it is easy to put it all together with 'dir supplying a list of arguments for the resulting function. Simplest but somewhat ugly is to replace @ above with the full 'make-'from nest and take the file name from 'dir, so when it has gone through all of the files you end up with a lot of database objects containing a full set of swedish metal bands known to Metal-Archives.

It is of course better to factor it into a few functions in a file instead, at least if one wants to reuse the basic 'make 'in 'from 'till structure, however, it is as an interactive tool pil really shines and sometimes this means sacrificing a bit of elegance for a more brutish approach to code style.

Primitive ad hoc queries

Querying the database can be done in several ways. We used a simple '+Ref index so tolerant and substring searches aren't possible right away, but we can simulate this with our old friend 'match and the 'printsp function, that prints something with a space.

: (for X (collect 'nm '+Band) 
    (with X
        (and 
            (match '("S" "t" "o" "c" "k" "h" "o" "l" @B) (chop (: loc)))
            (printsp (pack (: nm) " - " (: loc))) ]

Using a variable like '@B amounts to a wildcard, this pattern will be true for all location fields that begin with 'Stockhol' (i.e. the next line will fire and 'printsp name and location), also those that look like 'Stockholm/SomeOtherCity'. When doing this in practice it is likely one puts a '@A at the beginning, and in a file one would use 'use to declare these symbols and prevent them from leaking artifacts from other parts of the program.

The end

If you would like to see this done quickly by the REPL I saved a ten minute session here that is basically a repetition of the above.

If you thought this was worthwhile reading, consider donating a coffee or two, Stagling@Patreon.

The data used above might be copyrighted, if so I consider this to be fair use and decent advertisement for an excellent web service, but will of course remove the post if the right people tell me to. While doing this I listened to Temple of Baal, Verses of Fire.

Addendum: Sorry about the formerly broken links, they are fixed now. Thanks aw for pointing it out.

Building Factor language without X/GUI

PNS11 — Mon, 02 Apr 2018 11:06:09 +0000

Over the last weeks I've procrastinated with stack languages more than usual and dabbled a little in a practical one called Factor. It has a lot of batteries included but no fluff and some edges are as sharp as a you know what, then again, it is very powerful, at least in the way Paul Graham meant in this well known essay and others.

Ran into this issue when trying to run the prebuilt binary for the Factor language on a headless Debian server. Basically it expects to find a shared GTK library where there isn't any since neither X nor GTK is installed in this type of environment. One could place a file with the same name in one's PATH as a workaround but it seemed a little too ugly even for my taste.

Following the recommendation to

git clone git://factorcode.org/git/factor.git

I then added 'NO_UI=1' to './build.sh', basically right after the hashbang. Without any other changes to the script I ran

./build.sh update

and it compiles as well as runs as a CLI interface on the first try.

After having done some experimenting with this Forth-like tool I'm quite impressed, if you're tired of the sluggishness and ad hoc qualities of Python, come to the dark and stack oriented side where programming is concatenative, strange and powerful. The standard libraries and FFI are quite good and seem to be well enough for exploratory and dynamic programming in almost all the common domains, from systems to applications and web.

Happy hacking!

Stagling@Patreon

My first asciinema

PNS11 — Mon, 26 Mar 2018 10:32:17 +0000

Earlier today I found out about an alternative to screen recording, asciinema.

It was quite easy to install under Debian by grabbing the package python3-pip and and using it for fetching the application.

$ sudo apt-get install python3-pip
...
$ pip3 install asciinema

It's about 20 minutes back and forth in the REPL showing how to reach text files and the shell commands.

Here's the recording.

Let's build: 'load to get faster early web development

PNS11 — Fri, 23 Feb 2018 09:56:30 +0000

When trying stuff out or doing early work on a web site or service it can seem slow and fragmented to restart your application server over and over again.

Instead of churning out all that Ctrl+C or D and possibly also restarting httpGate or nginx at every adjustment you could use a simple little trick that is suitable when testing app modules or just getting started on a project.

dev.l

# libraries for the app server
(load "@lib/http.l" "@lib/xhtml.l" "@lib/form.l") 

(de work () 
    (load "app.l"))

(de go ()
    (server 8080 "!work"))

Once you have this you run and forget about it:

$ pil dev.l -go -wait

Now if you visit localhost:8080 it won't be spectacular.

Enter some GUI boilerplate into "app.l".

app.l

(app) # assuming you want session handling
(action 
    (html 0 "Test bed for savvy gravy" "@lib.css" NIL
        (<h1> NIL "It's alive!")))

Next visit will then look better, and as you work on and save "app.l" and reload localhost:8080 you'll see changes reflected immediately without restarting "main.l".

app.l

(app)
(setq *Awesome T)
(action 
    (html 0 "Test bed for savvy gravy" "@lib.css" NIL
        (if *Awesome
            (<h1> 'awesome "Apparently you're awesome!")
            (<h1> 'notsoawesome "Pretty sure you won't see this!") )))

Using this technique 'in production' isn't a good idea since file I/O performance is about as bad as it gets, but compared to restarting the same script over and over manually while allowing a separation of your development helper functions and globals in "dev.l" from the app code in "app.l" will feel like super powers.

There are of course other ways to achieve this or similar results, you could for example use coroutines or background tasks instead, laying ground for a more sophisticated integrated development and production administration environment. Or whatever it's called when you program your servers by pew-pew:ing messages at objects and watching web services grow out of it.

Next time we'll create a small guestbook application.

Stagling@Patreon for coffee donations, unless you'd perhaps, y'know, like to hire me or something, then I'm sure we could work something out.

Raw XML exploration with picolisp

PNS11 — Mon, 05 Feb 2018 13:40:01 +0000

Below is a short tutorial on using the pil VM, DB and OOP for basic data mining in XML structured data. The article assumes some knowledge in programming and the environment to be POSIX or at least unixlike. For information on installing picolisp, visit picolisp.com.

How to lisp some XML

This year we're having elections in Sweden. To prepare it might be a good idea to read up on the last election cycle back in 2014.

Luckily the Swedish government supplies us with a full set of voting results.

wget http://www.val.se/val/val2014/slutresultat/slutresultat.zip

unzip slutresultat.zip

This will unpack a lot of ISO-8859-1 encoded XML files in a new directory, as well as some metadata files about the rest. It is easy to convert them into UTF-8, like this.

Most of them contain voting results from a county, regarding either national parliament, regional or local assembly. The file names are constructed from a numeric county code and a character for each election, R for parliament ("riksdag"), L for region ("landsting"), K for local assembly ("kommun").

In the directory with the data we then start pil.

$ pil +
:

Not very exciting. Let's take a look at one of the files by attaching it to a symbol and do some simple processing.

:(length 
    (setq A 
        (make 
            (in "slutresultat_1407L.xml" 
                (until (eof)(link (line T]
-> 1832
:(head 10 A)

As 'head shows this actually is XML data. Since it is we also know that right after every '< there will be interesting information, and we can see that it usually is the first or second character in the data attached to our 'A symbol.

To extract this we can use an expression like this.

:(length 
    (setq *Subjects 
        (uniq 
            (make 
                (for X A 
                    (and 
                        (or 
                            (match '("<" @Subject " " @B ">") (chop X))
                            (match '(@C "<" @Subject " " @B ">") (chop X)))
                        (link (pack @Subject]
-> 17
:*Subjects

In a real mining session one would most likely use 'de and turn this into a function that takes a list (for 'match to use) as argument and has the side effect that the symbol and global variable *Subjects (which would probably have another name) is updated with the results from 'match. When doing interactive work globals can be like post-it notes of arbitrary size, holding anything from millions of text fragments to object references or the emptiness of NIL while the temporary nature of these sessions is resistant to the horrors of globals in production code.

Here 'length is used to make sure the expression returns something both short and useful, the amount of unique hits in our search. 'setq returns the contents of the symbol which means that it will fill the screen with junk unless our expression is sensible or the data set is very small (and who would want that?).

Picolisp 'match is often useful where one would otherwise use 'grep' or regexp. Its first argument is a list that it matches against its second argument, and when there's a fit the @Variables get the content that's in the data source at that place in the structure. This means it is extremely well suited for sophisticated exfiltration techniques as well as the basic ones shown here.

As you can see we now have a set of 17 unique XML tag names. These can be supplied to our earlier expression to extract more information. Let's have a look at what "PARTI" contains.

:(length 
    (setq *PARTI
        (uniq 
            (make 
                (for X A 
                    (and 
                        (or 
                            (match '("<" ~(chop "PARTI") " " @B ">") (chop X))
                            (match '(@C "<" "P" "A" "R" "T" "I" " " @B ">") (chop X)))
                        (link (pack @B]
-> 21
:*PARTI

The tilde (~) causes the expression to which it is attached to be evaluated as regular lisp, here 'chop turning a transient symbol like "PARTI" into the sequence "P" "A" "R" "T" "I", before matching the result against the list in the next argument. If it is easy to omit 'chop one should, or at least use it to prepare a list for 'match that gets reused, since it brings some overhead that gets very noticable with bigger data sets.

Apparently 21 parties ("PARTI" means 'party') ran in the regional election in this county (the name of which is in the "NAMN" attribute of the "KOMMUN" tag).

Let's write out their names.

:(mapcar 
    '((X)
        (and 
            (match '(@A " " @B "\"" @Party "\"" @C) (chop X))
            (printsp (pack @Party)))) 
    *PARTI)

This kind of expression is especially useful, since the first argument to 'mapcar could perform all sorts of side effects while 'mapcar itself will return a list of all successful matches regardless of those side effects.

One such potential use is saving the exfiltrated data or perhaps entire lines or documents in database objects, by simply putting a 'new! clause at the end and having set up some classes for holding and structuring the data of interest.

It could look something like this:

:(class +GeneralContainment +Entity)
-> +GeneralContainment
:(rel tag (+Ref +String))
-> +GeneralContainment
:(rel dta (+List +String))
-> +GeneralContainment
:(mapcar 
    '((X)
        (and 
            (match '(@A "=" "\"" @Short "\"" @B "=" "\"" @Full "\"" @C) (chop X))
            (new! '(+GeneralContainment) 
                'nm "Party names, (abbreviation . name)" 
                'dta (cons 
                    (pack @Short) 
                    (pack @Full))))) 
    *PARTI)

Such class names aren't particularly idiomatic, the point here is that these classes ought to be very generic and kept in a small utility library. Usually the structure above or one with two classes where one can hold many '+Joint relations to objects of the other class (in addition to a name/category field and one for data) is enough to structure most interactive mining.

Once one gets the hang of the basic list manipulations, text parsing techniques and pilog expressions it can be quite a bit easier to go straight to exfiltration into generic objects instead of dabbling with globals, especially with bigger data sets, but then one should also use some helper methods that do the exfiltration without blocking the REPL since DB operations will probably take some time.

Thanks for reading, questions and comments are welcome.

Stagling@Patreon

First post, picolisp script

PNS11 — Sat, 03 Feb 2018 13:39:09 +0000

Today I made a oneliner for converting a set of ISO-coded XML files in the current directory to UTF-8.

(for X (dir) (and (match '(@A "." "x" "m" "l") (chop X))(and (out "sh.sh" (prinl "#! /bin/sh"))(out "+sh.sh" (prinl (pack "iconv -f ISO-8859-1 -t UTF-8 " X " > utf8_" X)))(call 'sh "sh.sh"]

A more readable, commented version could look like this:

(for X (dir) # for every file name in this directory
    (and 
        (match '(@A "." "x" "m" "l") (chop X)) # check if *.xml
        (and # it is or this is skipped, so...
            (out "sh.sh" # we write a new file
                (prinl "#! /bin/sh")) # with a hashbang 
            (out "+sh.sh" # and then add an instruction
                (prinl 
                    (pack "iconv -f ISO-8859-1 -t UTF-8 " X " > utf8_" X)))
            (call 'sh "sh.sh"] # and finally run it

Now this isn't very lispy, but it can quickly be turned into a more functional style.

(de qconv (Lst) # '(FromCode ToCode OutPrefix InSuffix FileList) 
    (for X (last Lst)
        (and 
            (match '(@A ~(chop (cadddr Lst))) (chop X))
            (and 
                (out "sh.sh" 
                    (prinl "#! /bin/sh"))
                (out "+sh.sh" 
                    (prinl 
                        (pack "iconv -f " (car Lst) "-t " (cadr Lst) " " X " > " (caddr Lst) X)))
                )))
    (call 'rm "sh.sh"]

Still quite ugly though.