DEV Community: DiverSE

Testing your DSLs in Langium

DiverSE — Tue, 07 Nov 2023 14:18:09 +0000

Langium is a framework to build domain-specific languages (DSLs).
The principle is to write a grammar-like specification and obtains for ~~free~~ zero-effort a parser, an abstract syntax tree (AST), a customizable and advanced editor that can run on the Web or on any modern IDEs such as VSCode, and facilities to interpret or compile programs written in your DSL. Basically, engineering external, textual DSLs at the speed of light!
Langium is the successor of Xtext and is under active development.
Langium targets TypeScript, Language Server Protocol (LSP), VSCode, and Web technologies.
One "feature" missing from Xtext is the ability to programmatically test your DSL: test your syntax and conformant/illegal programs, test your interpreter or multiple compilers of your DSL, etc.
In this post, let's outline a possible setup for facilitating the testing of a DSL using Langium.

Running example

Let us consider a simple DSL for chess game.



grammar ChessGame

entry Game: "White:" whitePlayer=STRING "Black:" blackPlayer=STRING (moves+=Move)+;

Move: AlgebraicMove | SpokenMove;

AlgebraicMove: (piece=Piece)? source=Square (captures?='x')? dest=Square;

SpokenMove: piece=Piece 'at' source=Square (captures?='captures' capturedPiece=Piece 'at' | 'moves_to') dest=Square;

// regular expression ('a'..'h')('1'..'8');
terminal Square: /[a-h][1-8]/; 

Piece returns string: Piece_PAWN | Piece_KNIGHT | Piece_BISHOP | Piece_ROOK | Piece_QUEEN | Piece_KING;
Piece_PAWN returns string: 'P' | 'pawn';
Piece_KNIGHT returns string: 'N' | 'knight'; // not K, because of King
Piece_KING returns string: 'K' | 'king';
Piece_BISHOP returns string: 'B' | 'bishop';
Piece_QUEEN returns string: 'Q' | 'queen';
Piece_ROOK returns string: 'R' | 'rook';

hidden terminal WS: /\s+/;
terminal ID: /[_a-zA-Z][\w_]*/;
terminal STRING: /"[^"]*"/;

hidden terminal ML_COMMENT: /\/\*[\s\S]*?\*\//;
hidden terminal SL_COMMENT: /\/\/[^\n\r]*/;

It is actually based on the old tutorial of Xtext in 2009 (https://www.slideshare.net/HeikoB/xtext-at-eclipse-democamp-london-in-june-2009).
The DSL has limited, practical interest (PGN is better!), but why not for having a more human-readable DSL for chess games?

Out of the grammar, you can enjoy typing a program (a chess game) using the VS Code generated editor.

Playing with the editor is fun, but your tries and errors will go away: How to test more systematically and automatically the syntax of your DSL?

Testing the syntax

We have used Xtext for many years, as part of teaching and research, and enjoyed facilities to test DSL (see eg https://blog.mathieuacher.com/XtextStandaloneParsing/).
Xtext was oriented towards Java, Eclipse, and EMF as opposed to Langium that targets TypeScript, VSCode, and Web technologies.
A solution for testing DSLs in Langium is not straightforward, here is a possible approach.

We first need a helper function that loads any program written in your DSL and returns the AST.
In TypeScript and leveraging the Langium API:



import { describe, expect, test } from 'vitest';

import type { Game } from '../language/generated/ast.js';

import { AstNode, EmptyFileSystem, LangiumDocument } from 'langium';
import { parseDocument } from 'langium/test';
import { createChessGameServices } from '../language/chess-game-module.js';
const services = createChessGameServices(EmptyFileSystem).ChessGame; 
...

async function assertModelNoErrors(modelText: string) : Promise<Game> {
    var doc : LangiumDocument<AstNode> = await parseDocument(services, modelText)
    const db = services.shared.workspace.DocumentBuilder
    await db.build([doc], {validation: true});
    const model = (doc.parseResult.value as Game);
    expect(model.$document?.diagnostics?.length).toBe(0);
    return model;    
}

The imports and Langium facilities are not well documented right now.
The parseDocument function is key to load a program and obtain the AST.
Promise<Game> is the type of the AST and is specific to our DSL.
Same for const services = createChessGameServices(EmptyFileSystem).ChessGame;.
You should adapt to your grammar and main entry point (rule) or DSL name.

You can then call this function in a test, and puts further assertions on the AST, leveraging vitest.
For instance:



describe('Test basic game', () => {
    test('Two moves', async () => {
        const game = await assertModelNoErrors(`
        White: "INSA INFO5"
        Black: "ChatGPT"

        e2 e4
        e7 e5
        `)
        expect(game.whitePlayer).toBe("INSA INFO5");
        expect(game.moves.length).toBe(2);
    });
...
});

You can check some representative programs of your DSL that should be accepted or rejected by the parser.
More examples here: https://github.com/acherm/dsl-langium/blob/main/Chess/src/test/validation-test.ts

Testing interpreters and compilers

In the same spirit, you can test the interpreter or compiler of your DSL.

For instance:



import { convertMovesToPGNWithPython, generateMoves } from '../generator/pgn_converter.js';

...

test('Immortal game', async () => {

        const game = await assertModelNoErrors(`
        White: "Adolf Anderssen"
        Black: "Jean Dufresne"

        pawn at e2 moves_to e4 
        pawn at e7 moves_to e5        
        pawn at f2 moves_to f4 
        P at e5 captures pawn at f4 
        bishop at f1 moves_to c4
        queen at d8 moves_to h4
        king at e1 moves_to f1 
        P at b7 moves_to b5
        bishop at c4 captures pawn at b5 
        knight at g8 moves_to f6        
        `);

        const pgn_moves = generateMoves(game.moves);
        expect(pgn_moves).toBe("1. e4 e5 2. f4 exf4 3. Bc4 Qh4+ 4. Kf1 b5 5. Bxb5 Nf6");
        (async () => {
            const pgn_moves_with_python = await convertMovesToPGNWithPython(game.moves);
            expect(pgn_moves).toBe(pgn_moves_with_python);
        })();
    });

There are tests of an interpreter based on chess.js that translates a chess game into PGN format.
There are other tests for the translation into PGN, but this time generating and executing Python code (based on python-chess).
More details here: https://github.com/acherm/dsl-langium/blob/main/Chess/src/generator/pgn_converter.ts and https://github.com/acherm/dsl-langium/blob/main/Chess/src/test/pgn-test.ts

Other technicalities

To make it work, you have to set up a few things in your project:

adding a vite.config.ts to set up vite testing framework (eg see https://github.com/acherm/dsl-langium/blob/main/Chess/vite.config.ts)
in the package.json adding a script rule "test": "vitest" and devDependencies something like "vitest": "^0.27.3",, in such a way you can use npm run test (eg see https://github.com/acherm/dsl-langium/blob/main/Chess/package.json)

Conclusion

This post has shown how to test your DSLs in Langium, leveraging the Langium API and the vitest testing framework.
For sure, other testing frameworks can be used as well.
The solution presented, quite closed to ParseHelper in Xtext, can be systematized to any DSL.
Hopefully, new versions of Langium will generate such testing facilities out of the box.

Github of the illustrative project: https://github.com/acherm/dsl-langium/tree/main/Chess.
See also DSLs in Langium here: https://github.com/acherm/dsl-langium/.

End-user Programming of Flappy Bird with ChatGPT: A Reality Check

DiverSE — Tue, 19 Sep 2023 16:08:29 +0000

Is it possible to program non-trivial applications and customize code without knowing much about programming?
Impressive showcases of ChatGPT suggest a positive answer.
Some people have even claimed that programmers will be replaced or disappear.
So, can end-users create working and non-trivial applications, like games, just using instructions in natural language with ChatGPT?

We have considered the game Flappy bird, using Python.
We have employed various methods and strategies to interact with ChatGPT versions 3.5 and 4, using different prompts, and trying to play the role of an "end-user"

Method, prompts, code, observations, and results

Specifically, the five (prompting) methods we tried are as follows:

A short prompt. The prompt is given as a simple sentence that provides the overall (general) goal of the task. No specific information nor detailed functionalities are given. At lot have to be inferred by the Large Language Model (LLM) at generation time. For instance: > Write a Flappy Bird game in Python.
Providing a list of characteristics (this time written by a human). The user will come with a list of functions and/or characteristics that describe the game. The list is given all at once to ChatGPT and is used as the prompt.
A short description of the main features. This can be seen as a mix of previous methods, as the prompt is written in natural written language and not as a list of functionalities or characteristics. Yet, the text gives more information than the prompt proposed in the second method. For instance: > Write the code for a Flappy Bird game in python. I have a folder "assets" with "background.png", "pipe.png" and "bird.png". I would like to know my score and keep track of my highcore in a separate file. I would like to have a start screen when I first open the game, where I can see my highscore, and a game over screen where I can see my score, highscore and play again.
Giving ChatGPT an example of finished code doing what we want (in our case a complete Flappy Bird game written in Python), and asking it to return a prompt. The prompt returned was never the same (probably because of the temperature parameter), but was usually in the form of a list of characteristics.
A list of prompts (without having to look at the code in between the requests). This last method would mimic the behavior of an end-user that carefully thought about what they need. They do not really interact with the LLM and only give the instruction one after another without caring for the provided output. Only the final output is used and potentially asked to be improved, anything before that is only the user trying to build little by little what would be its codebase.

We obtained different codes and games, with varying qualities (let say).

Observation and synthesis

Do claims generalize? (Use the prompt and you'll get a game)

No, we did not find a method (or a magic prompt) that systematically works and would generate a feature-rich game.
Though we have been able to generate interesting and playable games without technical intervention, there are also several cases and sessions that lead to a dead-end situation, far from the ideal scenario usually promised.

First, it happens that we reach a dead "end-state" with a non-working or unusable game (mainly due to inability to fix an issue), see for instance:

2nd method, 2nd session (it is possible to continue, but just requiring skills to debug code "("Pipes only came 1 by 1 and not by pair. Trying to implement that only led to more problems that ChatGPT couldn't solve.")")
2nd method, 4th session (10 interactions, Too many problems (no top pipes, always at the same height, wrong collision detection...))

Second, despite the same original prompt (and the same "method"), we can have very different generated code.
This difference is in terms of:

issues in the code, requiring more or less fixing and debugging effort
features supported: ChatGPT takes the liberty to implement (or not) some functionalities, and the resulting game might be very different... It forces again to interact with ChatGPT, specifically with regard to what has been generated.

The consequence is that several specific interactions are needed, some leading to the worst situation (no working game), far from the ideal scenario.

Is it possible to program without expertise?

Yes, but again, it is not systematic.

Direct interventions in the code are sometimes needed (to fix issues!).
For example, we had to [...] change the values of the gravity, bird_movemement and pipe_gap to make it easier and more controllable.
ChatGPT does not seem to find how to fix the code by itself (when we just point out the problems).
Besides, it is more challenging to interact as end-user when the game was from the start in a very bad shape, and basically unplayable or missing critical features. In this case, it is not possible to get a visual observation that would help to formulate a proper feedback.

The style of ChatGPT is sometimes to decompose the problem and put placeholders in the code... but without the implementation!
From a developer perspective, it is an interesting style that forces to consider a step-by-step implementation.
However, from a end-user perspective, the game is in an incomplete shape.

There are sometimes ChatGPT explanations related to a "feature" that does not exist, leading to time-consuming effort.
For example, Implement a feature that has no meaning "There was a 'FLAP' event that made the bird jump every 200ms. That's not part of the game!"

Some examples of issues that needed to be fixed:

clock (pygame specifics?) => bird crashes immediately
gap placement between the pipes
collision detection
breaking of older functionality
missing a bit of diversity

Both fixes are time-consuming. It is possible to help, thanks to end-user, visual observations about the game.
Sometimes, there is the need to orient discussions and interactions towards code.

How are games different? What are the features?

In our git repository we share sessions and code.
Our observation is that you can get well a feature-rich Flappy bird, but also a game with closed pipes that make the progression of the game limited.

See the Youtube video for an excerpt of games you can get

Conclusion and perspectives

Formulating a prompt and systematically getting a comprehensive and playable game is not yet a reality.
ChatGPT can provide impressive results, but not all times. A magic prompt is missing to make ChatGPT reliable.
Many interactions are rather needed to fix issues or expand the features, sometimes in non-technical terms and with instructions out of observations of the current game. But the control of the code is not that far and seems inevitable.

Throughout our experiments, we notice several positive aspects of using ChatGPT:

inspiration, funny variants
discovering new features
good starting point for developers (or end-users)
sometimes it can work and end-users can create interesting games

There are several interesting directions to consider:

change the targeted programming language and/or the framework: instead of Python and pygame, it's possible to use JavaScript and p5 for instance... A hypothesis is that the targeted technological space can help ChatGPT filling the gap between the prompts and the intentions.
find a super prompt or a language (who says a domain-specific language?) that leads to more determinism and control of ChatGPT
improve the usability and the integration of ChatGPT outputs into development environment. The back and forth between the IDE and ChatGPT is time-consuming. Moreover, some (informal) instructions of ChatGPT can be automatically applied onto the code base, limiting the user effort or the technical expertise required. The hope is to have a better feedback-loop!

If you are interested, we are sharing 35 sessions, with prompts, code, observations, and results (videos) with ChatGPT-3.5 and ChatGPT-4 in this repository: https://github.com/diverse-project/enduserprogrammingLLM/

And we have open positions (for internships, engineers, PhDs, post-docs, etc.) about the use of ChatGPT in software engineering at DiverSE!