DEV Community

Rupal H Patel
Rupal H Patel

Posted on

Building a Custom Language Editor Leveraging Eclipse DLTK

#eclipse #java #ide #tutorial

Rupal Patel and Kaniska Mandal · written 2008-2009, revised 2026

Preface (2026)

This guide was written in 2008-2009 for the Eclipse wiki while we built a DLTK editor for a business modeling language. It was never published there; this is the first full version, lightly edited.

It targets the DLTK 1.0-era APIs. It is most useful if you maintain an existing DLTK-based editor (Eclipse PHP Development Tools still depends on DLTK) or want to see how IDE language support is structured. For a new language today, write a Language Server Protocol server instead; the same pipeline of parser, AST, structure model and completion still applies.

Code samples are excerpts. Helper methods and fields such as buildModel and completionToken are left out for brevity.

Introduction

This tutorial builds a full language editor for a custom language on top of the Eclipse Dynamic Languages Toolkit (DLTK): a nature, a language toolkit, a source parser, a source element parser and a completion engine.

By Rupal Patel and Kaniska Mandal. An earlier version was shared as Creating A Language Editor Using DLTK.

The problem. Tool vendors need a free-flow editor so business analysts and tool developers can write in tool-specific languages. That editor needs the full set of IDE features: syntax coloring, auto-completion, content assist, code folding, selection, problem markers, an outline view and an AST parser.

The example language. Throughout, we build an editor for a hypothetical Business Modeling Language (BML). A BML file looks like this:

Package com.bml.samples

BizModel {
    Declare {
        Customer bob = Customer.createCustomer(...);
        Account savings = Account.createAccount(...);
    }
    Get {
        String name = bob.getName();
    }
    Put {
        bob.setAccount(savings);
    }
}
Enter fullscreen mode Exit fullscreen mode

Versions. The code targets the DLTK 1.0-era APIs (2008-2009). Later DLTK releases changed some signatures, notably the sourceParsers extension point, which now takes a parser factory.

Background: Eclipse text editing and DLTK

Eclipse's text editing framework follows the same layered MVC design as the rest of the platform, and DLTK plugs language intelligence into it.

The four layers of the text editing framework are:

Layer Role Eclipse type
Model Document content core (IDocument)
View Rendering SWT StyledText
Controller Wires model to view JFace ISourceViewer
Presentation context Hosts the editor in the workbench ITextEditor

DLTK is a set of extensible frameworks for building IDEs for dynamically-typed languages. It models projects, code folders and libraries, source modules, types, functions, variables, and package and import declarations.

Type inference is what makes DLTK effective. Without inferred types you cannot build good content assist, go-to-declaration, reference search, code analysis or refactoring for a dynamic language.

On top of editing, DLTK provides:

  • Launch and debug: interpreter configuration, a full Eclipse debugger, remote launch and debug over the DBGp protocol.
  • Interactive console: a common console protocol, including remote consoles.
  • Views: Script Explorer, Outline, Call Hierarchy, Type Hierarchy, Quick Outline and Quick Hierarchy.
  • More UI: project properties, wizards, preference pages and search UI.

Three ways to contribute to DLTK

A custom language editor feeds language-specific behaviour into DLTK through three channels; this tutorial focuses on the first.

  1. Language toolkit (non-UI): parsers, checkers, selection and completion engines.
  2. UI language toolkit: label providers and other presentation pieces.
  3. Other extension points: documentation providers, view filters and similar.

Start with the language toolkit and its parsers. UI pieces can be added gradually once the model builds correctly.

Step 1: Create the core plugin

Create a plug-in project (for example com.bml.language.core) and add three dependencies in MANIFEST.MF:

Plug-in Provides
org.eclipse.dltk.core The DLTK core
org.eclipse.core.filesystem Eclipse file system
org.eclipse.core.resources Eclipse resources and workspace

Step 2: Define the project nature

DLTK treats a project as a script project when it carries your nature, so define com.bml.core.nature by extending ScriptNature, which already handles nature management.

Register it in plugin.xml:

<extension id="nature" point="org.eclipse.core.resources.natures">
    <runtime>
        <run class="com.bml.language.core.BMLNature"/>
    </runtime>
</extension>
Enter fullscreen mode Exit fullscreen mode

The nature class:

package com.bml.language.core;

import org.eclipse.dltk.core.ScriptNature;

public class BMLNature extends ScriptNature {
    public static final String NATURE_ID = BMLPlugin.PLUGIN_ID + ".nature";
}
Enter fullscreen mode Exit fullscreen mode

You can also attach an incremental builder to the nature at this point.

Step 3: Create the language toolkit

The language toolkit is the main bridge between DLTK and your language: it supplies most non-UI language behaviour and is bound to your nature.

<extension point="org.eclipse.dltk.core.language">
    <language class="com.bml.language.core.BMLLanguageToolKit"
              nature="com.bml.core.nature"/>
</extension>
Enter fullscreen mode Exit fullscreen mode

BMLLanguageToolKit implements IDLTKLanguageToolkit. Extending AbstractLanguageToolkit gives you a basic implementation to start from.

The toolkit must provide two things:

  • A source element parser, which builds the content of a source module so DLTK views such as the Script Explorer can show it (Steps 4 and 5).
  • Source module validation, which decides whether a file belongs to your language. A file-extension check is often enough; stricter versions can inspect file headers.

Step 4: Build the custom source parser

The source parser turns raw BML text into a DLTK AST rooted at a ModuleDeclaration; everything else in the editor reads that tree.

Register it per nature:

<extension point="org.eclipse.dltk.core.sourceParsers">
    <parser class="com.bml.language.internal.parsers.BMLSourceParser"
            nature="com.bml.core.nature"
            priority="0"/>
</extension>
Enter fullscreen mode Exit fullscreen mode

BMLSourceParser implements ISourceParser. It works in four stages:

  1. Tokenize into a script. BMLSourceParser hands the file buffer to SimpleBMLParser, which produces a BMLScript. A CodeScanner reads the buffer to EOF; the parser keeps adding BMLCommands until EOF, and each command gathers BMLWords until end of line.
  2. Form expressions. BMLSourceParser walks the script and checks each word for substitutions. A quoted substitution becomes a BMLBlockExpression; a plain command (add or delete operators) becomes a BMLExecuteExpression; anything else is a SimpleReference.
  3. Group into statements. The new expressions are added to a BMLStatement.
  4. Attach to the module. The statements are added to the BMLModuleDeclaration, which is the root ASTNode.

The scanner entry point:

public static BMLScript parse(String content) throws ParseException {
    CodeScanner scanner = new CodeScanner(content);
    BMLScript script = parse(scanner, false);
    return script;
}
Enter fullscreen mode Exit fullscreen mode

The module declaration delegates rebuilding to an AST builder:

import org.eclipse.dltk.ast.declarations.ModuleDeclaration;
import com.bml.language.internal.parsers.BMLASTBuilder;

public class BMLModuleDeclaration extends ModuleDeclaration {

    public BMLModuleDeclaration(int sourceLength) {
        super(sourceLength, true);
    }

    protected void doRebuild() {
        BMLASTBuilder.buildAST(this, getTypeList(), getFunctionList(), getVariablesList());
    }

    public void rebuildMethods() {
        BMLASTBuilder.rebuildMethods(this);
    }
}
Enter fullscreen mode Exit fullscreen mode

Step 5: Build the model with a source element parser

The source element parser runs when the editor opens a file: it fetches your source parser, gets the AST, and reports types, methods, fields and package declarations to DLTK's model.

<extension point="org.eclipse.dltk.core.sourceElementParsers">
    <parser class="com.bml.language.internal.parsers.BMLSourceElementParser"
            nature="com.bml.core.nature"
            priority="0"/>
</extension>
Enter fullscreen mode Exit fullscreen mode

DLTK passes an ISourceElementRequestor to the parser. It works like a visitor: the parser calls its methods to declare each model element.

public class BMLSourceElementParser implements ISourceElementParser {

    private ISourceElementRequestor fRequestor;

    public void setRequestor(ISourceElementRequestor requestor) {
        this.fRequestor = requestor;
    }

    public ModuleDeclaration parseSourceModule(char[] contents,
            ISourceModuleInfo astCache, char[] filename) {

        // 1. Look up the source parser registered for this nature
        ISourceParser sourceParser = null;
        try {
            sourceParser = (ISourceParser) DLTKLanguageManager
                    .getSourceParser(BMLNature.NATURE_ID);
        } catch (CoreException e) {
            if (DLTKCore.DEBUG) {
                e.printStackTrace();
            }
            return null;
        }

        // 2. Parse to an AST
        ModuleDeclaration moduleDeclaration = sourceParser.parse(null, contents, null);
        moduleDeclaration.disableRebuild();
        List statements = moduleDeclaration.getStatements();

        // 3. Report model elements to DLTK
        try {
            fRequestor.enterModule();
            namespacesLevel.push("::");
            buildModel(statements, TYPE_MODULE, "");
            fRequestor.exitModule(contents.length);
        } catch (Exception e) {
            if (DLTKCore.DEBUG_PARSER) {
                e.printStackTrace();
            }
        }
        return moduleDeclaration;
    }
}
Enter fullscreen mode Exit fullscreen mode

buildModel walks the statements and calls the requestor's enter and exit methods (enterType, enterMethod, enterField and so on) for each declaration it finds.

Step 6: Add code completion

Completion reuses the same source parser: an assist parser finds the node under the cursor, throws CompletionNodeFound, and the completion engine catches it and reports proposals.

<extension point="org.eclipse.dltk.core.completionEngine">
    <completionEngine class="com.bml.language.core.codeassist.BMLCompletionEngine"
                      nature="com.bml.core.nature"/>
</extension>
Enter fullscreen mode Exit fullscreen mode

The assist parser. BMLAssistParser implements IAssistParser and gets the source parser from DLTKLanguageManager, exactly as in Step 5. BMLCompletionParser extends it.

When the cursor is not inside a known element, handleNotInElement adds an empty statement at the cursor and parses the block around it. Once proposals are found, it wraps them in a CompletionOnKeywordOrFunction node and throws CompletionNodeFound.

public class BMLCompletionParser extends BMLAssistParser {

    /** Called when an element could not be found. */
    @Override
    public void handleNotInElement(ASTNode node, int position) {
        if (node instanceof ModuleDeclaration) {
            ModuleDeclaration unit = (ModuleDeclaration) node;
            List exprs = new ArrayList();
            exprs.add(new SimpleReference(position, position, ""));
            BMLEmptyCompleteStatement statement = new BMLEmptyCompleteStatement(exprs);
            unit.addStatement(statement);
            this.parseBlockStatements(statement, unit, position);
        }
    }

    public void parseBlockStatements(ASTNode node, ASTNode inNode, int position) {
        // when completion proposals are found:
        ASTNode nde = new CompletionOnKeywordOrFunction(
                completionToken, completionNode, node, keywords);
        throw new CompletionNodeFound(nde, ((ModuleDeclaration) inNode).scope, isMethod);
    }

    private static class BMLEmptyCompleteStatement extends BMLStatement {
        public BMLEmptyCompleteStatement(List expressions) {
            super(expressions);
        }
    }
}
Enter fullscreen mode Exit fullscreen mode

The terms:

Name Meaning Example: user types Pack + Ctrl+Space
node The AST node being completed the empty statement
completionNode The expression inside that node the Pack reference
completionToken The text typed so far Pack
keywords The candidate proposals Package

The completion engine. BMLCompletionEngine extends ScriptCompletionEngine. It parses the module, builds the type scope, runs the assist parser at the cursor, and catches CompletionNodeFound. It always reports a CompletionContext, even when nothing matched.

public class BMLCompletionEngine extends ScriptCompletionEngine {

    private BMLCompletionParser parser = new BMLCompletionParser();

    public void complete(ISourceModule sourceModule, int completionPosition, int pos) {
        this.requestor.beginReporting();
        boolean contextAccepted = false;
        try {
            this.fileName = sourceModule.getFileName();
            this.actualCompletionPosition = completionPosition;
            this.offset = pos;
            ModuleDeclaration parsedUnit = (ModuleDeclaration) this.parser.parse(sourceModule);
            if (parsedUnit != null) {
                try {
                    this.lookupEnvironment.buildTypeScope(parsedUnit, null);
                    if ((this.unitScope = parsedUnit.scope) != null) {
                        this.source = sourceModule.getSourceContents().toCharArray();
                        parseBlockStatements(parsedUnit, this.actualCompletionPosition);
                    }
                } catch (CompletionNodeFound e) {
                    if (e.astNode != null) {
                        contextAccepted = complete(e.astNode,
                                this.parser.getAssistNodeParent(), e.scope,
                                e.insideTypeAnnotation);
                    }
                }
            }
            if (this.noProposal && this.problem != null) {
                if (!contextAccepted) {
                    contextAccepted = true;
                    CompletionContext context = new CompletionContext();
                    context.setOffset(completionPosition);
                    context.setTokenKind(CompletionContext.TOKEN_KIND_UNKNOWN);
                    this.requestor.acceptContext(context);
                }
                this.requestor.completionFailure(this.problem);
            }
        } finally {
            if (!contextAccepted) {
                CompletionContext context = new CompletionContext();
                context.setTokenKind(CompletionContext.TOKEN_KIND_UNKNOWN);
                context.setOffset(completionPosition);
                this.requestor.acceptContext(context);
            }
            this.requestor.endReporting();
        }
    }

    private boolean complete(ASTNode astNode, ASTNode astNodeParent,
            Scope scope, boolean insideTypeAnnotation) {
        // Sort the proposals and append them to CompletionProposals,
        // which DLTK shows in the completion pop-up.
        // Return true once the completion context has been reported.
    }
}
Enter fullscreen mode Exit fullscreen mode

How the parser is invoked at runtime

Your source parser is called from several places, often on background threads, so it must be fast, thread-safe and free of UI calls.

WHO TRIGGERS A PARSE          DLTK EXTENSION YOU REGISTER      YOUR PARSER

Editor opens a file     --+
  ScriptEditor.doSetInput |
Reconciler (on edits)   --+
                          +-->  BMLSourceElementParser  --+
Indexer (background)    --+     reports model elements    |
                          |                               +-->  BMLSourceParser
Builder (auto-build)    --+                               |     via DLTKLanguageManager
                                                          |     returns ModuleDeclaration
Content assist          ----->  BMLCompletionEngine     --+
  Ctrl+Space                    via BMLCompletionParser
Enter fullscreen mode Exit fullscreen mode

All paths end in DLTKLanguageManager.getSourceParser(NATURE_ID), so registering the parser once in Step 4 serves every feature.

  • Editor and reconciler: opening a file and every edit trigger a reparse to refresh the outline, folding and problem markers.
  • Indexer: a separate thread parses changed files and fills the search index through SourceIndexerRequestor.
  • Builder: with auto-build on, Eclipse runs builders through the Jobs API.
  • Content assist: the completion engine parses on demand at the cursor (Step 6).

If you keep a custom model beside DLTK's, build it on demand and cache it rather than rebuilding it on every parse. DLTK developers suggested ISourceModuleInfoCache for this in a 2008 forum thread.

Next steps and references

With the parser, model and completion in place, the natural next steps are a UI language toolkit (label providers, syntax colouring), search via MatchLocatorParser, and validators for problem markers.

Top comments (0)