Building a Custom Language Editor Leveraging Eclipse DLTK

Rupal H Patel — Sat, 03 Oct 2026 02:46:04 +0000

Rupal Patel and Kaniska Mandal · written 2008-2009, revised 2026

Preface (2026)

This guide was written in 2008-2009 for the Eclipse wiki while we built a DLTK editor for a business modeling language. It was never published there; this is the first full version, lightly edited.

It targets the DLTK 1.0-era APIs. It is most useful if you maintain an existing DLTK-based editor (Eclipse PHP Development Tools still depends on DLTK) or want to see how IDE language support is structured. For a new language today, write a Language Server Protocol server instead; the same pipeline of parser, AST, structure model and completion still applies.

Code samples are excerpts. Helper methods and fields such as buildModel and completionToken are left out for brevity.

Introduction

This tutorial builds a full language editor for a custom language on top of the Eclipse Dynamic Languages Toolkit (DLTK): a nature, a language toolkit, a source parser, a source element parser and a completion engine.

By Rupal Patel and Kaniska Mandal. An earlier version was shared as Creating A Language Editor Using DLTK.

The problem. Tool vendors need a free-flow editor so business analysts and tool developers can write in tool-specific languages. That editor needs the full set of IDE features: syntax coloring, auto-completion, content assist, code folding, selection, problem markers, an outline view and an AST parser.

The example language. Throughout, we build an editor for a hypothetical Business Modeling Language (BML). A BML file looks like this:

Package com.bml.samples

BizModel {
    Declare {
        Customer bob = Customer.createCustomer(...);
        Account savings = Account.createAccount(...);
    }
    Get {
        String name = bob.getName();
    }
    Put {
        bob.setAccount(savings);
    }
}

Versions. The code targets the DLTK 1.0-era APIs (2008-2009). Later DLTK releases changed some signatures, notably the sourceParsers extension point, which now takes a parser factory.

Background: Eclipse text editing and DLTK

Eclipse's text editing framework follows the same layered MVC design as the rest of the platform, and DLTK plugs language intelligence into it.

The four layers of the text editing framework are:

Layer	Role	Eclipse type
Model	Document content	core (`IDocument`)
View	Rendering	SWT `StyledText`
Controller	Wires model to view	JFace `ISourceViewer`
Presentation context	Hosts the editor in the workbench	`ITextEditor`

DLTK is a set of extensible frameworks for building IDEs for dynamically-typed languages. It models projects, code folders and libraries, source modules, types, functions, variables, and package and import declarations.

Type inference is what makes DLTK effective. Without inferred types you cannot build good content assist, go-to-declaration, reference search, code analysis or refactoring for a dynamic language.

On top of editing, DLTK provides:

Launch and debug: interpreter configuration, a full Eclipse debugger, remote launch and debug over the DBGp protocol.
Interactive console: a common console protocol, including remote consoles.
Views: Script Explorer, Outline, Call Hierarchy, Type Hierarchy, Quick Outline and Quick Hierarchy.
More UI: project properties, wizards, preference pages and search UI.

Three ways to contribute to DLTK

A custom language editor feeds language-specific behaviour into DLTK through three channels; this tutorial focuses on the first.

Language toolkit (non-UI): parsers, checkers, selection and completion engines.
UI language toolkit: label providers and other presentation pieces.
Other extension points: documentation providers, view filters and similar.

Start with the language toolkit and its parsers. UI pieces can be added gradually once the model builds correctly.

Step 1: Create the core plugin

Create a plug-in project (for example com.bml.language.core) and add three dependencies in MANIFEST.MF:

Plug-in	Provides
`org.eclipse.dltk.core`	The DLTK core
`org.eclipse.core.filesystem`	Eclipse file system
`org.eclipse.core.resources`	Eclipse resources and workspace

Step 2: Define the project nature

DLTK treats a project as a script project when it carries your nature, so define com.bml.core.nature by extending ScriptNature, which already handles nature management.

<extension id="nature" point="org.eclipse.core.resources.natures">
    <runtime>
        <run class="com.bml.language.core.BMLNature"/>
    </runtime>
</extension>

The nature class:

package com.bml.language.core;

import org.eclipse.dltk.core.ScriptNature;

public class BMLNature extends ScriptNature {
    public static final String NATURE_ID = BMLPlugin.PLUGIN_ID + ".nature";
}

You can also attach an incremental builder to the nature at this point.

Step 3: Create the language toolkit

The language toolkit is the main bridge between DLTK and your language: it supplies most non-UI language behaviour and is bound to your nature.

<extension point="org.eclipse.dltk.core.language">
    <language class="com.bml.language.core.BMLLanguageToolKit"
              nature="com.bml.core.nature"/>
</extension>

BMLLanguageToolKit implements IDLTKLanguageToolkit. Extending AbstractLanguageToolkit gives you a basic implementation to start from.

The toolkit must provide two things:

A source element parser, which builds the content of a source module so DLTK views such as the Script Explorer can show it (Steps 4 and 5).
Source module validation, which decides whether a file belongs to your language. A file-extension check is often enough; stricter versions can inspect file headers.

Step 4: Build the custom source parser

The source parser turns raw BML text into a DLTK AST rooted at a ModuleDeclaration; everything else in the editor reads that tree.

<extension point="org.eclipse.dltk.core.sourceParsers">
    <parser class="com.bml.language.internal.parsers.BMLSourceParser"
            nature="com.bml.core.nature"
            priority="0"/>
</extension>

BMLSourceParser implements ISourceParser. It works in four stages:

Tokenize into a script. BMLSourceParser hands the file buffer to SimpleBMLParser, which produces a BMLScript. A CodeScanner reads the buffer to EOF; the parser keeps adding BMLCommands until EOF, and each command gathers BMLWords until end of line.
Form expressions. BMLSourceParser walks the script and checks each word for substitutions. A quoted substitution becomes a BMLBlockExpression; a plain command (add or delete operators) becomes a BMLExecuteExpression; anything else is a SimpleReference.
Group into statements. The new expressions are added to a BMLStatement.
Attach to the module. The statements are added to the BMLModuleDeclaration, which is the root ASTNode.

The scanner entry point:

public static BMLScript parse(String content) throws ParseException {
    CodeScanner scanner = new CodeScanner(content);
    BMLScript script = parse(scanner, false);
    return script;
}

The module declaration delegates rebuilding to an AST builder:

import org.eclipse.dltk.ast.declarations.ModuleDeclaration;
import com.bml.language.internal.parsers.BMLASTBuilder;

public class BMLModuleDeclaration extends ModuleDeclaration {

    public BMLModuleDeclaration(int sourceLength) {
        super(sourceLength, true);
    }

    protected void doRebuild() {
        BMLASTBuilder.buildAST(this, getTypeList(), getFunctionList(), getVariablesList());
    }

    public void rebuildMethods() {
        BMLASTBuilder.rebuildMethods(this);
    }
}

Step 5: Build the model with a source element parser

The source element parser runs when the editor opens a file: it fetches your source parser, gets the AST, and reports types, methods, fields and package declarations to DLTK's model.

<extension point="org.eclipse.dltk.core.sourceElementParsers">
    <parser class="com.bml.language.internal.parsers.BMLSourceElementParser"
            nature="com.bml.core.nature"
            priority="0"/>
</extension>

DLTK passes an ISourceElementRequestor to the parser. It works like a visitor: the parser calls its methods to declare each model element.

public class BMLSourceElementParser implements ISourceElementParser {

    private ISourceElementRequestor fRequestor;

    public void setRequestor(ISourceElementRequestor requestor) {
        this.fRequestor = requestor;
    }

    public ModuleDeclaration parseSourceModule(char[] contents,
            ISourceModuleInfo astCache, char[] filename) {

        // 1. Look up the source parser registered for this nature
        ISourceParser sourceParser = null;
        try {
            sourceParser = (ISourceParser) DLTKLanguageManager
                    .getSourceParser(BMLNature.NATURE_ID);
        } catch (CoreException e) {
            if (DLTKCore.DEBUG) {
                e.printStackTrace();
            }
            return null;
        }

        // 2. Parse to an AST
        ModuleDeclaration moduleDeclaration = sourceParser.parse(null, contents, null);
        moduleDeclaration.disableRebuild();
        List statements = moduleDeclaration.getStatements();

        // 3. Report model elements to DLTK
        try {
            fRequestor.enterModule();
            namespacesLevel.push("::");
            buildModel(statements, TYPE_MODULE, "");
            fRequestor.exitModule(contents.length);
        } catch (Exception e) {
            if (DLTKCore.DEBUG_PARSER) {
                e.printStackTrace();
            }
        }
        return moduleDeclaration;
    }
}

buildModel walks the statements and calls the requestor's enter and exit methods (enterType, enterMethod, enterField and so on) for each declaration it finds.

Step 6: Add code completion

Completion reuses the same source parser: an assist parser finds the node under the cursor, throws CompletionNodeFound, and the completion engine catches it and reports proposals.

<extension point="org.eclipse.dltk.core.completionEngine">
    <completionEngine class="com.bml.language.core.codeassist.BMLCompletionEngine"
                      nature="com.bml.core.nature"/>
</extension>

The assist parser. BMLAssistParser implements IAssistParser and gets the source parser from DLTKLanguageManager, exactly as in Step 5. BMLCompletionParser extends it.

When the cursor is not inside a known element, handleNotInElement adds an empty statement at the cursor and parses the block around it. Once proposals are found, it wraps them in a CompletionOnKeywordOrFunction node and throws CompletionNodeFound.

public class BMLCompletionParser extends BMLAssistParser {

    /** Called when an element could not be found. */
    @Override
    public void handleNotInElement(ASTNode node, int position) {
        if (node instanceof ModuleDeclaration) {
            ModuleDeclaration unit = (ModuleDeclaration) node;
            List exprs = new ArrayList();
            exprs.add(new SimpleReference(position, position, ""));
            BMLEmptyCompleteStatement statement = new BMLEmptyCompleteStatement(exprs);
            unit.addStatement(statement);
            this.parseBlockStatements(statement, unit, position);
        }
    }

    public void parseBlockStatements(ASTNode node, ASTNode inNode, int position) {
        // when completion proposals are found:
        ASTNode nde = new CompletionOnKeywordOrFunction(
                completionToken, completionNode, node, keywords);
        throw new CompletionNodeFound(nde, ((ModuleDeclaration) inNode).scope, isMethod);
    }

    private static class BMLEmptyCompleteStatement extends BMLStatement {
        public BMLEmptyCompleteStatement(List expressions) {
            super(expressions);
        }
    }
}

The terms:

Name	Meaning	Example: user types `Pack` + Ctrl+Space
`node`	The AST node being completed	the empty statement
`completionNode`	The expression inside that node	the `Pack` reference
`completionToken`	The text typed so far	`Pack`
`keywords`	The candidate proposals	`Package`

The completion engine. BMLCompletionEngine extends ScriptCompletionEngine. It parses the module, builds the type scope, runs the assist parser at the cursor, and catches CompletionNodeFound. It always reports a CompletionContext, even when nothing matched.

public class BMLCompletionEngine extends ScriptCompletionEngine {

    private BMLCompletionParser parser = new BMLCompletionParser();

    public void complete(ISourceModule sourceModule, int completionPosition, int pos) {
        this.requestor.beginReporting();
        boolean contextAccepted = false;
        try {
            this.fileName = sourceModule.getFileName();
            this.actualCompletionPosition = completionPosition;
            this.offset = pos;
            ModuleDeclaration parsedUnit = (ModuleDeclaration) this.parser.parse(sourceModule);
            if (parsedUnit != null) {
                try {
                    this.lookupEnvironment.buildTypeScope(parsedUnit, null);
                    if ((this.unitScope = parsedUnit.scope) != null) {
                        this.source = sourceModule.getSourceContents().toCharArray();
                        parseBlockStatements(parsedUnit, this.actualCompletionPosition);
                    }
                } catch (CompletionNodeFound e) {
                    if (e.astNode != null) {
                        contextAccepted = complete(e.astNode,
                                this.parser.getAssistNodeParent(), e.scope,
                                e.insideTypeAnnotation);
                    }
                }
            }
            if (this.noProposal && this.problem != null) {
                if (!contextAccepted) {
                    contextAccepted = true;
                    CompletionContext context = new CompletionContext();
                    context.setOffset(completionPosition);
                    context.setTokenKind(CompletionContext.TOKEN_KIND_UNKNOWN);
                    this.requestor.acceptContext(context);
                }
                this.requestor.completionFailure(this.problem);
            }
        } finally {
            if (!contextAccepted) {
                CompletionContext context = new CompletionContext();
                context.setTokenKind(CompletionContext.TOKEN_KIND_UNKNOWN);
                context.setOffset(completionPosition);
                this.requestor.acceptContext(context);
            }
            this.requestor.endReporting();
        }
    }

    private boolean complete(ASTNode astNode, ASTNode astNodeParent,
            Scope scope, boolean insideTypeAnnotation) {
        // Sort the proposals and append them to CompletionProposals,
        // which DLTK shows in the completion pop-up.
        // Return true once the completion context has been reported.
    }
}

How the parser is invoked at runtime

Your source parser is called from several places, often on background threads, so it must be fast, thread-safe and free of UI calls.

WHO TRIGGERS A PARSE          DLTK EXTENSION YOU REGISTER      YOUR PARSER

Editor opens a file     --+
  ScriptEditor.doSetInput |
Reconciler (on edits)   --+
                          +-->  BMLSourceElementParser  --+
Indexer (background)    --+     reports model elements    |
                          |                               +-->  BMLSourceParser
Builder (auto-build)    --+                               |     via DLTKLanguageManager
                                                          |     returns ModuleDeclaration
Content assist          ----->  BMLCompletionEngine     --+
  Ctrl+Space                    via BMLCompletionParser

All paths end in DLTKLanguageManager.getSourceParser(NATURE_ID), so registering the parser once in Step 4 serves every feature.

Editor and reconciler: opening a file and every edit trigger a reparse to refresh the outline, folding and problem markers.
Indexer: a separate thread parses changed files and fills the search index through SourceIndexerRequestor.
Builder: with auto-build on, Eclipse runs builders through the Jobs API.
Content assist: the completion engine parses on demand at the cursor (Step 6).

If you keep a custom model beside DLTK's, build it on demand and cache it rather than rebuilding it on every parse. DLTK developers suggested ISourceModuleInfoCache for this in a 2008 forum thread.

Next steps and references

With the parser, model and completion in place, the natural next steps are a UI language toolkit (label providers, syntax colouring), search via MatchLocatorParser, and validators for problem markers.

Creating A Language Editor Using DLTK: an earlier version of this article, Rupal Patel and Kaniska Mandal
DLTK IDE Guide and Step 2: Towards an Editor: the official Python example
DLTK Core Architecture: model building, search and type inference
DLTK Search Architecture: indexing and match locators
Using DLTK plugin framework for implementing language editor: the 2007 forum thread that started this work

DEV Community: Rupal H Patel