0. Prologue
Given the sheer size of Ghidra, the most effective way to understand its internals is to use a debugger such as Eclipse. To set up your development environment, run the following command (refer to the official Ghidra GitHub repository for details):
gradle prepdev eclipse buildNatives
I did it otherwise and it was painful to switch between all the directories and all the files. Using the Eclipse debbuger changed my life.
Another tool which changed my life is ripgrep.
rg 'extends PluginTool' -l
This command shows all the files which contain 'extends PluginTool' from the current directory. However, ripgrep offers far more features.
ripgrep is a command-line tool provided by Ghidra for performing regular expression searches within binary files.
To search for a simple string pattern (e.g., "hello") in a binary file: rg hello example.bin
To search for a hexadecimal pattern (e.g., 48 65 6c 6c 6f which corresponds to "Hello" in ASCII): rg '\x48\x65\x6c\x6c\x6f' example.bin
To search for a more complex pattern, such as a sequence of bytes that match a specific format:
rg '[\x00-\xFF]{4}' example.bin = Search for any 4-byte.
To output the matches in hex dump format: rg --hexdump '\x48\x65\x6c\x6c\x6f' example.bin
You have other Options like the following
--binary: Treat the pattern as a binary string.
--hexdump: Output matches in hex dump format.
--offsets: Show byte offsets of matches.
--context: Show context around matches.
1. Initialization
At the beginning of everything is the class GhidraRun.
This class, which can be found in ./Ghidra/Features/Base/src/main/java/ghidra/, performs several initializations (see below):
public void launch(GhidraApplicationLayout layout, String[] args) {
Runnable mainTask = () -> {
GhidraApplicationConfiguration configuration = new GhidraApplicationConfiguration();
Application.initializeApplication(layout, configuration);
log = LogManager.getLogger(GhidraRun.class);
log.info("User " + SystemUtilities.getUserName() + " started Ghidra.");
log.info("User settings directory: " + Application.getUserSettingsDirectory());
log.info("User temp directory: " + Application.getUserTempDirectory());
log.info("User cache directory: " + Application.getUserCacheDirectory());
writeLastRun();
initializeTooltips();
updateSplashScreenStatusMessage("Populating Ghidra help...");
GhidraHelpService.install();
ExtensionUtils.initializeExtensions();
updateSplashScreenStatusMessage("Checking for previous project...");
SystemUtilities.runSwingLater(() -> {
LaunchArguments launchArgs = processArguments(args);
openProject(launchArgs);
log.info("Ghidra startup complete (" + GhidraLauncher.getMillisecondsFromLaunch() +
" ms)");
checkForMissingNativeComponents();
});
};
// Start main thread in GhidraThreadGroup
Thread mainThread = new Thread(new GhidraThreadGroup(), mainTask, "Ghidra");
mainThread.start();
}
before opening a project (with the openProject method). If there is no project given in arguments, it will open the last opened project if still existing otherwise none.
The openProject method will instantiate 3 objects
- a DefaultProjectManager (DefaultProjectManager implements ProjectManager)
- a FrontEndTool which receives the DefaultProjectManager as a parameter of the constructor ( FrontEndTool extends PluginTool implements OptionsChangeListener)
- a ProjectLocator which is a base class
then will call another
- openProject method which will show a window and call
- doOpenProject which will get a Project from the ProjectManager ( Project activeProject = pm.openProject(projectLocator, true, false);) and then will call the
- openDomainFileInTool method which will get a ProjectData from the Project object ( ProjectData projectData = project.getProjectData(); ) which will give a DomainFile (DomainFile domainFile = projectData.getFile(domainFilePath); ) used to launch the default tool on this file ( PluginTool tool = toolServices.launchDefaultTool(List.of(domainFile)); ) using the ToolServices obtained from the project (see below)
private void openDomainFileInTool(Project project, String domainFilePath) {
// Ensure path starts with /
if (!domainFilePath.startsWith("/")) {
domainFilePath = "/" + domainFilePath;
}
ProjectData projectData = project.getProjectData();
DomainFile domainFile = projectData.getFile(domainFilePath);
if (domainFile == null) {
Msg.showError(GhidraRun.class, null, "File Not Found",
"Could not find file in project: " + domainFilePath);
return;
}
log.info("Opening file from command line: " + domainFilePath);
ToolServices toolServices = project.getToolServices();
PluginTool tool = toolServices.launchDefaultTool(List.of(domainFile));
if (tool == null) {
Msg.showError(GhidraRun.class, null, "Tool Launch Failed",
"Failed to launch tool for: " + domainFile.getName());
}
}
i.e the CodeBrowser tool
2. The CodeBrowser tool
As every tool it can be exported. Let's see what we find for this tool
For example the CodeBrowser tool accepts the following data types
<SUPPORTED_DATA_TYPE CLASS_NAME="ghidra.program.model.listing.Program" />
<SUPPORTED_DATA_TYPE CLASS_NAME="ghidra.program.model.listing.DataTypeArchive" />
it contains the following packages
<PACKAGE NAME="BSim" />
<PACKAGE NAME="Ghidra Core">
and the following plugins
<INCLUDE CLASS="ghidra.app.plugin.core.interpreter.InterpreterPanelPlugin" />
<PLUGIN_STATE CLASS="ghidra.app.plugin.core.navigation.GoToAddressLabelPlugin">
<PLUGIN_STATE CLASS="ghidra.app.plugin.core.functionwindow.FunctionWindowPlugin">
<PLUGIN_STATE CLASS="ghidra.app.plugin.core.datapreview.DataTypePreviewPlugin">
<PLUGIN_STATE CLASS="ghidra.features.base.memsearch.gui.MemorySearchPlugin">
<PLUGIN_STATE CLASS="ghidra.app.plugin.core.overview.OverviewColorPlugin">
<PLUGIN_STATE CLASS="ghidra.app.plugin.core.symboltree.SymbolTreePlugin">
<PLUGIN_STATE CLASS="ghidra.app.plugin.core.calltree.CallTreePlugin">
<PLUGIN_STATE CLASS="functioncalls.plugin.FunctionCallGraphPlugin">
<PLUGIN_STATE CLASS="datagraph.DataGraphPlugin">
<PLUGIN_STATE CLASS="ghidra.app.plugin.core.symtable.SymbolTablePlugin">
<PLUGIN_STATE CLASS="ghidra.app.plugin.core.datamgr.DataTypeManagerPlugin">
<PLUGIN_STATE CLASS="ghidra.app.plugin.core.script.GhidraScriptMgrPlugin">
<PLUGIN_STATE CLASS="ghidra.app.plugin.core.bookmark.BookmarkPlugin">
<PLUGIN_STATE CLASS="ghidra.app.plugin.core.byteviewer.ByteViewerPlugin">
<PLUGIN_STATE CLASS="ghidra.features.codecompare.plugin.FunctionComparisonPlugin">
<PLUGIN_STATE CLASS="ghidra.app.plugin.core.functiongraph.FunctionGraphPlugin">
<PLUGIN_STATE CLASS="ghidra.app.plugin.core.graph.GraphDisplayBrokerPlugin">
<PLUGIN_STATE CLASS="ghidra.app.plugin.core.codebrowser.CodeBrowserPlugin">
All plugins are important but the most important one is the CodeBrowserPlugin
The CodeBrowserPlugin is the Primary Controller of the Ghidra User Interface. If you think of Ghidra as a "game engine" for reverse engineering, the CodeBrowserPlugin is the "Level Controller" that manages everything you see and interact with when you are actually looking at a binary.
It is the "glue" that connects the Logic (the Program object) with all the UI (the Listing, Decompiler, and Function Graph windows).
Here is a breakdown of its specific responsibilities:
1. Orchestrating the "View" (The Orchestrator)
The CodeBrowser is not just one window; it is a collection of specialized windows (the "View"). The CodeBrowserPlugin is responsible for making sure all these windows stay in sync.
The Synchronization Problem: If you click on a specific instruction in the Listing View, the Decompiler View must immediately jump to the corresponding C code. If you click a node in the Function Graph, the Listing View must move to that address.
The Solution: The CodeBrowserPlugin manages the Context and the Selection. It listens to "Selection Events" from one window and broadcasts them to all other windows.
2. Managing the "Program" Lifecycle (The Controller)
The CodeBrowserPlugin is the lifecycle manager for the Program object within the UI.
Loading/Unloading: When you open a new program, the CodeBrowserPlugin handles the logic of "unloading" the old program (cleaning up the UI, clearing the memory) and "loading" the new program (populating the windows with the new address space).
The Active Program: It maintains a reference to the currentProgram. Every action you take (renaming a variable, adding a comment) is sent through the plugin to the currentProgram.
3. Handling User Interactions (The Event Handler)
The CodeBrowserPlugin is the primary listener for user input (Mouse and Keyboard) within the workspace.
Mouse Events: It intercepts clicks to handle things like "Double-click to follow jump," "Right-click for context menu," and "Click-and-drag to select a range of bytes."
Keyboard Events: It manages keyboard shortcuts (e.g., pressing 'c' to create a comment, or 'r' to rename a symbol) and routes them to the active window.
The Context Menu: It is responsible for generating the "Right-Click Menu." It looks at what you have selected (an instruction? a data byte? a function?) and decides which menu options are relevant to that specific selection.
4. Tool Integration (The Bridge)
The CodeBrowserPlugin acts as the bridge between the Core Engine and the User Interface.
It provides the Scripting Engine access to the UI. When you run a Python script that says currentProgram.getFunctionAt(addr), the plugin is the entity that ensures the "current" program is correctly passed to the script environment.
It manages the Tool object. It keeps track of which windows are currently "docked" in the CodeBrowser and ensures that the Tool's state is saved when the project is closed.
When Ghidra is launched, the CodeBrowserPlugin restores the "Active" program you were currently working on, and it "re-awakens" any other programs that were open in the background (the "Disconnected" providers). This is done by the readDataState method (see below)
public void readDataState(SaveState saveState) {
ProgramManager programManagerService = tool.getService(ProgramManager.class);
if (connectedProvider != null) {
connectedProvider.readDataState(saveState);
}
int numDisconnected = saveState.getInt("Num Disconnected", 0);
for (int i = 0; i < numDisconnected; i++) {
Element xmlElement = saveState.getXmlElement("Provider" + i);
SaveState providerSaveState = new SaveState(xmlElement);
String programPath = providerSaveState.getString("Program Path", "");
DomainFile file = tool.getProject().getProjectData().getFile(programPath);
if (file == null) {
continue;
}
Program program = programManagerService.openProgram(file);
if (program != null) {
CodeViewerProvider provider = createNewDisconnectedProvider();
provider.doSetProgram(program);
provider.readDataState(providerSaveState);
}
}
FieldSelection highlight = new FieldSelection();
highlight.load(saveState);
if (!highlight.isEmpty()) {
setConnectedProviderHighlight(highlight);
}
}
3. The Analysis process: Under the hood
The analysis process occurs when importing a binary into a project
When a first step is done, Ghidra shows the results.
when a project has been analyzed and saved in the database, a project is no longer analyzed unless you force the analysis
A. Importing a program into a Ghidra project
When clicking on the File > Import File menu or pressing the I key, it's the ImporterPlugin class which comes into play and more precisely its importFile method (see below)
public void importFile(DomainFolder folder, File file) {
System.out.println("ImporterPlugin importFile()");
if (handleSimpleDBUnpack(folder, file)) {
return;
}
FSRL fsrl = fsService().getLocalFSRL(file);
ProgramManager manager = tool.getService(ProgramManager.class);
ImporterUtilities.showImportDialog(tool, manager, fsrl, folder, null);
}
This importFile method calls the static ImporterUtilities showImportDialog method which calls the showImportSingleFileDialog method which instantiates an ImporterDialog object (see below)
public static void showImportSingleFileDialog(FSRL fsrl, DomainFolder destinationFolder,
String suggestedPath, PluginTool tool, ProgramManager programManager,
TaskMonitor monitor) {
System.out.println("ImporterUtilities showImportSingleFileDialog()");
try {
ByteProvider provider = fsService.getByteProvider(fsrl, true, monitor);
LoaderMap loaderMap = LoaderService.getAllSupportedLoadSpecs(provider, monitor);
SystemUtilities.runSwingLater(() -> {
ImporterDialog importerDialog = new ImporterDialog(tool, programManager, loaderMap,
provider, suggestedPath);
if (destinationFolder != null) {
importerDialog.setDestinationFolder(destinationFolder);
}
tool.showDialog(importerDialog);
});
}
catch (IOException ioe) {
Msg.showError(ImporterUtilities.class, tool.getActiveWindow(), "Error Importing File",
"Error when importing file " + fsrl, ioe);
}
catch (CancelledException e) {
Msg.info(ImporterUtilities.class, "Import single file " + fsrl + " cancelled");
}
}
As a result of all of this, the following dialog appears
At this stage, the system has already analyzed that the program was an ELF program and that the language was the x86:LE:64 and the compiler was default:gcc
How so ?
In fact the LoaderService class has a method getAllLoaders which lists all the available loaders
private synchronized static Collection<Loader> getAllLoaders() {
List<Loader> loaders = new ArrayList<>(ClassSearcher.getInstances(Loader.class));
Collections.sort(loaders);
return loaders;
}
ClassSearcher.getInstances(Loader.class):
This line of code is used to retrieve all instances of classes that implement or extend the Loader interface in the runtime classPath of the Ghidra's application. When Ghidra is running, it has a specific classpath that includes all the necessary libraries and classes required for its operation. This classpath is determined by how Ghidra was launched and configured.
The ClassSearcher.getInstances(Loader.class) method searches within this runtime classpath to find all classes that implement or extend the Loader interface.
Most of the loaders can be found in the directory ./Ghidra/Features/Base/src/main/java/ghidra/app/util/opinion/ (see below)
Available Loaders:
Loader Class: ghidra.app.util.opinion.GzfLoader
Loader Class: ghidra.app.plugin.core.debug.utils.GztLoader
Loader Class: ghidra.app.util.opinion.GdtLoader
Loader Class: ghidra.app.util.opinion.DecompileDebugXmlLoader
Loader Class: sarif.SarifLoader
Loader Class: ghidra.app.util.opinion.XmlLoader
Loader Class: ghidra.app.util.opinion.TenetLoader
Loader Class: ghidra.app.util.opinion.TenetPlusPlusLoader
Loader Class: ghidra.app.util.opinion.DyldCacheExtractLoader
Loader Class: ghidra.app.util.opinion.MachoFileSetExtractLoader
Loader Class: ghidra.app.util.opinion.ApkLoader
Loader Class: ghidra.app.util.opinion.CoffLoader
Loader Class: ghidra.app.util.opinion.DyldCacheLoader
Loader Class: ghidra.app.util.opinion.DexLoader
Loader Class: ghidra.app.util.opinion.DbgLoader
Loader Class: ghidra.file.formats.dump.DumpFileLoader
Loader Class: ghidra.app.util.opinion.ElfLoader
Loader Class: ghidra.app.util.opinion.JavaLoader
Loader Class: ghidra.app.util.opinion.MSCoffLoader
Loader Class: ghidra.app.util.opinion.MachoLoader
Loader Class: ghidra.app.util.opinion.DefLoader
Loader Class: ghidra.app.util.opinion.NeLoader
Loader Class: ghidra.app.util.opinion.Omf51Loader
Loader Class: ghidra.app.util.opinion.ComLoader
Loader Class: ghidra.app.util.opinion.PeLoader
Loader Class: ghidra.app.util.opinion.PefLoader
Loader Class: ghidra.app.util.opinion.MapLoader
Loader Class: ghidra.app.util.opinion.OmfLoader
Loader Class: ghidra.app.util.opinion.SomLoader
Loader Class: ghidra.app.util.opinion.UnixAoutLoader
Loader Class: ghidra.app.util.opinion.MzLoader
Loader Class: ghidra.app.util.opinion.IntelHexLoader
Loader Class: ghidra.app.util.opinion.MotorolaHexLoader
Loader Class: ghidra.app.util.opinion.CDexLoader
Loader Class: ghidra.app.util.opinion.BinaryLoader
Identifying which loader should be used to load the binary is done by the LoaderService method getSupportedLoadSpecs (see below)
public static LoaderMap getSupportedLoadSpecs(ByteProvider provider,
Predicate<Loader> loaderFilter, TaskMonitor monitor) {
initializeLanguageService(monitor);
LoaderMap loaderMap = new LoaderMap();
List<Loader> fallback = new ArrayList<>();
for (Loader loader : getAllLoaders()) {
if (loaderFilter.test(loader)) {
if (!loader.isFallback()) {
tryLoadSpecs(loader, provider, loaderMap);
}
else {
fallback.add(loader);
}
}
}
// Only try fallback loaders if no other loaders matched (ignoring the BinaryLoader)
boolean matches = loaderMap.keySet()
.stream()
.map(Loader::getName)
.anyMatch(Predicate.not(BinaryLoader.BINARY_NAME::equals));
if (!matches) {
fallback.forEach(loader -> tryLoadSpecs(loader, provider, loaderMap));
}
return loaderMap;
}
There is always one loader which matches: the raw binary loader however, if the program has a known header, ghidra is going to identify the corresponding loader as the first loader to choose ( for example the ELF Loader).
Then it will identify the corresponding language , the compiler and the compiler options
This is done by the ProgramLoader object in methods like the following one
private LoadSpec getLoadSpec(ByteProvider p)
throws LanguageNotFoundException, LoadException {
LoaderMap loaderMap = LoaderService.getSupportedLoadSpecs(p, loaderFilter, monitor);
LoadSpecChooser loadSpecChooser =
languageId != null ? new LcsHintLoadSpecChooser(languageId, compilerSpecId)
: (compilerSpecId != null ? new CsHintLoadSpecChooser(compilerSpecId)
: LoadSpecChooser.CHOOSE_THE_FIRST_PREFERRED);
LoadSpec loadSpec = loadSpecChooser.choose(loaderMap);
if (loadSpec == null) {
String name = Objects.requireNonNullElse(p.getName(), "???");
Msg.info(ProgramLoader.class, "No load spec found for import file: " + name);
throw new LoadException("No load spec found");
}
return loadSpec;
}
c. Example: Loading a ELF program
The class to load an ELF program is ElfLoader and more specifically the ElfLoader load method
@Override
public void load(Program program, ImporterSettings settings)
throws IOException, CancelledException {
System.out.println("ElfLoader->load()");
try {
ElfHeader elf =
new ElfHeader(settings.provider(), msg -> settings.log().appendMsg(msg));
ElfProgramBuilder.loadElf(elf, program, settings.options(), settings.log(),
settings.monitor());
}
catch (ElfException e) {
throw new IOException(e.getMessage());
}
}
the ElfLoader load method calls the ElProgramBuilder loadElf (see below) which at the end calls the ElProgramBuilder load method.
And this is the ElProgramBuilder load method which does the main part of the loading process (see below)
static void loadElf(ElfHeader elf, Program program, List<Option> options, MessageLog log,
TaskMonitor monitor) throws IOException, CancelledException {
ElfProgramBuilder elfProgramBuilder = new ElfProgramBuilder(elf, program, options, log);
elfProgramBuilder.load(monitor);
}
protected void load(TaskMonitor monitor) throws IOException, CancelledException {
System.out.println("ElfProgramBuilder->load()");
monitor.setMessage("Completing ELF header parsing...");
monitor.setCancelEnabled(false);
elf.parse();
monitor.setCancelEnabled(true);
int id = program.startTransaction("Load ELF program");
boolean success = false;
try {
addProgramProperties(monitor);
setImageBase();
program.setExecutableFormat(ElfLoader.ELF_NAME);
ByteProvider byteProvider = elf.getByteProvider();
createFileBytes(byteProvider, monitor);
adjustSegmentAndSectionFileAllocations(byteProvider, monitor);
// process headers and define "section" within memory elfProgramBuilder
processProgramHeaders(monitor);
processSectionHeaders(monitor);
// resolve segment/sections and create program memory blocks
resolve(monitor);
if (elf.getSectionHeaderCount() == 0) {
// create/expand segments to their fullsize if no sections are defined
expandProgramHeaderBlocks(monitor);
}
if (memory.isEmpty()) {
// TODO: Does this really happen?
success = true;
return;
}
markupElfHeader(monitor);
markupProgramHeaders(monitor);
markupSectionHeaders(monitor);
monitor.setIndeterminate(true);
markupDynamicTable(monitor);
markupInterpreter(monitor);
monitor.setIndeterminate(false);
processStringTables(monitor);
processSymbolTables(monitor);
monitor.setIndeterminate(true);
elf.getLoadAdapter().processElf(this, monitor);
monitor.setIndeterminate(false);
processRelocations(monitor);
processEntryPoints(monitor);
processImports(monitor);
monitor.setIndeterminate(true);
monitor.setMessage("Processing PLT/GOT ...");
elf.getLoadAdapter().processGotPlt(this, monitor);
markupHashTable(monitor);
markupGnuHashTable(monitor);
markupGnuXHashTable(monitor);
processGNU(monitor);
adjustReadOnlyMemoryRegions(monitor);
markupElfInfoProducers(monitor);
success = true;
}
finally {
program.endTransaction(id, success);
}
}
At the end the program is loaded and its code is ready to be disassembled
d. The disassembly process
When a program is imported in a project, there is an auto analysis made by the AutoAnalysisManager class.
This auto analysis contains several steps including a disassembling of the binary
INFO -----------------------------------------------------
ASCII Strings 0.497 secs
Apply Data Archives 0.702 secs
Call Convention ID 0.548 secs
Call-Fixup Installer 0.006 secs
Create Address Tables 0.004 secs
Create Function 0.017 secs
DWARF 0.410 secs
Data Reference 0.008 secs
Decompiler Switch Analysis 0.879 secs
Demangler GNU 0.052 secs
Disassemble 0.017 secs
Disassemble Entry Points 0.130 secs
ELF Scalar Operand References 0.011 secs
Embedded Media 0.003 secs
External Entry References 0.000 secs
External Symbol Resolver 0.010 secs
Function ID 0.037 secs
Function Start Pre Search 0.017 secs
Function Start Search 0.068 secs
Function Start Search After Code 0.003 secs
Function Start Search After Data 0.001 secs
GCC Exception Handlers 0.083 secs
Non-Returning Functions - Discovered 0.003 secs
Non-Returning Functions - Known 0.005 secs
Reference 0.018 secs
Shared Return Calls 0.012 secs
Source Language Support 0.037 secs
Stack 0.038 secs
Subroutine References 0.006 secs
Subroutine References - One Time 0.000 secs
x86 Constant Reference Analyzer 0.120 secs
-----------------------------------------------------
Total Time 3 secs
-----------------------------------------------------
(AutoAnalysisManager.java:1273)
The disassembling process is made by the Disassembler class and precisely by the disassemble method.
This disassemble method uses the SleighLanguage class to get Instruction Alignment and process recursively
public AddressSet disassemble(AddressSetView startSet, AddressSetView restrictedSet,
RegisterValue initialContextValue, boolean doFollowFlow) {
System.out.println("Disassembler disassemble()");
AddressSet disassembledAddrs;
disassembledAddrs = new AddressSet();
int alignment = language.getInstructionAlignment();
AddressRangeIterator addressRanges = startSet.getAddressRanges();
for (AddressRange addressRange : addressRanges) {
if (monitor.isCancelled()) {
break;
}
if (disassembledAddrs.contains(addressRange.getMinAddress(),
addressRange.getMaxAddress())) {
continue;
}
AddressSet todoSubset = new AddressSet(addressRange);
while (!todoSubset.isEmpty() && !monitor.isCancelled()) {
Address nextAddr = todoSubset.getMinAddress();
// Check if location is already on disassembly list
if (disassembledAddrs.contains(nextAddr)) {
AddressRange doneRange = disassembledAddrs.getRangeContaining(nextAddr);
todoSubset.delete(doneRange);
continue;
}
todoSubset.delete(nextAddr, nextAddr);
// must be aligned
if (nextAddr.getOffset() % alignment != 0) {
continue;
}
Data data = listing.getUndefinedDataAt(nextAddr);
if (data == null) {
AddressSetView undefinedRanges = null;
try {
undefinedRanges =
program.getListing().getUndefinedRanges(todoSubset, true, monitor);
todoSubset = new AddressSet(undefinedRanges);
}
catch (CancelledException e) {
break;
}
}
else {
AddressSet currentSet =
disassemble(nextAddr, restrictedSet, initialContextValue, doFollowFlow);
if (!currentSet.isEmpty()) { // nothing disassembled
todoSubset.delete(currentSet);
disassembledAddrs.add(currentSet);
}
}
if (monitor.isCancelled()) {
break;
}
}
}
return disassembledAddrs;
}
the SleighLanguage class uses the initialize method to get all the files implied in the disassembly of a binary
private void initialize(boolean forceCompile, TaskMonitor monitor) throws SleighException {
long startTS = System.currentTimeMillis();
this.defaultSymbols = new ArrayList<>();
this.defaultMemoryBlocks = new MemoryBlockDefinition[0];
this.compilerSpecDescriptions = new LinkedHashMap<>();
for (CompilerSpecDescription compilerSpecDescription : description
.getCompatibleCompilerSpecDescriptions()) {
this.compilerSpecDescriptions.put(compilerSpecDescription.getCompilerSpecID(),
(SleighCompilerSpecDescription) compilerSpecDescription);
}
compilerSpecs = new HashMap<>();
additionalInject = null;
SleighLanguageValidator.validatePspecFile(description.getSpecFile());
readInitialDescription(); // process pspec file
// Should addressFactory and registers initialization be done at construction time?
// For now we'll assume yes.
contextcache = new ContextCache();
SleighLanguageFile langFile = description.getLanguageFile();
if (!langFile.getSlaSpecFile().exists()) {
throw new SleighFileException("Missing slaspec: " + langFile.getSlaSpecFile());
}
// check .sla file freshness inside lock, and recompile if necessary before releasing lock.
// if can't lock, it's single jar mode and we can't recompile anyways
AtomicLong lockElapsed = new AtomicLong();
if (langFile.canLock()) {
try (PreserveStateWrappingTaskMonitor tm =
new PreserveStateWrappingTaskMonitor(monitor)) {
tm.setCancelEnabled(true);
tm.setShowProgressValue(true);
langFile.withLock(SleighLanguageProvider.LANGUAGE_LOCK_TIMEOUT, tm, () -> {
tm.setCancelEnabled(false);
long lockStartTS = System.currentTimeMillis();
if (forceCompile || langFile.needsCompilation(SlaFormat.FORMAT_VERSION)) {
langFile.compileSlaFile(monitor);
}
lockElapsed.set(System.currentTimeMillis() - lockStartTS);
});
}
catch (TimeoutException e) {
throw new SleighFileLockException(
"Timeout waiting for Sleigh language file lock: %s"
.formatted(langFile.getSlaFile()),
e);
}
catch (IOException e) {
throw new SleighFileException(
"Error locking Sleigh language file %s".formatted(langFile.getSlaFile()), e);
}
}
// Read in the sleigh specification
try (PackedDecode decoder = SlaFormat.buildDecoder(langFile.getSlaFile())) {
decode(decoder);
}
catch (IOException | DecoderException e) {
throw new SleighException("Error decoding", e);
}
registerBuilder = new RegisterBuilder();
loadRegisters(registerBuilder);
readRemainingSpecification();
buildVolatileSymbolAddresses();
xrefRegisters();
instructProtoMap = new ConcurrentHashMap<>();
initParallelHelper();
int maxLength =
getPropertyAsInt(GhidraLanguagePropertyKeys.MAXIMUM_INSTRUCTION_LENGTH, -1);
if (maxLength > 0) {
maxInstructionLength = OptionalInt.of(maxLength);
}
long initElapsed = System.currentTimeMillis() - startTS;
Msg.debug(this, "Took %dms (%dms inside lock) to initialize language %s"
.formatted(initElapsed, lockElapsed.get(), langFile));
}
There are 3 kind of files
- the pspec file
- the slaspec file
- the sla file
The .pspec file (Processor Specification): This is indeed a specification file linked to the processor architecture. It defines elements such as default registers, specific stack locations, calling conventions, or hardware starting addresses for a given architecture.
<?xml version="1.0" encoding="UTF-8"?>
<processor_spec>
<programcounter register="PC"/>
<default_symbols>
<symbol name="NMI" address="FFFA" entry="true" type="code_ptr"/>
<symbol name="RES" address="FFFC" entry="true" type="code_ptr"/>
<symbol name="IRQ" address="FFFE" entry="true" type="code_ptr"/>
</default_symbols>
<default_memory_blocks>
<memory_block name="ZERO_PAGE" start_address="0x0000" length="0x0100" initialized="false"/>
<memory_block name="STACK" start_address="0x0100" length="0x0100" initialized="false"/>
</default_memory_blocks>
</processor_spec>
The .slaspec file (SLEIGH Specification): This is the main textual source file that describes the assembly syntax and processor architecture semantics (e.g., x86, ARM, MIPS). It is written by the creators of the architecture (or emulator) and is compiled once and for all.
# sleigh specification file for MOS 6502
define endian=little;
define alignment=1;
define space RAM type=ram_space size=2 default;
define space register type=register_space size=1;
define register offset=0x00 size=1 [ A X Y P ];
define register offset=0x20 size=2 [ PC SP ];
define register offset=0x20 size=1 [ PCL PCH S SH ];
define register offset=0x30 size=1 [ N V B D I Z C ]; # status bits
#TOKENS
define token opbyte (8)
op = (0,7)
aaa = (5,7)
bbb = (2,4)
cc = (0,1)
;
define token data8 (8)
imm8 = (0,7)
rel = (0,7) signed
;
define token data (16)
imm16 = (0,15)
;
macro popSR() {
SP = SP + 1;
local ccr = *:1 SP;
N = ccr[7,1];
V = ccr[6,1];
B = ccr[4,1];
D = ccr[3,1];
I = ccr[2,1];
Z = ccr[1,1];
C = ccr[0,1];
}
The .sla file (Compiled SLEIGH): SLA stands for SLEIGH Language Architecture format. It is a binary/intermediate file loaded by Ghidra to quickly decode instructions for a specific architecture.
4. Save into the database
if you create a test.gpr project in some directory (example XXXX), Ghidra is going to create a test.rep directory in XXXXX
In the test.rep directory, you will find the following files and directories
ls -la
total 28
drwxr-x--- 5 daniel daniel 4096 Jul 28 16:05 .
drwxrwxr-x 9 daniel daniel 4096 Jul 29 11:06 ..
drwxr-x--- 3 daniel daniel 4096 Jul 29 11:06 idata
-rw-r----- 1 daniel daniel 159 Jul 26 20:21 project.prp
-rw-r----- 1 daniel daniel 602 Jul 29 11:06 projectState
drwxr-x--- 3 daniel daniel 4096 Jul 29 11:06 user
drwxr-x--- 2 daniel daniel 4096 Jul 29 11:06 versioned
Here is a breakdown of what each of those files and directories represents:
project.prp (Project Properties)
What it is: This is the "Identity Card" of your project.
Role: It contains the configuration and metadata specific to the project itself. If you rename your project or change project-wide settings (like certain script paths or project-wide preferences), that information is recorded here. It is essentially the "header" for the .gpr project.projectState
What it is: The "Session Tracker."
Role: This file tracks the current "state" of your work within the project. It remembers which programs were recently opened, which windows were positioned where, and the overall "context" of your last session. This is why, when you reopen Ghidra, you can pick up exactly where you left off.idata (Internal Data)
What it is: The "Shared Library" of the project.
Role: This directory holds data that is shared across all programs within this specific project. If you create a custom Data Type (a struct, for example) and save it so that it can be used by multiple different binaries in the same project, that structure definition is stored here. It prevents duplication.user
What it is: The "User Preference" override.
Role: This directory contains settings or configurations that are specific to the user interacting with this particular project. It allows for user-specific customizations that don't affect the "core" project properties but are relevant to how you interact with this specific analysis.versioned
What it is: The "History/Snapshot" directory.
Role: The presence of this folder indicates that you (or a script you ran) have used Ghidra's Versioning/Snapshot feature.
How it works: Ghidra allows you to take "snapshots" of a program's state. This folder stores the "diffs" or the historical states of the programs in the project. It allows you to revert the analysis to a previous point in time.
We will study more precisely this subject in another article
How it is handled in Ghidra ?
The Databae Engine is handled by the following files
Framework/FileSystem/bin/main/ghidra/framework/store/db/package.html
Framework/FileSystem/src/main/java/ghidra/framework/store/db/PackedDBHandle.java
Framework/FileSystem/src/main/java/ghidra/framework/store/db/package.html
Framework/FileSystem/src/main/java/ghidra/framework/store/db/PrivateDatabase.java
Framework/FileSystem/src/main/java/ghidra/framework/store/db/PackedDatabase.java
Framework/FileSystem/src/main/java/ghidra/framework/store/db/VersionedDBListener.java
Framework/FileSystem/src/main/java/ghidra/framework/store/db/VersionedDatabase.java
Framework/FileSystem/src/main/java/ghidra/framework/store/db/PackedDatabaseCache.java
Framework/FileSystem/src/test/java/ghidra/framework/store/db/PackedDatabaseTest.java
Framework/FileSystem/src/test/java/ghidra/framework/store/db/VersionFailureRecoveryTest.java
Framework/FileSystem/src/test/java/db/RecoveryDBTest.java
Framework/FileSystem/src/test.slow/java/db/buffers/RecoveryFileTest.java
Framework/DB/src/main/java/db/ObjectStorageAdapterDB.java
Framework/DB/src/main/java/db/TranslatedRecordIterator.java
Framework/DB/src/main/java/db/BinaryCodedField.java
Framework/DB/src/main/java/db/KeyToRecordIterator.java
Framework/DB/src/main/java/db/InteriorNode.java
Framework/DB/src/main/java/db/BTreeNode.java
Framework/DB/src/main/java/db/ShortField.java
Framework/DB/src/main/java/db/BinaryDataBuffer.java
Framework/DB/src/main/java/db/RecordTranslator.java
Framework/DB/src/main/java/db/DatabaseUtils.java
Framework/DB/src/main/java/db/TableRecord.java
Framework/DB/src/main/java/db/DBLongIterator.java
Framework/DB/src/main/java/db/LongKeyInteriorNode.java
Framework/DB/src/main/java/db/VarRecNode.java
Framework/DB/src/main/java/db/VarKeyRecordNode.java
Framework/DB/src/main/java/db/LongKeyNode.java
Framework/DB/src/main/java/db/DBFileListener.java
Framework/DB/src/main/java/db/TerminatedTransactionException.java
Framework/DB/src/main/java/db/VarKeyInteriorNode.java
Framework/DB/src/main/java/db/LongField.java
Framework/DB/src/main/java/db/FixedField.java
Framework/DB/src/main/java/db/TestSpeed.java
Framework/DB/src/main/java/db/VarKeyNode.java
Framework/DB/src/main/java/db/TableStatistics.java
Framework/DB/src/main/java/db/BinaryField.java
Framework/DB/src/main/java/db/IllegalFieldAccessException.java
Framework/DB/src/main/java/db/NoTransactionException.java
Framework/DB/src/main/java/db/NodeMgr.java
Framework/DB/src/main/java/db/DBRecord.java
Framework/DB/src/main/java/db/RecordNode.java
Framework/DB/src/main/java/db/DBHandle.java
Framework/DB/src/main/java/db/LongKeyRecordNode.java
Framework/DB/src/main/java/db/DBInitializer.java
Framework/DB/src/main/java/db/MasterTable.java
Framework/DB/src/main/java/db/Field.java
Framework/DB/src/main/java/db/FixedRecNode.java
Framework/DB/src/main/java/db/ConvertedRecordIterator.java
Framework/DB/src/main/java/db/DBListener.java
Framework/DB/src/main/java/db/FixedKeyRecordNode.java
Framework/DB/src/main/java/db/Table.java
Framework/DB/src/main/java/db/FixedField10.java
Framework/DB/src/main/java/db/FieldKeyRecordNode.java
Framework/DB/src/main/java/db/Buffer.java
Framework/DB/src/main/java/db/ByteField.java
Framework/DB/src/main/java/db/FixedKeyVarRecNode.java
Framework/DB/src/main/java/db/IndexTable.java
Framework/DB/src/main/java/db/DBBuffer.java
Framework/DB/src/main/java/db/IntField.java
Framework/DB/src/main/java/db/SparseRecord.java
Framework/DB/src/main/java/db/DBParms.java
Framework/DB/src/main/java/db/FixedKeyNode.java
Framework/DB/src/main/java/db/IndexField.java
Framework/DB/src/main/java/db/Transaction.java
Framework/DB/src/main/java/db/util/ErrorHandler.java
Framework/DB/src/main/java/db/FieldKeyInteriorNode.java
Framework/DB/src/main/java/db/ConstrainedForwardRecordIterator.java
Framework/DB/src/main/java/db/FixedKeyFixedRecNode.java
Framework/DB/src/main/java/db/FieldIndexTable.java
Framework/DB/src/main/java/db/Database.java
Framework/DB/src/main/java/db/ChainedBuffer.java
Framework/DB/src/main/java/db/FieldKeyNode.java
Framework/DB/src/main/java/db/DBFieldIterator.java
Framework/DB/src/main/java/db/LegacyIndexField.java
Framework/DB/src/main/java/db/DBRollbackException.java
Framework/DB/src/main/java/db/buffers/LocalBufferFile.java
Framework/DB/src/main/java/db/buffers/RecoveryMgr.java
Framework/DB/src/main/java/db/buffers/VersionFile.java
Framework/DB/src/main/java/db/buffers/DataBuffer.java
Framework/DB/src/main/java/db/buffers/LocalManagedBufferFile.java
Framework/DB/src/main/java/db/buffers/BufferNode.java
Framework/DB/src/main/java/db/buffers/VersionFileHandler.java
Framework/DB/src/main/java/db/buffers/InputBlockStream.java
Framework/DB/src/main/java/db/buffers/BufferFile.java
Framework/DB/src/main/java/db/buffers/ManagedBufferFileHandle.java
Framework/DB/src/main/java/db/buffers/BufferFileBlock.java
Framework/DB/src/main/java/db/buffers/IndexProvider.java
Framework/DB/src/main/java/db/buffers/BufferMgr.java
Framework/DB/src/main/java/db/buffers/ChangeMapFile.java
Framework/DB/src/main/java/db/buffers/ManagedBufferFileAdapter.java
Framework/DB/src/main/java/db/buffers/BufferFileHandle.java
Framework/DB/src/main/java/db/buffers/BlockStreamHandle.java
Framework/DB/src/main/java/db/buffers/RemoteManagedBufferFileHandle.java
Framework/DB/src/main/java/db/buffers/ManagedBufferFile.java
Framework/DB/src/main/java/db/buffers/RecoveryFile.java
Framework/DB/src/main/java/db/buffers/BlockStream.java
Framework/DB/src/main/java/db/buffers/BufferFileAdapter.java
Framework/DB/src/main/java/db/buffers/RemoteBufferFileHandle.java
Framework/DB/src/main/java/db/buffers/OutputBlockStream.java
Framework/DB/src/main/java/db/buffers/BufferFileManager.java
Framework/DB/src/main/java/db/buffers/ChangeMap.java
Framework/DB/src/main/java/db/RecordIterator.java
Framework/DB/src/main/java/db/StringField.java
Framework/DB/src/main/java/db/FixedKeyInteriorNode.java
Framework/DB/src/main/java/db/BooleanField.java
Framework/DB/src/main/java/db/Schema.java
Framework/DB/src/main/java/db/DBChangeSet.java
Framework/DB/src/main/java/db/PrimitiveField.java
The files above you are looking at are the source code for the custom database engine that Ghidra uses to power its Repository (.rep) system.
The files are divided into two distinct layers: the API/Abstraction Layer and the Engine/Implementation Layer.
Layer 1: The Abstraction Layer
Path: ghidra/framework/store/db/...
This layer provides the "Java-friendly" way for the rest of Ghidra to interact with the database. If you are writing a Ghidra plugin and you want to save something to the repository, you use these classes.
PrivateDatabase.java & PackedDatabase.java: These are the "Entry Points." They represent a logical database that a user can open. They hide the complexity of the raw bytes and present a structured object.
VersionedDatabase.java: This is the logic that allows the .rep folder to have snapshots. It manages the "History" of the database.
PackedDatabaseCache.java: This manages memory. It ensures that frequently accessed parts of the database stay in RAM so that Ghidra doesn't have to hit the disk every time you click a function.
PackedDBHandle.java: This is the "Pointer" or "Cursor" used to navigate through the database records.
Layer 2: The Engine Layer (The "Heavy Lifters")
Path: db/...
This is the "low-level" code. This is the code that actually manipulates bits and bytes on your hard drive. This is a highly optimized implementation of a B-Tree Database (similar to Berkeley DB).
A. The Structural Layer (The B-Tree)
These classes define the "Shape" of the data.
BTreeNode.java, InteriorNode.java, VarKeyInteriorNode.java: These implement the B-Tree structure. A B-Tree is a self-balancing tree data structure that maintains sorted data and allows searches, sequential access, insertions, and deletions in logarithmic time.
MasterTable.java & IndexTable.java: These are the "Metadata" tables. They store the "Map" of the database (e.g., "Table X exists, and it has 5 columns").
B. The Schema Layer (The Data Types)
These classes define what a "Field" is. Without these, the database is just a blob of random bytes.
Field.java: The base class for all data types.
ByteField.java, IntField.java, StringField.java, BooleanField.java, LongField.java: These define the actual types of data the database can hold. They handle the logic of "How do I turn a Java int into 4 bytes on the disk?"
C. The Transaction & Safety Layer (The ACID Properties)
This is the most critical part for preventing data corruption.
Transaction.java: This manages the "All or Nothing" principle. When you perform a write, this class ensures that either the entire write succeeds, or none of it does.
RecoveryMgr.java & RecoveryFile.java: This is the Write-Ahead Log (WAL) logic. As we discussed earlier, these files record changes before they are committed to the main database, allowing the engine to recover after a crash.
DBListener.java & VersionFileHandler.java: These notify the system when changes occur, allowing for the "Versioned" feature to trigger.
D. The I/O Layer (The Buffer Manager)
This is the bridge between your RAM and your Hard Drive.
BufferMgr.java & BufferFile.java: These manage the "Buffer Pool." They decide which parts of the database files are currently loaded into your computer's RAM.
DataBuffer.java & ChainedBuffer.java: These handle the raw byte arrays that are being streamed from the disk.
InputBlockStream.java & OutputBlockStream.java: These are the "Pipes" that stream data into and out of the database files.
Summary: How they work together
A user clicks "Save" in Ghidra.
The PrivateDatabase (Layer 1) receives the command.
It tells the Transaction (Layer 2) to start a new operation.
The Field classes determine how the new data should be encoded into bytes.
The BTreeNode logic determines where in the tree this new data should be inserted.
The RecoveryMgr writes the change to a log file to ensure safety.
The BufferMgr pushes those bytes into the DataBuffer.
The BufferFile finally writes those bytes to the physical .db file on your disk.
6. Epilogue
This analysis would have been far more difficult if the creators of Ghidra had not done an incredible job standardizing their files and maintaining consistent design rules. Kudos to them! Right now, I am still scratching the surface and there's a lot more to learn. I will continue to document my journey in the next articles.
Top comments (0)