No bundle, no problem: running Ghidra scripts without OSGi

Most reverse engineers know Ghidra, the software reverse engineering framework the NSA released as open source in 2019. If you have used it for more than an afternoon, you have probably written a script to automate something tedious: renaming functions from debug strings, hunting for byte patterns, bulk exporting decompiled output.

Ghidra gives you two languages for that: Java, which is what the API itself is written in, and Python, either through Jython (stuck on Python 2.7, now deprecated) or through PyGhidra, which bridges to a real CPython 3 interpreter using JPype. Python is fine for prototyping, until you are analyzing thousands of binaries and interpreter startup becomes the bottleneck.

That was our situation on SightHouse, where we wrote a custom analyzer meant to go through dozens of executables as fast as possible. So we wrote it in Java. And then, seemingly at random, we started getting this:

ERROR REPORT SCRIPT ERROR: TestScript.java : The class could not be found.
 It must be the public class of the .java file: Failed to get OSGi bundle containing script: /home/user/TestScript.java
 (HeadlessAnalyzer) ghidra.app.script.GhidraScriptLoadException: The class could not be found.
 It must be the public class of the .java file: Failed to get OSGi bundle containing script: /home/user/TestScript.java
 at ghidra.app.script.JavaScriptProvider.getScriptInstance(JavaScriptProvider.java:110)
 at ghidra.app.util.headless.HeadlessAnalyzer.runScriptsList(HeadlessAnalyzer.java:912)
 at ghidra.app.util.headless.HeadlessAnalyzer.analyzeProgram(HeadlessAnalyzer.java:993)
 at ghidra.app.util.headless.HeadlessAnalyzer.processFileNoImport(HeadlessAnalyzer.java:1172)
 at ghidra.app.util.headless.HeadlessAnalyzer.processFolderNoImport(HeadlessAnalyzer.java:1344)
 at ghidra.app.util.headless.HeadlessAnalyzer.processNoImport(HeadlessAnalyzer.java:1373)
 at ghidra.app.util.headless.HeadlessAnalyzer.processLocal(HeadlessAnalyzer.java:454)
 at ghidra.app.util.headless.AnalyzeHeadless.launch(AnalyzeHeadless.java:198)
 at ghidra.GhidraLauncher.launch(GhidraLauncher.java:81)
 at ghidra.Ghidra.main(Ghidra.java:54)
Caused by: java.lang.ClassNotFoundException: Failed to get OSGi bundle containing script: /home/user/TestScript.java
 at ghidra.app.script.JavaScriptProvider.loadClass(JavaScriptProvider.java:159)
 at ghidra.app.script.JavaScriptProvider.getScriptInstance(JavaScriptProvider.java:96)
 ... 9 more

rm -rf ~/.config/ghidra/*/osgi/

Read the first line again. “The class could not be found. It must be the public class of the .java file.” When you get his error, you immediately go and check that TestScript.java declares public class TestScript. It does. It always did. The same script ran fine an hour earlier 🤬. That sentence is a generic fallback message from JavaScriptProvider, the part that matters that Ghidra never got as far as looking for the class because the bundle supposed to contain it was never successfully built in the first place.

Which left me with two questions I had somehow never asked in years of using Ghidra. How does Ghidra run a .java file at all, given that Java is a compiled language? And what exactly is an OSGi bundle?

Compiling on the fly

#

Since JDK 11 and JEP 330, java HelloWorld.java compiles to bytecode in memory and runs it in a single step. Doing it in two separate steps also works:

# Compile HelloWorld.java into HelloWorld.class
$ javac HelloWorld.java
# Run the class file
$ java HelloWorld

So far so good. Now let’s write something that actually needs the Ghidra API:

// TestGhidraScript.java
import ghidra.app.script.GhidraScript;
import ghidra.util.Msg;

public class TestGhidraScript extends GhidraScript {

    @Override
    protected void run() throws Exception {
        Msg.info(this, "Hello World");
    }
}

We can compile it the same way:

$ javac TestGhidraScript.java
TestGhidraScript.java:2: error: package ghidra.app.script does not exist
import ghidra.app.script.GhidraScript;
                        ^
TestGhidraScript.java:3: error: package ghidra.util does not exist
import ghidra.util.Msg;
                  ^
TestGhidraScript.java:5: error: cannot find symbol
public class TestGhidraScript extends GhidraScript {
                                      ^
  symbol: class GhidraScript
3 errors

This is expected: javac has no idea where the Ghidra packages live. Ghidra solves the problem by compiling scripts on the fly against its own jars, in GhidraSourceBundle.compileToExplodedBundle:

/**
 * Compile a source directory to an exploded bundle.
 *
 * @param writer for updating the user during compilation
 * @return a summary of the work performed
 * @throws IOException for source/manifest file reading/generation and binary deletion/creation
 * @throws OSGiException if generation of bundle metadata fails
 */
private String compileToExplodedBundle(PrintWriter writer) throws IOException, OSGiException {

    Files.createDirectories(binaryDir);

    Summary summary = new Summary();

    List<String> options = new ArrayList<>();
    options.add("-g");
    options.add("-d");
    options.add(binaryDir.toString());
    options.add("-sourcepath");
    options.add(getSourceDirectory().toString());
    options.add("-classpath");
    options.add(
        System.getProperty("java.class.path") + File.pathSeparator + binaryDir.toString());
    options.add("-proc:none");

  ...
}

Which is roughly equivalent to running:

$ javac -g \
    -d /path/to/binaries \
    -sourcepath /path/to/scripts \
    -classpath "<ghidra's own java.class.path>:/path/to/binaries" \
    -proc:none \
    TestGhidraScript.java

Two details worth noting: the classpath is Ghidra’s own runtime classpath, read from the java.class.path system property, which is why the JVM running your script can see the API while a standalone javac cannot. And the output directory is appended to that classpath, so scripts living side by side can reference each other. That explains how a .java file gets executed, but not yet why it sometimes doesn’t work.

What is an OSGi bundle?

#

OSGi is a specification for building modular Java applications whose components can be loaded, updated and unloaded at runtime. Ghidra embeds an implementation of it (Apache Felix) and uses it to manage both extensions and scripts.

The unit of modularity is the bundle: a jar plus manifest metadata declaring which packages it imports and which it exports, which the framework wires together at load time. Ghidra extends the idea to directories of source files, which it calls source bundles, and compiles on your behalf, and that is exactly what compileToExplodedBundle above is doing. Before any class inside a bundle can be loaded, though, that bundle must reach the ACTIVE state, a transition Ghidra triggers when the bundle is added, enabled, or when a script inside it is run.

The compiled result is cached on disk under <user settings>/osgi/compiled-bundles/<hash>/. The <user settings> part is platform-dependent: on Linux it lives in ~/.config/ghidra/<ghidra_version>/, on Windows under %USERPROFILE%.

Putting those pieces together, here is what happens when analyzeHeadless runs a .java script:

sequenceDiagram
    participant HA    as HeadlessAnalyzer
    participant JSP   as JavaScriptProvider
    participant GSB   as GhidraSourceBundle
    participant Felix as OSGi framework
    HA->>JSP:    getScriptInstance("TestScript.java")
    JSP->>GSB:   build bundle for the script *directory*
    GSB->>Felix: compare sources against cached binaries
    Felix-->>GSB: missing or stale entries
    GSB->>GSB:   javac -g -d ... -proc:none
    GSB->>Felix: install and activate bundle
    Felix-->>JSP: bundle ACTIVE
    JSP->>Felix: loadClass("TestScript")
    Note over JSP,Felix: If activation failed, this throws
ClassNotFoundException

The last step is where our stack trace comes from. JavaScriptProvider.loadClass asks the framework for a class inside a bundle that never reached ACTIVE, gets nothing back, and reports it as a missing class.

What actually corrupts the cache

#

Let’s be honest: I never nailed this down. The failure was intermittent, correlated with cache state in a way I could not reproduce on demand, and went away after deleting <user settings>/osgi/ (most of the time). At some point the cost of understanding it exceeded the cost of routing around it, and I stopped digging.

# This code is not fixable.
# Increment the counter to warn the next person.
# hours_wasted = 254

Finding another way

#

I started looking around for a way that avoid the bundle entirely. It turns out there is one, possibly the worst-kept secret the NSA has ever had, hiding in plain sight: Ghidra can run a compiled .class file directly, with no bundle involved. Look at HeadlessAnalyzer.runScriptsList:

private HeadlessContinuationOption runScriptsList(List<Pair<String, String[]>> scriptsList,
        Map<String, ResourceFile> scriptFileMap, GhidraState scriptState,
        HeadlessContinuationOption continueOption) {

    try {
        for (Pair<String, String[]> scriptPair : scriptsList) {
            scriptName = scriptPair.first;
            String[] scriptArgs = scriptPair.second;

            // For .class files, there is no ResourceFile mapping. Need to load from the
            // stored 'classLoaderForDotClassScripts'
            if (scriptName.endsWith(".class")) {
                ...
            } else {
                // GhidraScriptProvider case
            }
        }
    }
}

Two entirely separate code paths, selected depending on the file extension:

graph TD
    A["-preScript Foo"] --> B{"name ends\nwith .class?"}
    B -->|"no"| C["GhidraScriptProvider"]
    C --> D["GhidraSourceBundle\ncompile sources"]
    D --> E["OSGi cache\ncompiled-bundles/hash"]
    E --> F["Felix: activate bundle"]
    B -->|"yes"| G["classLoaderForDotClassScripts\nplain URLClassLoader over -scriptPath"]
    F --> H["GhidraScript instance"]
    G --> H

The right-hand branch bypasses GhidraScriptProvider, GhidraSourceBundle and the OSGi layer entirely and instead loads the class with an ordinary URLClassLoader built from the -scriptPath entries. No cache, and therefore nothing to corrupt.

Taking that path means doing ourselves the work Ghidra was doing for us. The awkward part is the classpath: we can’t read Ghidra’s java.class.path from the outside, and its jars are scattered across Ghidra/Features/*/lib, Ghidra/Framework/*/lib and several other places, so rather than enumerate them, we glob the whole installation. Not very elegant, but it survives Ghidra rearranging its own directory layout across updates (If it works, it works):

def build_script(ghidradir: Path, script_dir: Path) -> None:
    """Compile Ghidra scripts to .class files to speed up the loading process
    and avoid random OSGi errors
    """
    ...
    # Find all jars of Ghidra and build the classpath argument
    jars = os.pathsep.join(map(str, ghidradir.rglob("*.jar")))
    classpath = f".{os.pathsep}{jars}"

    for source_file in script_dir.rglob("*.java"):
        javac_command = [
            "javac",
            "-g",
            "-d", str(script_dir),
            "-sourcepath", str(script_dir),
            "-cp", classpath,
            "-proc:none",
            str(source_file),
        ]

        returncode, stdout, stderr = run_process(javac_command)
        if returncode != 0:
            raise Exception(
                f"Failed to compile '{source_file}': {stdout.decode()}\n{stderr.decode()}"
            )

The flags mirror compileToExplodedBundle with one difference: -d points at the script directory itself rather than a separate binary directory, because the .class file has to end up somewhere the runtime class loader will look, and that means inside a directory we pass as -scriptPath.

Running it is then a matter of handing analyzeHeadless the compiled file:

def run_ghidra_script(ghidradir: Path, script: Path, args: List[str], ...):
    """Run a given Ghidra script"""
    ...

    script_path: Path = script.parent
    compiled_script: Path = script.with_suffix(".class")
    # Compile script if not already done
    if not compiled_script.exists():
        build_script(ghidradir, script_path)

    # Project is stored in temp directory
    with TemporaryDirectory() as tmpdirname:
        process_args: List[str] = [
            str(ghidradir / "support" / "analyzeHeadless"),
            tmpdirname,
            "tmpproj",
            "-scriptPath",
            str(script_path),
            "-preScript",
            compiled_script.name,
        ]
        process_args += args

        # Run process without a timeout
        return run_process(process_args, env=env, capture_output=capture_output)

This is a workaround, not a solution. The compiled bytecode is tied to the Ghidra version it was built against. If you are shipping a tool rather than a script and you control the deployment, the better option is to package your code as a Ghidra extension built with Gradle. We stayed with regular scripts because SightHouse has to work against whatever Ghidra installation the user already has, and a per-version was too much of a requirement.

Conclusion

#

Narrow problem, admittedly and I’m probably the only one who cares. But that’s the fun of reading Ghidra’s source: you end up understanding something you’d normally never think about. “Java is compiled” and “Ghidra runs .java files” are both true statements, and sitting between them is an entire OSGi framework that quietly rebuilds your scripts folder into a bundle whenever you hit run.