Improving IDLs Discoverability: Accelerating Solana Development, Integration and Composability

What is this Article About?

Solana has over time garnered the reputation as the fastest and most scalable blockchain on the the market, leveraging its unique architecture to achieve high throughput and low latency periods. However, associated with a massive growth of this scale, is an important component of the ecosystem that requires constant attention: Developer Experience (DX). Improving the developer experience is always on the spotlight as developers and builders are the lifeline of Solana.
Of the issues that hamper a smooth development experience on Solana, IDLs discovery and implementation for existing infrastructure appears to provide the most encumbrance to Solana developers and developers from other ecosystems looking to build on Solana at the present time with two concerns predominantly echoed:

• Reading data on the network is difficult and even more so for non-anchor programs
• Obscurity in CPI with programs as security concerns

In this article, we will explore the IDL landscape,and analyze how they are a vital part of improving the development experience on Solana. This article aims to provide a comprehensive understanding of the role IDLs play across the Solana development landscape, and why further understanding into upgrading its discoverability will bolster and accelerate development experience on Solana.

Interface Description Language(IDL)

An Interface Definition Langauge (IDL) in Solana refers to the public interface of a program. It represents the structure of programs accounts, instructions as well as error codes. They are usually in a JSON and Typescript file format and they are used to generate client-side code enabling clients to easily interact with a Solana program. They are automatically generated for programs written in anchor. IDLs can be compared to ABI files for EVM, which gives users easy understanding of program structures.

Why are IDLs very useful

While the applications of IDLs can be enumerated to varying use cases, IDLs are very useful for the main reasons:

• They make program interaction human readable: Through IDLs, we can demystify the concatenation of transaction and accounts as just a bunch of bytes and provide developers a medium to easily parse instructions or accounts in order to make sense of what the program is doing. This attribute of IDLs enable indie and professional developers to get smooth integration on explorers, consumer UIs, and data platforms (like Dune). Below is an example of a parsed transaction instruction on Solana:

Here is an example of how Solana explorer display public IDLs:

• Providing Standard Interfacing: By leveraging a standard interface, tools and frameworks can take advantage of this to generate instructions to interact with a program much more easily therefore enabling easier composability between program interfaces, making building client-side code far simpler (A good example would be the case of embedded wallet providers that whitelist specific ix discriminant). Below we have an example of two Hello World frontends that create a greeting account, one using Native Solana and the other using Anchor:

Using Native Solana:

const GREETING_SEED = 'hello'
const greetedPubkey = await PublicKey.createWithSeed(
    payer.publicKey,
    GREETING_SEED,
    programId
)

// Check if the greeting account has already been created
const greetedAccount = await connection.getAccountInfo(greetedPubkey)
if (greetedAccount === null) {
    console.log('Creating account', greetedPubkey.toBase58(), 'to say hello to')
    const lamports = await connection.getMinimumBalanceForRentExemption(
        GREETING_SIZE
    )

    const transaction = new Transaction().add(
        SystemProgram.createAccountWithSeed({
            fromPubkey: payer.publicKey,
            basePubkey: payer.publicKey,
            seed: GREETING_SEED,
            newAccountPubkey: greetedPubkey,
            lamports,
            space: GREETING_SIZE,
            programId,
        })
    )
    await sendAndConfirmTransaction(connection, transaction, [payer])
}
// ...
export async function sayHello(): Promise<void> {
    console.log('Saying hello to', greetedPubkey.toBase58())
    const instruction = new TransactionInstruction({
        keys: [{ pubkey: greetedPubkey, isSigner: false, isWritable: true }],
        programId,
        data: Buffer.alloc(0), // All instructions are hellos
    })
    await sendAndConfirmTransaction(
        connection,
        new Transaction().add(instruction),
        [payer]
    )
}

Source: https://github.com/solana-labs/example-helloworld/blob/master/src/client/hello_world.ts

Using Anchor:

const anchor = require('@project-serum/anchor')

async function greet(message: string) {
    anchor.setProvider(anchor.Provider.local())
    // Here we load the HelloAnchor example IDL
    const program = anchor.workspace.HelloAnchor
    let greetKey = Keypair.generate()
    await program.methods
        .initialize(message)
        .accounts({
            newAccount: greetKey.publicKey,
            signer: wallet.publicKey,
            systemProgram: web3.SystemProgram.programId,
        })
        .signers([wallet, greetKey])
        .rpc()
}

greet('Hello World')

By utilizing IDLs, Anchor is able to abstract the process so that is so much easier to read, quicker to deployed and has a better composability for CPI interactions. To know more about CPIs and why they are so important to Solana development, refer to this article Cross-Program Invocations and PDAs- the combination of two powerful mechanism on Anchor

• Dependency-free SDKs: IDLs provide dependency-free SDKs for long-tail of Solana programs with declare_program!()

Creating IDLs

With Anchor, IDLs are automatically generated when you build because they are so integral to the entire anchor programming landscape. From the anchor project directory, enter the following in your terminal:

anchor build --idl <IDL_OUTPUT_DIRECTORY>

However by default, idl.json will be saved in ./target/idl folder

For Solana programs written in native Rust, IDLs can be generated using tooling like Metaplex Shank or Codama Kinobi

Publishing IDLs

Anchor also makes it easy to publish IDLs on-chain. To do this, you simply need to enter the following in your terminal:

anchor idl init --filepath <FILEPATH> <PROGRAM_ID> --provider.cluster <CLUSTER> --provider.wallet <WALLET>

Make sure the wallet being used is the program authority and has SOL for the on-chain transaction! After the IDL is loaded, you should be able to see the IDL on Solana Explorer at https://explorer.solana.com/address/YOUR_PROGRAM_ID/anchor-program.

The anchor idl command has several other handy tools that enables developers to fetch, update, or freeze an IDL. For information on each of those commands, you can enter the following in your terminal:

anchor idl -h

Interfacing IDLs on Client-Side

Programs defined in your IDL are mapped to corresponded to functions are provided by the generated Typescript module which is integrated into the front-end of our application. The general structure of this module is outlined below:

import { Program, AnchorProvider, setProvider } from '@project-serum/anchor'
import { Connection, KeyPair } from '@solana/web3.js'
import { PROGRAM_ID, IDL } from './your-program-dir'
// where IDL is the .json created by anchor build
// and PROGRAM_ID is your on-chain program ID

export const yourFunction = async () => {
    const wallet = KeyPair.generate()
    const connection = new Connection(QN_ENDPOINT)
    const provider = new AnchorProvider(connection, wallet, {})
    setProvider(provider)
    const program = new Program(IDL, PROGRAM_ID)
    // ... your code
    // e.g. await program.methods.yourMethod(YOUR_PARAMETERS).accounts({YOUR_ACCOUNTS}).rpc();
}

Due to the program instructions and account embedded in the IDL, the code editor is able to automatically detect callable programs via program.methods... A type guard is put in place to ensure all the necessary accounts are passed into the instruction. See Intro to client-side Anchor development

Improving IDL Discoverability

The majority of programs use anchor today, and for non-anchor frameworks, codama coupled with verified builds can fill the gaps. These IDLs provides efficient reading and a starting point for calling programs with declare_program!() but they are currently not discoverable. It is not yet ascertained whether IDLs can cover all possibility on programs, however the majority can be covered.

Current solutions being proposed to improved discoverability has to satisfy the following requirements:

• Must allow a generalized reading from the network.

• There should be a significant reduction in time taken to understand how to CPI into other programs.

• Program discovery should be as frictionless as possible.

Conclusion

We have been able to explore what IDLs are and their numerous benefits. We have also examined IDLs on anchor and non-anchor frameworks and how developers can utilize IDLs to improve program composability.

However existing infrastructure do not have versioned IDLs based on deployment time which makes it difficult for developers concerned with consistent data integration to understand what IDL to use at what slot.

The benefits to improving IDL discoverability is huge; developers can easily CPI into programs which can expand composability within the general developer landscape, unlocking new use cases that may have not been thought of.

Discoverable IDLs also provide more insight into security deficiencies as even seasoned developers on Solana tends to secure their program through obscurity which can often times prove to be a bug rather than a feature.

Over the coming years, IDLs are beginning to take a place within the spotlight as targets for more investment and development to enable building on Solana a more framework and language agnostic process, therefore accelerating developer experience across all development landscapes on Solana.

Additional Resources/Further Reading

• Anchor
• Metaplex shank
• Intro to client-side Anchor development