getProgramAccounts Solana Developer Guide

Learn how to fetch every account owned by a Solana program with getProgramAccounts — dataSize and memcmp filters, reading a program IDL to find byte offsets, and cutting a 10 GB scan down to a few kilobytes with changedSinceSlot and dataSlice.

Reference documentation: getProgramAccounts

Before We Begin

At this stage, you should have mastered fetching single accounts with getAccountInfo, and up to 255 accounts with getMultipleAccounts. The next step is learning to fetch ALL the Solana accounts belonging to a program, instead of just one or some.

Our free API keys are sufficient to complete this guide. You can get one by logging in to FluxRPC, navigating to the API Keys page, and selecting "Create API Key".

There is no need to copy-paste it — just navigate back to this page while logged in, and we will automatically use the key you select.

In this guide, we will accomplish the following:

Discuss when getProgramAccounts is useful, and its limitations
Deal with missing and out-of-date data
Basic optimization techniques
Some more advanced optimizations

We will provide examples in Python, TypeScript, Go, and cURL.

Why Fetch Every Account in a Solana Program?

Fetching all the accounts owned by a Solana program can indeed be a lot of data (over 10 GB for some programs), but there are many situations where this is useful:

Building a local DB (for example) of all Meteora pools, then keeping it up to date with a streaming protocol, instead of making repeated RPC requests
Searching for accounts that match certain criteria
Storing data for later analysis

Since FluxRPC charges based on the bandwidth you use, we are going to spend extra time on optimization in this guide.

Programs You Cannot Query on FluxRPC

Before we continue, there are some Solana programs that you cannot run getProgramAccounts against on FluxRPC, due to their size. These are:

The System program
The Metadata program
Both Token and Token-2022 programs
Address Lookup Table program

We're working on enabling getProgramAccounts on these programs eventually. Although be advised that running it in these cases would expend the bandwidth in your account very quickly!

Missing Data

Unlike getAccountInfo or getMultipleAccounts where you are attempting to fetch specific public keys, here you are requesting everything. So you will receive an array of responses, without any null entries. If there are no accounts to return, getProgramAccounts will simply return an empty array.

If you make an error in the public key of the program in your request, the call will also return an empty set. In some cases, you might want to catch this mistake and return an error — in this case you could follow up your getProgramAccounts call that returned an empty set with a getAccountInfo request for the program public key itself. If it returns null, your getProgramAccounts call was for a program that does not exist on-chain.

How to Use getProgramAccounts

A getProgramAccounts RPC call has more parameters available than any RPC call we have used so far.

The response is most similar to getMultipleAccounts. You will receive an array of accounts — often a very large array of accounts. In the simplest case, it looks like the below, which fetches all Fluxbeam pools (some ~800 megabytes). There's no need to run this RPC call, we'll provide more efficient and interesting examples later.

curl "https://eu.fluxrpc.com?key=<Your-API-Key>" -s -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "jsonrpc": "2.0",
    "id": 1,
    "method": "getProgramAccounts",
    "params": [
      "FLUXubRmkEi2q6K3Y9kBPg9248ggaZVsoSFhtJHSrm1X",
      {
        "encoding": "base64"
      }
    ]
  }'

However, unless you really need all the accounts that belong to a program, it's generally more sensible to apply filters to limit the number of accounts returned (and therefore vastly reduce the bandwidth usage!).

For getProgramAccounts, it only makes sense to use base64 encoding. Due to the large amounts of data it can return, other encoding schemes are not really useful (although FluxRPC supports them).

The following example is much more practical — it searches for and returns the data in the FLUXB-USDT Fluxbeam pool account. We'll explain how this works in a moment, but for now, here it is:

curl "https://eu.fluxrpc.com?key=<Your-API-Key>" -s -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "jsonrpc": "2.0",
    "id": 1,
    "method": "getProgramAccounts",
    "params": [
      "FLUXubRmkEi2q6K3Y9kBPg9248ggaZVsoSFhtJHSrm1X",
      {
        "encoding": "base64",
        "filters": [
          { "dataSize": 324 },
          {
            "memcmp": {
              "offset": 131,
              "bytes": "FLUXBmPhT3Fd1EDVFdg46YREqHBeNypn1h4EbnTzWERX"
            }
          },
          {
            "memcmp": {
              "offset": 163,
              "bytes": "Es9vMFrzaCERmJfrF4H2FYD4KCoNkY11McCe8BenwNYB"
            }
          }
        ]
      }
    ]
  }'

Very large result sets
If a filtered query still returns more accounts than you want to handle in one response, FluxRPC supports a cursorFilter parameter for paginating getProgramAccounts. See the getProgramAccounts reference for the exact shape.

How to Use dataSize and memcmp Filters

In the example above, we use two filters: dataSize and memcmp.

dataSize: will only return accounts that have an exact match for the number of bytes they store. This is one way to limit your request to a certain type of account.
memcmp: This filter checks the bytes, starting at a given offset, for an exact match to the bytes given (base58 encoded by default). In other words, you're specifying a position in the account data, and instructing the RPC to only return accounts that contain specific data at that position.

We know that Fluxbeam pools are always 324 bytes in size, because we can open a few up on any blockchain explorer and check.

The next step is a bit more difficult, and requires that we introduce program IDLs. IDLs are essentially guides to the structure of transactions that can interact with a program, as well as the structure of the data that is stored in the accounts owned by that program. Not every program on Solana publishes an IDL, but Fluxbeam does! You can find it here.

Looking at that IDL, we can see the structure of V1 swap accounts is defined here:

{
  "accounts": [
    {
      "name": "SwapV1",
      "type": {
        "kind": "struct",
        "fields": [
          {
            "name": "isInitialized",
            "type": "bool"
          },
          {
            "name": "bumpSeed",
            "type": "u8"
          },
          {
            "name": "tokenProgramId",
            "type": "publicKey"
          },
          {
            "name": "tokenA",
            "type": "publicKey"
          },
          {
            "name": "tokenB",
            "type": "publicKey"
          },
          {
            "name": "poolMint",
            "type": "publicKey"
          },
          {
            "name": "tokenAMint",
            "type": "publicKey"
          },
          {
            "name": "tokenBMint",
            "type": "publicKey"
          },
          {
            "name": "poolFeeAccount",
            "type": "publicKey"
          },
          {
            "name": "fees",
            "type": {
              "defined": "Fees"
            }
          },
          {
            "name": "swapCurve",
            "type": {
              "defined": "SwapCurve"
            }
          }
        ]
      }
    }
  ]
}

The discriminator is inferred — all instructions and accounts on Solana are identified by a discriminator. In the case of Fluxbeam, we use 1-byte discriminators. Some programs use 8-byte discriminators, we'll show an example of that later on — you can tell which it is by looking at the program IDL.

All the 8-bit integers and booleans above are stored in 1 byte on-chain, and all public keys are 32 bytes, so we can tally up the amounts from left to right and figure out how to slice the data we want out of the account:

Byte offset	Size	Field
0	1	discriminator (inferred)
1	1	`isInitialized` (bool)
2	1	`bumpSeed` (u8)
3	32	`tokenProgramId`
35	32	`tokenA`
67	32	`tokenB`
99	32	`poolMint`
131	32	`tokenAMint`
163	32	`tokenBMint`
195	32	`poolFeeAccount`

In our case, we are interested in finding the pool (if it exists) that has the Fluxbot token (FLUXBmPhT3Fd1EDVFdg46YREqHBeNypn1h4EbnTzWERX) as mint A and USDT (Es9vMFrzaCERmJfrF4H2FYD4KCoNkY11McCe8BenwNYB) as mint B. From the table, mint A starts at byte index 131, and mint B at byte index 163 — which is exactly what the filters in the previous request match against.

As another example, if we wanted to adjust our RPC call so that we fetch all pools containing the Fluxbot token, we would have to combine the following two requests — because Fluxbot could be token A or token B!

First request:

curl "https://eu.fluxrpc.com?key=<Your-API-Key>" -s -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "jsonrpc": "2.0",
    "id": 1,
    "method": "getProgramAccounts",
    "params": [
      "FLUXubRmkEi2q6K3Y9kBPg9248ggaZVsoSFhtJHSrm1X",
      {
        "encoding": "base64",
        "filters": [
          { "dataSize": 324 },
          {
            "memcmp": {
              "offset": 131,
              "bytes": "FLUXBmPhT3Fd1EDVFdg46YREqHBeNypn1h4EbnTzWERX"
            }
          }
        ]
      }
    ]
  }'

Second request:

curl "https://eu.fluxrpc.com?key=<Your-API-Key>" -s -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "jsonrpc": "2.0",
    "id": 1,
    "method": "getProgramAccounts",
    "params": [
      "FLUXubRmkEi2q6K3Y9kBPg9248ggaZVsoSFhtJHSrm1X",
      {
        "encoding": "base64",
        "filters": [
          { "dataSize": 324 },
          {
            "memcmp": {
              "offset": 163,
              "bytes": "FLUXBmPhT3Fd1EDVFdg46YREqHBeNypn1h4EbnTzWERX"
            }
          }
        ]
      }
    ]
  }'

Interlude

Let's take a short break to review the significance of what we've just done:

We can run getProgramAccounts on the whole Fluxbeam program and get all the pools, but this uses quite a bit of bandwidth (around 800 megabytes)
We can search for a specific pool on a DEX, by querying raw account data (uses under 700 bytes of bandwidth)
We can return all pools for a token, by querying raw account data (uses around 650 bytes per pool found)

That's not bad, but it's still not a useful program. We could build on this and optimize so that we fetch the best price of arbitrary tokens on-chain. For this example, we'll start with the Meteora WSOL-USDC pool.

How Does Meteora Store Account Data?

Meteora publishes its DLMM SDK. We can grab their IDL from that repository. It's of course perfectly valid to use their SDK, but we'll avoid doing so in order to show how everything works at depth, with a minimum of dependencies.

The Meteora IDL is also integrated into Solscan, which will save us some effort!

First, looking at their IDL, we see from lines like this that they use 8-byte discriminators:

{
  "discriminator": [
    181,
    157,
    89,
    67,
    143,
    182,
    52,
    72
  ]
}

On Solana, discriminators can be 1 or 8 bytes. In the example of the Fluxbeam program, it was 1 byte. In Meteora DLMM it's 8 bytes — notice how the IDLs are different! This will help you spot this difference in the future.

We can select the table view in Solscan to save us some effort counting bytes from the IDL. Expanding all data fields, we can see here that static parameters occupy 32 bytes, and variable parameters occupy the next 32 bytes. Then, there are 16 bytes of other fields before "Token X Mint" and "Token Y Mint". That's 88 bytes when we include the 8-byte discriminator! So we could fetch all the WSOL-USDC pools on Meteora as follows (this will return more than one pool!):

curl "https://eu.fluxrpc.com?key=<Your-API-Key>" -s -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "jsonrpc": "2.0",
    "id": 1,
    "method": "getProgramAccounts",
    "params": [
      "LBUZKhRxPF3XUpBCjp4YzTKgLccjZhTSDM9YuVaPwxo",
      {
        "encoding": "base64",
        "filters": [
          { "dataSize": 904 },
          {
            "memcmp": {
              "offset": 88,
              "bytes": "So11111111111111111111111111111111111111112"
            }
          },
          {
            "memcmp": {
              "offset": 120,
              "bytes": "EPjFWdd5AufqSSqeM2qN1xzybapC8G4wEGGkZwyTDt1v"
            }
          }
        ]
      }
    ]
  }'

Skipping Dead Pools with changedSinceSlot

Now that we can fetch all the WSOL-USDC pools on Meteora, we'll probably want to filter out the dead ones, then find the one with the best price!

One shortcut we can take to reduce the amount of data we'll need to fetch is to use changedSinceSlot. Specifying this parameter will make FluxRPC only return accounts that have changed since the slot you specify. Setting this to a recent slot (say, current slot - 1000) is sufficient to remove a lot of dead pools, and reduce the cost of this RPC call:

curl "https://eu.fluxrpc.com?key=<Your-API-Key>" -s -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "jsonrpc": "2.0",
    "id": 1,
    "method": "getProgramAccounts",
    "params": [
      "LBUZKhRxPF3XUpBCjp4YzTKgLccjZhTSDM9YuVaPwxo",
      {
        "encoding": "base64",
        "changedSinceSlot": 435696000,
        "filters": [
          { "dataSize": 904 },
          {
            "memcmp": {
              "offset": 88,
              "bytes": "So11111111111111111111111111111111111111112"
            }
          },
          {
            "memcmp": {
              "offset": 120,
              "bytes": "EPjFWdd5AufqSSqeM2qN1xzybapC8G4wEGGkZwyTDt1v"
            }
          }
        ]
      }
    ]
  }'

Slicing Out Only the Bytes We Need

This is not a bad result, and reasonably usable. However, ask an engineer if a glass is half empty or half full, and we'll tell you it's twice as big as it needs to be. We can still do significantly better.

Like we did in our getMultipleAccounts guide, we can use byte slicing to only return the data we need, and nothing else. There is a lot of data stored in a Meteora pool, and fully covering all of it is beyond the scope of this guide. The Meteora docs have more information if you're curious.

To keep things simple we will focus on one fact: we can calculate the price using two items: the bin step, and the active ID. We'll do this with the following formula:

price = 1000.0 * pow(1.0 + (binStep / 10000.0), activeID)

That is to say, the price is 1000 times (1 + binStep/10000) raised to the power of the active ID. Conveniently, the active ID is an int32 stored in bytes 76-80 (a signed integer!), and the bin step is a uint16 stored in bytes 80-82 — clearly the engineers at Meteora had this use case in mind when they placed these next to each other (thanks!). So to calculate the price of a pool, we can slice out only bytes 76-82 like so:

Heads up
This will return more slowly than the RPC calls we've made thus far.

curl "https://eu.fluxrpc.com?key=<Your-API-Key>" -s -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "jsonrpc": "2.0",
    "id": 1,
    "method": "getProgramAccounts",
    "params": [
      "LBUZKhRxPF3XUpBCjp4YzTKgLccjZhTSDM9YuVaPwxo",
      {
        "encoding": "base64",
        "changedSinceSlot": 435696000,
        "dataSlice": { "offset": 76, "length": 6 },
        "filters": [
          { "dataSize": 904 },
          {
            "memcmp": {
              "offset": 88,
              "bytes": "So11111111111111111111111111111111111111112"
            }
          },
          {
            "memcmp": {
              "offset": 120,
              "bytes": "EPjFWdd5AufqSSqeM2qN1xzybapC8G4wEGGkZwyTDt1v"
            }
          }
        ]
      }
    ]
  }'

Now, this is pretty well optimized! At the time of writing, this returns under 5 kilobytes of data in the result, compared to the 10+ gigabytes stored in all the Meteora pools. Again, FluxRPC charges by byte of bandwidth you use, so it's worth it for you to build efficiently like this.

Building the Price Program

With our RPC call optimized, it only remains to build a program around it:

import requests
import json
import base64

APIKey = "<Your-API-Key>"
RPCRegion = "eu"

def fetchMeteoraWSOLPools():
    headers = {'content-type': 'application/json'}
    url = "https://" + RPCRegion + ".fluxrpc.com?key=" + APIKey
    data = {
        "jsonrpc": "2.0",
        "id": 1,
        "method": "getProgramAccounts",
        "params": [
            "LBUZKhRxPF3XUpBCjp4YzTKgLccjZhTSDM9YuVaPwxo",
            {
                "encoding": "base64",
                "changedSinceSlot": 435696000,
                # Only return the 6 bytes holding the active ID and the bin step
                "dataSlice": {"offset": 76, "length": 6},
                "filters": [
                    {"dataSize": 904},
                    {"memcmp": {"offset": 88, "bytes": "So11111111111111111111111111111111111111112"}},
                    {"memcmp": {"offset": 120, "bytes": "EPjFWdd5AufqSSqeM2qN1xzybapC8G4wEGGkZwyTDt1v"}}
                ]
            }
        ]
    }
    response = requests.post(url, data=json.dumps(data), headers=headers)
    print("This RPC call consumed: " + str(len(response.content)) + " bytes!")
    return response

response = fetchMeteoraWSOLPools()
if response.status_code != 200:
    # Handle HTTP errors from the RPC.
    raise Exception("RPC returned HTTP error code " + str(response.status_code))
responseJSON = json.loads(response.text)
# print(responseJSON)  # Print the raw data from getProgramAccounts, uncomment if you want to see it

results = responseJSON["result"]  # All the results are in the result object
prices = []  # We'll store prices here for analysis once we're done fetching them

for account in results:
    thisPubkey = account["pubkey"]
    try:
        # Decode the data slice into bytes
        priceBytes = base64.b64decode(account["account"]["data"][0])
        # First 4 bytes are the activeID, a SIGNED integer
        activeID = int.from_bytes(priceBytes[0:4], byteorder='little', signed=True)
        # Bin step is an unsigned integer
        binStep = int.from_bytes(priceBytes[4:6], byteorder='little')
        # Calculate WSOL price for this pool based on active ID and bin step
        thisPrice = 1000.0 * pow(1.0 + (binStep / 10000.0), activeID)
        print("Pool with public key " + thisPubkey + " has a WSOL price of: " + str(thisPrice))
        prices.append(thisPrice)  # Store the price to later determine which price is the highest
    except Exception:
        # Something went wrong, abort with exception
        raise Exception("Failed to process pool with public key: " + thisPubkey)

highest_index = prices.index(max(prices))  # Find the index of the highest price
print("The highest SOL price found on Meteora was " + str(prices[highest_index]) +
      " at pool with public key: " + str(results[highest_index]["pubkey"]))

A word of caution
There are some limitations to this program — namely, it doesn't check how much liquidity is in the pool. So don't go using this to try and make an arbitrage bot without accounting for that! You'll likely get destroyed by slippage, and probably other things besides.

(Still) Further Optimizations

If your program is making a lot of getProgramAccounts calls, or needs an answer very quickly, the typical approach is to fetch the entire program, and store it in a local database. Then keep that database up to date using Yellowstone gRPC streaming. With this approach, you can build a much faster application that makes fewer RPC calls, because it has the data it needs locally.

It's a bit more complex to build than anything we've discussed so far. Luckily, we've built something like that (but better!) for you. Lantern is an application that handles all of that complexity for you — you just tell it what program to store, and it will take care of keeping it up-to-date for you. Then you just point your RPC calls to localhost!

Not only is this more convenient, but the underlying protocol we use to update Lantern's cache is much more efficient than Yellowstone gRPC. So you save even more bandwidth.

Next Steps

If you skipped ahead, the two guides this one builds on are the getAccountInfo guide — response fields, encoding, and account data parsing — and the getMultipleAccounts guide, which introduces dataSlice and batching. When you're ready to serve these queries locally, get started with Lantern.

← All developer guides