Yellowstone gRPC - Compressed Filters

Update Yellowstone account subscriptions without re-sending the full key list using a compressed Cuckoo filter on FluxRPC.

Previously, changing a Yellowstone subscription involved re-sending the full list of public keys you subscribed to.

We've added a compressed filter (a Cuckoo filter) that allows you to efficiently update your existing Yellowstone subscription by adding and removing public keys.

This works by creating a small (but unique) fingerprint for each public key in your subscription. These fingerprints take up about 90% less space than storing the public keys themselves.

This lets us support Yellowstone subscriptions to a larger number of concurrent accounts, without compromising on speed.

Application-side, you can now update your subscription without re-sending the entire public key list.

Using Compressed Yellowstone Filters

First, you store the public keys that you wish to subscribe to:

Store public keys Go

import (
    "fmt"
    "github.com/Gealber/cuckoo"
    "github.com/mr-tron/base58"
    pb "yourmodule/proto"
)
func buildCuckooFilter(pubkeys []string) (*pb.CuckooFilter, error) {
    capacity := max(uint(len(pubkeys)), 10_000)
    cf := cuckoo.New(capacity)
    for _, pk := range pubkeys {
        raw, err := base58.Decode(pk)
        if err != nil || len(raw) != 32 {
            return nil, fmt.Errorf("invalid pubkey %q", pk)
        }
        if !cf.Insert(raw) {
            return nil, fmt.Errorf("filter full, increase capacity")
        }
    }
    data := cf.Bytes()
    return &pb.CuckooFilter{
        Data:             data,
        BucketCount:      uint32(len(data) / 8),
        EntriesPerBucket: 4,
        FingerprintBits:  16,
        HashSeed:         cf.Seed(),
        HashAlgorithm:    pb.CuckooHashAlgorithm_SIP_HASH,
    }, nil
}
tracked := []string{
    "EPjFWdd5AufqSSqeM2qN1xzybapC8G4wEGGkZwyTDt1v", // USDC mint
    "TokenkegQfeZyiNwAJbNbGKPFXCWuBvf9Ss623VQ5DA",  // Token Program
    // ... thousands more
}
// Exact set for client-side false-positive filtering
exact := make(map[string]struct{}, len(tracked))
for _, pk := range tracked {
    exact[pk] = struct{}{}
}
filter, err := buildCuckooFilter(tracked)
if err != nil {
    return err
}

Then you can update your subscription like this:

Update subscription Go

import (
    "context"
    "crypto/tls"
    "google.golang.org/grpc"
    "google.golang.org/grpc/credentials"
    "google.golang.org/grpc/metadata"
    pb "yourmodule/proto"
)
conn, err := grpc.NewClient("https://yellowstone.eu.fluxrpc.com",
    grpc.WithTransportCredentials(credentials.NewTLS(&tls.Config{})))
if err != nil {
    return err
}
defer conn.Close()
ctx := metadata.AppendToOutgoingContext(context.Background(), "x-token", "YOUR_FLUX_RPC_API_KEY")
stream, err := pb.NewGeyserClient(conn).Subscribe(ctx)
if err != nil {
    return err
}
commitment := pb.CommitmentLevel_PROCESSED
req := &pb.SubscribeRequest{
    Accounts: map[string]*pb.SubscribeRequestFilterAccounts{
        "tracked_accounts": {CuckooAccountsFilter: filter},
    },
    Commitment: &commitment,
}
if err := stream.Send(req); err != nil {
    return err
}

To add a new account to your subscription at any time:

Add pubkey Go

// Keep exact set in memory (for false-positive filtering)
exact := make(map[string]struct{})
cf := cuckoo.New(10_000)
// Add new account
newPubkey := "So11111111111111111111111111111111111111112" // wSOL
exact[newPubkey] = struct{}{}
raw, _ := base58.Decode(newPubkey)
cf.Insert(raw)  // O(1) operation!
// Rebuild filter and send updated subscription
data := cf.Bytes()
filter := &pb.CuckooFilter{
    Data:             data,
    BucketCount:      uint32(len(data) / 8),
    EntriesPerBucket: 4,
    FingerprintBits:  16,
    HashSeed:         cf.Seed(),
    HashAlgorithm:    pb.CuckooHashAlgorithm_SIP_HASH,
}
commitment := pb.CommitmentLevel_PROCESSED
req := &pb.SubscribeRequest{
    Accounts: map[string]*pb.SubscribeRequestFilterAccounts{
        "tracked_accounts": {CuckooAccountsFilter: filter},
    },
    Commitment: &commitment,
}
stream.Send(req)  // Send on same stream — no reconnect!

To remove an account from your subscription:

Remove pubkey Go

// Remove account
oldPubkey := "EPjFWdd5AufqSSqeM2qN1xzybapC8G4wEGGkZwyTDt1v" // USDC
delete(exact, oldPubkey)
raw, _ := base58.Decode(oldPubkey)
cf.Delete(raw)  // O(1) operation!
// Send updated subscription (same as above)
stream.Send(req)

Finally, if you would like to check membership of a public key in your local set, you can:

Check membership Go

// When you receive an update, check against exact set
for {
    update := stream.Recv()
    acc := update.GetAccount()
    if acc == nil {
        continue
    }
    pubkey := base58.Encode(acc.Account.Pubkey)
    // First: check if it matches the filter (may have false positives)
    raw, _ := base58.Decode(pubkey)
    inFilter := cf.Lookup(raw)
    // Second: check exact set (definitive)
    _, inExact := exact[pubkey]
    if !inExact {
        if inFilter {
            log.Printf("⚠ FALSE POSITIVE: %s (dropped)", pubkey)
        }
        continue  // Drop it
    }
    // Exact match — process it
    log.Printf("✓ MATCH: %s", pubkey)
}

The Cuckoo filter is probabilistic, it can only tell you one of the following:

  1. The account is definitely not in the set
  2. The account might be in the set

However, the full list of accounts is stored client-side, so when performing this membership check, it will not return false positives. It will tell you with certainty if it is present or not.

Open the section overview for the broader context, live navigation, and interactive code examples.

Continue in this section

← Back to documentation home