DEV Community: Justin Weiss

Ruby enumerables considered helpful

Justin Weiss — Thu, 12 Oct 2023 19:36:12 +0000

Ruby's Enumerable methods help you make powerful code simple — by filtering, transforming, and processing data like the best engineers do. These methods are available on Arrays, Hashes, and many (many) other objects, and similarly-named methods are available on even more. If you don't already know these methods well, then the most valuable time you can spend in Ruby is on mastering them.

But not all data can take advantage of Enumerable methods, at least not directly. What if you don't have an Array or Hash? Can you still use the Enumerable methods? With all the time you spent becoming an Enumerable master, wouldn't it be nice if you could treat more things like Enumerables — like lists of items across multiple pages, or even items that slowly stream in from an API over time?

Ruby's Enumerator class is exactly what you need to do this. It's not the easiest thing to understand, but a few examples can show you how to use it in your own applications.

Automatic pagination

Sometimes your data seems like a natural fit for an Enumerable, but it doesn't quite work. What if you're using an API that paginates? The best you can get is an Enumerable per page, which makes it annoying to work with multiple pages at once. You could fetch all the pages and then flatten the list of lists… But now you're always fetching all the data (even if you don't need it) and you have to wait for every page to load before you can do anything.

Instead, wrap your API calls in an Enumerator:

def paginated_list(initial_url, params = {})
  url = initial_url

  Enumerator.new do |yielder|
    # ...
  end
end

This returns an Enumerator, an object that can be used like an Enumerable, responding to all of the core Enumerable methods you know and love: map, filter, first, and all the rest.

When you call a method like map on that Enumerator, that method will ask for the next object as it needs it. When asked, the body of the Enumerator yields the next object using the yielder:

def paginated_list(initial_url, params = {})
  url = initial_url

  Enumerator.new do |yielder|
    loop do
      response = get(url, params)

      body = response.body

      # Yield each result to the caller
      body["results"].each { |result| yielder.yield(result) }

      break if last_page?(body)

      # get ready to fetch the next page
      url = next_page_url(body)
    end
  end
end

This Enumerator makes the request for the first page. Then, any time a record is needed from the first page, the Enumerator yields it to the caller. When you run out of records on the first page, the Enumerator fetches the next page and so on until you run out. If you don't need the next page, it won't fetch it.

How does this look for the caller?

paginated_list("/things").filter { |t| t["selected"] == "true" }.sort_by { |t| t["name"].downcase }

Easy — you don't even have to care that the data spans multiple pages across multiple API requests. From the caller's point of view, it's just an ordinary list.

Save a block for later

While working on the Aha! AI writing assistant, we had two different ways of working with AI responses. For development and testing, the response should be returned all at once because it is easier to work with. For a user, though, it feels better for the response to arrive incrementally — streaming in so you can know early on if you're getting the result you wanted.

Everything else was exactly the same, except for how the response was dealt with. In one case, a string was returned. In another, bits of a string were yielded to the caller as they came in:

client = Client.new(stream: stream?)

# Burst response
response = client.get(query: params[:query], context: params[:context] ...)
render json: { text: response }

# Stream response

# all kinds of streaming setup work
client.get(query: params[:query], context: params[:context] ...) do |chunk|
  # stream chunk to browser
end
# all kinds of streaming teardown work, stream error handling, etc.

This is fine. But the API method calls look exactly the same, except one takes a block and the other doesn't. And for streaming, you have extra work to do before and after the method call.

It's hard to untangle this. You could pass around lambdas, get could ignore a lambda parameter in stream mode, you could wrap the setup and teardown in its own block:

client = Client.new(stream: stream?)

response = streaming_setup(stream?) do
  client.get(query: params[:query], context: params[:context]) do |chunk|
    # stream chunk to browser
  end
end

render json: { text: response } unless stream?

Maybe this will work. Again, it's fine. But this is Ruby, so we can do better than fine.

Object#to_enum is a method available on any object. You give it a method name and arguments and it returns an Enumerator. Whenever that Enumerator is enumerated, it will call the method you gave it, on the object you called it on, with the arguments you gave it, and it will pass along anything your method yields as the next value of the Enumerator.

At first, it's hard to see how this is useful. In practice, this means you can capture the arguments a method was called with, but defer needing the block the method yields to until the part that processes your data needs it.

Using to_enum, the caller can now look like this:

def show
  client = Client.new(stream: stream?)

  response = client.get(query: params[:query], context: params[:context])

  if stream?
    stream_response response
  else
    render json: { text: response }
  end
end

def stream_response(response)
  # all kinds of setup work
  response.each do |chunk|
    # stream each chunk to browser as it arrives
  end
  # all kinds of teardown work, stream error handling, etc.
end

client.get now returns either a string or an Enumerator. If it's an Enumerator, it will yield each part of the response as it's received. Because this code no longer needs the block at the same time as the arguments, all of the logic around streaming a response can go somewhere else. This is much less tangled up than the last version. How is this possible? By using to_enum in the client when it's not given a block:

class Client

  def initialize(stream:)
    @stream = stream
  end

  def get(query:, context:, &block)
    if @stream
      get_stream(query, context, &block)
    else
      get_burst(query, context)
    end
  end

  def get_stream(query, context, &block)
    return to_enum(:get_stream, query, context) unless block_given?

    some_api_request do |chunk|
      # ...
      yield chunk
    end
  end

  def get_burst(query, context)
    # nothing interesting here, just a basic request and response
    response
  end
end

If the client is in streaming mode and isn't given a block, get_stream returns an Enumerator that calls get_stream with a block. The yielded chunk is passed back through to the Enumerator when the caller, or any other code that the caller hands the Enumerator to, uses it.

This pattern — decoupling the place where you need the arguments from the place where you have the block — becomes even more powerful the further apart those places are separated. It's a nice pattern to understand, and it's possible because of Enumerators.

Enumerators can be tricky at first. It's hard to see the use when you read about them in the docs. But they have an amazing ability to wrap Ruby code into objects that take advantage of the full power of Enumerable methods. It's valuable to use the knowledge you already have in more places, and Enumerators will help you do exactly that.

Building a composable query generator for GraphQL

Justin Weiss — Fri, 13 Jan 2023 18:15:56 +0000

In a lot of newer projects, we use our GraphQL API. This is the same API you use when you're building Aha! Develop extensions. GraphQL has given us a more consistent, more flexible, and often more efficient way to access the data inside an Aha! account.

GraphQL queries are just strings:

const query = `{
  feature(id: "DEMO-29") {
    id
    name
  }
}`;

const response = await fetch("/api/v2/graphql", {
  "headers": {
    "Content-Type": "application/json",
  },
  "method": "POST",
  "body": JSON.stringify({ query }),
});

Strings are easy to start with and work well for simple, mostly static queries. But what about something more complex? What if you wanted the name of the person assigned to each feature but not when you're looking at a list of your own features? What if only some teams used sprints and you didn't want sprint information if a team didn't use them? How could you easily make wildly customizable views and only fetch the data you needed for that specific view?

Simple interpolation is one option:

const sprintQuery = sprintsEnabled ? `{
  sprint {
    id
    name
    startDate
    duration
  }
}` : "";

const query = `{
  features {
    id
    name
    ${ sprintQuery }
  }
}`;

But this can quickly get out of hand. You need to know in advance how a query can be modified so you can be sure to leave room for strings to be interpolated inside of them. Handling optional arguments can be a pain. So if strings aren't the right option for very dynamic queries, what is?

The next step beyond strings

There's a common solution to this problem: Store the information you need to make your final decisions in a more raw form that you can look at. Keep them in a structure — an introspectable form — until you're ready to generate a result.

What does that look like, though?

Say a part of your code wanted a few fields from a Feature API, "id", and "name". Some other code wants the reference number too. Instead of building a query as a string and interpolating fields into it, like this:

const query = `{
  features {
    id
    name
    ${ maybeReferenceNumber }
  }
}`;

You could keep a list of fields:

const fields = ["id", "name"];

Other parts of your system could add to it:

fields.push("referenceNumber");

And when you generate your query, the query has all of the information it needs right at hand:

const query = `{
  features {
    ${ fields.join("\n") }
  }
}`;

This seems like a minor change, and it is, but it's powerful. An object, which is easy to work with, keeps track of the decisions you've made up to this point. Your query generation is simple — it just does what the object tells it to do. Here, that's turning an array of names into fields in the query. This opens up a lot of possibilities but first we can clean it up.

How to store the query state

If you've ever used Rails, you'll know that it's great at building queries over time. You call methods to build up a query and only fire it off when you're done with it:

query = account.features
query = query.where(project_id: project.id) if project
query = query.where(release_id: release.id) if release
# ...

I've been surprised that more languages or frameworks haven't been influenced by this query building. To me, it's one of the best parts of Rails.

You can build something like this with GraphQL, though, keeping your decisions in an object and generating a query at the last minute. Imagine a Query class. We'll keep it simple to start, just holding a query type and a list of fields:

class Query {
  constructor(type) {
    this.type = type;
    this.fields = new Set();
  }
}

Next, you can implement a simple "select" method to select ID and name:

class Query {
  // ...
  select(fields) {
    fields.forEach(field => {
      this.fields.add(attr);
    });

    return this;
  }
}

let query = new Query("features");
query.select(["id", "name"]);

You can pass that query object around and other functions can add to that query object:

query.select(["referenceNumber"]);

Now, when you need it, the Query object can generate a query string. Just like the example above:

class Query {
  // ...
  toString() {
    return `{
      ${ this.type } {
        ${ Array.from(this.fields).join("\n") }
      }
    }`;
  }
}

Once you have this string, you can hand it to whichever GraphQL library you're using as if you wrote it yourself.

With that simple pattern, you can go further. You can add arguments to filter the results:

class Query {
  constructor(type) {
    // ...
    this.filters = {};
  }

  // ...
  where(filters = {}) {
    this.filters = { ...this.filters, ...filters };
    return this;
  }

  processFilters(filters) {
    // Very basic conversion, can be improved as necessary
    return Object.keys(filters)
      .map(k => `${k}: ${JSON.stringify(filters[k])}`)
      .join(', ');
  }

  toString() {
    let args = '';
    if (Object.keys(this.filters).length > 0) {
      args = `(filters: {${this.processFilters(this.filters)}})`;
    }

    return `{
      ${ this.type }${ args } {
        ${ Array.from(this.fields).join("\n") }
      }
    }`;
  }
}

let query = new Query("features");
query.select(["id", "name"]).where({ assigneeId: "12345" }).toString();

You can add subqueries to select information from child objects:

class Query {
  constructor(type) {
    // ...
    this.subqueries = [];
  }

  // ...
  merge(subquery) {
    this.subqueries.push(subquery);
    return this;
  }

  toString(options = {}) {
    const { asSubquery } = options;

    let args = '';
    if (Object.keys(this.filters).length > 0) {
      args = `(filters: {${this.processFilters(this.filters)}})`;
    }

    const subqueryString = `${this.type}${args} {
        ${Array.from(this.fields).join('\n')}
        ${this.subqueries.map(s => s.toString({ asSubquery: true })).join('\n')}
      }`;

    if (asSubquery) {
      return subqueryString;
    } else {
      return `{
        ${subqueryString}
      }`;
    }
  }}

And now you can build arbitrarily complicated queries based on whatever state you have:

let query = new Query('features');
query.select(['id', 'name']).where({ assigneeId: '12345' });

if (projectId) {
  query.where({ projectId });
}

if (includeRequirements) {
  let requirementsQuery = new Query('requirements');
  requirementsQuery.select(['id', 'name', 'position']);

  query.merge(requirementsQuery);
}

query.toString();

From now on, any time you want to change a query based on information you have someplace else, it's as easy as calling a method.

What's next?

This query generator is simple and is still missing a lot, but the core idea is easy to extend to fit your API:

You can add functionality for sorting, passing query arguments, pagination, etc.
You can have it generate GraphQL fragments and variables instead of a single string.
You can add another layer. For example, have your query framework generate Apollo documents instead of strings and you can get better editor integration and error messages.
You can create a mutation builder, tracking changes you make to your JavaScript objects and generating mutations based on those changes.

We've done all of those things, and found a lot of value in a flexible generator like this. Building a query generator ourselves also allowed us to integrate it more closely with other parts of our application framework. It's a nice simple base we could build off of easily.

And once you have this base — an introspectable object that can generate the strings you need — your code no longer needs to worry about the actual queries. It's just responsible for the decision-making and the generator does the rest.

Aha! is happy, healthy, and hiring. Join us!

We are challenged to do great work and have fun doing it. If you want to build lovable software with this talented and growing team, apply to an open role.