DEV Community: Steve Coffman

#go

Steve Coffman — Wed, 07 Jan 2026 00:25:59 +0000

Authorization (authz) and GraphQL

Steve Coffman — Wed, 18 Dec 2024 19:50:20 +0000

Hi! I maintain gqlgen a popular GraphQL library for Go. I've noticed many people stumbling on GraphQL authentication regardless of programming language. This is partly because they don't know what the possibilities are, what the tradeoffs of those possibilities are, or even what to ask for advice about.

Hopefully, this article will clarify choices, and help you make informed decisions.

Securing dynamic data access is hard

GraphQL APIs are appealing because they are flexible, but this makes adding authorization difficult. In the REST world, you can (at a minimum) authorize individual endpoints, but in GraphQL, you have to find a way to authorize each query and mutation generically.

In addition to the client/server behavior changes, GraphQL also introduces features like federation that make it possible to deploy many different GraphQL services and expose them via a single unified API to clients. The big issue you face when building authorization in a distributed architecture is not having local access to all of the data required to make authorization decisions.

The correct solution is (as always) dependent on your expected scale and complexity budget.

Authorization models in GraphQL

Imagine a user/subject with an ID like id_1234567890 that makes an HTTP POST request with a GraphQL mutation to rename a District with ID of "47b3f42a-65c2-46b1-93d5-47ff4e92cf5b". That looks like this:

mutation {
  updateDistrict(input: {name: "Test", id: "47b3f42a-65c2-46b1-93d5-47ff4e92cf5b"}) {
    id
  }
}

How could you handle authorization (allow or deny) for this request in different authorization (authz) models?

Role-Based Access Control (RBAC)
- Only the subject’s role determines whether to allow or deny access. e.g. DevAdmin Role (“Can Edit District”)
Attribute-Based Access Control (ABAC)
- The user/subject, action, and resource are combined to determine whether to allow or deny access. e.g. userID + updateDistrict + district:ID would be allowed for users who have the Attribute of owner for the district:ID of 47b3f42a-65c2-46b1-93d5-47ff4e92cf5b

Policy-Based Access Control (PBAC)

Like ABAC but the rules are kept in policy documents. Attributes in rules can be on subject, object, or action.

[request_definition]
r = sub, obj, act

[policy_definition]
p = sub, obj, act

[policy_effect]
e = some(where (p.eft == allow))

[matchers]
m = r.sub == r.obj.Owner

Relationship-Based Access Control (ReBAC)
- Relationship-based access control (ReBAC) is both a subset of ABAC and a superset of RBAC. ReBAC builds a relationship graph between subjects and objects via relations. These relations include data ownership, parent-child relationships, groups, and hierarchies (or relation chains). Google’s Zanzibar proved this could be extremely low latency at a huge scale. Zanzibar’s relations are built out of tuples defined like:
```
tuple := (object, relation, user)
object := namespace:id
user :=  object | (object, relation)
relation := string
namespace := string
id := string
```
  For instance, "User id_1234567890 owns District 47b3f42a-65c2-46b1-93d5-47ff4e92cf5b" is represented as a tuple of <resource>#<relation>@<subject>like:
```
district:47b3f42a-65c2-46b1-93d5-47ff4e92cf5b#owner@user:id_1234567890
```
  The policy that operates on the graph of such tuples would always have both a check_permission and check_relation similar to:
```
allowed {
  ds.check_permission({
    "subject": {"id": input.user.id},
    "permission": "can-edit",
    "object": {"key": input.resource.district_id, "type": "district"},
  })
}

allowed {
  district = ds.object({"key": input.resource.district_id, "type": "district"})

  ds.check_relation({
    "subject": {"id": input.user.id},
    "relation": {"name": "manager_of", "type": "user"},
    "object": {"id": district.properties.owner_id},
  })
}
```

Graph-Based Access Control (GBAC)

Similar to how ReBAC builds a restricted graph, GBAC uses arbitrary queries of an unrestricted graph. For instance, in DGraph you can add schema directives with @auth rules that are arbitrary GraphQL queries:

mutation {
  updateDistrict @auth(
    query: { rule: """
        query($USER: String!) {
            queryDistrict {
                owner(filter: { username: { eq: $USER } }) {
                    __typename
                }
            }
        }"""})(input: {name: "Test", id: "47b3f42a-65c2-46b1-93d5-47ff4e92cf5b"}) {
    id
  }
}

Ad hoc ACLs or Centralized System?

Access Control List (ACL): An access control list (ACL) is a list of rules that specify which users or systems are granted or denied access to a particular object or system resource.

For our example mutation, would we want the access control list of rules for it to be unique (bespoke or ad hoc), or compose it from some standard rules? These are some common ones that might be standardized:

OpenAccess is an ACL that is a no-op: it marks that this resolver is open-access. (Add a comment if it’s not obvious why!)
IsLoggedIn checks if the request is made by some sort of authenticated user.
IsCurrentUser checks that the given user (target) and the current user (actor) are the same.
ActorHasPermission checks that the actor has a specified permission (TODO link).
IsManagedByActor checks that the current user (actor) manages the resource (target).
ValidatesSecret checks that the request includes the correct shared secret.

Roles vs. Scopes

Scopes are usually associated with API access. An API defines what scopes are available (what services it provides). For example, a user account management API might define scopes like read:user, create:user, update:user. These are the capabilities the API provides, but not necessarily what any given user can do. In the Role & Scope model, Roles are defined, and users are given a Role. Individual Scopes are associated with a given Role, combining all these elements together. For example, you might have:

Role: Audit, API: user_manager, Scopes: read:user
Role: Access Control, API:  user_manager, Scopes: create:user, update:user, read:user

So if a user, Alice has the Audit role, while Bob has the Access Control role, then a GraphQL query of users might have the scope read:user and allow them both, but a mutation to edit a user might have the scope update:user and only Bob should be allowed to make that mutation request.

Where to perform authorization in a GraphQL architecture?

There’s a great article on https://www.osohq.com/post/graphql-authorization that you should just read. Assuming that authentication (authn) has already happened and the user has some sort of authenticated identity token (by the way, this is an excellent article on the different token types), where should you perform authorization (authz)?

GraphQL resolver

Often people put code into the beginning of their resolver like this:

if !(acl.IsCurrentUser(ctx, userId) ||
    acl.ActorHasPermission(ctx, capabilities.CanChangeUserData, acl.UserScope(userId)) ||
    acl.IsManagedByActor(ctx, userId)) {
    // return an unauthorized error
}

Directives

You could add directives to your schema like in wundergraph or DGraph:

mutation @rbac(requireMatchAll: [superadmin]) {
  updateDistrict(input: {name: "Test", id: "47b3f42a-65c2-46b1-93d5-47ff4e92cf5b"}) {
    id
  }
}

Middleware

Middleware can decouple your authorization logic from your schema as with GraphQL Shield. Middleware authorization works best for rules that apply to your whole schema at once i.e. every query and resolver, since middleware doesn’t have more specific information for more complicated domain rules. Good middleware rules could be used to filter out invalid tokens, reject non-safelisted queries, or to use calculated query complexity scores and reject overly expensive queries (e.g. see compgen).
Data Access Layer

Since GraphQL can return partial results, authorization for reading data can be pushed below resolvers. If you are using PostgreSQL you can even use row and column-based access to push it all the way out of your app into your database! However, authorizing this deep is awkward:
- without request and user-specific variables
- for write access control
- if your data is in more than one place.
Federated Gateway

Like middleware, this is either a great or terrible place for authorization. If there is no way to bypass a federated gateway, and you have only a few simple rules without needing domain or application-specific information then it can greatly simplify things. However, for more complicated needs (e.g. ABAC) trying to do authorization at the gateway is an example of https://www.thoughtworks.com/radar/platforms/overambitious-api-gateways that encourages designs that are difficult to test and deploy. You can still use it like a middleware to filter out obvious bad actors and filter out invalid tokens, reject non-safelisted queries, or to use calculated query complexity scores and reject overly expensive queries
External Authorization System

Using Policy engines like SpiceDB, OpenFGA, ORY Keto, OpenPolicy Agent (OPA), let you put your

ReBAC rules in an external system and references them from your queries. The main benefit you get from the centralized relationships model is it makes it possible to manage authorization centrally. This means that development teams can create new applications and add new relationships without needing to update any application code.

However, the downside is that you are constraining your application to use a very specific data model and you need to design your application around that data store.

At a certain scale, the balance tips towards centralization.
Identity token claims

Tokens are great since HTTP is stateless. If you have a few roles (or claims), you can put them in the token, and your authorization logic is done. A claim of mutation:* and query:* would give full access, just as a role of admin would. However, cookies are headers, and you are limited in the total size of any one header, as well as the total size of all your headers together.

Microsoft specifically calls out Windows authentication (NTLM/Kerberos/Negotiate) as not supported on HTTP/2 due to HPACK performance issues that those large headers cause.

HTTP/2 sets a default 4K limit on all headers together. Beyond that, plan to set your relationship status to "It's Complicated" as you will need to negotiate increasing it with all your intervening infrastructure (load balancers, proxies, CDNs).
- nginx: link --> 8KB for 1 header max
- envoy (used in istio): link --> 60 KB for headers
- node.js: link --> 16KB for headers
- traefik: link --> re-uses go/http: 1MB for headers

Conclusion: Wait, what is your advice?

Start with RBAC, and use schema directives, codegen and identity tokens. Then you can grow out of it to more complicated setups.

Move as much authorization into the schema, so everything is more transparent and visible. It helps you to set good boundaries.

Parting Questions

What are your favorite authorization schema directives? What is your favorite token claim, attribute, policy, or role?

HTTP/2 streaming and you, HTTP/3 streaming and wheee!

Steve Coffman — Sat, 23 Nov 2024 19:59:21 +0000

HTTP/3 is used by 30.1% of all the websites.

HTTP/2 is used by 34.7% of all the websites.

Some methodologies put these higher, but we can safely say:

A large chunk of current Internet traffic already uses HTTP/3 today.

However, this change has been a quiet revolution in our industry.

HTTP/2 (and HTTP/3) changes the rules of the game for web development!

This is a summary of a number of other articles and my own experience building a jackbox clone using HTTP/3 Server Streaming using ConnectRPC (gRPC).

People have become long accustomed to HTTP/1.1 and the best practices associated that have grown up around it, but a lot of these are worth re-evaluating now that things have changed, and it’s important to understand why.

In HTTP/1.1 downloading multiple smaller files is slower than downloading a single file of the same combined size, because of the lookups, handshakes, and “queuing process” that happens.

In HTTP/1.1 each connection is a single unary request/response cycle.

Many Best Practices under HTTP/1.1 are actually counterproductive in HTTP/2 or HTTP/3 usage. For example:

In HTTP/1.1 it is a best practice to open several HTTP 1.x connections in parallel to speedup the page loading (up to 6 max per domain shard).
In HTTP/1.1 it is a best practice to concatenate JavaScript and CSS files, sprite images and inline resources.
In HTTP/1.1 it is a best practice to design your APIs to batch multiple queries to minimize round trips.

In contrast, HTTP/2 (and HTTP/3) it is almost as if both gravity is reversed and all friction ceases to exist, as these well worn practices degrade performance instead of enhancing performance!

HTTP/2 TLS 1.3 hugely reduces the number of round trips required for setting up an HTTPS connection. For clients that have ever visited a site before, the data that traverses this connection is cryptographically protected, and there's no need to share key exchange protocol messages before transmission.

HTTP/2 also supports multiplexing, which allows the Browser to fire off multiple requests at once on the same connection and receive the requests back in any order.

HTTP/2 supports streams, a bidirectional flow of frames between the client and the server. A single client request can have multiple responses (e.g. subscribe to updates).

HTTP/2 is full duplex across streams, in that one stream may be sending while another is receiving. But HTTP/2 is still very much half duplex within a given stream.

HTTP/2 supports 100 max number of concurrent requests that a browser can make to a server up from 6-8.

HTTP/3 is pretty much HTTP/2 but over UDP (yeah, they also improved and simplified things like stream prioritization). This prevents TCP's Head-Of-Line blocking problem, when a queue of TCP packets is held up by the first packet in the queue..

Websockets and HTTP/2

HTTP/2 obsoletes websockets for all use cases except for pushing binary data from the server to a JS webclient. HTTP/2 fully supports binary bidi (bidirectional) streaming (read on), but browser JS doesn't have an API for consuming binary data frames and AFAIK such an API is not planned. Once the client opens a stream by sending a request, both sides can send DATA frames across a persistent socket at any time - full bidi.

For every other application of bidi streaming, HTTP/2 is as good or better than websockets, because (1) the spec does more work for you, and (2) in many cases it allows fewer TCP connections to be opened to an origin.

That's not much different from websockets: the client has to initiate a websocket upgrade request before the server can send data across, too.

The biggest difference is that, unlike websockets, HTTP/2 defines its own multiplexing semantics: how streams get identifiers and how frames carry the id of the stream they're on. HTTP/2 also defines flow control semantics for prioritizing streams. This is important in most real-world applications of bidi.

If you need to build a real-time chat app, let's say, where you need to broadcast new chat messages to all the clients in the chat room that have open connections, you can (and probably should) do this without websockets.

You would use Server-Sent Events to push messages down and the Fetch api to send requests up. Server-Sent Events (SSE) is a little-known but well supported API that exposes a message-oriented server-to-client stream. Although it doesn't look like it to the client JavaScript, under the hood your browser (if it supports HTTP/2) will reuse a single TCP connection to multiplex all of those messages. There is no efficiency loss and in fact it's a gain over websockets because all the other requests on your page are also sharing that same TCP connection. Need multiple streams? Open multiple EventSources! They'll be automatically multiplexed for you.

Besides being more resource efficient and having less initial latency than a websocket handshake, Server-Sent Events have the nice property that they automatically fall back and work over HTTP/1.1 (but the 6 connection per domain limit). But when you have an HTTP/2 connection they work incredibly well.

Here's a good article with a real-world example of accomplishing the reactively-updating SPA.

It is worth mentioning that websockets over HTTP/2 and even websockets over HTTP/3 (UDP QUIC) are also possible, and do notably improve performance.

HTTP/2 GraphQL impact is a mixed bag

With HTTP/2 (and HTTP/3) batching requests like this is now counter-productive to performance:

GET /widgets/5382894223,35223231,534232313,5231332435 HTTP/1.1
Accept: application/widgets+json
Host: api.widgets.org
Accept-Encoding: gzip, deflate
User-Agent: BobsWidgetClient/1.5

The above approach has a number of downsides. Both the service and clients need to understand a new endpoint, and a new list-based format, bloating the API – especially if there are many different types of resources that need similar treatment. Furthermore, this approach seriously impacts cache efficiency, creating further server load and client-perceived latency.

If you can describe precisely what you want to the server in an efficient format, it can now reply with just what you want.

Without too much exaggeration, another way to think of HTTP/2 is as a new query language – one that lets you encode a very complex set of requests into a small amount of data that is heavily optimized for transmission, while still allowing standard HTTP components – especially caches – to work with the individual requests. However, this is still server-centric as it currently requires that the server backend provides endpoints that are exactly as granular as the client needs, rather than allowing the client to independently decide granularity.

By using HTTP/2, it fixes most problems caused by compound documents and sparse fieldsets based formats such as GraphQL and JSON:API:

Because each pushed resource is sent in a separate HTTP/2 stream (HTTP/2 multiplexing), related resources can be sent in parallel to the client.
Consequently, clients and network intermediates (such as a CDN or proxy), can store each resource in a specific cache, while resource embedding only allows to have the full big JSON document in cache, cache invalidation is then more efficient, and can be done at the HTTP level.

Specifically with GraphQL, using cache mechanisms provided by the HTTP protocol isn't easy (POST requests cannot be cached using just HTTP semantics).

HTTP/2 and GraphQL Federation

Apollo Federation lets you declaratively combine multiple APIs into a single, federated graph. This federated graph enables clients to interact with your APIs through a single request. That’s no longer desirable from a performance standpoint in HTTP/2 or HTTP/3. In federation, the slowest subquery becomes the bottleneck for the entire federated query. Federation also increases the difficulty of independent deployability and testability. It also makes it hard to provide the client with meaningful partial success and partial errors when subqueries return errors. It may be that there are other specific concerns — such as authentication and rate limiting — where a GraphQL gateway is still more useful than not.

Apollo Federation also does not support GraphQL Subscriptions, and they have no plans to add support

HTTP method QUERY and GraphQL

The IETF WHATWG has a draft HTTP extension that would add a new HTTP method, QUERY, as a safe, idempotent request method that can carry request content. (which would apply to HTTP/1.1, HTTP/2, and HTTP/3). This specification defines an HTTP method is safe if it doesn't alter the state of the server. In other words, a method is safe if it leads to a read-only operation.

This extension would allow complicated client-centric queries like GraphQL to avoid using POST and be able to enjoy more standard HTTP caching behavior. When combined with the other performance benefits of HTTP/2 and HTTP/3 it should open new opportunities for dynamic client interactions.

As to how active the proposal is, you can see a lot of revisions in the GitHub history of the source document here: https://github.com/httpwg/http-extensions/commits/main/draft-ietf-httpbis-safe-method-w-body.xml

And use the IETF data tracker to see the other activity on mailing lists:

https://datatracker.ietf.org/doc/draft-ietf-httpbis-safe-method-w-body/

Node 22.2.0 supports HTTP Method Query https://github.com/nodejs/node/issues/51562

GraphQL Subscriptions over HTTP/2

Summary of the Problems with GraphQL Subscriptions over WebSockets

WebSockets make your GraphQL Subscriptions stateful
WebSockets cause the browser to fall back to HTTP/1.1
WebSockets cause security problems by exposing Auth Tokens to the client
WebSockets allow for bidirectional communication, but GraphQL subscriptions do not
WebSockets are not ideal for SSR (server-side-rendering)

GraphQL Subscriptions over WebSockets cause a few problems with performance, security and usability.

GraphQL Subscriptions over HTTP/2 with Server Sent Events (SSE)

HTTP/2 SSE (Server-Sent Events) / Fetch is stateless
HTTP/2 SSE (Server-Sent Events) / Fetch can easily be secured
HTTP/2 SSE (Server-Sent Events) / Fetch disallows the client to send arbitrary data
HTTP/2 SSE (Server-Sent Events) / Fetch can easily be used to implement SSR (Server-Side Rendering) for GraphQL Subscriptions

Sources and further reading

Similar Event De-duplication per Period

Steve Coffman — Fri, 06 Sep 2024 13:43:19 +0000

Event orientation is great!

Whether you are using GCP Pub/Sub, Kafka, Kinesis, RabbitMQ, NATS JetStream, Redis Pub/Sub or any of the myriad alternatives, the Patterns you learn apply to them all.

Similar Event de-duplication per period

Even if you are enjoying exactly-once-delivery, you will still get similar events that you don't really want to react to multiple times.

A great example of this is actionable alerts. The first time something notices a problem, that's great to escalate to get someone's attention that an action is required. The 700th time is just noise.

If you are sending an event you have fields (whether it is JSON/protobuf/struct whatever), and you just need to identify which fields to group things by to sort them into the same bucket for your time period.

You can take a hash of any arbitrary set of those event fields and calculate a hash of them be the key to some persistence source (key value storage, SQL, whatever) For instance, in Go: https://go.dev/play/p/Ain8FIJiDit

Then store that hash with an expiration timestamp and. If you encounter any more "similar" events before the expiration timestamp, ignore them, as they have already been brought to someone's attention.

Real Life Example

At work, we deal with school districts, and they provide us with their roster, which synchronize with on a regular basis (nightly). But sometimes a human at the school district will make a mistake like accidentally deleting all the students in their roster. Whoops! However, if they haven't fixed the problem, we actually don't need to continue to be reminded more than once.

The district, and the reason we were unable to roster it, are thecomposite unique keys.

Whenever we would file a JIRA ticket (or send an alert to Slack), we first compute the hash of those two keys and see if a matching hash has already been sent, and if so if the last alert has expired (24 hrs or something) before sending a new one and replacing the old one. So for instance, a roster failure for a district could have something a generic as:

v := map[string]any{
        "district":  "00be2b9c-ef18-4c27-8fa9-087dd5f39f27",
        "attention": "rosterops",
        "reason": "roster change exceeds threshold",
    }

and you wouldn’t hear about that district failing to roster again until the next day. But if someone manually tried to sync up and that district had a different type of failure that occurred (re-running now gives "service unavailable" instead of exceeding the change tolerance threshold:

v := map[string]any{
        "district":      "00be2b9c-ef18-4c27-8fa9-087dd5f39f27",
        "attention": "rosterops",
        "reason": "roster provider is unavailable",
    }

Again, We can take a hash of any arbitrary set of those fields and make that an indexed datastore field on an alert entity. https://go.dev/play/p/Ain8FIJiDit

Extra bonus

If you also persist another field like "status" you can see if someone is already handling it and make it into a nice action item tracker for any arbitrary alert system.

Browser Client to gRPC Server Routing options: Connect, gRPC-web, gRPC-gateway and more!

Steve Coffman — Fri, 09 Aug 2024 21:39:05 +0000

Browser Client to gRPC Server Routing options: Connect, gRPC-web, gRPC-gateway and more!

These days, gRPC is commonly used and well-supported enough that many options exist for communicating to a gRPC backend from the browser.

For gRPC apps, I quite like using Connect. Connect is a family of libraries for building browser and gRPC-compatible HTTP APIs: you write a short Protocol Buffer schema and implement your application logic, and Connect generates code to handle marshaling, routing, compression, and content type negotiation. It also generates an idiomatic, type-safe client in any supported language.

Connect lets you choose from gRPC, gRPC-Web, and Connect's own protocol, but each protocol has a different set of production routing options.

Choosing a protocol and how to route it!

In Connect: Choosing a protocol shows connect supports gRPC-web as well, but I thought I'd expand on how routing works from the browser for these different protocol choices. As a refresher, this is from Connect's docs:

Connect has Seamless multi-protocol support

Connect servers and clients support three protocols: gRPC, gRPC-Web, and
Connect's own protocol.

Connect fully supports the gRPC protocol, including streaming, trailers, and error details. Any gRPC client, in any language, can call a Connect server, and Connect clients can call any gRPC server. We validate our gRPC compatibility with an extended version of Google's own interoperability tests.

Connect also offers direct support for the gRPC-Web protocol used by grpc/grpc-web, without relying on a translating proxy like Envoy.

Finally, Connect supports its own protocol: a straightforward HTTP-based protocol that works over HTTP/1.1, HTTP/2, and HTTP/3. It takes the best parts of gRPC and gRPC-Web, including streaming, and packages them into a protocol that's equally at home in browsers, monoliths, and microservices. By default, implementations support both JSON- and binary-encoded Protobuf. You can call our live demo service with cURL:
  curl \
      --header "Content-Type: application/json" \
      --data '{"sentence": "I feel happy."}' \
      https://demo.connectrpc.com/connectrpc.eliza.v1.ElizaService/Say
By default, Connect servers support ingress from all three protocols. Clients
default to using the Connect protocol, but can switch to gRPC or gRPC-Web with
a configuration toggle — no further code changes required. The APIs for
errors, headers, trailers, and streaming are all protocol-agnostic.

Connect protocol HTTP Host and Request Path Routing

We get to use traditional HTTP Routing!

The Connect protocol is a simple, POST-only protocol that works over HTTP/1.1 or HTTP/2 (or HTTP/3!). It takes the best portions of gRPC and gRPC-Web, including streaming, and packages them into a protocol that works equally well in browsers, monoliths, and microservices. The Connect protocol is what we think the gRPC Protocol should be. By default, JSON- and binary-encoded Protobuf is supported. Calling a Connect API is as easy as using curl:




# Try it out! This is a live demo!
curl \
    --header "Content-Type: application/json" \
    --data '{"sentence": "I feel happy."}' \
    https://demo.connectrpc.com/connectrpc.eliza.v1.ElizaService/Say

If you have a load balancer, you can use all the familiar HTTP/1.1 or HTTP/2 (or HTTP/3) routing rules to get to the backend (Host or Host+Path). This is my preferred option, and you can use your favorite load balancer. I like Envoy these days.

And inside your server, HTTP/1.1 or HTTP/2 (or HTTP/3) for the Connect protocol would be based on the URL /connectrpc.eliza.v1.ElizaService.

gRPC Routing

In Kubernetes, the common Gateway API spec has experimental support GRPCRoute. AFAICT Envoy Gateway has the most mature support for Gateway API GRPCRoute of the implementations. At least GKE specifically recommends using Envoy as your gRPC gateway.

The source code for that tutorial is here.

Note: Browsers do not support gRPC protocol natively because they do not support HTTP/2 Trailers, nor does the Browser's Fetch API support client streaming. The fetch API does specify streaming request bodies, but unfortunately, browser vendors have not come to an agreement to support streams from the client—see this WHATWG issue on GitHub. This means you can use streaming from the browser, but only server streaming.

gRPC-Gateway: JSON all the things

The gRPC-Gateway project started out after the original author,Yuki Yugui Sonoda, left Google and wanted to use a tool they had been using internally at Google, but couldn’t find an open-source version. So they wrote one!

At its core the gRPC-Gateway is a translation layer between the HTTP/JSON REStful API paradigm and the gRPC/Protobuf API paradigm.

Johan Brandhorst-Satzkorn wrote an excellent article on the ins and outs of gRPC-Gateway.

Vanguard: JSON all the things, but also Connect

Vanguard is a nearly seamless replacement for gRPC-Gateway. It replaces the JSON/REST to gRPC translation, but given that it also can translate JSON/REST to Connect, it opens up a few more options.

gRPC-web Routing options

There are two different options, and both of them involve using a proxy like Envoy.

grpc-web option 1: grpc-web

In grpc-web, traditional HTTP URL routing works. For instance:



curl 'http://localhost:8080/grpc.gateway.testing.EchoService/Echo' 
-H 'Pragma: no-cache' 
-H 'X-User-Agent: grpc-web-javascript/0.1' 
-H 'Origin: http://localhost:8081' 
-H 'Content-Type: application/grpc-web+proto' 
-H 'Accept: */*' 
-H 'X-Grpc-Web: 1' 
-H 'Connection: keep-alive' --data-binary $'\x00\x00\x00\x00\x05\n\x03abc' --compressed

The URL /grpc.gateway.testing.EchoService would be routed to the service.

grpc-web option 2: gRPC bridging

Envoy also supports gRPC bridging where gRPC-web is translated to gRPC.

See https://github.com/connectrpc/envoy-demo/issues/2

Envoy notes

From Envoy HTTP/3 overview:

While HTTP/3 downstream support is deemed ready for production use, improvements are ongoing, tracked in the area-quic tag.

HTTP/3 upstream support is alpha - key features are implemented but have not been tested at scale.

https://github.com/envoyproxy/gateway/issues/422

From Gophers Slack regarding gRPC routing in GKE:

No particular opinions, it's fine. Google gave up on actually fixing their ingress controller and came up with the Gateway spec instead, It's not super widely supported in the rest of the k8s ecosystem yet. That said, it works very well on GKE for a lot of things.

For anything http1.1 it should work out of the box as an L7 LB.

For http2 to work properly it's likely that between the LB and app it needs to be configured to do TLS, because GCP LBs (regardless if configured with ingress or gateway) does not support h2c (http2 over cleartext) afaik. GKE only implements the HTTPRoute part of the Gateway spec also, so can't do anything protocol-aware for grpc or have tcp vs udp routes either.

Envoy Terminology

Downstream: A downstream host connects to Envoy, sends requests, and receives responses.
Upstream: An upstream host receives connections and requests from Envoy and returns responses.
xDS configuration API overview - gRPC/REST based configuration provider APIs known as “xDS” (* discovery service)
- fully-static
- EDS
- CDS
- RDS
- VHDS
- SRDS
- LDS
- SDS
- RTDS
- ECDS
- Aggregated xDS (“ADS”)
- Delta gRPC xDS
- xDS TTL

GitHub Actions and checking Tokens for expiration

Steve Coffman — Sun, 14 Jul 2024 15:08:50 +0000

What is a GitHub token?

GitHub lets you create a Personal Access Token (PAT) for your GitHub user. These types of tokens are used pretty much the same as the GitHub Action default token.

A GitHub Action default GITHUB_TOKEN expires when a job finishes or after a maximum of 24 hours.

Your GitHub Personal Access Token (PAT) may expire when it's past its expiration date, but GitHub may also expire or revoke it for a variety of other reasons. It is pretty tedious to tell which if any of those other reasons have happened.

In either case, both the official GitHub Actions (like checkout) and the marketplace ones will often fail with a very unhelpful error:

Error: fatal: could not read Username for 'https://github.com': terminal prompts disabled

The layers of their JavaScript between you and a git command will never be updated to make more friendly error messages upon failure.

Instead, if you add a step to your own workflow you can emit helpful action items for when these errors occur and save your future self from wasting time.

Verify GitHub Token has access (and is not expired)

You can verify your Personal Access Token is not expired and has proper permissions:

GitHub:

curl -XGET -H 'authorization: token <personal token>' '<https://api.github.com/repos/><owner>/<repo>'

GitHub Enterprise:

curl -XGET -H 'authorization: token <personal token>' 'https://<GHE url>/api/v3/repos/<owner>/<repo>'

So for instance:

curl -XGET -H 'authorization: token '"${GHBOT_TOKEN}" https://api.github.com/repos/MyOrg/MyRepo

So write a quick script to use as a step in your workflow:

echo "Checking GitHub Token for expiry and/or access"
# prepend some arguments, but pass on whatever arguments this script was called with
output="$(curl -XGET \
--write '\n%{http_code}\n'  \
--silent \
--fail -H 'Content-Type:application/json' \
-H 'Accept:application/json' \
-H 'authorization: token '"${GHBOT_TOKEN}" https://api.github.com/repos/MyOrg/MyRepo \
)"
return_code=$?
if [ 0 -eq $return_code ]; then
    # remove the "http_code" line from the end of the output, and parse it
    echo "Yay! Your personal access code is not expired and has access to this repo"
else
    echo "Boo! Your personal access code is expired or has no access"
   # comment the next line if you don't want to see the HTTP status code and return messages from GitHub
   echo "Failure: code=$output"
   exit 1
fi

So you can put this in your GitHub Action workflow:

    steps:
      - name: Check Personal Access Token
        run: |
          echo "Checking GitHub Token for expiry and/or access"
          # prepend some arguments, but pass on whatever arguments this script was called with
          output="$(curl -XGET \
          --write '\n%{http_code}\n'  \
          --silent \
          --fail -H 'Content-Type:application/json' \
          -H 'Accept:application/json' \
          -H 'authorization: token '"${GHBOT_TOKEN}" https://api.github.com/repos/MyOrg/MyRepo \
          )"
          return_code=$?
          if [ 0 -eq $return_code ]; then
              # remove the "http_code" line from the end of the output, and parse it
              echo "Yay! Your personal access code is not expired and has access to this repo"
          else
              echo "Boo! Your personal access code is expired or has no access"
             # comment the next line if you don't want to see the HTTP status code and return messages from GitHub
             echo "Failure: code=$output"
             exit 1
          fi

If you are using the default GitHub token, change the above ${GHBOT_TOKEN} (which is sourced by from a shell environment variable you set in your action) to instead be ${{ secrets.GITHUB_TOKEN }} which the action will replace before it passes the script to the shell.

Or write your own layers of JavaScript to check if the token is expired. You do you.

Bonus content

Setting your git config to use a the token in a GitHub action can be done with the token:

      - name: Setup git config via HTTPS using token
        run: |
          echo "Configuring git for HTTPS using token"
          git config --global user.name "github-actions[bot]"
          git config --global user.email "41898282+github-actions[bot]@users.noreply.github.com"
          git config --global url."https://oauth2:${{ secrets.GITHUB_TOKEN }}@github.com/".insteadOf "https://github.com/"

Or you can decide this whole token/HTTPS thing is dumb and use SSH keys (like a deploy key). Warning: GitHub will mark any commits as being authored by the same user who created the deploy key:

name: My cool workflow
env:
  SSH_AUTH_SOCK: /tmp/ssh_agent.sock
  GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
jobs:
  cooljobnamehere:
    runs-on: ubuntu-latest
    timeout-minutes: 30
    steps:
    - name: Setup Git with SSH Keys and known_hosts
      run: |
        ssh-agent -a "${SSH_AUTH_SOCK}" > /dev/null
        ssh-add - <<< "${{ secrets.MY_SSH_PRIVATE_KEY }}"
        echo "Configuring git..."
        git config --global user.name "github-actions[bot]"
        git config --global user.email "41898282+github-actions[bot]@users.noreply.github.com"
        git config --global url."git@github.com:".insteadOf "https://github.com/"

Sometimes when you use:

echo "${{ secrets.MY_SSH_PRIVATE_KEY }}" | ssh-add - > /dev/null

You will sometimes get a failure:

Error loading key "(stdin)": error in libcrypto

You might have some carriage returns in there so try:

echo "${{ secrets.MY_SSH_PRIVATE_KEY }}" | tr -d '\r' | ssh-add - > /dev/null

Also for some reason(?), this appears more reliable:

ssh-add - <<< "${{ secrets.MY_SSH_PRIVATE_KEY }}"

Testing your SSH Deploy Key

You likely have your own developer SSH config and key, but if you want to verify an SSH Deploy key (or a bot GitHub account SSH key):

ssh -F /dev/null -o"IdentitiesOnly=yes" -o"StrictHostKeyChecking=no" -i id_ed25519 -T git@github.com

Just ensure that the key and its directory have proper permissions!

Typically you want the permissions to be:

Item	Sample	Numeric	Bitwise
SSH folder	`~/.ssh`	`700`	`drwx------`
Public key	`~/.ssh/id_rsa.pub`	`644`	`-rw-r--r--`
Private key	`~/.ssh/id_rsa`	`600`	`-rw-------`
Home folder	`~`	`755`	`drwxr-xr-x`

So when you are in the deploy key directory:

chmod 700 .
chmod 600 id_ed25519
chmod 600 id_ed25519.pub

Comparison golang stacktrace error library output

Steve Coffman — Wed, 24 Mar 2021 18:06:03 +0000

Note: Runnable examples are available here

Golang is great. I mostly love it. However, collecting an error with relevant context and a nicely formatted stacktrace is kind of a mess of competing approaches. This this feature comparison is a good overview of the landscape.

I wanted to compare the output of a few different "nicely formatted" approaches with two different kinds of wrapped
errors, Sentinel errors and custom error types.

Some libraries capture the exact state of the stack when an error happens, including every function call.

Some try to attach relevant contextual information (messages, variables) at strategic places along the call stack,
keeping stack traces compact and maximally useful.

Some eliminate stacktrace duplication from wrapped errors.

jba/errfmt

  {Msg:This is a message Detail:Important detail Err:reading "file"
    cmd/prog/reader.go:122
  parsing line 23
    iff x > 3 {
    cmd/prog/parser.go:85
  syntax error
    cmd/prog/parser.go:214
  }

emperror

  something went wrong
  main.foo
  /Users/steve/Documents/git/comparerr/emperror/main.go:23
  main.main
  /Users/steve/Documents/git/comparerr/emperror/main.go:31
  runtime.main
  /Users/steve/.asdf/installs/golang/1.14.15/go/src/runtime/proc.go:203
  runtime.goexit
  /Users/steve/.asdf/installs/golang/1.14.15/go/src/runtime/asm_amd64.s:1373

  starting bar

  something went wrong
  got an error in bar
  main.bar
  /Users/steve/Documents/git/comparerr/emperror/main.go:27
  main.main
  /Users/steve/Documents/git/comparerr/emperror/main.go:39
  runtime.main
  /Users/steve/.asdf/installs/golang/1.14.15/go/src/runtime/proc.go:203
  runtime.goexit
  /Users/steve/.asdf/installs/golang/1.14.15/go/src/runtime/asm_amd64.s:1373

cockroachdb/errors

  Sentinel Something Went Wrong
  (1) attached stack trace
  -- stack trace:
  | main.foo
  |     /Users/steve/Documents/git/comparerr/cockroachdb-errors/main.go:23
  | main.main
  |     /Users/steve/Documents/git/comparerr/cockroachdb-errors/main.go:31
  | [...repeated from below...]
  Wraps: (2) attached stack trace
  -- stack trace:
  | main.init
  |     /Users/steve/Documents/git/comparerr/cockroachdb-errors/main.go:10
  | runtime.doInit
  |     /Users/steve/.asdf/installs/golang/1.14.15/go/src/runtime/proc.go:5480
  | runtime.main
  |     /Users/steve/.asdf/installs/golang/1.14.15/go/src/runtime/proc.go:190
  | runtime.goexit
  |     /Users/steve/.asdf/installs/golang/1.14.15/go/src/runtime/asm_amd64.s:1373
  Wraps: (3) Sentinel Something Went Wrong
  Error types: (1) *withstack.withStack (2) *withstack.withStack (3) *errutil.leafError


  starting bar

  got an error in bar: bar something went wrong
  (1) attached stack trace
  -- stack trace:
  | main.bar
  |     /Users/steve/Documents/git/comparerr/cockroachdb-errors/main.go:27
  | main.main
  |     /Users/steve/Documents/git/comparerr/cockroachdb-errors/main.go:39
  | runtime.main
  |     /Users/steve/.asdf/installs/golang/1.14.15/go/src/runtime/proc.go:203
  | runtime.goexit
  |     /Users/steve/.asdf/installs/golang/1.14.15/go/src/runtime/asm_amd64.s:1373
  Wraps: (2) got an error in bar
  Wraps: (3) bar something went wrong
  Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) main.ErrMyError

palantir/stacktrace

  foo returned error
  --- at /Users/steve/Documents/git/comparerr/palantir-err/main.go:22 (foo) ---
  Caused by: Sentinel Something Went Wrong
  --- at /Users/steve/Documents/git/comparerr/palantir-err/main.go:9 (init) ---


  starting bar

  got an error in bar
  --- at /Users/steve/Documents/git/comparerr/palantir-err/main.go:26 (bar) ---
  Caused by: bar something went wrong

pkg/errors

  Sentinel Something Went Wrong
  main.init
  /Users/steve/Documents/git/comparerr/pkg-errors/main.go:10
  runtime.doInit
  /Users/steve/.asdf/installs/golang/1.14.15/go/src/runtime/proc.go:5480
  runtime.main
  /Users/steve/.asdf/installs/golang/1.14.15/go/src/runtime/proc.go:190
  runtime.goexit
  /Users/steve/.asdf/installs/golang/1.14.15/go/src/runtime/asm_amd64.s:1373
  main.foo
  /Users/steve/Documents/git/comparerr/pkg-errors/main.go:23
  main.main
  /Users/steve/Documents/git/comparerr/pkg-errors/main.go:31
  runtime.main
  /Users/steve/.asdf/installs/golang/1.14.15/go/src/runtime/proc.go:203
  runtime.goexit
  /Users/steve/.asdf/installs/golang/1.14.15/go/src/runtime/asm_amd64.s:1373


  starting bar

  bar something went wrong
  got an error in bar
  main.bar
  /Users/steve/Documents/git/comparerr/pkg-errors/main.go:27
  main.main
  /Users/steve/Documents/git/comparerr/pkg-errors/main.go:39
  runtime.main
  /Users/steve/.asdf/installs/golang/1.14.15/go/src/runtime/proc.go:203
  runtime.goexit
  /Users/steve/.asdf/installs/golang/1.14.15/go/src/runtime/asm_amd64.s:1373

You can clone this repo, cd into any of the directories and run go run main.go to see an example

jba errfmt output example
emperror output example
cockroackdb/errors playground times out? Not sure what that is all about.
pkg/errors
palantir/stacktrace

Don't Panic: Catching Panics in Errgroup Goroutines

Steve Coffman — Sat, 16 Jan 2021 16:04:37 +0000

Image by Nicosmos, Public domain, via Wikimedia Commons

TL;DR version

StevenACoffman/errgroup is a drop-in alternative to Go's wonderful sync/errgroup but it converts goroutine panics to errors.

Why you want this

While net/http installs a panic handler with each request-serving goroutine,
goroutines do not and cannot inherit panic handlers from parent goroutines,
so a panic() in one of the child goroutines will kill the whole program.

So in production, whenever you use an sync.errgroup, you have to have the discipline to always remember to add a
deferred recover() to the beginning of every new goroutine, and convert any panics to errors.



            defer func() {
                if rec := recover(); rec != nil {
                    err = FromPanicValue(rec)
                }
            }()

You can see this in action in the Go Playground here. You also want to be careful not to lose the stack trace. The CollectStack function I used here is overly simplified, so it adds a little noise because it doesn't the skip the FromPanicValue and CollectStack frames. 🤷‍♂️



func FromPanicValue(i interface{}) error {
    switch value := i.(type) {
    case nil:
        return nil
    case string:
        return fmt.Errorf("panic: %v\n%s", value, CollectStack())
    case error:
        return fmt.Errorf("panic in errgroup goroutine %w\n%s", value, CollectStack())
    default:
        return fmt.Errorf("unknown panic: %+v\n%s", value, CollectStack())
    }
}

func CollectStack() []byte {
    buf := make([]byte, 64<<10)
    buf = buf[:runtime.Stack(buf, false)]
    return buf
}

A co-worker of mine co-worker Ben Kraft, wrote some handy wrapper code around sync/errgroup to avoid that boilerplate (and required discipline). With his permission, I lightly modified it to
lift it out of our private work repository for the more general Go community.

StevenACoffman/errgroup is a drop-in alternative to Go's wonderful
sync/errgroup with the difference that it converts goroutine panics to errors.

You can see it in use in the playground or here:



package main

import (
    "fmt"

    "github.com/StevenACoffman/errgroup"
)

func main() {
    g := new(errgroup.Group)
    var urls = []string{
        "http://www.golang.org/",
        "http://www.google.com/",
        "http://www.somestupidname.com/",
    }
    for i := range urls {
        // Launch a goroutine to fetch the URL.
        i := i // https://golang.org/doc/faq#closures_and_goroutines
        g.Go(func() error {

            // deliberate index out of bounds triggered
            fmt.Println("Fetching:", i, urls[i+1])

            return nil
        })
    }
    // Wait for all HTTP fetches to complete.
    err := g.Wait()
    if err == nil {
        fmt.Println("Successfully fetched all URLs.")
    } else {
        fmt.Println(err)
    }
}

Counterpoint

There is an interesting discussion which has an alternative view that,
with few exceptions, panics should crash your program. I'm ok with that in development and testing, but would rather sleep soundly at night.

Prior Art

With only a cursory search, I found a few existing open source examples.

Kratos errgroup

Kratos Go framework for microservices has a similar errgroup
solution.

PanicGroup by Sergey Alexandrovich

In the article Errors in Go:
From denial to acceptance,
(which advocates panic based flow control 😱), they have a PanicGroup that's roughly equivalent:



type PanicGroup struct {
  wg      sync.WaitGroup
  errOnce sync.Once
  err     error
}

func (g *PanicGroup) Wait() error {
  g.wg.Wait()
  return g.err
}

func (g *PanicGroup) Go(f func()) {
  g.wg.Add(1)

  go func() {
    defer g.wg.Done()
    defer func(){
      if r := recover(); r != nil {
        if err, ok := r.(error); ok {
          // We need only the first error, sync.Once is useful here.
          g.errOnce.Do(func() {
            g.err = err
          })
        } else {
          panic(r)
        }
      }
    }()

    f()
  }()
}

I would appreciate feedback or suggestions for improvement!

Tripperwares: http.Client Middleware - chaining RoundTrippers

Steve Coffman — Wed, 22 Jul 2020 00:36:30 +0000

http.Client Middleware or "Tripperwares" adhere to func (http.RoundTripper) http.RoundTripper signature, hence the name. As such they are used as a Transport for http.Client, and can wrap other transports.

This means that the composition is purely net/http-based.

While server middleware is pretty commonly discussed (and there's lots to pick from), I found the documentation a little sparse on how (and why) to do this for clients. Luckily the amazing @PeterBourgon helped me on the #gophers slack.

A couple good use cases for TripperWare are request logging, caching, and certain kinds of auth (e.g. bearer token renewal).

However, instead of making one complicated Tripperware to do all this, it's nice to compose several more tightly focussed ones.

package main

import (
    "encoding/base64"
    "fmt"
    "io"
    "net/http"
    "os"
    "time"
)

// thanks to peter bourgon on Gophers slack

func main() {
    jiraUserID := os.ExpandEnv("${JIRA_USERID}")
    jiraPassword := os.ExpandEnv("${JIRA_API_TOKEN}")
    jiraBaseURL := "https://example.atlassian.net/rest/agile/1.0/board/"

    var rt http.RoundTripper
    rt = http.DefaultTransport
    rt = NewDelayRoundTripper(rt, 123*time.Millisecond)
    rt = NewLoggingRoundTripper(rt, os.Stderr)
    header := make(http.Header)
    header.Set("Accept", "application/json")
    header.Set("Content-Type", "application/json")
    hrt := NewHeaderRoundTripper(rt, header)
    hrt.BasicAuth(jiraUserID, jiraPassword)

    client := &http.Client{
        Transport: hrt,
    }

    _, err := client.Get(jiraBaseURL)
    fmt.Printf("err=%v\n", err)
}

//
//
//

type DelayRoundTripper struct {
    next  http.RoundTripper
    delay time.Duration
}

func NewDelayRoundTripper(next http.RoundTripper, delay time.Duration) *DelayRoundTripper {
    return &DelayRoundTripper{
        next:  next,
        delay: delay,
    }
}

func (rt *DelayRoundTripper) RoundTrip(req *http.Request) (resp *http.Response, err error) {
    time.Sleep(rt.delay)
    return rt.next.RoundTrip(req)
}

//
//
//

type LoggingRoundTripper struct {
    next http.RoundTripper
    dst  io.Writer
}

func NewLoggingRoundTripper(next http.RoundTripper, dst io.Writer) *LoggingRoundTripper {
    return &LoggingRoundTripper{
        next: next,
        dst:  dst,
    }
}

func (rt *LoggingRoundTripper) RoundTrip(req *http.Request) (resp *http.Response, err error) {
    defer func(begin time.Time) {
        fmt.Fprintf(rt.dst,
            "method=%s host=%s status_code=%d err=%v took=%s\n",
            req.Method, req.URL.Host, resp.StatusCode, err, time.Since(begin),
        )
    }(time.Now())

    return rt.next.RoundTrip(req)
}

//
//
//

type HeaderRoundTripper struct {
    next http.RoundTripper
    Header http.Header
}

func NewHeaderRoundTripper(next http.RoundTripper, Header http.Header) *HeaderRoundTripper {
    if next == nil {
        next = http.DefaultTransport
    }
    return &HeaderRoundTripper{
        next:   next,
        Header: Header,
    }
}

func (rt *HeaderRoundTripper) RoundTrip(req *http.Request) (resp *http.Response, err error) {
    if rt.Header != nil {
        for k, v := range rt.Header {
            req.Header[k] = v
        }
    }
    fmt.Println("HeaderRoundTrip")
    return rt.next.RoundTrip(req)
}

func (rt *HeaderRoundTripper) BasicAuth(username, password string) {
    if rt.Header == nil {
        rt.Header = make(http.Header)
    }

    auth := username + ":" + password
    base64Auth := base64.StdEncoding.EncodeToString([]byte(auth))
    rt.Header.Set("Authorization", "Basic "+ base64Auth)
}

func (rt *HeaderRoundTripper) SetHeader(key, value string) {
    if rt.Header == nil {
        rt.Header = make(http.Header)
    }
    rt.Header.Set(key, value)
}

This works fine, but if you're less into receivers and prefer functional composition, here's an functional alternative:

// NewRoundTripper returns an http.RoundTripper that is tooled for use in the app

func NewRoundTripper(original http.RoundTripper) http.RoundTripper {
    if original == nil {
        original = http.DefaultTransport
    }

    return roundTripperFunc(func(request *http.Request) (*http.Response, error) {
        response, err := original.RoundTrip(request)
        return response, err
    })
}

type roundTripperFunc func(*http.Request) (*http.Response, error)

func (f roundTripperFunc) RoundTrip(r *http.Request) (*http.Response, error) { return f(r) }

I found that code in evepraisal/go-evepraisal where @sudorandom used it to instrument external calls with New Relic monitoring.

Also, I found that there's a several tripperwares and some helper libraries here: improbable-eng/go-httpwares

It's worth mentioning that changing the request is a bit of a no-no. Cloning the request, modifying that, and then passing that on is a workaround

Here's an article about an alternative for header manipulation.

When Now is not the time (in Go)

Steve Coffman — Tue, 21 Jul 2020 22:10:29 +0000

When testing two time.Time variables for equality in Go, you may find that there are (ahem) times when you get surprising results if you don't use time.Equal():

package main

import (
    "fmt"
    "time"
)

func main() {

    now := time.Now()
    nowUTC := now.UTC()

    fmt.Println("now:   ", now.Location(), now.Format("15:04"))
    fmt.Println("nowUTC:", nowUTC.Location(), " ", nowUTC.Format("15:04"))
    fmt.Println("-----")

    fmt.Printf("now == nowUTC:     %t\n", now == nowUTC)

    fmt.Printf("now.Equal(nowUTC): %t\n", now.Equal(nowUTC))

}

// Output:
// now:    Local 23:00
// nowUTC: UTC   23:00
// -----
// now == nowUTC:     false
// now.Equal(nowUTC): true

Run it in the Go Playground!

Handling Go modules replacement directive version lag and fork maintenance

Steve Coffman — Thu, 11 Jun 2020 15:11:22 +0000

At Khan Academy, we have a public fork of a public repo, which we are carrying a patch for in a branch. The latest release tag in the upstream is 1.27.0

In our other repositories, we have a replace directive:

replace github.com/golangci/golangci-lint => github.com/Khan/golangci-lint v1.21.1-0.20200609223024-b6a17227ca75

Even though the date and git commit SHA1 portion were correct, we found the pseudo-version prefix of v1.21.1 continuing to lag the upstream v1.27.0 confusing, and we were wondering what actions we could do in our public fork to be able to get that to update without running afoul of the dreaded "version before v1.27.0 would have negative patch number" or "unknown revision" errors.

It turns out that it was because the branch with our patch was merging upstream changes, rather than rebasing. This meant the last common tag between upstream and our fork's branch was v1.21.1, not v1.27.0. Darn computers always just doing what you asked them, right?

So to fix this we had to rebase against the upstream's master's up to the last release commit (we don't want to use unreleased code). This is a little tricky to get correct, so here's the recipe in glorious bash:

export WORKING_BRANCH=custom-autofix
export REBASED_BRANCH=autofix-custom
git clone git@github.com:Khan/golangci-lint.git
cd golangci-lint
git remote add upstream git@github.com:golangci/golangci-lint.git
git pull upstream master
git fetch --tags upstream
# Push the tags from my local to our fork's master branch (not sure if required)
git push -f --tags origin master
git checkout $WORKING_BRANCH
git co -b $REBASED_BRANCH
# Rebase against upstream latest tag
git reset --soft $(git log $(git tag | sort -V | tail -1) -1 --pretty=%H)
git commit -s -S -a
git push origin $REBASED_BRANCH -u
export LATEST_GOLANGCI_COMMIT=$(git log -1 --pretty=%H)
# Ta-da! All done. Now go to our private repo that uses this:
cd ~/khan/privaterepo
go mod edit -replace=github.com/golangci/golangci-lint=github.com/khan/golangci-lint@${LATEST_GOLANGCI_COMMIT}
# This will create/update the replace directive to github.com/Khan/golangci-lint v1.27.2-0.20200610182323-d6b2676ecc9b
# I could also have used the branch name instead of commit SHA1.

WTF was all that about?

Now there's some stuff in that bash-fu to unpack:

git tag | sort -V | tail -1

This will give the latest tag, which also will be upstream's latest tag unless you've tagged a release in your fork with a semantic version that's later (tip: Don't do that).

git log $(git tag | sort -V | tail -1) -1 --pretty=%H

This will give the full commit SHA1 of the latest tag.

Wait, why does is it start with `v1.27.2` instead of `v1.27.0` or `v1.27.1` ?

go mod will take the fork's latest release tag's semver and increment the patch number by one (because your replacement fork is a patch, see?).

The incrementing the semver prefix is the only thing wrong with calculating it by hand in your fork using:

echo "$(git tag | sort -V | tail -1)-$(git --no-pager show\
  --quiet\
  --abbrev=12\
  --date='format-local:%Y%m%d%H%M%S'\
  --format="%cd-%h")"
# output: v1.27.1-20200609183024-b6a17227ca75

To test things out, I made a newer release tag than upstream in our fork, v1.27.1, and go mod chose v1.27.2 which tells me it's not just looking at upstream's tags. It seems less confusing to me to just keep syncing upstream's tags to your fork.

Ok, I don't know if that is useful to anyone else!

Grace - graceful shutdown made simple

Steve Coffman — Sun, 05 Jan 2020 00:41:42 +0000

I wrote a tiny library called Grace to gracefully shutdown your application by catching the OS signals using sync.errgroup.

I often find I have invoked one or more persistent blocking methods, and some other method is needed be invoked in another goroutine to tell it to gracefully shut down when an interrupt is received.

For instance, when ListenAndServe() is invoked, Shutdown needs to be called.

This library allows you to start zero or more concurrent goroutines, and trigger a graceful shutdown when an interrupt is received. You can bring your own cleanup function to listen for the context's Done channel to be closed, so it will work flexibly in a variety of scenarios.

For example, you probably only want your cleanup function to close your database connection after you have drained your http connections or grpc connections.

        func() error { 
            //cleanup: on interrupt, shutdown server
            <-ctx.Done()
            // We received an interrupt signal, shut down.
            if err := srv.Shutdown(ctx); err != nil {
            // Error from closing listeners, or context timeout:
                log.Printf("HTTP server Shutdown error: %v", err)
            }
            return db.Close()
        })

For reference:

Go net/http package offers Shutdown function to gracefully shutdown your http server.
Go database/sql package offers Close function to gracefully close the connection to your SQL database.
Google google.golang.org/grpc package offers Server.GracefulStop, stops accepting new connections, and blocks until all the pending RPCs are finished

Alternatively, this library also allows you to invoke zero or more concurrent goroutines with an optional timeout.

Documentation

Installation

go get -u github.com/StevenACoffman/grace

Usage

Simple Run until Interrupt signal received

package main

import (
    "log"
    "time"

    "github.com/StevenACoffman/grace"
)

func main() {

    wait, ctx := grace.NewWait()

    err := wait.WaitWithFunc(func() error {
        ticker := time.NewTicker(2 * time.Second)
        for {
            select {
            case <-ticker.C:
                log.Printf("ticker 2s ticked\n")
                // testcase what happens if an error occured
                //return fmt.Errorf("test error ticker 2s")
            case <-ctx.Done():
                log.Printf("closing ticker 2s goroutine\n")
                return nil
            }
        }
    })

    if err != nil {
        log.Println("finished clean")
    } else {
        log.Printf("received error: %v", err)
    }
}

Usage with a default timeout:

package main

import (
    "log"
    "time"

    "github.com/StevenACoffman/grace"
)

func main() {

    wait, ctx := grace.NewWait()

    err := wait.WaitWithTimeoutAndFunc(15*time.Second, func() error {
        ticker := time.NewTicker(2 * time.Second)
        for {
            select {
            case <-ticker.C:
                log.Printf("ticker 2s ticked\n")
                // testcase what happens if an error occured
                //return fmt.Errorf("test error ticker 2s")
            case <-ctx.Done():
                log.Printf("closing ticker 2s goroutine\n")
                return nil
            }
        }
    })

    if err != nil {
        log.Println("finished clean")
    } else {
        log.Printf("received error: %v", err)
    }
}

Usage with cleanup on shutdown

Bring your own cleanup function!

package main

import (
    "fmt"
    "log"
    "net/http"
    "time"

    "github.com/StevenACoffman/grace"
)

func main() {

    wait, ctx := grace.NewWait()
    var httpServer *http.Server

    err := wait.WaitWithFunc(
        func() error {
            http.HandleFunc("/", healthCheck)
            httpServer = newHTTPServer()

            if err := httpServer.ListenAndServe(); err != http.ErrServerClosed {
                return err
            }
            return nil
        },
        func() error { 
            //cleanup: on interrupt, shutdown server
            <-ctx.Done()
            log.Printf("closing http goroutine\n")
            return httpServer.Shutdown(ctx)
        })

    if err != nil {
        log.Println("finished clean")
    } else {
        log.Printf("received error: %v", err)
    }
}

func healthCheck(w http.ResponseWriter, r *http.Request) {
    w.Header().Set("Content-Type", "text/plain")
    w.Header().Set("Content-Length", "0")
    w.WriteHeader(200)
}

func newHTTPServer() *http.Server {
    httpServer := &http.Server{
        Addr:         fmt.Sprintf(":8080"),
        ReadTimeout:  10 * time.Second,
        WriteTimeout: 10 * time.Second,
    }
    log.Printf("HTTP Metrics server serving at %s", ":8080")
    return httpServer
}

Prior Art and Alternatives

This library uses errgroup, but I found a number of other libraries that use other mechanisms:

death (sync.WaitGroup)
graceful (context cancellation)
finish (context + mutexes)
waitabit (sync.WaitGroup)
Gist using errgroup
Gist using GRPC GracefulStop

Comparing them is pretty instructive. I wish I'd used some of their testing techniques!

DEV Community: Steve Coffman

#go

Authorization (authz) and GraphQL

Securing dynamic data access is hard

Authorization models in GraphQL

Ad hoc ACLs or Centralized System?

Roles vs. Scopes

Where to perform authorization in a GraphQL architecture?

More Reading

Conclusion: Wait, what is your advice?

Parting Questions

HTTP/2 streaming and you, HTTP/3 streaming and wheee!

HTTP/2 (and HTTP/3) changes the rules of the game for web development!

Websockets and HTTP/2

HTTP/2 GraphQL impact is a mixed bag

HTTP/2 and GraphQL Federation

HTTP method QUERY and GraphQL

GraphQL Subscriptions over HTTP/2

Summary of the Problems with GraphQL Subscriptions over WebSockets

GraphQL Subscriptions over HTTP/2 with Server Sent Events (SSE)

Sources and further reading

Similar Event De-duplication per Period

Similar Event de-duplication per period

Real Life Example

Extra bonus

Browser Client to gRPC Server Routing options: Connect, gRPC-web, gRPC-gateway and more!

Browser Client to gRPC Server Routing options: Connect, gRPC-web, gRPC-gateway and more!

Choosing a protocol and how to route it!

Connect has Seamless multi-protocol support

Connect protocol HTTP Host and Request Path Routing

gRPC Routing

gRPC-Gateway: JSON all the things

Vanguard: JSON all the things, but also Connect

gRPC-web Routing options

grpc-web option 1: grpc-web

grpc-web option 2: gRPC bridging

Envoy notes

From Gophers Slack regarding gRPC routing in GKE:

Envoy Terminology

GitHub Actions and checking Tokens for expiration

What is a GitHub token?

Verify GitHub Token has access (and is not expired)

GitHub:

GitHub Enterprise:

Bonus content

Testing your SSH Deploy Key

Comparison golang stacktrace error library output

Don't Panic: Catching Panics in Errgroup Goroutines

TL;DR version

Why you want this

Counterpoint

Prior Art

Kratos errgroup

PanicGroup by Sergey Alexandrovich

Tripperwares: http.Client Middleware - chaining RoundTrippers

When Now is not the time (in Go)

Handling Go modules replacement directive version lag and fork maintenance

WTF was all that about?

Wait, why does is it start with v1.27.2 instead of v1.27.0 or v1.27.1 ?

Grace - graceful shutdown made simple

Documentation

Installation

Usage

Simple Run until Interrupt signal received

Usage with a default timeout:

Usage with cleanup on shutdown

Prior Art and Alternatives

Wait, why does is it start with `v1.27.2` instead of `v1.27.0` or `v1.27.1` ?