Enough has been said recently about why having a semantic layer is a good thing. Unfortunately, each of them has its own data model and corresponding storage format for storing those models, so migrating from one to another was a nontrivial endeavour.
Apache Ossie sets out to change that. It is an open-source standard for describing semantic layer models, with a lot of heavy hitters behind it. It's early days yet, and it only defines the semantic model interchange format, but with the kind of backers it has, it seems very likely it will get broad adoption (the obligatory xkcd comes to mind).
This leads to an interesting question: if porting semantic layer definitions from one format to another becomes free thanks to Apache Ossie, what are reasons to choose one semantic layer over the other?
The easy answer is to use whatever is bundled with a solution already in use for other purposes, such as Snowflake or dbt Cloud - and that is a reasonable option. However, their providers have a vested interest in promoting exactly the kind of lock-in that Apache Ossie is supposed to minimize; also, semantic layers that are offered as features of other products have a way of not being quite as polished as standalone products that live and die on their quality as semantic layers alone.
If the config side is made fungible by Apache Ossie, then the differentiating factor becomes the query-time behavior. The old-school semantic layers, such as Looker, Cube, or Metricflow (the dbt semantic layer) were optimized for a world where they were mostly called with the same query over and over, possibly with a different set of filters and groupbys (because only data people could author unusual queries, and they just wrote SQL). This made it natural to optimize for performance in such scenarios, which meant caching of pre-aggregations, which in turn pushed towards pretty static model configs.
In the world where agents are doing the querying, neither of those scenarios quite fits. Having SQL generated by semantic layers, using governed metrics and deterministic guarantees on join correctness, is a great source of reliability for the agent world. However, it's useless if to calculate a ratio of two metrics, or a time shifted version of one, the agent needs to modify the semantic model configs and wait for (at best) seconds for that to feed through - if it is allowed to do that at all.
The other gap that old-school semantic layers tend to have is discoverability - among hundreds, or thousands, of tables, with over a hundred columns each, how does the agent discover what is pertinent for just the query it wants to make? That extends to discoverability of related business rules that mention a metric, but also mention others so don't neatly fit into a single metric's description field.
These, then, are the two properties that distinguish an agent-ready semantic layer in a post-Apache Ossie world: expressiveness and flexibility at query time, and discoverability of both single dimensions/metrics and related business rules.
Naturally, those are what Motley has invested in, and also made available in its open source SLayer engine. On the expressiveness side, the vision is clear: the agent should be able to specify what it's looking for in as concise and natural way as possible, for example directly specifying in a simple syntax dimensions or filters that are themselves aggregates, or transforms such as time shift, no matter what dataset they all come from - and the semantic layer engine should figure out the necessary CTEs (subqueries) and join patterns, and generate the correct SQL in the right database dialect.
Current SLayer is closer to this vision than any other semantic layer on the market, and the upcoming 0.10 release next week will take it all the way there.
On the discoverability side, SLayer supports arbitrary chunks of context stored as memories, with references to the specific entities mentioned (such as metrics or dimensions), and a 3-channel search (embeddings + full-text + entity overlap, merged via Reciprocal Rank Fusion) over both memories and entities, so the agent can ask a freeform question and get a list of the likely relevant memories and entities. In contrast, the best the open source versions of the old school semantic layers can offer is keyword search.
So when evaluating a semantic layer for agent-friendliness, two things matter: query-time expressiveness, and entity and context discoverability. If they don't come out of the box, they'll need to be built before the semantic layer is much help to an agent - is that hassle really necessary?
Top comments (0)