A few weeks back we showed that an agent using our semantic layer (SLayer) beat a raw-SQL agent by 30%+ on a random selection of 265 live-sqlbench-large tasks. Fair follow-up question: how does that hold up against Cube, the incumbent open-core semantic layer?
So we put open-source Cube behind an agent. There's no MCP server for it, so we wrapped its REST API for introspection and queries, auto-generated models from the DB schemas, adding dimensions and measures off the JSON columns' leaves, and handed the agent the same knowledge-base tools the raw-SQL agent had.
Then we picked 10 tasks the SLayer-using agent had already solved — to give Cube an easy start. The raw-SQL agent had cleared 5 of them.
The Cube agent solved zero.
The first guess was a bug in the harness. There wasn't one. The logs showed two hard blockers:
Cube's schema metadata is so verbose it often blew straight past Claude Code's 25K tool-output limit. The agent couldn't see the whole schema, so it sat there hallucinating queries.
Nine of the ten tasks needed custom measures or dimensions built at solve time — ratios, derived formulas, CASE groupings that were provided in the knowledge base. Open-source Cube gives you no reasonable way to edit models on the fly.
It would be possible to build around it: an external model store, a validate-on-edit compile loop, smart schema retrieval to dodge the token limit. That's a serious chunk of engineering to turn Cube into something it isn't — and it'd still take seconds to recompile on every single edit.
Those are some of the reasons why we left Cube and built SLayer in the first place.
So if you're planning to run Cube behind an agent, go in with your eyes open: you're writing the scaffolding yourself. Batteries not included.
(Or use an agent-native semantic layer and skip the hassle. But we would say that.)
Top comments (2)
It’s fascinating to see how the verbosity of Cube's schema metadata can become a significant obstacle, especially when it leads to hallucinated queries due to the token limit. Your approach to wrapping the REST API for introspection is a clever workaround, but as you noted, it can introduce considerable overhead. If you’re considering enhancements for SLayer, exploring dynamic model updates could make it even more robust. I'm open to discussing paid collaboration if you need extra hands on this kind of engineering challenge! Have you considered any specific user feedback to guide future iterations of SLayer?
Luis, thanks for your comment! What do you mean by dynamic model updates? SLayer MCP already has tools to permanently edit models; and the query also allows to inject temporary fields, joins, etc into an existing model, just for the purpose of the current query.
We're always happy to integrate user feedback, including from paying users of the motley.ai platform, and welcome contibutions to the SLayer repo!