A few days ago, a user opened a ticket that started like this: "I can log in, I can see the home page, but 'almost' every list is empty."
That 'almost' tickled my mind. The user had the right roles. The same user could open the model-driven app in the same environment and see everything. And, the most confusing part, one list on the home page was populated. All the others weren't.
Then another user came with the same problem. Than another. The common factor? They were all guest users: Entra B2B accounts, the ones with #EXT# in their UPN. Internal users were fine.
This article tells the story of that bug, why it happened, how we're fixing it, and the shortcut that didn't make it. Spoiler: it's a story about two different roads to Dataverse that look identical from inside a Code App... until they aren't.
π Why we needed the Dataverse connector in the first place
When you build a Code App, the "standard" way to talk to Dataverse is to add your tables as data sources:
pac code add-data-source -a dataverse -t account
The CLI generates a typed model and a service for the table, and from then on you call retrieveMultipleRecordsAsync, createRecordAsync and friends. The calls go through the native Dataverse channel of the Power Apps host, with the host's token. It's clean, it's typed, it requires zero configuration from the user.
For CRUD, it's perfect. But the moment your app starts to behave like a real business application, you hit the walls:
-
No FetchXML. The query options are OData:
select,filter,orderBy,top,skip,count,expand,apply. That's it. There's nofetchXmloption. - Only the tables you registered. Want to read the audit history? Resolve the display name of a lookup that may point to any table? Each of those tables must be a data source of the app first.
-
$expanddidn't work for us, so fetching related records data cost a second query (I know it's in the options list: I'm telling you what happened on our tables, not what the type definition promises).
Now, why does FetchXML matter so much? Because views are FetchXML. In a previous article I wrote that the capabilities users take for granted are the ones nobody puts in the requirements: personal views, view selectors, filters, counters. To give our users views that behave like the model-driven ones, we built a small view engine where every grid is a FetchXML query plus a layout. User-defined views are stored as FetchXML too. And FetchXML gives you things that are painful (or impossible) to express in OData from a Code App:
-
ownerid eq-useroruserteams: "records owned by me or by one of my teams", resolved server-side; - filters on linked entities;
- aggregates, for the KPI tiles on the dashboard;
- paging cookies, for reading large sets without skipping rows.
When we built it, the SDK we were using (@microsoft/power-apps 1.0.x) offered no way at all to run FetchXML on the native channel.
Why didn't Microsoft provide, by default, a simple and basic api just like the one offered by
IOrganizationServiceas a first class citizen of code app SDK library, or the one provided by the Model Driven app Client SDK, and had to invent a completely brand new way to connect to Dataverse?
"I guess we'll never know..." (cit.)
So, to overcome those limits, we did what anyone would do: we used the Dataverse connector (commondataserviceforapps), which does accept FetchXML.
const result = await client.executeAsync({
connectorOperation: {
tableName: "commondataserviceforapps",
operationName: "ListRecordsWithOrganization",
parameters: {
organization, // e.g. "contoso.crm4.dynamics.com"
entityName: "accounts",
fetchXml,
prefer: 'odata.include-annotations="*"',
},
},
});
Notice the operation name: ListRecordsWithOrganization, not ListRecords, and the organization parameter that comes with it. Keep it in mind: it's the key of this whole story, and we'll come back to it. For now, all you need to know is that the app has no clean way to know its own organization URL, so we stored it in an environment variable, one value per environment, and moved on.
It worked. On every environment. Until production.
π₯ Enter the guest users
In production we have Entra ID Guest users, that we don't in the other environments. Here's what the guest users were seeing:
- they signed in without problems;
- the home page rendered, and the "Recent records" list was populated;
- every KPI tile in the home page failed;
- every other grid was empty.
Every failing call answered with the same HTTP 403:
{
"error": {
"code": "0x80072560",
"message": "The user is not a member of the organization."
}
}
"Not a member of the organization"? The user is a member. The systemuser record is there, enabled, licensed, synced with Entra, with the right object id. No duplicates. The environment variable points to the right organization. The user's connection belongs to the right identity and its status is Connected. And again: the same user reads the same table a few pixels above, in the "Recent records" list.
That was the clue. Let's look at which road each call takes:
| What | How it reads Dataverse | Guest users |
|---|---|---|
| "Recent records" list | Native data source (retrieveMultipleRecordsAsync), host token |
β Works |
| KPI tiles, every view-based grid, related grids, export | FetchXML β Dataverse connector, user's own connection | β 403 |
Same user. Same table. Same app. Two roads. One of them doesn't recognize the user.
π Why the connector road breaks
ListRecordsWithOrganization belongs to the "(environment)" flavour of the Dataverse connector: the operations designed to reach other environments. Microsoft documents that these operations don't use the native integration but go through the connectors platform, and that they can reach environments of the same tenant or of other tenants, "based on the connection used" (Connect to other environments using the Microsoft Dataverse connector).
For a B2B guest, the identity that this road presents to Dataverse is evidently not mapped to the guest systemuser of the host tenant. Hence: "The user is not a member of the organization."
So why on earth were we using the "talk to another environment" flavour? Not by choice, believe me.
The connector also has a plain ListRecords operation, the one meant for the current environment. That's the first one we tried, and on paper it was perfect: its schema doesn't even declare an organization parameter. "It figures out the organization from the context", we thought. Then we ran it inside the Power Apps host, and the answer was:
400 Bad Request
Invalid organization URL 'null' provided.
null. The operation that declares it doesn't need an organization fails because it doesn't have one. Without being told, the connector simply doesn't know which organization to query: the API hub does not infer it from the connection.
Fine, we thought, let's give it one. That's when we tried every alternative we could think of, running the SDK live inside the app's iframe:
| Attempt | Result |
|---|---|
ListRecords, no organization
|
β 400 Invalid organization URL 'null'
|
ListRecordsWithOrganization + environment id |
β fails |
ListRecordsWithOrganization + host name without https://
|
β fails |
GetOrganizations, to discover the URL at runtime |
β not even available among the app's data sources |
ListRecordsWithOrganization + full organization URL
|
β works |
And where do you get the full organization URL from, inside a Code App? Nowhere. It's not in getContext(). It's not in power.config.json. The SDK doesn't expose it. It doesn't even show up in the browser's Resource Timing, because the iframe never talks to Dataverse directly: the host does it on its behalf. Hence the environment variable.
Let that sink in for a second: the only connector operation that worked was the one built to reach any organization, and the only way to make it work was to hand it, explicitly, the address of the organization we were already running in. From the outside it looked like a harmless detail, a parameter more. In reality it meant that every FetchXML query of the app was taking the "other environment" road, the one that goes through the connectors platform and resolves the user through the connection, instead of the native integration.
I want to be honest here: this is the most likely explanation, not a proof. We couldn't observe the connector internals (the admin API doesn't tell you which account issued the token stored in a connection). But the evidence is consistent: every guest we checked fails, every member works, and the same guest succeeds on the native road.
In other words: we were using an operation built to talk to another environment to talk to our own environment. It worked for members, so nobody noticed. Guests are simply the first users for whom "another environment's road" and "our road" stop being equivalent.
π οΈ The solution: a Custom API that runs the FetchXML server-side
The goal is clear: take FetchXML off the connector and run it on the native channel. In our code, every FetchXML query goes through a single function, so the change is confined there and its callers don't move.
The road we're taking is a Custom API that receives the FetchXML, runs it on the server and returns the rows in the same OData shape the app already expects.
Why would this work for guests? Because in a Code App Custom APIs don't go through the connector. Add the Custom API as a Dataverse data source, call it through its generated service, and under the hood the SDK runs executeAsync with dataverseRequest: { action: 'customapi' } on the native channel, the same road as retrieveMultipleRecordsAsync.
Two warnings, if you go this way too:
- Call it through its generated service, not through a connector action like "Perform an unbound action" of the Dataverse connector: otherwise you're back on the road you were trying to leave.
- Run the query in the caller's context, not with the system service:
// The caller's service: RBAC applies, and eq-userid / eq-useroruserteams
// resolve to the calling user, not to SYSTEM.
var service = serviceFactory.CreateOrganizationService(context.UserId);
var rows = service.RetrieveMultiple(new FetchExpression(fetchXml));
Use the system service and you get two bugs for the price of one: security roles are bypassed, and "my records" suddenly means "SYSTEM's records".
It's the robust option: fully supported APIs only, and real paging, with page, count and paging cookie honoured by RetrieveMultiple on the server: exactly what the shortcut below couldn't give us. But it's also more work: a plug-in, the serialization of entities and formatted values into the OData shape, deployment, tests. Not exactly what you want to write while users are blocked in production. But here we are.
And there's one more thing that, honestly, bothers me more than the effort: performance.
Think about what a single grid page becomes. Today, ideally, it's one GET: Dataverse runs the FetchXML and streams back the JSON the grid renders. With a Custom API in the middle, the same page takes a detour through a whole extra layer:
- the app calls the Custom API, which enters the plug-in pipeline and runs your code in the sandbox;
- the plug-in calls
RetrieveMultiple, and Dataverse materializes the result as anEntityCollectionof .NET objects; - the plug-in walks that collection and serializes it to JSON, rebuilding by hand the OData shape the app expects: formatted values, lookup annotations, the count, the paging cookie;
- since the Code App SDK doesn't handle complex Custom API parameters, that JSON travels back as a string inside the JSON response of the Custom API;
- the SDK deserializes the response, and the app deserializes the string again to get its rows.
That's two extra serialization round trips for every query, on top of the plug-in execution itself. And it's not a query you run once in a while: it's every grid page, every counter on the dashboard, every page of an Excel export (up to 5000 rows each, hence a few megabytes of JSON escaped inside a string).
Will it be noticeably slower? My gut feeling said yes: a layer that doesn't add any business logic, just re-packages what Dataverse already knows how to return, is a layer you pay for on every click, forever.
So we measured it. Same query on all three roads: a typical "active records" view with 9 columns (lookups, choices, dates, money, a boolean), filtered, sorted, with the total count. Calls issued from the browser straight to the organization (not through the Player, whose overhead is the same for every native road), one warm-up round, then 15 alternated repetitions. Times in milliseconds:
| Query | Rows | Payload | min | median | p90 |
|---|---|---|---|---|---|
Full list, OData $select / $filter / $orderby
|
251 | 275 KB | 121 | 132 | 166 |
Full list, Web API ?fetchXml=
|
251 | 269 KB | 145 | 167 | 275 |
| Full list, Custom API | 251 | 249 KB | 161 | 179 | 255 |
| Page of 50, OData | 50 | 56 KB | 81 | 85 | 106 |
Page of 50, Web API ?fetchXml=
|
50 | 55 KB | 86 | 96 | 148 |
| Page of 50, Custom API | 50 | 50 KB | 109 | 123 | 234 |
How I read it:
- the Custom API costs roughly 40β50 ms more per call than a plain OData GET, and 10β30 ms more than the very same FetchXML run directly on the Web API. That's the price of the plug-in pipeline;
- the response is even a bit lighter, since it carries no
@odata.etagand no navigation property annotations; - the rows are the same: an automated comparison between the Web API and the Custom API on the same FetchXML (33 system views, the aggregate queries behind the KPIs, link-entities,
all-attributes, page 2 with the paging cookie) found no difference in keys, values, totals ormorerecords.
So: less than I feared, but not zero. A few tens of milliseconds on a grid page that already takes a hundred is hardly something a user will notice. Two caveats, to be fair: the test environment had 251 records, so a 5000-row export page is still unmeasured, and the old connector road can't be called from the browser, so there's no before/after comparison with what users had until now.
π§ͺ I also tried another road...
Before settling for the Custom API, I tried a shortcut. In the meantime we had upgraded the SDK to @microsoft/power-apps 1.6.0, and that seemed to open a door that simply didn't exist when we built the view engine.
In 1.6.0, retrieveMultipleRecordsAsync accepts a nextLink: the continuation URL Dataverse gives you to read the next page. Before using it, the SDK validates it, and here's the interesting part. Looking at the SDK code, the validation compares only the origin and the path of the URL with <instance url>/api/data/v9.0/<entity set> of the current environment. The query string is not checked.
And what is a FetchXML query in the Dataverse Web API? A GET on the entity set with a fetchXml query string parameter.
// β οΈ Relies on undocumented behaviour of @microsoft/power-apps 1.6.0:
// the nextLink is validated on origin and path only.
const nextLink =
`${instanceUrl}api/data/v9.0/${entitySet}` +
`?fetchXml=${encodeURIComponent(fetchXml)}`;
const result = await client.retrieveMultipleRecordsAsync<T>(entitySet, {
nextLink,
count: true, // without it, the total record count doesn't come back
maxPageSize: 5000,
} as IOperationOptions); // nextLink is not in the public options type
The FetchXML runs on the native channel, with the host's token, with odata.include-annotations=*, so formatted values for choices and lookups still arrive. No connector, no user connection, no "other environment" road, no plug-in. A handful of lines.
On paper, it was perfect. So I tried it.
Why it doesn't work: pagination
No error. The rows came back, formatted values included. The total record count was right. Everything looked fine... until I moved to page 2.
On a test table with 251 records, the grid proudly showed "Page 6 of 6 (251 records)". And on page 6 there were the same 50 rows of page 1. Every page returned the first rows. I tried passing the paging cookie of page 1 too: same result.
The culprit is not the nextLink: the SDK passes it through untouched. The culprit is a header. retrieveMultipleRecordsAsync always sends:
Prefer: odata.maxpagesize=500,odata.include-annotations=*
500 is the default: maxPageSize changes the number, but there's no way to remove the header. And when that header is there, Dataverse ignores page, count and paging-cookie of the FetchXML, and simply returns the first maxpagesize rows. We verified it calling the Web API directly, outside the SDK:
Prefer: odata.maxpagesize= |
What Dataverse returns |
|---|---|
50, FetchXML on page 2 |
rows 1β50. Again. |
5000, FetchXML with count="50"
|
all 251 rows |
0 or -1
|
all the rows |
| empty | HTTP 500 |
| not a number | HTTP 405, "Prefer Odata.maxpagesize value should be 32 bit integer" |
| no header at all | β the right page (page 6 = 1 row) |
The response does carry morerecords and the fetchxmlpagingcookie (but no @odata.nextLink), and the SDK throws both away anyway. So: we can't remove the header, we can't page by number, we can't page by cookie. Game over.
The consequences would have been nasty: grids showing the wrong rows from page 2 on, Excel exports past 5000 rows repeating the first 5000, "read everything" loops full of duplicates. And all of it silently: no error, no warning, just wrong data. A hotfix that shows the wrong rows without telling anyone is worse than the bug it's supposed to fix.
Shame, because this road had a nice side effect: with no FetchXML left on the connector, the Dataverse connection reference could have been removed from the app, and with it the "allow this app to use your Dataverse connection" consent every user has to click the first time. Oh well.
π― Conclusion
Let's be honest: this bug is not really about guest users. Guest users just pulled the curtain.
The real problem is that, today, a Code App has no first-class way to run FetchXML on its own environment. Views, counters, "my team's records": everything that makes a business app feel like a business app lives in FetchXML. And to get there we had to choose between a connector operation built for other environments, an undocumented trick in a nextLink, or a plug-in that re-packages what Dataverse already knows how to return.
About that trick: smuggling FetchXML into a nextLink was never an officially supported road. It relied on a validation that happened to be lenient, in an SDK that never promised to run FetchXML at all. It looked like a good idea (native channel, a few lines of code, no plug-in to deploy)... and it showed its limits the moment we asked it for something as basic as page 2. That's the thing with undocumented behaviours: they don't fail when you read the code, they fail when you rely on them.
None of these workarounds should be necessary. Model-driven apps have had Xrm.WebApi.retrieveMultipleRecords with FetchXML for years. Code Apps deserve the same.
Until then, the Custom API it is. The extra layer turned out to cost a few tens of milliseconds per call: a price I'm willing to pay to give guest users their grids back. I'll keep an eye on the big pages once it's live: if the 5000-row exports tell a different story, you'll read it here.
In the meantime, I'm genuinely curious: have you hit the same wall? Did you write a Custom API too, or did you find a road we haven't thought of? Let me know in the comments. π
Top comments (0)