1. The Problem
The authors start from a realistic problem.
Most Federated Learning research assumes that clients already have a labeled dataset.
But what happens when data is continuously arriving at the client without labels?
For example:
Hospital
↓
Patient image arrives
↓
No label yet
↓
Doctor / expert must annotate it
↓
Then it can be used for training
Imagine a client continuously receiving new data.
The problem is that getting an expert to label every single sample is expensive.
So you can't simply label everything.
This creates the question:
Which incoming samples should the client spend its labeling budget on?
This is the data-querying problem that the paper is trying to solve.
2. The Deeper Problem
Now, a deeper problem arises.
Suppose Client 1 receives:
A, B, C, D, E
But it only has enough budget to label 2 samples.
The client might think:
"A and B are the most useful for my local model."
That sounds reasonable.
But there is a problem.
Federated Learning is not trying to optimize only Client 1's model.
It is trying to optimize the global model across all clients.
So the samples that are most useful from Client 1's local perspective might not necessarily be the samples that are most useful for the global training objective.
The authors describe this conflict between the local data-access perspective and the global training objective as a fundamental challenge.
And this is the key problem that LeaDQ tries to attack.
3. The System Architecture
SERVER
↓
Global Model
↓
┌───────────┴───────────┐
↓ ↓
Hospital 1 Hospital K
↓ ↓
Unlabeled images Unlabeled images
↓ ↓
Query policy Query policy
↓ ↓
Select samples Select samples
↓ ↓
Doctor / expert Doctor / expert
labels labels
↓ ↓
Local dataset Local dataset
↓ ↓
Local training Local training
└───────────┬───────────┘
↓
SERVER
↓
Global Model
The overall process has two main phases.
Phase 1 — Data Querying
Unlabeled data
↓
Selected Samples
↓
Obtain labels
Clients receive incoming unlabeled data.
The querying policy decides which samples are worth sending to an oracle for labeling.
Phase 2 — Federated Training
So the overall idea is basically:
Unlabeled data
↓
Selected Sample
↓
Get labels
↓
FedAvg
↓
New Global model
↓
Repeat
The selected samples are labeled and then used for federated training.
After training, more unlabeled data arrives and the process repeats.
4. How Does the Client Decide What to Select?
This is where LeaDQ becomes a reinforcement learning problem.
For each incoming sample, the policy makes a binary decision.
1 = query this sample
0 = don't query this sample
The decision is basically:
Select the sample?
- Yes → query the oracle for its label
- No → don't query it
So if 5 samples arrive:
x1 x2 x3 x4 x5
The policy might make decisions like:
1 0 1 0 0
and then:
x1 and x3
Are then sent to the oracle for labeling.
But there is an important constraint.
The client only has a limited querying budget.
So:
Number of selected samples = Nq
The policy therefore has to decide which samples are worth spending that limited budget on.
5. What Information Does the Query Policy See?
The policy needs some information before deciding whether to select an incoming sample.
The observation it receives includes the model's predictive logits.
Compute Predicitive logits
↓
local Observation
↓
Query Policy
↓
Q-Values
↓
Choose Samples
So the policy gets information about:
"How does the current model respond to this incoming data?"
For example, if the model is very uncertain about a sample, that information can potentially influence the querying decision.
The important thing here is that the policy isn't simply looking at the raw sample and randomly deciding whether to select it.
It is using information from the current model's response to the incoming data.
6. Why Reinforcement Learning?
This is probably the part that made the most sense to me once I understood the problem.
The client doesn't immediately know:
"Was choosing this sample a good decision?"
For example:
Client Selects:[](
)
Sample X
↓
Get labels
↓
Trains
↓
Global model Change
↓
Global Accuracy Improves
Suppose the client chooses sample A.
At that exact moment, it doesn't know whether A will actually help the global model.
The consequence of that decision might only become visible after the sample is labeled and used during federated training.
So there is a delay between:
Decision → Training → Global model performance
That is exactly the kind of problem reinforcement learning can model.
The querying policy takes an action, and later receives information about the consequences of that action.
The authors therefore formulate the querying problem as a:
Decentralized Partially Observable Markov Decision Process (Dec-POMDP).
7. Why "Partially Observable"?
This part is important.
Each client only sees its own information.
Client 1 doesn't know exactly what data Client 2 has.
Client 2 doesn't know exactly what Client 3 has.
And so on.
But the objective they're ultimately trying to optimize depends on the global federated learning system.
So each client only has a partial view of the overall environment.
Hence:
Decentralized + Partially Observable → Dec-POMDP
Decentralized
+
Partially Observable
+
Multiple Decision-Makers
8. They Use MARL
Now we have another problem.
There isn't just one client making decisions.
There are multiple clients making querying decisions.
So the authors use:
Multi-Agent Reinforcement Learning (MARL).
Each client can be treated as an agent.
Client 1 → Policy y1
Client 2 → Policy y2
Client K → Policy yk
The policies make their decisions locally.
So Client 1 has its own policy.
Client 2 has its own policy.
Client 3 has its own policy.
And they are all making decisions based on their own local observations.
9. But How Do Local Policies Learn About the Global Objective?
This is where Centralized Training with Decentralized Execution (CTDE) comes in.
The basic idea is:
During training
The system can use global information to help train the policies.
During actual operation
Each client has to make decisions using only its local information.
So:
Training can be centralized, but execution remains decentralized.
This makes sense for Federated Learning because we don't want every client to require access to all the other clients' private data just to make a querying decision.
10. Their Biggest Technical Component: QMIX
The authors adapt QMIX for the multi-agent policy training.
Each client has its own local Q-function:
Q₁, Q₂, Q₃, ..., Qₖ
These local Q-functions represent the value of actions from the perspective of each individual agent.
But the overall goal is still the global objective.
QMIX is used to combine these individual value functions into a global value function.
Individual client decisions
↓
Individual Q-functions
↓
QMIX
↓
Global Q-value
The important idea is that the clients can learn policies that make local decisions while still being trained toward a shared global objective.
11. What I Think Is Interesting About This Paper
What I found interesting is that the paper doesn't treat data selection as simply:
"Which samples are the most useful to this client?"
Instead, it asks a much harder question:
"Which samples should this client query so that they ultimately help the global Federated Learning objective?"
That difference is important.
A sample can be very useful for one client's local model but not necessarily useful for the overall federated system.
LeaDQ tries to bridge that gap by treating the querying process as a multi-agent decision-making problem.
12. Limitations
The approach is interesting, but it also introduces quite a lot of complexity.
Instead of simply running Federated Learning, we now have:
Data querying
→ Reinforcement Learning
→ Multi-Agent Reinforcement Learning
→ Dec-POMDP
→ CTDE
→ QMIX
→ Federated Training
That is a lot of machinery around the original problem of deciding which data to label.
There is also the practical question of how well these policies can be trained and how much additional computation and coordination the approach requires.
So while the idea is interesting, implementing something like LeaDQ is definitely more complicated than simply running a standard FedAvg system.
Final Thoughts
The biggest thing I took away from this paper is that data selection in Federated Learning is not just a local optimization problem.
When data is unlabeled and labeling is expensive, the client has to decide where to spend its limited labeling budget.
But the client's decision should ultimately help the global model, not just its own local model.
LeaDQ approaches this by turning the data-querying problem into a multi-agent reinforcement learning problem, where clients make decisions locally while being trained toward a global objective.
And that was my main takeaway from Paper 5.
There is a lot more technical detail in the paper, especially around the QMIX formulation and how the querying policy is trained, but this was my attempt to understand the core idea behind the work before diving deeper into the implementation.
Paper: Learn How to Query from Unlabeled Data Streams in Federated Learning
Top comments (0)