![\"\"](\"https:\/\/blogs.kcl.ac.uk\/kclip\/files\/2022\/11\/Fig1.jpg\")
Fig. 1. The hard decision regions of an optimal demodulator (dashed lines) and of a data-driven demodulator trained on few pilots (solid lines) are displayed in panel (a), while the corresponding probabilistic predictions for some outputs are shown in panel (b).<\/p><\/div>\n
<\/p>\n
The property of \u201cknowing what the AI knows\/ does not know\u201d is very useful when the AI module is used as part of a larger engineering system. In fact, well-calibrated decisions should be treated differently depending on their confidence level. Furthermore, well-calibrated models enable monitoring \u2013 by tracking the confidence of the decisions made by an AI \u2013 and other functionalities, such as anomaly detection [2].<\/p>\n
In a recent paper from our group published on the IEEE Transaction on Signal Processing [3], we proposed a methodology to develop well-calibrated and efficient AI modules that are capable of fast adaptation. The methodology builds on Bayesian meta-learning.<\/p>\n
To start, we summarize the main techniques under consideration.<\/p>\n
- \n
- Conventional, frequentist, learning<\/strong> ignores epistemic uncertainty \u2013 uncertainty caused by limited data \u2013 and tends to be overconfident in the presence of limited training samples.<\/li>\n
- Bayesian learning <\/strong>captures epistemic uncertainty by optimizing a distribution in the model parameter space, rather than finding a single deterministic value as in frequentist learning. By obtaining decisions via ensembling, Bayesian predictors can account for the \u201copinions\u201d of multiple models, hence providing more reliable decisions. Note that this approach is routinely used to quantify uncertainty in established fields like weather prediction [4].<\/li>\n
- Frequentist meta-learning<\/strong> [5], also known as learning to learn, optimizes a shared training strategy across multiple tasks, so that it can easily adapt to new tasks. This is done by transferring knowledge from different learning tasks. As a communication system example, see Fig. 2 in which the demodulator adapts quickly with only few pilots for a new frame. While frequentist meta-learning is well-suited for adaptation purpose, its decisions tend to be overconfident, hence not improving monitoring in general.<\/li>\n
- Bayesian meta-learning<\/strong> [6,7] integrates meta-learning with Bayesian learning in order to facilitate adaptation to new tasks for Bayesian learning.<\/li>\n
- Bayesian active meta-learning<\/strong> [8] Active meta-learning can reduce the number of meta-training tasks. By considering streaming-fashion of availability of meta-training tasks, e.g., sequential supply of new frames from which we can online meta-learn the AI modules, we were able to effectively reduce the time required for satisfiable meta-learning via active meta-learning.<\/li>\n<\/ol>\n
<\/p>\n
![\"\"](\"https:\/\/blogs.kcl.ac.uk\/kclip\/files\/2022\/11\/Fig2.jpg\")
Fig. 2. Through meta-learning, a learner (e.g., demodulator) can be adapted quickly using few pilots to new environment, using hyperparameter vector optimized over related learning tasks (e.g., frames with different channel conditions).<\/p><\/div>\n
<\/p>\n
Some Results<\/h1>\n
We first show the benefits of Bayesian meta-learning for monitoring purpose by examining the reliability of its decisions in terms of calibration. In Fig. 3, reliability diagrams for frequentist and Bayesian meta-learning are compared. For an ideal calibrated predictor, the accuracy level should match the self-reported confidence (dashed line in the plots). In can be easily checked that AI modules designed by Bayesian meta-learning (right part) are more reliable than the ones with Frequentist meta-learning (left part), validating the suitability of Bayesian meta-learning for monitoring purpose. Experimental results are obtained by considering a demodulation problem.<\/p>\n
<\/p>\n
<\/p>\n
![\"\"](\"https:\/\/blogs.kcl.ac.uk\/kclip\/files\/2022\/11\/Fig3.jpg\")
Fig. 3. Bayesian meta-learning (right) yields reliable decisions as compared to frequentist meta-learning (left) which can be captured via reliability diagrams [9].<\/p><\/div>Fig. 4 demonstrates the impact of Bayesian active meta-learning that successfully reduces the number of required meta-training tasks. The results are obtained by considering an equalization problem.<\/p>\n
![\"\"](\"https:\/\/blogs.kcl.ac.uk\/kclip\/files\/2022\/11\/Fig4.jpg\")
Fig. 4. Bayesian active meta-learning actively searches for meta-training tasks that are most surprising (left), hence increasing the task efficiency as compared to Bayesian meta-learning which randomly chooses tasks to be meta-trained.<\/p><\/div>\n
<\/p>\n
References<\/h1>\n
[1] O-RAN Alliance, \u201cO-RAN Working Group 2 AI\/ML Workflow Description and Requirements,\u201d ORAN-WG2. AIML. v01.02.02, vol. 1, 2.<\/p>\n
- Bayesian learning <\/strong>captures epistemic uncertainty by optimizing a distribution in the model parameter space, rather than finding a single deterministic value as in frequentist learning. By obtaining decisions via ensembling, Bayesian predictors can account for the \u201copinions\u201d of multiple models, hence providing more reliable decisions. Note that this approach is routinely used to quantify uncertainty in established fields like weather prediction [4].<\/li>\n