{"id":508,"date":"2021-06-23T05:48:07","date_gmt":"2021-06-23T05:48:07","guid":{"rendered":"https:\/\/blogs.kcl.ac.uk\/kclip\/?p=508"},"modified":"2021-06-23T05:48:07","modified_gmt":"2021-06-23T05:48:07","slug":"how-similar-should-similar-tasks-be-in-meta-learning-ask-an-information-theorist","status":"publish","type":"post","link":"https:\/\/blogs.kcl.ac.uk\/kclip\/2021\/06\/23\/how-similar-should-similar-tasks-be-in-meta-learning-ask-an-information-theorist\/","title":{"rendered":"How similar should \u201csimilar\u201d tasks be in meta-learning? (Ask an information theorist)"},"content":{"rendered":"

Problem<\/strong><\/p>\n

Conventional learning optimizes model parameters using a training algorithm, while meta-learning optimizes the hyperparameters of a training algorithm. A meta-learner has access to data from a class of tasks, and its goal is to ensure that the resulting training algorithm, also called base-learner, performs well on any new tasks from the same class. For example, the base-learner could be a stochastic gradient descent (SGD) algorithm with hyperparameters like initialization or learning rate.<\/p>\n

The tasks observed during meta-training are conventionally assumed to belong to a task environment, which defines a distribution over the class of tasks, where each task has an associated data distribution. The statistical properties of the task environment then determine the similarity between the tasks. Intuitively, if the average \u201cdistance\u201d between data distributions of any two tasks in the task environment is small, the meta-learner should be able to learn a suitable shared hyperparameter by observing fewer number of tasks.<\/p>\n

In our recent work<\/a> accepted to ISIT 2021, we build on the above observation and address the following questions for a fixed base-learner and meta-learner: How to measure task similarity? Given the level of similarity of the tasks in the environment, how many tasks and how much data per task should be observed to guarantee that the target average population loss for new tasks can be well approximated using the available meta-training data?<\/p>\n

The difference between the average population loss on a new, previously unseen, meta-test task and the meta-training loss on the data gathered from the meta-training tasks is the meta-generalization gap<\/strong>, and is a measure of the generalization capability of the meta-learner. Our main contribution is a novel information theoretic bound<\/strong> on the average absolute value of the meta-generalization gap<\/strong>, that explicitly captures the impact of task relatedness, the number of tasks, and the number of data samples per task on meta-generalization.<\/p>\n

 <\/p>\n

Results<\/strong><\/p>\n

\u00a0<\/strong>Although information-theoretic bounds on generalization performance of meta-learning have been previously studied \u2013 in both average<\/a> and high probability PAC-Bayesian settings, they fail to capture the impact of task similarity in meta-generalization gap. \u00a0We identify the following distinguishing components of our analysis that enable the explicit characterization of task similarity:<\/p>\n