{"id":956,"date":"2025-05-02T08:45:56","date_gmt":"2025-05-02T08:45:56","guid":{"rendered":"https:\/\/blogs.kcl.ac.uk\/kclip\/?p=956"},"modified":"2025-05-02T08:46:37","modified_gmt":"2025-05-02T08:46:37","slug":"adaptive-learn-then-test","status":"publish","type":"post","link":"https:\/\/blogs.kcl.ac.uk\/kclip\/2025\/05\/02\/adaptive-learn-then-test\/","title":{"rendered":"Adaptive Learn-Then-Test"},"content":{"rendered":"

Motivation<\/strong><\/h2>\n
\n

Hyperparameter selection is a fundamental step in deploying machine learning models, aimed at assessing whether a model meets specified requirements in terms of performance, robustness, or safety. Recent approaches based on the Learn-Then-Test <\/b>(LTT) [1] framework formulate this task as a multiple hypothesis testing procedure<\/b>. For each candidate hyperparameter, LTT tests whether the corresponding model meets a target reliability level by evaluating it on multiple instances of the task (e.g., deploying the model in real-world scenarios). Despite its theoretical guarantees, LTT supports only non-adaptive testing<\/b>, where all evaluation decisions and the length of the testing phase must be fixed in advance. This rigidity limits its practical utility in safety-critical environments, where minimizing the cost of testing is essential.<\/p>\n

E-process-based testing<\/strong><\/h2>\n
\n

To overcome this limitation, our recent work<\/a>\u2014accepted at ICML 2025\u2014introduces adaptive Learn-Then-Test (aLTT)<\/b><\/span>, a statistically rigorous, sequential testing framework that enables efficient, data-driven hyperparameter selection with provable reliability guarantees. The core innovation behind aLTT is its use of e-process-based multiple hypothesis testing [2]<\/b><\/span>, which replace the traditional p-value-based testing employed in LTT. E-processes support sequential, data-adaptive hypothesis testing while maintaining formal statistical guarantees.<\/p>\n<\/div>\n

\n

Practically speaking, as illustrated in Figure 1, this means that at each testing round, the experimenter can decide\u2014based on the accumulated evidence\u2014whether to continue testing specific hyperparameters or to stop if a sufficiently large set of reliable candidates has been identified. All of this is achieved without sacrificing<\/b><\/span> the statistical guarantees of the procedure in terms of family-wise error rate (FWER)<\/b><\/span> or false discovery rate (FDR)<\/b><\/span> control. This stands in sharp contrast to p-value-based approaches, where such flexibility would invalidate the statistical guarantees of the procedure. An insidious problem known as p-hacking.<\/p>\n

\"\"

Figure 1: aLTT enables data-adaptive testing and flexible termination rules. At each testing round, based on the accumulated evidence, it is possible to decide which hyperparameters to test next and whether to continue testing.<\/p><\/div>\n

Automated Prompt Engineering<\/h2>\n
\n

The aLTT framework is broadly applicable to any setting where reliable configuration must be achieved under limited testing budgets. In our paper, we demonstrate its effectiveness in three concrete domains: configuring wireless network policies, selecting offline reinforcement learning strategies, and optimizing prompts for large language models. In the prompt engineering<\/b><\/span> setting [3], the goal is to identify instructions (prompts) that consistently lead an LLM to generate accurate, relevant, or high-quality responses across tasks. Since each prompt must be tested by running the LLM\u2014often a costly operation\u2014efficiency is critical. aLTT enables the sequential testing of prompts, adaptively prioritizing those that show early promise and terminating the process once enough reliable prompts are found. As shown in Figure 2<\/b><\/span>, this not only reduces the computational burden (yielding a higher true discovery rate under the same testing budget), but also leads to the discovery of shorter, more effective prompts<\/b><\/span>\u2014a valuable property in latency-sensitive<\/b><\/span> or resource-constrained<\/b><\/span> environments. The result:<\/b><\/span> fewer evaluations, higher-quality prompts, and rigorous statistical reliability.<\/p>\n

\"\"

(Left) True positive rate as a function of the testing horizon attained by aLTT with $\\epsilon$-greedy exploration and LTT. (Right) Length of the shortest prompt in the predicted set of reliable hyperparameters retuned by aLTT and LTT. aLTT needs fewer testing round to return high quality and short prompts<\/p><\/div>\n<\/div>\n

References<\/h3>\n

[1] Angelopoulos AN, Bates S, Cand\u00e8s EJ, Jordan MI, Lei L. Learn then test: Calibrating predictive algorithms to achieve risk control. arXiv preprint arXiv:2110.01052. 2021 Oct 3.<\/p>\n

[2] Xu Z, Wang R, Ramdas A. A unified framework for bandit multiple testing. Advances in Neural Information Processing Systems. 2021 Dec 6;34:16833-45.<\/p>\n

[3] Zhou Y, Muresanu AI, Han Z, Paster K, Pitis S, Chan H, Ba J. Large language models are human-level prompt engineers. InThe Eleventh International Conference on Learning Representations 2022 Nov 3.<\/p>\n<\/div>\n<\/div>\n","protected":false},"excerpt":{"rendered":"

Motivation Hyperparameter selection is a fundamental step in deploying machine learning models, aimed at assessing whether a model meets specified requirements in terms of performance, robustness, or safety. Recent approaches based on the Learn-Then-Test (LTT) [1] framework formulate this task as a multiple hypothesis testing procedure. For each candidate hyperparameter, LTT tests whether the corresponding […]<\/p>\n","protected":false},"author":1311,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-956","post","type-post","status-publish","format-standard","hentry","category-uncategorized","post-preview"],"_links":{"self":[{"href":"https:\/\/blogs.kcl.ac.uk\/kclip\/wp-json\/wp\/v2\/posts\/956","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blogs.kcl.ac.uk\/kclip\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blogs.kcl.ac.uk\/kclip\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blogs.kcl.ac.uk\/kclip\/wp-json\/wp\/v2\/users\/1311"}],"replies":[{"embeddable":true,"href":"https:\/\/blogs.kcl.ac.uk\/kclip\/wp-json\/wp\/v2\/comments?post=956"}],"version-history":[{"count":7,"href":"https:\/\/blogs.kcl.ac.uk\/kclip\/wp-json\/wp\/v2\/posts\/956\/revisions"}],"predecessor-version":[{"id":965,"href":"https:\/\/blogs.kcl.ac.uk\/kclip\/wp-json\/wp\/v2\/posts\/956\/revisions\/965"}],"wp:attachment":[{"href":"https:\/\/blogs.kcl.ac.uk\/kclip\/wp-json\/wp\/v2\/media?parent=956"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blogs.kcl.ac.uk\/kclip\/wp-json\/wp\/v2\/categories?post=956"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blogs.kcl.ac.uk\/kclip\/wp-json\/wp\/v2\/tags?post=956"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}