The EU Commission’s Proposal for the Regulation of Artificial Intelligence and “human-centric” Regulation for AI as Medical Device

*Daria Onitiu


The EU Commission’s proposal for the Regulation of Artificial Intelligence (the ‘AI Act’) will bring significant changes to standard-setting for high-risk systems, including medical AI. The proposal’s risk-based approach intends to balance the socio-economic benefits of medical AI and the need for harmonised standards for safety-critical applications in healthcare. From medical diagnostic systems to the robot surgeon, medical AI illustrates the need for interdisciplinary perspectives for the formal governance of these novel tools in a dynamic healthcare setting.

Setting the scene regarding ‘high-risk’ systems regarding medical AI

A critical debate is that the AI Act considers almost all AI tools as ‘high-risk’. High-risk systems are those AI systems that, by their nature, such as increased autonomy and opacity, require enhanced mandatory obligations under the Act (Title III).  This risk-based classification has been criticised for enabling over-regulation of AI systems in healthcare. For instance, the AI Act’s broad-brush definition of “AI” including statistical and logical-based applications of algorithms ‘could also [encompass] systems that are not commonly considered AI being regulated by the Act, potentially impacting innovation’.[1]

Nevertheless, ‘the classification as high-risk does not only depend on the function performed by the AI system, but also on the specific purpose and modalities for which that system is used’, as rightly acknowledged in Title III of the proposal.  Therefore, it is not the proposal’s prescriptive although non-exhaustive nature of the risk-based framework which tips the balance regarding innovation and formal governance of AI systems. The EU’s vision is to promote holistic alignment of EU values with a product safety approach.

Accordingly, the real question is the AI Act’s consistency of values when considering the governance of software of AI as medical device. The EU Commission White Paper is clear in this regard in that the system’s ‘the sector and the intended use’ may create ‘distinct’ risks to fundamental rights and safety in healthcare, such as ‘safety considerations and legal implications concerning AI systems that provide specialized medical information to physicians, AI systems providing medical information directly to the patient and AI systems performing medical tasks themselves directly on a patient…’ (p. 17). The reasoning for this varied approach to AI governance is the need for ‘human-centric’ regulation to achieve trust- such as entailing the design of AI systems in healthcare with the involvement of human control, as well as enhanced transparency requirements addressing algorithmic opacity (p. 21). These values regarding the ‘systemic, individual, and societal aspects’ of technology ultimately shape the balance between innovation and regulation (p. 11).

Accordingly, the current discourse on AI governance is framed as a balancing exercise, considering the EU’s future efforts regarding the EU New Legislative Framework (Annex II of the AI Act). However, this process of value-alignment currently stagnates, considering the role of transparency of medical AI systems, based on efforts to align the regulation of medical AI and its procedural alignment with other sectoral legislation, including the Medical Device Regulation (MDR).

Medical AI: a balancing exercise

The current EU Commission’s proposal follows the spirit of other sectoral legislation, including the MDR, reinforcing the modalities including the associated problems of medical AI to shape human usage and decision-making beyond the laboratory setting. The proposal, just as the MDR, is a legal tool protecting product safety. With the AI Act, the EU Commission’s vision of human-centric regulation becomes an “ecosystem of protecting functionality and intended uses of AI” as medical devices.

Consequently, this perspective brings about two interesting problems that deserve further attention. One aspect is that formal governance remains tied to the system’s performance, intended use, and functionality. The European Coordination Committee of the Radiological, Electromedical and Healthcare IT Industry does recognise this, claiming that international standards need to consider lifecycle changes to software as medical device on the ground, and subsequently inform and update the MDR (p.11-13). The problems of technical documentation tied to the lack of standarisation still exist and the AI Act does not include the requirement to verify the system supporting clinical outcomes including patient-centred care. Second, the AI Act’s progressive outlook on transparency in Articles 13-14 stagnates concerning the contribution covering only the algorithms’ functional revelations of foreseeable risks, leaving out the subject’s perception of the nature of risk undermining shared decision-making in a healthcare setting.  This further reinforces gaps in the MDR to verify software as medical device via the levels of explanations for meaningful clinical outcomes.

Therefore, the considerations about AI technologies’ inherent, as well as distinct risks for fundamental rights and safety are transferred to the umbrella of the system’s innovative approach to outperform human judgement. Most requirements in the AI Act, including the ‘appropriate type and degree of transparency’, as well as the identification of technical safeguards for oversight (Article 13 (1); Article 14 (3) (a)) are left to the manufacturer. There is no appropriate involvement of the user, and the subject impacted by these novel technologies. Follow-up measures, such as post-market surveillance under both the AI Act and the MDR (Articles 61 and 89 respectively), will fulfil the function of monitoring life-cycle changes in product development, but do not provide the necessary confidence to develop safe and trustworthy systems considering EU values.

Value-alignment is key for legal certainty

What this shows is that we should not downplay the socio-economic impact of AI to a question of legislative competence but consider the question of safety-critical systems as being a task of value-alignment. Indeed, the significant overlap between the AI Act and MDR induces risks of double-standards at the cost of legal certainty, concerning the governance and enforcement of safety considerations. Nevertheless, what we need first is a risk-based approach that considers an interdisciplinary perspective of EU values into the modalities of AI systems, such as the use of Machine Learning approaches in healthcare. This way the focus on the analysis of prescriptive regulation of AI will shift to the formal governance of novel technologies in the long-term.

AI governance and medical AI: an interdisciplinary approach

The modalities of AI systems necessitate a new approach of standard-setting, which goes beyond a vision regarding the EU’s proactive approach restricted to an AI system’s functionality (p. 2). Restricting AI governance this way creates a false dichotomy stifling innovation, as well as rapid advancements of AI in “narrow” domains. An interdisciplinary approach to AI governance is tools testing a system’s operation on the ground, considering user perspectives of the tool’s reliability, a patient’s perception of risk, as well as core ethical values in decision-making including patient-centred care. This outlook will eventually provide a more consistent approach to AI governance in healthcare, as well as legal certainty.

(This blogpost is the author’s current work in progress. Please contact the author for the latest version of the work).

Details about the author

*Daria Onitiu is a Research Associate based at Edinburgh Law School. She researches at the Governance & Regulation Node within the UKRI Trustworthy Autonomous Systems Project. Her work intends to identify the transparency goals of medical diagnostic systems, and how to translate notions of accountability into a healthcare setting. Twitter @DariaOnitiu

[1] Huw Roberts, Josh Cowls, Emmie Hine, Francesca Mazzi, Andreas Tsamados, Mariarosaria Taddeo and Luciano Floridi, ‘Achieving a ‘Good AI Society’: Comparing the Aims and Progress of the EU and the US’ (2021) 27 (6) Science and Engineering Ethics 1, 6.


The editors of the KSLR EU Law Blog are pleased to announce recent publications in the field of EU law, with a discount attached for our readers.

Rightful Relations with Distant Strangers

Kant, the EU, and the Wider World

Aravind Ganesh

This book provides a philosophical critique of legal relations between the EU and ‘distant strangers’ neither located within, nor citizens of, its Member States. Starting with the EU’s commitment in Articles 3(5) and 21 TEU to advance democracy, human rights, and the rule of law in ‘all its relations with the wider world’, Ganesh examines in detail the salient EU and international legal materials and thereafter critiques them in the light of a theory of just global legal relations derived from Kant’s philosophy of right. In so doing, Ganesh departs from comparable Kantian scholarship on the EU by centering the discussion not around the essay Toward Perpetual Peace, but around the Doctrine of Right, Kant’s final and comprehensive statement of his general theory of law.

The book thus sheds light on areas of EU law (EU external relations law, standing to bring judicial review), public international law (jurisdiction, global public goods) and human rights (human rights jurisdiction), and also critiques the widespread identification of the EU as a Kantian federation of peace.

Aravind Ganesh is Vice Chancellor’s Research Fellow in Law at Oxford Brookes University, UK.

Mar 2021   |   9781509941315   |   352pp   |   Hbk   |    RSP: £70

Discount Price: £56

Order online at – use the code UG7 at the checkout to get 20% off your order!


Torture, Inhumanity and Degradation under Article 3 of the ECHR

Absolute Rights and Absolute Wrongs

Natasa Mavronicola

This book theorises and concretises the idea of ‘absolute rights’ in human rights law with a focus on Article 3 of the European Convention on Human Rights (ECHR). It unpacks how we might understand what an ‘absolute right’ in human rights law is and draws out how such a right’s delimitation may remain faithful to its absolute character. From these starting points, it considers how, as a matter of principle, the right not to be subjected to torture or inhuman or degrading treatment or punishment enshrined in Article 3 ECHR is, and ought, to be substantively delimited by the European Court of Human Rights (ECtHR). Focusing on the wrongs at issue, this analysis touches both on the core of the right and on what some might consider to lie at the right’s ‘fringes’: from the aggravated wrong of torture to the severity assessment delineating inhumanity and degradation; the justified use of force and its implications for absoluteness; the delimitation of positive obligations to protect from ill-treatment; and the duty not to expel persons to places where they face a real risk of torture, inhumanity or degradation.

Few legal standards carry the simultaneous significance and contestation surrounding this right. This book seeks to contribute fruitfully to efforts to counter a proliferation of attempts to dispute, circumvent or dilute the absolute character of the right not to be subjected to torture or inhuman or degrading treatment or punishment, and to offer the groundwork for transparently and coherently (re)interpreting the right’s contours in line with its absolute character.

Natasa Mavronicola is a Reader in Law and Deputy Head of Research at Birmingham Law School, University of Birmingham, United Kingdom.

Feb 2021   |   9781509902996   |   224pp   |   Hbk   |    RSP: £75

Discount Price: £60

Order online at – use the code UG7 at the checkout to get 20% off your order!

Cassis de Dijon

40 Years On

Edited by Albertina Albors-Llorens, Catherine Barnard and Brigitte Leucht

Why is the 1979 the Court of Justice judgment in Cassis de Dijon so famous and so significant in the evolution of EU trade law?. As this landmark judgment approaches middle age, this book revisits this decision with the benefit of hindsight: why did the Court of Justice decide Cassis de Dijonas it did? How has the decision been developed by the EU? And, looking forward, how has the decision been used to develop international trade? This book brings together some of the leading writers in the field of EU trade law, constitutional law and European history for a fresh examination of his ground-breaking judgment, looking at it from the perspective of its past (who, what and why); its present (is it making a difference?); and its future (how does it fit in international trade agreements, including the future UK-EU FTA?).


Albertina Albors-Llorens is Professor of EU Law at the University of Cambridge, United Kingdom.

Catherine Barnard is Professor of European Union and Labour Law at the University of Cambridge, United Kingdom.

Brigitte Leucht is Senior Lecturer in German and European Studies at the University of Portsmouth, United Kingdom.

Feb 2021   |   9781509936632   |   304pp   |   Hbk   |    RSP: £85

Discount Price: £68

Order online at – use the code UG7 at the checkout to get 20% off your order!

EMU Integration and Member States’ Constitutions

Edited by Stefan Griller and Elisabeth Lentsch

In this book, legal scholars from the EU Member States (with the addition of the UK) analyse the development of the EU Member States’ attitudes to economic, fiscal, and monetary integration since the Treaty of Maastricht.

The Eurozone crisis corroborated the warnings of economists that weak economic policy coordination and loose fiscal oversight would be insufficient to stabilise the monetary union. The country studies in this book investigate the legal, and in particular the constitutional, pre-conditions for deeper fiscal and monetary integration that influenced the past and might impact on the future positions in the (now) 27 EU Member States.

The individual country studies address the following issues:

– Main characteristics of the national constitutional system, and constitutional culture;

– Constitutional foundations of Economic and Monetary Union (EMU) membership and related instruments;

– Constitutional obstacles to EMU integration;

– Constitutional rules and/or practice on implementing EMU-related law; and

– The resulting relationship between EMU-related law and national law

Offering a comprehensive and detailed assessment of the legal and constitutional developments concerning the Economic and Monetary Union since the Treaty of Maastricht, this book provides not only a study of legal EMU-related measures and reforms at the EU level, but most importantly sheds light on their perception in the EU Member States.


Stefan Griller is Professor of European Law at the University of Salzburg, Austria.

Elisabeth Lentsch was formerly Post-Doc Researcher in the Horizon 2020 Project ‘The choice for Europe since Maastricht’, from 2015 to 2019.

Feb 2021   |   9781509935789   |   704pp   |   Hbk   |    RSP: £150

Discount Price: £120

Order online at – use the code UG7 at the checkout to get 20% off your order!

The Principle of Mutual Trust in EU Criminal Law

Auke Willems

This book develops a conceptual framework of the principle of mutual trust in EU criminal law. Mutual trust is a household term in the EU criminal law vocabulary and is widely regarded to be a prerequisite for a successful application of mutual recognition. But despite its importance, the parameters of the concept are not clear. The book demonstrates that mutual trust is multi-faceted: combining the elements essential to a successful EU criminal law, as part of the Area of Freedom, Security and Justice.

The book approaches trust from multiple angles. First, a study of social science literature. Second, a meticulous assessment of mutual trust in EU criminal law. Third, a study of trust in US interstate criminal justice cooperation. Finally, the book identifies a comprehensive approach to tackle trust related difficulties in EU criminal law.

This timely book will be of great interest to anyone looking to gain a full picture of this core principle in EU criminal law.


Auke Willems is Lecturer in EU Law at the University of Liverpool and a Visiting Fellow at the London School of Economics and Political Sciences.

Feb 2021   |   9781509924547   |   352pp   |   Hbk   |    RSP: £80

Discount Price: £64

Order online at – use the code UG7 at the checkout to get 20% off your order!

Call for Papers

 The KSLR EU Law Blog hereby invites you to submit abstracts for blog posts on

 any area of EU law

 (including but not limited to judicial protection, internal market, external relations, financial regulation, data-protection, environmental policies).

Submissions covering any of the following topics are particularly welcome:

  • The future of the EU: reflections and reforms
  • The external dimension of EU policies
  • The EU Charter of Fundamental Rights and its application at EU and national level
  • Protecting and enhancing the rule of law in the EU
  • Issues of law or policy of Member States with relevance for the EU

We also invite submissions on:

  • Coverage of EU law-related events
  • Reviews of recently published EU law-related books as well as
  • Recent developments of EU case law

We are looking for 800 – 2000 words articles.

Please refer to our style guidelines.

Please send abstracts or full articles to and by 20 February 2018.

We look forward to hearing from you!

The KSLR EU Law Blog Editorial Team

Europe.think.again – Call for Ideas

Dear All,

The KSLR EU Law Blog is pleased to share with you a EU-related project managed by “Foraus Global”, a Swiss think tank on foreign policy.

In view of the 60th anniversary of the Treaty of Rome establishing the European Economic Community on 25 March 2017, Foraus Global is publishing a short collection of ideas to reform European cooperation.

For more information, please visit the following page: europe.think.again project.

If you are interested in participating or you have further questions, please contact Laura Knopfel ( 

We look forward to hearing from you!

The KSLR EU Law Blog editorial team


KSLR Brexit Series

The KSLR EU Blog is pleased to announce that over the next few weeks it will publish a series of articles on the EU referendum and the possibility of Britain leaving the EU (‘Brexit’).  We’ll look at issues such as Ireland’s experience with referendums on EU matters, how other Union members and the USA view ‘Brexit’, the legal technicalities of the referendum which are causing arguments between the ‘In’ and ‘Out’ camps and a review of the new book Britain Alone: The Implications and Consequences of the UK Exit from the EU. 
We’ll kick off today with Anne Wesemann’s post : ‘IN or OUT – WIN or LOSE? Who is really going to feel the Brexit?’