Contemporary AI governance typically treats ethics and law as complementary normative domains with different levels of enforcement. This paper argues that this distinction alone is insufficient given the evolving role of AI systems as artificial moral agents. Conceptualizing ethics and law as functionally distinct yet interdependent domains, the authors argue that the notion of functional agency to describe how AI systems generate normatively relevant outcomes through decision substitution, the embedding of societal norms in the design of AI systems, and behavioural steering. Drawing on case studies of recommender systems, large language models, autonomous driving, and care robotics, the paper demonstrates the systematic displacement of micro-level ethical reasoning by standardized, societal logics that tend to prioritize macro-level considerations such as beneficence over individual autonomy. This shift poses a structural challenge to human rights, which explicitly protect individual moral deliberation. The paper therefore calls for a paradigm shift in AI governance from embedding societal norms in AI systems to governing the construction of normativity itself grounded with an emphasis on the protection of individual ethical reasoning and stronger acknowledgement of moral pluralism.