Theory and Practice of Social Media’s Content Moderation by Artificial Intelligence in Light of European Union’s AI Act and Digital Services Act

After a brief, general introduction to AI, the present article will discuss whether AI itself has freedom of expression or whether it only entitles to tech companies that own AI. All this is relevant in the context of whether we can consider AI a separate legal actor in the context of content creation on social media, which is the second main issue of this article. Here, negative and positive cases of content moderation will be presented, i.e., whether the full spectrum of content moderation in social media can be entrusted to AI as a separate actor with respect to liability, or whether moderation requires some form of human control, direction or supervision.

Introduction

Although Gartner’s so-called Hype Cycle curve exploring the rise of technology shows that generative AI has already entered the disappointing phase in August 2024 (Gartner, 2024), we are seeing more and more software with some form of AI. This trend cannot escape social media platforms, for example, Meta, the company behind Facebook and Instagram, is developing its own AI model called Llama, which users can feel free to tailor to their preferences (Meta, 2021). In addition, Meta is openly committed to using AI to filter content (Meta, 2021), as are X/Twitter and TikTok.

The rise of artificial intelligence, maybe seeming sudden to the public, has not escaped the attention of legislators. The problem, however, is that–in the words of Kinga Pázmándi– “on one plate of the sliding scale are the protection goals (...) that the law is supposed to safeguard, while on the other end is the socially beneficial innovation, and the »adjustment of the values of the sliding scale« is accompanied by constant (...) uncertainty” (Pázmándi, 2018, p. 11). At the time of writing the present study, within the last year, the European Union has adopted the EU AI Act¹ 1 Regulation (EU) No 2024/1689 of the European Parliament and of the Council of 13 June 2024, laying down harmonised rules on artificial intelligence and amending Regulations (EC) No 300/2008, (EU) No 167/2013, (EU) No 168/2013, (EU) 2018/858, (EU) 2018/1139, (EU) 2019/2144 and Directives 2014/90/EU, (EU) 2016/797 and (EU) 2020/1828 (the Artificial Intelligence Regulation), PE/24/2024/REV/1, OJ L 2024/1689, 12.7.2024, 12.7.2024. (hereinafter: AI Act) and the accompanying Proposal for a Directive on adapting non-contractual civil liability rules to artificial intelligence² 2 Directive of the European Parliament and of the Council on the adaptation of the rules on non-contractual civil liability to artificial intelligence (Directive on liability in the field of artificial intelligence) COM(2022) 496 final 2022/0303 (COD). and the President of the United States has issued an Executive Order on the Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence (hereinafter: Executive Order 14110).³ 3 Executive Order (E.O.) 14110 on Safe, Secure and Trustworthy Development and Use of Artificial Intelligence, 88 FR 75191.

Although there is no generally accepted definition of artificial intelligence in the current literature (Nikolinakos, 2023), both the EU and the US define what is meant by artificial intelligence. According to the AI Act, means “a machine-based system that is designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment, and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments.”⁴ 4 AI Act Article 3(1). And according to Executive Order 14110, artificial intelligence is a “machine-based system that can, for a given set of human-defined objectives, make predictions, recommendations, or decisions influencing real or virtual environments. Artificial intelligence systems use machine- and human-based inputs to perceive real and virtual environments; abstract such perceptions into models through analysis in an automated manner; and use model inference to formulate options for information or action.”⁵ 5 E.O. 14110 Section 3 (b).

In contrast to legislators, the literature is more sentimental, focusing on similarities or differences with human thinking.⁶ 6 For a summary of the characteristics of strong and weak artificial intelligences, see Boryet al. (2024). When the literature examines the freedom of expression of AI–and the related matter of personality or the lack thereof–it tends to focus on the less technical, more human aspects, which is the focus of the first part of this article. The article then examines the general issues and regulation of content moderation in the EU, and the role and potential–positive and negative–of AI in connection with content moderation practices.

The Freedom of Expression of Artificial Intelligence

In case of some artificial intelligences, we have seen examples of them being given some kind of right separate from their author, as for example in the case of Sophia (Fuchs, 2024)⁷ 7 For example, Sophia has been granted citizenship in Saudi Arabia. and Mirai (Alqodsi & Gura, 2023).⁸ 8 Mirai was granted a permanent establishment permit independently of the operating company. However, due to the increasing and widespread use of AI, liability issues are in the forefront of relevant literature,⁹ 9 E.g., see Montagnaniet al. (2024). while, if we leave aside the physical dimension, robot law has long been concerned with the possible rights of man-made artificially intelligent machines, including the right to communicate (Vadymovych, 2017).

When it comes to artificial intelligence and freedom of expression it is worth examining who is the subject to freedom of expression. The subject could be artificial intelligence, or an underlying subject, namely (a) the programmer (author), (b) the owner/operator, or (c) the user.

From the possible subjects, the personality of AI itself is the most controversial. The possibility of recognising AI as a subject of freedom of expression seems to be a valid possibility in the academic literature (Koopset al., 2010), for which different criteria are defined by different authors, ranging from quite technical, relatively simple to abstract conditions. One simple criterion, for example, is that the recognition of AI’s own freedom of expression is conditional on the outcome being indeterminate and unpredictable (Świerczyński & Więckowski, 2023). A similar position is taken by Tim Wu, who argues that one can speak of a distinct personality if (1) one is capable of conceptual thinking and (2) one is able to express one’s formed opinions (Wu, 2013).¹⁰ 10 See also in this context Autronic AG v Switzerland App no. 12726/87 (ECtHR, 22 May 1990). Moving on to more abstract criteria, Vadymovych (2017) argues that in order to recognise AI’s freedom of expression, it is necessary to consider whether the subject of the inquiry has a socially complex functionality that pushes the boundaries of human autonomy, and Schwitzgebel and Garza (2015) argue that adequate social status may be sufficient. Noteworthy, however, that AI may also be given legal personality–and thus also certain rights–and that the granting of legal personality is a discretionary decision of the legislator and, as such, the decision may also follow considerations of practicality (Ribeiroet al., 2024).

From another perspective, however, AI is not intrinsic, some kind of human activity or input is always required it (Kaminski & Jones, 2024), so if we do not attribute freedom of expression to AI, we need to look at the person behind it (Hines, 2019; Pizzetti, 2021). In case of the programmer, the operator and the user, the roles are intersecting and there is no sharp boundary between them. With regard to moderation, the following examples can illustrate possible scenarios:

a) The programmer, the operator and the user are the same. In this case, the same company programs, operates (troubleshoots, provides support) and uses (prompts, parameterises, makes decisions based on output),

b) The company outsources the development of artificial intelligence to another company, but it operates and uses it itself,

c) The company just acquires and uses the AI, while the programming and operation remains with someone else.

Whereas in case a) the situation is clear, the rights and responsibilities are all being linked to the company under scrutiny, in case c) the question is not only how much of the software is compact, so-called commercial off-the-shelf software, but how much of the software is custom-developed. With regards to the role of the operator, the AI Act can be followed as a guideline, which considers the service provider, the user, the authorised representative, the importer and the distributor as falling within this role, if the AI is considered to be hazardous operation.¹¹ 11 AI Act Article 3 (3–7).

However, from these actors, we will presumably consider the one who set the parameters necessary to generate (or in this case moderate) the content as the speaker (Hines, 2019). What the roles of the programmer, operator and user have in common, however, is that they all have a natural and/or legal personality, which gives them freedom of expression. With regard to social media providers, due to the proprietary development, the activity of AI will presumably be attributed to the providers, especially given the fact that the policies and guidelines that need to be implemented are also drafted by the provider (Robison, 2024).

Other authors, however, do not address the question of legal personality, but opt for a different justification for defending the output of AI, partly because in case of AI, precisely because of its unpredictability, its activity may not be attributable to humans, it may not be easily traceable back to human activity (Higby, 2021). These arguments include:

1. There is no doctrine that would explicitly exclude artificial intelligence from the scope of the First Amendment (Koopset al., 2010).

2. The output of AI may contain elements that are of interest or importance to people, or it may contain communications that are otherwise protected by freedom of expression (Koopset al., 2010). In turn, they participate in democratic debates and information dissemination (Kaminski & Jones, 2024). As part of this argument, the right of the audience to access information as part of freedom of expression should be highlighted.¹² 12 European Convention on Human Rights (ECHR) Article 10.

In case of AI-generated messages, there are consequences both for accepting that AI has freedom of expression and for not accepting as well. If AI does not fall within the scope of freedom of expression simply because it does not come directly from a human being, then it can be used to censor otherwise protected speech (Higby, 2021). If, on the other hand, it is not excluded from the scope of freedom of expression, then there may be a problem in judging speech that is not otherwise protected, depending on the speaker’s intent, and there may be a question of liability (Higby, 2021). In case of social media, where proprietary artificial intelligence is actually used for content moderation, the situation is clear: both the development and the parameterisation (the drafting of the policies to be implemented and the way they are implemented) are united under one hand (Robison, 2024).

General Issues and Regulation of Content Moderation in the EU

Content Moderation and DSA, Particularly the Relevant Transparency Requirements

In March 2021, Mark Zuckerberg told during his congressional hearing that 95% of hate speech content and, to the best of his knowledge, 98%–99% of terrorist content is now being identified by artificial intelligence, instead of humans (House Energy and Commerce Subcommittee on Communications & Technology, 2021). Regulating the moderation practices and policies of Facebook, X/Twitter and other social media platforms around the world is a major challenge. Regulation (EU) 2022/2065 of the European Parliament and of the Council of 19 October 2022 on a Single Market for Digital Services and amending Directive 2000/31/EC (hereinafter: the DSA)¹³ 13 Regulation (EU) 2022/2065 of the European Parliament and of the Council of 19 October 2022 on a Single Market for Digital Services and amending Directive 2000/31/EC (Digital Services Act), PE/30/2022/REV/1, OJ L 277, 27.10.2022, p. 1–102. primarily regulates intermediary service providers, including online platforms, complementing other EU efforts to regulate AI through provisions on content moderation and algorithmic transparency and accountability requirements (Brown, 2023).

In this context, the DSA also seeks to address some of the problematic issues of content moderation, with the objectives of ensuring transparency and accountability¹⁴ 14 DSA Recital 49. and the effective remedy of content moderation decisions by online platform providers.¹⁵ 15 DSA Recitals 44, 109.

Although the DSA defines the concept of content moderation, it has not previously had a–and still does not have in literature universally accepted–definition, but it is rather considered an umbrella-term (Gosztonyi, 2023). From a scientific standpoint, content moderation can be grasped as encompassing all actions of social media service providers that raise fundamental rights issues, in all cases involving user content by restricting access to it by human resources or artificial intelligence, but which do not presuppose the illegality of the content concerned, since their primary “legal basis” is to be found in the contractual terms of the platforms rather than in legislation and which is usually used by the platform to pursue its own economic or social agenda (Koltay, 2024) or to avoid liability, thus lacking transparency and accountability when it comes to decision-making. Moderating due to the platform’s own economic interest (in order to avoid users encountering offensive, outrageous or disturbing content) is more predominant the more frequent the exchanges of views between users and the more intense the communication is (Koltay, 2024). By contrast, content moderation under the Article 3 of the DSA means: “the activities, whether automated or not, undertaken by providers of intermediary services, that are aimed, in particular, at detecting, identifying and addressing illegal content or information incompatible with their terms and conditions, provided by recipients of the service, including measures taken that affect the availability, visibility, and accessibility of that illegal content or that information, such as demotion, demonetisation, disabling of access to, or removal thereof, or that affect the ability of the recipients of the service to provide that information, such as the termination or suspension of a recipient’s account.”¹⁶ 16 DSA Article 3. t).

Comparing the above two definitions, we can clearly see that there are substantial differences between the two definitions, the DSA does not make the fundamental rights aspect a conceptual element, although these decisions always involve–at least–a restriction of the users’ freedom of expression. The other key difference between the definitions is that, while content moderation under the DSA also includes the process of detecting content affected by moderation, detection can hardly be considered as moderation in the sense that it does not affect fundamental rights in the absence of actual interference with the content. For the purposes of this study, given its fundamental rights approach, the act of non-interference detection, despite the definition of the DSA, will not be considered as moderation.

To counteract the moderation decisions of platforms based on largely automated decision-making and to safeguard fundamental rights restrictions,¹⁷ 17 DSA Recital 9. the DSA contains five main tools: (i) comprehensible report on content moderation–terms and conditions; (ii) transparency reports; (iii) statement of reasons; (iv) internal complaint-handling system with the supervision of appropriately qualified staff; and (v) out-of-court dispute settlement.

Ad (i) Article 14 obliges intermediary service providers to include in their contractual terms and conditions on content moderation in a “clear, plain, intelligible, user-friendly and unambiguous language”, and “shall be publicly available in an easily accessible and machine-readable format”, including information on any policies, procedures, measures and tools used for the purpose of content moderation, including algorithmic decision-making and human review, as well as the rules of procedure of their internal complaint handling system (Husovec, 2023).

Ad (ii) Among the fundamental characteristics of AI are its complexity, its dependence on data,¹⁸ 18 European Commission, Proposal for a Regulation of the European Parliament and of the Council Laying Down Harmonised Rules on Artificial Intelligence (Artificial Intelligence Act) and Amending Certain Union Legislative Acts (hereinafter: EC Proposal) 3.5. its inherent opacity (Vasiliki, 2022), its unpredictability and its unknowability (Yampolskiy, 2024), from which it logically follows that, in order to counterbalance these factors (Vasiliki, 2022), one of the main moral and fundamental rights objectives when it comes to the regulation of AI, is transparency (Stahlet al., 2023). The opacity of AI is also likely to hinder, for example, the exercise of the fundamental procedural rights of social media users, in particular the right to an effective remedy and to fair trial (Kattnig, 2024).

At the same time, autonomous behaviour is another fundamental characteristic of AI,¹⁹ 19 EC Proposal 3.5. which makes ensuring transparency challenging, for example, especially with regard to machine learning. This problem is also reinforced by the so-called “black box effect”, which essentially refers to the characteristic of AI systems that autonomous AI systems operate in a way that is inherently unintelligible to humans (users) (Wulf & Seizov, 2020). Moving along this line of thought, we are faced with the paradoxical situation (information overload or transparency paradox (Vasiliki, 2022) that full transparency, i.e., the disclosure of the exact programming code or algorithm, is not able to achieve the desired objective of transparency, since users are unable to understand the systems’ operation due to a lack of knowledge, or only with a disproportionate expenditure of time and energy, which has the exact opposite effect: it potentially misleads users rather than inform them about the reliability of the system. Furthermore, transparency in the ordinary sense could also mean that knowledge about AI is systematically transferred from one stakeholder to another (Blackman & Ammanath, 2022), which could, however, result in imposing transparency requirements on AI systems to an inappropriate extent and in an inappropriate manner, which would result in a disproportionate damage by obliging the disclosure of know-how and business information (trade secrets).²⁰ 20 EC Proposal point 3.5.

The DSA imposes transparency reporting obligations on service providers to counteract moderation decisions based on automated decision making.²¹ 21 DSA Recital 65. These obligations are set out cumulatively at three levels: firstly, in the context of minimum content elements for all intermediary service providers,²² 22 DSA Article 15. secondly, for service providers operating an online platform,²³ 23 DSA Article 24. and thirdly, for service providers operating a very large online platform or a very popular online search engine.²⁴ 24 DSA Article 42.

According to Articles a5 and 42 of the DSA, the most popular social networking service providers that are considered to be a very large platform are required to include the following information in their transparency reports on automated decision-making:

a) “Meaningful and comprehensible information about the content moderation engaged in at the providers’ own initiative, including the use of automated tools, the measures taken to provide training and assistance to persons in charge of content moderation, the number and type of measures taken that affect the availability, visibility and accessibility.”²⁵ 25 DSA Article 15 (1) c).

b) “Any use made of automated means for the purpose of content moderation, including a qualitative description, a specification of the precise purposes, indicators of the accuracy and the possible rate of error of the automated means used in fulfilling those purposes, and any safeguards applied.”²⁶ 26 DSA Article 15 (1) e).

c) “The indicators of accuracy and related information” referred to in point (b).²⁷ 27 DSA Article 42 (2) c).

According to Ad (iii), one of the main objectives of the DSA was to improve the availability of effective remedy mechanisms.²⁸ 28 DSA Recital 9. In order to ensure that users can effectively exercise their right to remedy, all hosting providers are required to provide adequate justification for their decisions (not only when deleting content, but also, for example, when demonetising or demotioning content) when moderating content, as justification is an instrumental element for the exercise of the right to effective remedy. The obligation of social media service providers to state the reasons of their decisions applies to any restrictions imposed on the grounds of illegality or incompatibility with the contractual terms of the content (information) shared by the user, and even if with limited content,²⁹ 29 DSA Article 16 (5). the intermediary service provider must also state the reasons for not acting on any requests or notifications. The DSA follows the structure and logic prescribed for judicial decisions, with the exception that the legal consequence applied is also set out in the statement of reasons instead of in the operative part. In addition, the statement of the facts, the legal basis or contractual terms and information on the use of automated (AI-assisted) tools in reaching the decision are mandatory elements. The statement of reasons should also include a clause on possible appeals, as in judicial decisions.

According to Ad (iv), while the obligation to state reasons under Article 17 of the DSA applies largely when the service providers are active, the electronic and free effective complaint mechanism³⁰ 30 DSA Article 20 (1). under Article 20 of the DSA, which must be guaranteed for at least six months, applies to a wider range of decisions, also providing the possibility to lodge a complaint even if the platform remains passive in case of a notification under Article 16.³¹ 31 DSA Article 17. The complaint procedure necessarily requires staff intervention, and the service provider must inform the complainant without undue delay of its reasoned decision and of the possibilities for out-of-court dispute resolution or other means of redress. However, the phrase “without undue delay” is as broadly formulated as the immediate removal obligation in the E-Commerce Directive.³² 32 Directive 2000/31/EC of the European Parliament and of the Council of 8 June 2000 on certain legal aspects of information society services, in particular electronic commerce, in the Internal Market (Directive on electronic commerce), OJ L 178, 17.7.2000, 1–16. In the absence of further specification, this expression is likely to be a source of uncertainty and fragmentation, like the assessment of the immediacy of the removal obligation.³³ 33 See Delfi AS v Estonia App no. 64569/09 (ECtHR, 16 June 2015); Magyar Tartalomszolgáltatók Egyesülete and Index.hu Zrt. v Hungary App no. 22947/13 (ECtHR, 2 February 2016); Pihl v Sweden App no. 74742/14 (ECtHR, 9 March 2017).

Regarding Article 17, the question rightly arises: to what extent can a notifier lodge a complaint, whose notification did not incide an action, therefore the service provider was not obliged to state reasons for the decision under Article 17. The difference in the obligation to state reasons in the case of a restriction and the lack of such obligations in case of the passivity of the intermediary service provider (Ortolani, 2023) may result in that a user whose notice had no consequences (service provider’s inaction) does not have the same chances of a successful redress a user whose content has been removed by the service provider with a reasoned decision.³⁴ 34 DSA Article 16 (5). However, looking at the entirety of obligations, it can be seen that the service provider is obliged to provide a statement of reasons covering the circumstances and the facts of the complaint decided under the internal complaint handling mechanism, thus complementing the lack of reasoning obligation in case of passivity. As the exhaustion of the complaint mechanism is a prerequisite for access to out-of-court dispute resolution (Lendvai, 2024) this obligation enables the user to challenge a reasoned decision before the Article 21 out-of-court dispute resolution forum. (Husovec, 2023).

Ad (v) Out-of-court dispute resolution is of paramount importance because in the past, excluding Facebook (Meta) Oversight Board’ almos four years of activity, the only way to resolve disputes between users and social media platforms was through national (and possibly arbitration) courts. As of 17 February 2024, following the implementation of the DSA, users have the right to out-of-court dispute resolution, which, due to the form of the regulation, must be provided by all Member States, with a significant share of the costs to be borne by social media service providers. (Ruschemeieret al., 2024) The out-of-court dispute resolution mechanism outlined the Regulation is most similar to the alternative fora provided for by the German Netzwerkdurchsetzungsgesetz³⁵ 35 Gesetz zur Verbesserung der Rechtsdurchsetzung in sozialen Netzwerken (Netzwerkdurchsetzungsgesetz, NetzDG), BGBl. I S. 3352. (Act on the Improvement of Enforcement in Social Networks, hereinunder: NetzDG). While the NetzDG only applies to social media platforms with more than two million German users,³⁶ 36 NetzDg § 1 (2). the obligation under the DSA applies to all providers of online platforms larger than micro and small enterprises that are located or established in the EU, irrespective of the place of establishment of these intermediary service providers.³⁷ 37 DSA Article 21.

Although the decisions resulting from the procedure before the panel under Article 21 of the DSA are not binding, except in the case of a submission, they allow a panel of experts to decide in a timely and cost efficient way, to settle disputes outside the judicial organisation, that the users of the online platform could not resolve directly with the service provider.³⁸ 38 DSA Article 21 (2). These procedures do not affect the right to go to court, but they certainly have the advantage of having a panel of experts instead of a lay court.

Content Moderation, Transparency and the AI Act

The AI Act brings significant changes to the way online platforms use artificial intelligence in content moderation, and brings both opportunities and challenges. The AI Act takes a risk-based approach, classifying AI systems into four categories: unacceptable, high, systemic and minimal risk (Ebers, 2024) Content moderation tools used by online platforms generally fall into the category of high or limited risk systems, depending on their potential impact on fundamental rights such as freedom of expression and non-discrimination.

The AI Act prohibits the use of artificial intelligence systems that pose an unacceptable risk to fundamental rights. This includes AI systems that manipulate human behaviour or exploit vulnerable persons in ways that could cause them harm.³⁹ 39 AI Act Article 5. Regarding content moderation, this category would include AI systems that intentionally or systematically censor lawful content in order to manipulate public opinion (Veale & Zuiderveen Borgesius, 2021).

High-risk artificial intelligence systems are subject to the strictest rules.⁴⁰ 40 AI Act Articles 6–27. Content moderation systems used by very large online platforms such as Facebook, YouTube and X/Twitter can be classified as high-risk because of their potential impact on users’ fundamental rights. These platforms often use artificial intelligence to automatically detect and remove content that violates community standards, such as hate speech, misinformation and extremist content. Because ofthe sensitivity of these functions, the EU imposes specific obligations on high-risk AI systems, in particular with regard to transparency, supervision and accuracy (Wagner, 2021).

Content moderation tools that have less impact on users’ rights are considered systemic and minimal risk.⁴¹ 41 AI Act Articles 51–55. These may include basic automatic filtering systems that, for example, block spam or detect certain copyright infringements. For limited risk systems, the requirements are less stringent but still include transparency obligations. Artificial intelligence with minimal risk, such as general-purpose algorithms used in recommender systems or language filters with limited moderation functionality, are subject to even fewer regulatory obligations (Floridi, 2021).

Several articles of the AI Act directly address the responsibility of platforms using AI to moderate content, focusing on transparency, data management, accountability and human supervision.

Proper data management is essential to ensure the fair and unbiased operation of the artificial intelligence systems used to moderate content (Boone, 2023). Article 10 of the AI Act requires that the data used to train high-risk AI systems should be of high quality, free from bias and be representative of all relevant groups. This is particularly important in content moderation, where biased training data may result in unfair treatment of certain groups or overly strict supervision of marginalised communities (Binns, 2018). For example, if an AI system used to flag hate speech is trained on data that is predominantly from dialects or languages, it may disproportionately flag content from these groups, which may raise concerns about discrimination and freedom of expression.

In order to prevent over-reliance on automated decisions, Article 14 of the AI Act subjects high-risk AI systems to human supervision. In the context of content moderation, this means that although AI can automate the detection of harmful content, human moderators should be involved in the final decisions, especially those affecting users’ fundamental rights. This human approach is essential to address the limitations of AI, such as its inability to fully grasp or appropriately understand context. In case of content moderation, this means that AI tools reliably detect harmful content while minimising false positives, where legitimate content is wrongly flagged or removed, and false negatives, where harmful content is missed (Gillespie, 2018). The provisions on human supervision reinforce the importance of balancing automated decisions and human judgement. While artificial intelligence can effectively moderate large amounts of content, human moderators remain essential to deal with complex issues requiring contextual understanding.

The MI Act also contains relatively strict transparency requirements and definitions, similar to the DSA, in order to reduce risks relating to certain content moderation issues. To clarify the meaning of transparency, for example, Recital 27 states that:

Transparency in the AI Act means that AI systems should be developed and used in a way that allows for adequate traceability and explainability, while making people aware that they are communicating or interacting with an AI system, while adequately informing users of their rights, as well as of the capabilities and limitations of the AI system.⁴² 42 AI Act Recital 27.

Building on this, Article 50 of the AI Act includes transparency obligations for AI systems that (a) interact with people, (b) are used to detect emotions or to determine (social) categories based on biometric data, or (c) create or manipulate content, in order to balance the rights and interests of different stakeholders and to reduce information asymmetry.⁴³ 43 AI Act Article 50. Persons should be informed when interact with AI systems or have their emotions or characteristics recognised with automated tools. Where an AI system is used to create or manipulate images, audio or video content that bears a perceptible resemblance to authentic content, it is mandatory to disclose that the content was created by automated means, with the exception falling under the legitimate purpose exception. This transparency allows individuals to make informed decisions or to withdraw from a given situation.⁴⁴ 44 EC Proposal 5.2.4.

Artificial Intelligence in the Practice of Content Moderation

Artificial intelligence has become an essential tool for the content moderation of online platforms, where managing the huge amount of user-generated content is a critical task. Platforms such as Facebook, X/Twitter, YouTube and Instagram process huge volumes of posts, images, videos and comments on a daily basis, and AI offers an effective way to automate the monitoring, flagging and removal of content that violates community guidelines. While AI can greatly improve content moderation, it is not without challenges. Its application has resulted in both successful and flawed content classification, leading to numerous debates around accuracy, fairness and ethical implications. In this chapter we examine the effectiveness, limitations and specific cases where AI has performed well or poorly in content moderation.

Artificial intelligence usually works by combining machine learning algorithms, natural language processing and computer vision to moderate content. These technologies allow AI to quickly examine and analyse large amounts of data and identify patterns or signals that may indicate violations of community guidelines.

Algorithms are trained on large data sets containing labelled examples of acceptable and unacceptable content. Artificial intelligence then uses these learned patterns to predict whether new content meets community guidelines (Gorwaet al., 2020). These models are constantly evolving as they encounter new data, enabling dynamic improvements in content recognition. Natural Language Processing (NLP) is essential for analysing text-based content and detecting inappropriate phrases. NLP models can sometimes recognise the context of a word or phrase, distinguishing between benign and harmful uses (Schmidt & Wiegand, 2017). For example, X/Twitter uses NLP to flag tweets containing offensive language or hate speech. To moderate images and videos, computer vision algorithms analyse visual content to detect inappropriate images such as nudity, graphic violence or terrorist propaganda (Araujoet al., 2020).

Artificial intelligence is very effective in identifying harmful content in many cases. The ability to process huge amounts of data quickly is essential for platforms with billions of users. Facebook, for example, has reportedly invested heavily in AI to detect and remove terrorism-related content. The company reported in 2024 that more than 99% of terrorism-related posts were flagged by AI before they were reported by users (Facebook, 2024). During the COVID-19 pandemic, YouTube used AI to combat the spread of health misinformation, and AI was used to identify 94% of videos flagged for policy violations (Parabhoiet al., 2021). Twitter’s AI proactively handled more than 99% of offending accounts in 2024 without relying on user reports (X, 2024).

Despite these successes, AI-based content moderation still faces significant limitations, which have led to high-profile failures. These often stem from AI’s inability to understand the full context of content, cultural nuances and language complexities. While these systems are constantly evolving, flawed content moderation can also harm the reputation of companies. In 2016, Facebook’s AI incorrectly removed the iconic Vietnam War Napalm Girl photo, claiming that its nudity violated its community guidelines (Gillespie, 2018). AI failed to recognise the historical significance of the image and treated it as inappropriate content based solely on its nudity. During the COVID-19 pandemic, YouTube relied heavily on AI to moderate content, as human moderators worked from home. This led to a significant increase in false positives when educational and news-related content about COVID-19 was removed as they were classified as misinformation. Satire also often poses a challenge to AI, it is often identified as infringing content, because AI is oblivious to its context and tone of voice, which is different from normal speech (Lymn & Bancroft, 2021).

Problems of content moderation stems from that despite the huge amount of data fed to AI, it is still a challenge to fully understand contexts, different cultures and language differences. For example, it would be important to be aware that language is constantly evolving, new slang and colloquial expressions are emerging rapidly, especially in social media. Artificial intelligence models trained on outdated datasets may not be able to capture new terms, which can lead to inaccurate moderation. In addition, language varies significantly across cultures, making it difficult for global platforms to develop moderation systems that can be used uniformly and consistently in all countries. It should be remembered that AI models are not better than the data sets they are trained on. If the data is biased or incomplete, the end result will produce biased results.

Summary, or the Relationship between Moderation by Artificial Intelligence and Humans

Given the limitations of artificial intelligence, many platforms use a hybrid approach to content moderation, combining the speed of artificial intelligence with human supervision. Human moderators review tagged content to ensure that contextual and linguistic nuances and cultural differences are taken into account. This collaborative approach allows platforms to balance the power of AI with the human ability to make informed decisions. As AI evolves, content moderation systems are likely to become more sophisticated. Deep learning, contextual analysis and AI’s improved ability to understand cultural differences could increase the accuracy of moderation systems.

EU’s Artificial Intelligence Regulation and DSA will play an important role in shaping the future of AI-driven content moderation on online platforms. These Regulations impose strict requirements on AI-powered systems and aim to ensure that content moderation tools are transparent, fair and accountable. They offer a way to ensure that the quality of content moderation is improved, and fundamental rights are protected in the digital age.

In summary, while AI plays a vital role in moderating content on online platforms, its use and efficiency is not unlimited (Oversight Board, 2024). The balance between artificial intelligence and human moderation will continue to evolve as online platforms adapt to new regulations, in particular as platforms are forced to mitigate the problems that arise. However, human oversight will remain key in the foreseeable future to manage the subtleties and complexities that AI cannot yet understand (Lendvai, 2024).