This is some text inside of a div block.
This is some text inside of a div block.
At Discord, we believe safety isn't just a standalone feature - it’s a foundational part of how we design our platform.
As AI becomes a bigger asset for how the industry moderates content, Discord is committed to developing these tools thoughtfully, in addition to continuing to invest in our content moderation practices with human reviewers. That's why we partnered with the European Center for Not-for-Profit Law (ECNL) to ensure Discord is incorporating a diverse range of stakeholder input to inform how we build these AI systems. Our partnership focused on exploring how AI models can better support our player communities by addressing bullying and harassment.
Players are at the heart of Discord. Whether they’re strategizing Valorant mid-match or simply hanging out with friends playing Farm Merge Valley, our users rely on Discord to create a sense of community and to build meaningful connections around the joy of playing games. But as AI-powered tools become more critical to keeping spaces safe, we face a critical challenge: how do we build with AI responsibly while considering the nuances of gaming culture?
To answer that, we worked with ECNL to bring together a diverse group of global stakeholders, including a range of casual players, teen representatives from global partner organizations, civil society and human rights advocates, and AI/ML experts and researchers. Across six events, we heard directly from people with lived experience of both the benefits and risks of AI moderation.
Our ECNL partnership reinforced that building effective AI moderation for players requires a deep understanding of how players communicate. Rather than deploying one-size-fits-all solutions, we need to design our AI systems with player-specific considerations from the ground up.
Player communication can be fast, full of in-jokes, and closely tied to specific titles or communities. What could sound like aggressive banter to an outsider may actually be a core part of how a team bonds. Meanwhile, subtle harassment can slip by, if AI content moderation tools if they aren’t designed with these nuances in mind.
Teens and young adults we spoke to highlighted how quickly language evolves online. Teen slang changes weekly, with layers of sarcasm that can completely flip a message's meaning. Accounting for linguistic fluidity by continuously updating our training data and testing our models against evolving communication patterns is crucial.
When we build models and policies, we consider the player context. Additionally, before deploying any AI moderation tool, we acknowledge the importance of preventing false positives that could disrupt gameplay or frustrate or harm users. We’re especially mindful of communities speaking languages that may be underrepresented in training datasets and work to ensure our models understand context, dialectic and slang across different cultures and identities, especially marginalized groups.
That's why collaborating with communities is central to our AI development process. We know we need to work with multiple stakeholders to continue to build out our moderation support across the globe, and are committed to continuing to do so. By incorporating these insights into our model development, we aim to create AI tools that enhance safety without compromising the authentic player experience our users love.
We embed safety into our product development through our "Safety by Design" framework. For every new feature, we ask four key questions:
In 2024, we expanded this work by creating a dedicated AI/ML safety workstream within our policy team, focused on incorporating user and expert feedback early in our AI/ML development process. We are eager to evaluate how automated systems can better reflect the realities of player communication and reduce harm, without overreach, by extending the concept of safety by design to AI/ML model development lifecycle.
By collaborating with our community when designing our AI systems from the start, we strive to build tools that:
Our partnership with ECNL shows how meaningful collaboration can improve the way companies evaluate AI systems based on human rights. This is one more way that we continue to gather input from players, developers, and diverse community voices to ensure our AI tools serve everyone effectively.
We're actively reviewing these findings to find ways to incorporate these insights into our Safety by Design processes as we expand our AI offerings.
Safety isn't just a feature on Discord - it's the foundation of how we build everything on our platform. By incorporating these voices into our AI development, in addition to continuing to invest in human content moderation efforts, we can better create digital spaces where everyone can connect, feel supported, and have fun more safely.
.png)
.svg)
