research
List of published and ongoing research work
ongoing
- ArXivAnalysis of Indic Language Capabilities in LLMs2025
This report evaluates the performance of text-in text-out Large Language Models (LLMs) to understand and generate Indic languages. This evaluation is used to identify and prioritize Indic languages suited for inclusion in safety benchmarks. We conduct this study by reviewing existing evaluation studies and datasets; and a set of twenty-eight LLMs that support Indic languages. We analyze the LLMs on the basis of the training data, license for model and data, type of access and model developers. We also compare Indic language performance across evaluation datasets and find that significant performance disparities in performance across Indic languages. Hindi is the most widely represented language in models. While model performance roughly correlates with number of speakers for the top five languages, the assessment after that varies.
published
- NAACL 2024The Uli Dataset: An Exercise in Experience Led Annotation of oGBVArnav Arora, Maha Jinadoss, Cheshta Arora, and 5 more authors🏆 Outstanding Paper AwardIn The 8th Workshop on Online Abuse and Harms at Annual Conference of the North American Chapter of the Association for Computational Linguistics 2024
Online gender-based violence has grown concomitantly with the adoption of the internet and social media. Its effects are worse in the Global majority where many users use social media in languages other than English. The scale and volume of conversations on the internet have necessitated the need for automated detection of hate speech and, more specifically, gendered abuse. There is, however, a lack of language-specific and contextual data to build such automated tools. In this paper, we present a dataset on gendered abuse in three languages- Hindi, Tamil and Indian English. The dataset comprises of tweets annotated along three questions pertaining to the experience of gender abuse, by experts who identify as women or a member of the LGBTQIA+ community in South Asia. Through this dataset, we demonstrate a participatory approach to creating datasets that drive AI systems.
- CODS-COMADAnalysing the Spread of Toxicity on TwitterAatman Vaidya, Seema Nagar, and Amit A NanavatiIn Proceedings of the 7th Joint International Conference on Data Science & Management of Data (11th ACM IKDD CODS and 29th COMAD) 2024
The spread of hate speech on social media platforms has become a rising concern in recent years. Understanding the spread of hate is crucial for mitigating its harmful effects and fostering a healthier online environment. In this paper, we propose a new model to capture the evolution of toxicity in a network – if a tweet with a certain toxicity (hatefulness) is posted, how much toxic a social network will become after a given number of rounds. We compute a toxicity score for each tweet, indicating the extent of the hatefulness of that tweet. Toxicity spread has not been adequately addressed in the existing literature. The two popular paradigms for modelling information spread, namely the Susceptible-Infected-Recovered (SIR) and its variants, as well as the spreading-activation models (SPA), are not suitable for modelling toxicity spread. The first paradigm employs a threshold and categorizes tweets as either toxic or non-toxic, while the second paradigm treats hate as energy and applies energy-conversion principles to model its propagation. Through analysis of a Twitter dataset consisting of 19.58 million tweets, we observe that the total toxicity, as well as the average toxicity of original tweets and retweets in the network, does not remain constant but rather increases over time. In this paper, we propose a new method for toxicity spread. First, we categorize users into three distinct groups: Amplifiers, Attenuators, and Copycats. These categories are assigned based on the exchange of toxicity by a user, with Amplifiers sending out more toxicity than they receive, Attenuators experiencing a higher influx of toxicity compared to what they generate, and Copycats simply mirroring the hate they receive. We perform extensive experimentation on Barabási–Albert (BA) graphs, as well as subgraphs extracted from the Twitter dataset. Our model is able to replicate the patterns of toxicity.
- ICON 2023Overview of the 2023 ICON Shared Task on Gendered Abuse Detection in Indic LanguagesAatman Vaidya, Arnav Arora, Aditya Joshi, and 1 more authorThe 20th International Conference on Natural Language Processing 2023
This paper reports the findings of the ICON 2023 on Gendered Abuse Detection in Indic Languages. The shared task deals with the detection of gendered abuse in online text. The shared task was conducted as a part of ICON 2023, based on a novel dataset in Hindi, Tamil and the Indian dialect of English. The participants were given three subtasks with the train dataset consisting of approximately 6500 posts sourced from Twitter. For the test set, approximately 1200 posts were provided. The shared task received a total of 9 registrations. The best F-1 scores are 0.616 for subtask 1, 0.572 for subtask 2 and, 0.616 and 0.582 for subtask 3. The paper contains examples of hateful content owing to its topic.
- CoGMI 2023Forecasting the Spread of Toxicity on TwitterAatman Vaidya, Seema Nagar, and Amit A NanavatiIn IEEE 5th International Conference on Cognitive Machine Intelligence (CogMI) 2023
In this paper, we explore the question of whether it is possible to forecast the spread of hate on Twitter. Unlike most prior work which models the spread of Twitter over a network with the goal of predicting the future “state" of a user (typically, as being "hateful" or not), here we are interested in how "hateful" (or toxic) the network as a whole becomes. We pose toxicity spread as a forecasting problem, and use ARIMA to find out whether the spread is forecastable. We find that toxicity spread is indeed forecasted by ARIMA. Given that it is forecastable, we ask two follow-up questions: (a) How well can we forecast it? and (b) What role, if any, does the structure of the retweet network play in forecasting? In order to answer these questions, we employ several techniques including Spatio-Temporal Graph Convolution Network (STGCN) and several variants of transformers. To determine the role that structure might play, we re-purpose the dataset in three ways: the network as a whole (the structure is ignored), communities interconnected with each other, and neighbourhoods of a set of individuals. Experiments with the network as a whole informs us how well we can forecast hate spread at a global level, while the latter experiments tell us whether and how network effects affect the forecasting. In an effort to tease out the effect of the network, we use two distinct techniques: STGCN, which requires the explicit connections in the form of a graph as input; and Transformers, where no explicit graph is given as input. Instead, we pose it as a multivariate analysis problem. We find that the PatchTST transformer performs the best at all levels. STGCN performs better than ARIMA suggesting that network structure matters. Somewhat interestingly, STGCN does not perform as well as PatchTST, suggesting that (a) PatchTST is able to implicitly learn the associations (flow of influence), and (b) the presence of explicit connections may not always imply influence.