Wikimedia Research

Addressing knowledge gaps

We develop models and insights using scientific methods to identify, measure, and bridge Wikimedia’s knowledge gaps.

A deeper understanding of content contributions

With the availability of scalable edit types library, our research on Wikipedia Edit Types has entered the next stage. We have started experimenting with large language models to auto-generate edit summary recommendations for edits that changed textual content on English Wikipedia. We submitted a paper with our initial results which is now under review. (Learn more)

An improved readership experience through better new user onboarding

We worked on making the link recommendation model more accurate and scalable across different languages. We started by improving and releasing a new version of the mwtokenizer library to support tokenization across Wikipedia languages to scale the link recommendation model. (Learn more)

A model to increase the visibility of articles

Our paper characterizing orphan articles in Wikipedia has been accepted for publication at ICWSM 2024. We developed an experimental tool for developers and researchers to experiment with the output of the research. This technology can be used to support editors in de-orphanizing Wikipedia articles and to support readers by recommending orphan articles they can read next. (Learn more)

Metrics to measure knowledge gaps

Our focus was on three primary areas: ensuring the adoption and usage of the knowledge gap metrics and measurements we have researched and developed over the past years; improving the accessibility of the measurements we publish; continuing research to develop metrics for the knowledge gaps that currently do not have a metric.

One of the barriers for the adoption of the gender and geography content gap metrics in the Wikimedia Foundation was that the Foundation needed a more nuanced metric that takes into account not only the number of articles but also their quality. In response we developed a new knowledge gaps metric: standard quality , the percentage of articles meeting a standard quality threshold. The standard quality metric for the gender and geography gaps is now used by the Wikimedia Foundation to make strategic decisions. This work was part of our contributions to one of the Foundation’s top annual plan goals, Infrastructure: Building the infrastructure of knowledge as a service.

We advanced our efforts to make the Knowledge Gap Index data more accessible. First, we created a public repository which will include productionized knowledge gap metrics as they become available. We started developing an architecture and documenting the code for data pipelines that provide measurements for the content gap metrics. Lastly, to lower the barrier for utilizing this data, we also developed notebooks that can help visualize the gender and geography data gaps and act as a starting point to answer more questions with using the data.

We continued research towards developing more content gap metrics for the identified gaps and advocated for the productionization of the metrics we have developed. We prioritized our attention on three fronts: readability gap, structured data gap, and language gap. You can learn more below.

The multilingual readability model which can be used to develop a readability gap metric is now in production (Get readability prediction via querying the service or use the interface to test it). We also completed the first iteration of developing a metric for the structured data gap. The metric calculates how complete a Wikidata item's labels and descriptions are. (Notebook)

We started investigating the development of metrics for the language gap, to study how well Wikimedia projects cover languages from around the world. We compiled data from different existing sources to create a “state of languages” table: a semi-automated table that shows which languages currently have which free-knowledge projects. (Learn more)

Finally with the survey expertise that are now in the Research team, we used the opportunity to rethink how we approach survey development and data collection to serve the Movement. As a result we prioritized work on three fronts that you can learn more about below.

Revamping Reader Surveys. Inspired by our 2019 research, we launched the Global Reader Survey, a representative survey of Wikipedia readers, fielded in 23 languages (representing at least 90% of current Wikipedia readership). The data from the survey will provide measurements for 9 of the Readers’ gaps. (Task)

Refocusing the Community Insights survey. The survey is run by WMF to learn about the communities. We reflected on the usage of the data from this survey and considered the needs of the organization for data. As a result of this exercise we made a few changes in the survey’s roadmap. Specifically we decided to reduce the scope of the survey to two types of questions related to the contributors’ representation gaps and community health considering the importance of the topics for WMF. With the clarity of scope, we prioritized work on the operations side to prepare for the launch of the updated survey in 29-languages in March 2024. The data from the survey will provide measurements for 7 of the Contributors’ gaps.

Streamlining survey analyses. With the goal of reducing the number of weeks needed for analyzing survey data, we continued building repeatable frameworks for analyzing survey data. We are using the opportunity of supporting the December 2023 Developer Satisfaction Survey as a use-case that can help us improve our framework.

A deeper understanding of the role of visual knowledge in Wikimedia Projects

Due to other priorities this project was on a pause during the period of this report.

A model for image recommendation

We concluded our support to Wikimedia Foundation’s Product teams for the “section-level image suggestion” structured task. At the end of November 2023, 4629 images were added on 14 Wikipedia languages through this task. One limitation of the current algorithm for image suggestion is its limited coverage, because it is based on section-image associations that are already existing in some Wikipedia languages. Addressing this limitation requires highly precise multimedia retrieval frameworks. To advance research on this front, we organized the AToMiC (Authoring Tools for Multimedia Content) track for the annual TREC conference.

A deeper understanding of reader navigation

We are happy to share that our paper on temporal regularities of Wikipedia consumption has been accepted for publication at ICWSM 2024 (paper. This concludes the portion of readership research dedicated to understanding how readers navigate within the encyclopedia. (Learn more)

We started a new line of research to understand how readers enter Wikipedia. More specifically, we are looking at the relevance of Wikipedia to users of search engines, which are one of the main drivers of traffic to Wikipedia. Specifically, we aim to study what proportion of search engine queries lead to a user's visit to Wikipedia. As a first step towards this goal, we are building a new dataset that combines Google Trends (what users on Google are searching for) and Wikipedia clickstream (how readers reach Wikipedia), two publicly available resources. (Learn more)

A unified framework for equitable article prioritization

We concluded the work on our experimental framework to test the balance between personalization and content equity within recommender systems. The learnings from this research are accepted for publication in ICWSM 2024. (Paper, Learn more)

We continued work on integrating our content tagging models into existing product workflows with the goal of promoting more equitable article improvements. We worked on the adoption of the model on three fronts. As of July 2023, all Growth features in all languages are now using our multilingual topic models to suggest articles for newcomers, greatly improving the coverage of topic labels. Additionally, after a successful pilot, we are assisting the Android team as they prepare to launch a model for AI-generated article description recommendations. We are also receiving and recording feedback for our List-Building tool, as it is being tested by campaign organizers to more easily build article worklists that are relevant to their topic areas. (Learn more).

Models for content tagging

We concluded our work on this front, and we are now working on integrating our content tagging tools in the context of other projects and product workflows. See the previous paragraphs update for work done on this space.

Large language models for text simplification

We started exploratory work on developing a model to automatically simplify Wikipedia article text using large language models. The model aims to improve the readability of articles and is a follow-up to our work on measuring readability as part of the knowledge gap metrics. We started by conducting a literature review to understand existing solutions and best practices for systematically evaluating the model’s performance. Based on these insights, we developed a working prototype by fine-tuning a pre-trained language model (T5) and evaluating it on the D-Wikipedia benchmark dataset. (Learn more)

Improve Knowledge Integrity

We develop models and insights using scientific methods to support the technology and policy needs of the Wikimedia projects in the areas of misinformation, disinformation, and content integrity.

Enhanced models for content patrolling

We primarily focused our work on the Language Agnostic Revert Risk model improvements which have been in production for some time now. We improved the model for better accuracy and serving time. Furthermore we debugged the model based on feedback from the usage of the model by Wikimedia Enterprise. On another hand, we developed and released an annotation tool through which we can collect ground truth labels for evaluating and improving our models. Lastly, we launched a labeling campaign to evaluate and improve the Revert Risk model for Wikidata. (Learn more)

Wikipedia Knowledge Integrity Risk Observatory

We focused on the adoption and usage of the Observatory. As a result of this work the Observatory’s data and inferred insights are now being used on two fronts. First by the Trust & Safety Disinformation team in monitoring the Wikimedia projects at the time of upcoming elections in countries across the world; Second by the Moderator Tools team for characterizing high-risk revisions for the Automoderator tool, a tool that allows moderators automate the prevention or reversion of bad edits. (Learn more)

A project to help develop critical readers

The research is now concluded. A preprint is available and is currently under review.

Infrastructure for more efficient machine learning research and development

We have been investing on the engineering front over the past couple of years to better align the research environment with WMF’s production environment with the ultimate goal of facilitating the use of research outputs in products. During the period of this report, we invested on three fronts.

First, we started developing pipelines that streamline dataset generation with a particular focus on the datasets our team relies on for developing machine learning models. Our goal is to streamline dataset creation processes from data labeling to storage, versioning, quality assurance, and human feedback loop to create high quality training and evaluation datasets. We currently have two datasets as part of this new pipeline and expect to add more in the coming months.

Second, we developed tooling to support working with embeddings (special codes represented by a sequence of numbers that help computers understand and represent words, sentences, or objects in a way that is useful for various tasks, such as comparing the similarity of two Wikipedia articles or two images). In particular, in support of automatically building lists for campaigns, we built a vector database index in WMF's compute infrastructure and hosted the vector database service on WMF’s open cloud computing infrastructure (Cloud VPS).

Third, we started experimenting with and building exploratory tooling for using Cloud GPUs for machine learning training tasks, with a focus on fine-tuning LLMs.

Conducting foundational work

Wikimedia projects are created and maintained by a vast network of individual contributors and organizations. We focus part of our efforts on strengthening part of this network: the Wikimedia research community.

A Wikimedia Research course

A few instructors developed their sessions content. We continued fine-tuning what is in and out of every module and invited some more instructors to develop the course’s content. However, the progress on the course was slower than expected during the period of this report due to unforeseen need for change in priorities for the Head of Research. If you are interested to receive more frequent updates about this project, please subscribe to the relevant epic task.

Research Showcases

Our July showcase focused on improving knowledge integrity in Wikimedia projects, with presentations of recent work by the Wikimedia Foundation Research team on reference quality in Wikipedia and multilingual approaches to revert prediction. We explored the role of rules on Wikipedia in our September showcase with talks covering their implications for experiential epistemology and a comparative study of rule-making activity in the five largest wikis. The topic of our October showcase was data privacy featuring works on synthetic data of Wikipedia reader navigation patterns and differential privacy approaches for datasets released by the Wikimedia Foundation. In November, two different pieces of bibliometrics research were presented to connect this showcase with the GLAM Wiki 2023 conference. Our final showcase of 2023 was a panel titled "A year of Generative AI: Future directions for Wikimedia". (Learn more)

Office Hours

We continued offering 1:1 consultations with members of our team. In the past six months, most office hours were focused on assisting prospective applicants with their Research Fund submissions. (Learn more)

Research Fund

We launched the third cycle of the Wikimedia Research Fund and have received 76 submissions. (Learn more)

Wikimania

We participated in the first in-person Wikimania in Singapore after a few years of virtual Wikimanias. Wikimania is the largest annual gathering of the Wikimedia Movement celebrating all the free knowledge projects hosted by the Wikimedia Foundation. We led a session titled “10 Research findings and how you can use them in your works (Slides, Video) and participated in two panels: AI advancements and the Wikimedia Projects (Video) and ChatGPT vs. WikiGPT (Video). Wikimania was a great opportunity for us to connect with volunteers and other Movement participants and discuss Wikimedia research and much more.

Wiki Indaba

We participated in this year’s Wiki Indaba which took place in Morocco. We presented a talk titled “Reading Wikipedia in Swahili, Yoruba, French, and English: Insights from Sub-Saharan Africa” (Video) and engaged with Wikimedians from across the African region.

Presentations and keynotes

We engaged with research audiences through the following presentations and keynotes during the past six months.

In July, we participated in a panel discussion on The Sociopolitical Impact of AI in the Digital Humanism Summit on AI and Democratic Sustainability in Vienna organized by the Digital Humanism Initiative, the OSUN Hannah Arendt Humanities Network at Bard College and the Institute for Human Sciences (IWM). Our contribution focused on featuring the work and principles of AI/ML in the Wikimedia Research ecosystem.

In July, we gave an invited talk as part of The Weaponization of Knowledge Conference at the University of Tübingen. We were invited by research collaborators from the University of North Carolina – Chapel Hill to share findings of ongoing research on historical revisionism in Japanese Wikipedia. (Extended abstract)

In August, we presented a paper entitled “Fair multilingual vandalism detection system for Wikipedia” at the XXIX Conference in On Knowledge Discovery and Data Mining (KDD’23). This paper describes our scientific approach to create the Revert Risk Models.

In October, we moderated a panel discussion on Ethical Challenges of the Democratisation of Artificial Intelligence at Decidim Fest 2023 Conference in Barcelona with panelists from Queer In AI and Universitat Oberta de Catalunya.

In October, we co-organize the CSCW Workshop on epistemic injustice in online communities. The workshop discussed the systemic silencing, exclusion, or delegitimization of certain knowledge contributions.

In November, we hosted the AToMiC track at the TREC 2023 conference. We received a total of 27 runs from 3 different teams (11 for the image promotion task, and 16 for the image suggestion task). A second edition of the AToMiC Track has been accepted and will be held with TREC 2024.

In November we participated on a panel organized by Wikimedia Argentina, discussing the usage of AI-based technologies in the Wikimedia projects. (Video [Spanish])

In December, we participated in the “Recentering Platform Governance” Workshop hosted at Yale University’s Law school. The workshop considered ways to center not-for-profit initiatives and community-based models for platform governance in policy discussions.

Mentorship through Outreachy

We continue to participate in Outreachy and are excited to have a new intern onboard. Shriya Kamat Tarcar will be working with our Lead Design Researcher, Mike Raish, on a Multilingual Wikipedia Editor Survey project.

Mentorship through internships

All of our past internships have been completed prior to the period of this report and no new internships have been taken on. We remain committed to mentoring through internships in the future.

Ethical AI

We have continued our commitment to formalizing our approach to ethical development and deployment of AI technologies on the Wikimedia projects through a number of initiatives.

We continued collaborating with the Human Rights team to develop a stable Human Rights AI Checklist. We are wrapping up our first round of testing of the checklist with model developers to get feedback on the ease of using it and to identify whether any major components are missing. (Task)

The pilot of the machine-assisted article descriptions tool on the Wikipedia Android app wrapped up successfully. The results were largely positive and it identified one failure case with hallucination of dates that we were able to address ahead of a broader release. The model is being put on our Machine Learning platform and in preparation for the expected release to the 25 language editions that the model supports in early 2024.

Survey support for WMF and the affiliates

Every year Wikimedia Foundation and the affiliates send a large number of surveys to different Wikimedia communities to gather valuable data that can inform decision making or support developing insights about the state of the communities and the projects. Developing and implementing good surveys requires knowledge about how to form survey questions, ethics, how to sample, where to expose the survey, how to analyze the data from the survey, and more. Through a new service we aim to provide methodological and question writing feedback, and assist CentralNotice administrators in reviewing community surveys distributed through banners. We further manage WMF survey tools, and provide access to Qualtrics for affiliates who may need more advanced survey tooling. We supported Wiki Women Camp, the Wikimania 2024 Core Organizing Team, Wikimedia Deutschland, and Wikimedia Chile on their survey projects. If you are a Wikimedia affiliate, you can request survey support by writing to surveys@wikimedia.org with the details of your project.

Additionally

Frameworks for metric definition and dissemination

We contributed to Infrastructure: Building the infrastructure of knowledge as a service, one of the top-line goals of Wikimedia Foundation’s annual plan for 2023-2024. Our contributions on this front do not fall under existing roadmaps of our team, as a result, we are going to report them in this dedicated section.

Our primary contributions were two-fold. First, we organized and led a committee of experts from across WMF who developed criteria for metrics that strategic decision making and understanding of the projects rely on (essential metrics). Second, we developed a process for computing, visualizing, and presenting business critical annual plan metrics (core metrics) for decision makers.