SAMbot Methodology: Municipal Elections

/
June 18, 2023
  
min. read
Uncategorized
Share
Share on Twitter
Share on Facebook
Share on Linkedin
Copy Link
SAMbot Methodology: Municipal Elections
An arrow pointing left
View all of our work

Throughout the fall of 2022, the Samara Centre for Democracy deployed SAMbot in eight municipal elections across Canada to measure abusive content received by candidates and political parties on Twitter.

SAMbot tracked each of these elections, in their respective regions & time zones, from the evening that nominations closed until the evening of election day.  

  • The Toronto, Brampton, and Ottawa elections were tracked for 65 days from August 20, 2022 at 12:00 a.m. ET to October 24, 2022 at 11:59 p.m. ET. Candidates for city council and relevant school boards were monitored in these elections.
  • The Vancouver and Surrey elections were tracked for 35 days from September 10, 2022 at 12:00 a.m. PT to October 15, 2022 at 11:59 p.m. PT. Candidates for city council and relevant school boards were monitored as well as official political party Twitter accounts in both elections. The Vancouver election included monitoring of park board candidates.
  • The Winnipeg election was tracked for 35 days from September 21, 2022 at 12:00 a.m. CT to October 26, 2022 at 11:59 p.m. CT. Candidates for city council and relevant school boards were monitored in this election.
  • The Charlottetown election was tracked for 17 days from October 22, 2022 at 12:00 a.m. AT to November 7, 2022 at 11:59 p.m. AT. Candidates for city council were monitored in this election. Unlike in other elections, school board trustee candidates were not monitored, as Prince Edward Island school board trustee elections are held separately and at different times than city council elections. 
  • No Yellowknife candidates had public and active Twitter accounts by the end of the election’s nomination period. As a result no candidates were tracked.

SAMbot tracks the Twitter mentions of all candidates who possessed a public and active Twitter account as of the end of each election’s respective nomination deadlines.

Since data from each election was collected during different time periods, different lengths, and with different totals of tracked candidates, it is not useful, nor advised, to compare municipal SAMbot data across elections.

SAMbot does not track or store retweets, as counting the same tweet more than once can distort the analysis. SAMbot only evaluates text within a tweet; content such as images, audio, or videos that may spread abuse cannot be evaluated by SAMbot. Twitter data was collected and used in line with Twitter’s acceptable terms of use.

How does SAMbot detect abuse?

SAMbot is a machine learning bot — a software application that runs automated tasks. SAMbot tracks all English and French tweets sent to candidates. Each message that SAMbot tracked, whether it’s a reply, quote tweet or mention, was analyzed and sorted based on five abuse categories:

Abuse categories are not exclusive. Tweets can possess more than one attribute (for example, a tweet could contain a threat and an identity attack simultaneously).

SAMbot makes a confidence prediction to assess whether a tweet meets an abuse category. When a tweet is evaluated, SAMbot gives the tweet a score from 0% to 100% for each category, based on how certain SAMbot is that the tweet meets that abuse category. If SAMbot assesses a tweet as >=51% likely to meet an abuse category, we determine that the tweet has met the criteria.

If a tweet is scored as >=51% in any of those five abuse categories, it would also be categorized as an abusive tweet. The abusive tweet category serves to aggregate all comments that meet at least one abuse category.

Using machine learning allows us to analyze tweets at a massive scale. SAMbot can evaluate millions of tweets for how likely abusive or toxic they are. However, we willingly sacrifice a degree of precision in the pursuit of scale through our use of machine learning. There is no explicit qualitative analysis conducted by human researchers to evaluate these tweets before they are collected; SAMbot, as a piece of software, conducts all of this sentiment analysis. Thus, SAMbot-related findings are not intended to act as precise findings, but instead to help point us in the direction of where and how online abuse manifests in Canadian political discussions.

How SAMbot is always improving

SAMbot uses a machine learning model to assess abusive language. These models are ever-evolving which means that each time SAMbot is deployed, it becomes more accurate and informed. In addition to measuring volume of abuse, SAMbot also provides insight into the type of abuse received by candidates (e.g. identity attacks, sexually explicit content, threats).

However, this also means that data collected during different SAMbot election periods must be siloed. Data collected in different periods (say for example, our 2021 federal election data versus our 2022 municipal election data) can not, and should not be compared because the methodology behind how SAMbot evaluates language has changed significantly between periods, thus, the data is incompatible, and will likely be skewed in certain capacities.

Click here for:

This material is part of the Online Abuse in Local Elections: The SAMbot Municipal Report.

How to cite: The Samara Centre for Democracy, 2023 SAMbot Municipal Elections Report, (Toronto: The Samara Centre for Democracy, 2023), http://www.samaracentre.ca/project/online-abuse-in-local-elections-sambot-municipal-report.

Explore our work

Explore Our Work