Friends with Black Hats? Understanding the Dark Communication Potential of Chatbots

Modern User Experience (UX) is not solely concerned with understanding users’ needs and designing efficient, enjoyable interactions. It is also a form of communication which uses visual and textual means to communicate the functionality of systems to users, and guide them through the process of completing their tasks.

While UX typically seeks to help users by improving their interactions with software and companies, it can also serve less noble purposes with various UX strategies being used to create addictive products (Eyal, 2014), or to manipulate users in some way (Nodder, 2013). This phenomenon is known as "Black Hat UX" and is defined as "the intentional act of creating [a] suboptimal user experience" (Szabo, 2015), generally with the aim of furthering a company's business goals or agendas, and often at the expense of the users' needs.
Against this backdrop, there is cause for concern at the current interest in conversational user interfaces particularly chatbots. While the aim is ostensibly to improve the user experience with more natural, intuitive, and efficient interactions, the mechanisms behind such technologies are hidden from users and are open to abuse. Taken together with the potential for humans to develop intense relationships with virtual communication agents (e.g. Weizenbaum, 1966) and to be influenced by them (Rizoiu et al., 2018), the potential role of chatbots in dark communication clearly merits further discussion.

After examining various business cases for chatbots, we will explore various commercial, ethical, legal, psychological, and social issues involved in their use. In doing so we can understand the ways in which chatbots might be used for nefarious purposes and, armed with this knowledge, highlight ways of preventing this.