Provocative Objects: Smart Caption Glasses

13 November 2024 -

Smart caption glasses (Figure 1) emerged from advancements in augmented reality (AR) technology, aiming to provide real-time captions for hearing-impaired individuals by converting sound to text and projecting it onto the glasses. Initially hailed as a revolutionary communication aid, these glasses promised barrier-free access to spoken content (1). However, as a hearing-impaired researcher in inclusive design, I found that despite the high expectations, the glasses revealed significant shortcomings in real-world use, leading many in the hearing-impaired community to abandon them.

The roots of subtitling for the hearing impaired go back to the mid-20th century, with Emerson Romero, a Cuban-American deaf actor (Figure 2), who pioneered subtitling sound films in 1947. When silent films gave way to sound films, deaf viewers like Romero lost access due to a lack of captions. Romero responded by manually inserting text frames into film reels, allowing deaf audiences to follow the dialogue. His innovative approach laid the foundation for today’s subtitling technology, and his spirit is echoed in modern tools, like the subtitle glasses used at the National Theatre (NT）in London, designed to make performances more accessible (3).

In the 21st century, AR technology has enabled further developments in smart caption glasses. In 2022, Google unveiled a prototype of AR glasses with real-time translation (Figure 3) and support for multiple languages, including American Sign Language (4). This is, after all, just a concept, and Google has not spoken about this product since. Another company, XRAI Glass, developed glasses capable of real-time conversation transcription, allowing users to engage more naturally in conversations without handheld devices (5). It represents a hope that key information will no longer be missed in various situations due to a lack of real-time subtitling support. However, this expectation also poses a provocative experiment: will it really achieve long-term communication convenience?

While smart caption glasses aim to deliver captions in various environments, technical limitations and practical challenges have led to their rejection by many users. Inclusive design should eliminate barriers and offer a “seamless” experience (6), but smart caption glasses unintentionally introduce new obstacles:

1. Caption accuracy. In noisy or complex settings, the speech recognition accuracy of smart caption glasses is low, leading to delayed or incorrect captions. This hinders understanding, especially in multi-person conversations, making it hard for hearing-impaired users to trust the device fully.

2. Wearing comfort. Despite efforts to improve comfort and portability, the weight and screen strain make extended use uncomfortable. This discomfort limits the glasses’ practicality for daily use.

3. Social awkwardness. In public, Smart caption glasses draw attention to the hearing loss of the hearing-impaired user, especially when conversation requires the user to switch their gaze between the captions and the speaker. This can cause a sense of social isolation, highlight the wearer as different, and potentially alienate them in social situations. A sense of dependency on the device may inadvertently highlight rather than remove barriers, prompting hearing-impaired users to question whether the technology facilitates or hinders communication.

Smart caption glasses exemplify a “provocative object” that exposes gaps in inclusive design by adding unintended emotional and social barriers. Inclusive design should naturally integrate into users’ lives, addressing both functional and emotional needs without creating new issues. Instead, smart caption glasses, with their “rigid” approach to real-time captioning, risk reducing accessibility by overemphasizing the functional at the expense of the emotional.

Zhixin Zhang is a PhD candidate in Design at the Edinburgh College of Art, University of Edinburgh. He is a hearing-impaired designer and researcher, as well as a coordinator for the hearing-impaired community and hearing people, working to create a more equal and fairer inclusive society. He has been focusing on researching participatory design, collaborative design, communication design, disability studies, design justice, inclusive design, to help hearing designers on how to communicate better and design better with hearing-impaired people.

References
(1) M. R. Mirzaei, S. Ghorshi and M. Mortazavi, "Combining Augmented Reality and Speech Technologies to Help Deaf and Hard of Hearing People," 2012 14th Symposium on Virtual and Augmented Reality, Rio de Janeiro, Brazil, 2012, pp. 174-181, DOI: 10.1109/SVR.2012.10
(2) “1900 - 1972: Emerson Romero”. History of Deaf People in Europe.
Accessed Nov 4, 2024 https://deafhistory.eu/index.php/component/zoo/item/emerson-romero
(3) “National Theatre, Enhancing the theatre experience for the hard of hearing”. accenture.
Accessed Nov 4, 2024 https://www.accenture.com/au-en/case-studies/artificial-intelligence/enhancing-theatre-experience-hard-hearing
(4) Sundar Pichai. “Google I/O 2022: Advancing knowledge and computing”. Google. May 11, 2022.
Accessed Nov 4, 2024 https://blog.google/technology/developers/io-2022-keynote/
(5) XRAI Glass. XRAI AR One: All-in-One Wireless Glasses, Made for XRAI Glass. 18 Jun 2024.
Accessed Nov 4, 2024 https://xrai.glass/blog/introducing-the-xrai-ar-one/
(6) “Microsoft Inclusive Design”. Microsoft.
Accessed Nov 4, 2024 https://inclusive.microsoft.design/#InclusiveDesignPrinciples

Started by the DHS Ambassadors in 2022, the Design History Society’s Provocative Objects and Places blog series looks at spaces and objects that challenge and confront us as design historians.

Past topics have ranged from the ancient Colosseum in Rome to the ultramodern Antilia in Mumbai; pink razors and Barbies to Lalique’s Bacchantes vase and nineteenth-century asylum photography. The full collection of previous posts can be found here: https://www.designhistorysociety.org/blog/category/provocative-objects-spaces

We invite submissions for guest blog posts from students, early career researchers, and established academics to those with a general interest in design history. Post can be on any object or place from any era, anywhere in the world, which in some way incites discussion and debate.Post should be 500-800 words in length, accompanied by at least one image with associated credits and clearances, and a short bio.

Please send to the DHS Senior Administrator, Dr Jenna Allsopp designhistorysociety@gmail.com