Saying “Alexa, open Mary’s Domain” or “OK Google, talk to Mary’s Domain” (depending on the voice assistant of your choice) will cause your device to begin reading today’s menus at Sharples, Essie Mae’s, and other campus dining locations. But how did this come to be? Who’s Mary, anyway, and why her domain?
2019: What’s in an API?
While looking through a series of help articles for using the computer science (CS) department’s systems, I found an article called “Sharples” containing just one sentence: “to see what’s on the menu, try running our sharples command.” And so I connected to a CS machine, entered “sharples” at its text-based command-line interface, and the menu appeared.
I was born totally blind, so on the computer I navigate exclusively by keyboard rather than with a mouse. As such, text-based interfaces like the one provided by the CS machines are often far faster for me to use than graphical interfaces that require clicking on icons. This was one of those instances: entering “Sharples” to return the menu instantly is more efficient than finding my web browser, navigating to The Dash, and searching the page for the particular menu of interest.
This discovery made me wonder if there might be even faster ways to read the menu. If I could have Alexa, the voice assistant on the Amazon Echo speaker in my dorm, read the Sharples menu as well, I could check, just by voice, what’s on offer and decide if I want to eat at Sharples or at Essie’s. Yes, I admit, this is all a bit of a “first world” problem: I’m fully capable of navigating websites, and The Dash is reasonably accessible. But this “problem” presented a learning opportunity. Driven both by curiosity and the sort of relentless pursuit of efficiency stereotypical of the computer science and software engineering community, my investigation went on.
I started by examining the source code that makes the Sharples command work – it must be accessing the menu by some means, and I might be able to do something similar for Alexa. The source code, available at /usr/swat/bin/sharples on the CS system, revealed a seemingly hidden (or at least not well-publicized) application programming interface (API) for accessing the menus. To understand what an API is, I offer a metaphor. In keeping with the dining theme of this article, imagine that you’re at the counter in Essie’s: you request food using particular words (“could I please have a hamburger?”) and get your food, but you don’t necessarily know, or even care about, all the particular details of how your food is prepared. An API is, in essence, a machine equivalent. Your computer places a special request for today’s menus via the API, and the menus are returned in a standard format that your computer can readily process without worrying about how the menu is displayed visually on The Dash. My task, then, was to connect Alexa to this API, taking particular care to think about how to represent the menu in a spoken format.
Those who know me well are undoubtedly familiar with the various nicknames I’ve given to notable people, places, and things. I affectionately refer to Sharples as “Mary’s Domain” after Mary Kassab. Officially an associate residential manager at Swarthmore, Mary coordinates the preparation of food for special diets and often assists me in getting around the dining hall. As such, I chose the name Mary’s Domain for an initial prototype. After a bit of work on the programming side, I succeeded in getting Alexa to read the menu. As a fun aside, while you can ask “what’s at Essie Mae’s?”, asking the current version of Mary’s Domain “what’s at Mare’s?” will also read the Essie’s menu: “Mare” was an Essie’s manager.
2020: Growing to Google, publication postponed by pandemic
The first versions of the Mary’s Domain Alexa integration, or “skill” as Amazon would refer to it, met my requirements. Before heading to eat, I could ask Alexa for the menu to determine conveniently where I’d like to go. I quietly used the skill for a few months and showed it to a few friends. My skill was received positively, with use in cases beyond my previous consideration: while Mary’s Domain solves a “first world” problem for me, dyslexic students and those with concussions that make them temporarily unable to read screens found that usage of a voice assistant would greatly improve their access. I frequently received requests to write a Google Assistant version for Android phones and Google Home speakers. I initially targeted Alexa for convenience (it’s what I had, and it seemed far easier to connect it to campus Wi-Fi), but I was willing to expand to Google if the skill would receive usage. I posted a short video demonstration around this time to the Swarthmore College 2019–2020 Facebook group to quite positive reception and decided to begin developing the skill for Google Home.
Around February 2020, I found a good-priced Google Home Hub (a Google Assistant speaker with a small touchscreen) on eBay. I purchased it as a test device for a future Google Assistant version of Mary’s Domain. Developing for Google Assistant posed a few additional technical challenges. Chief among these difficulties is that the Google Dialogflow developer console, when compared with the Amazon equivalent, is not as accessible to my screen reader (a program that reads out the contents of the computer screen in a synthesized voice). I worked through the challenges during small blocks of free time and, just before spring break, an initial Google Assistant version of Mary’s Domain was up and running. At first, I planned to release both versions later that semester. However, the coronavirus pandemic resulted in what I like to call “the semester in which the world went mad”: the sudden closure of campus, displacement of various members of the Swarthmore community, and a shift to remote instruction. I thought it best to delay the release of the skill until our return to campus to permit time for further testing and release at a time when Mary’s Domain would receive more usage.
2021: Mary’s goes live
Given the return to campus of many third and final year students (and opening of dining facilities) in Spring 2021, I decided to clean up the Mary’s Domain skill and prepare for general release. As part of this effort, I attempted to change the skill’s name to “Swarthmore Dining”, a name that would more clearly identify the skill’s purpose and be easier to remember for those outside my immediate “inner circle”. During testing, however, it became clear that the voice assistants couldn’t reliably understand the word “Swarthmore”! As such, I ultimately decided to go ahead with the release under the skill’s original name.
Supporting both Amazon and Google meant that, at least at first, there was a lot of duplicated code: I was doing a lot of the same things (converting menus into speech), but in slightly different ways depending on platform. To ease maintenance going forward, I created “Marys”, a software library in the Python programming language to access the Swarthmore menus and convert them into text and Speech Synthesis Markup Language, a common speech format that both Alexa and Google Assistant could understand. Then, going forward, I could write only a little bit of platform-specific code for either Amazon or Google, and whenever the format changes slightly (which it does somewhat regularly), I’d only have to update one place and both assistants could easily pick up the changes. At the start of 2021, I released Mary’s on GitHub as open-source software, so that other programmers, not just me, could display the Sharples menus in their own apps. After a few more changes, testing, and filling in several forms, Mary’s Domain went live on both the Amazon Alexa and Google Assistant app directories in the spring semester. After release, I demonstrated to Mary Kassab her namesake skill — she was quite impressed!
2022: The new Dash and beyond
At the start of the Spring 2022 semester, a redesigned Dash was released. Given my work on voice assistants and usage of assistive technology, the dining staff asked me to participate in a closed beta of the website a few months before its release. I immediately attempted to access the API to verify the continued functioning of the voice assistants when the new site went live but realized that it had been removed. After I asked for the API to be restored, a fairly similar (though slightly different) one was added to the new website. Now that the new website is officially live, I’ve made a few updates to ensure strong compatibility with the latest format of the menus. As the new design of The Dash is somewhat controversial, I suspect that the Mary’s Domain skill will see increases in usage as community members seek an alternative means of quickly and conveniently finding out what’s available to eat at Swarthmore.
As this is my final semester, I’ll be handing off maintenance of Mary’s Domain. The Swarthmore College Computer Society (SCCS), among others, have expressed interest in continuing development and maintenance of the voice assistant skills and library beyond my graduation.
Oh boy. You’re good kid! You’ve been like this since you were 6 months old. Congrats!