Accessibility and AI: Help or Hindrance?
Artificial Intelligence in Everything!
I am sure like me, you can’t go more than 10 minutes hearing or seeing something about AI. It is everywhere and the hype is intense. Is this technology the modern day industrial revolution, or is it just noise? Can it in fact help out with things that we struggle with, or will it be more annoyance than insightful?
The answer I’m going to give is a qualified “yes!”
Hang on a Moment!
Okay, so I said “yes” to what should have been an either or selection, but hear me out.
I currently use two AI powered tools in my every day accessibility journey, and they are interesting, entertaining, helpful, and wrong. The “wrong” is the part that allows me to get away with my snarky “Yes!”
AI in Assistive Tech
The first tool that I use a great deal is a feature in the JAWS screen reader, and this one can truly make a difference in getting things done that were previously beyond my reach.
The screen reader has a built-in feature called “Picture Smart” that can take a screenshot or picture, send it off to AI engines, and ask that engine to describe the image. It allows follow-up queries to be asked about the result and it can then clarify and provide further information. It utilizes multiple models though it defaults to ChatGPT.
I have successfully used this technology to accomplish tasks which are rather challenging for the screen reader. As an example, I’m a bit of a nerd and wanted to install an old version of Debian Linux to get access to a very ancient compiler for C++. The complexity to that is the version of Linux simply can’t run on modern hardware as it doesn’t know about things like SATA. I did mention old, right? Well, the only way I could think of achieving this goal was to set up a virtual machine on a modern Linux host using QEMU and then accessing the booted OS via a VNC client, specifically TightVNC. I hope this isn’t too nerdy, but I’m about to get to the important bit so hang in there.
TightVNC is not accessible. I don’t know of any VNC client that is. Normally at this stage I’m bugging my wife and asking her to read the screen over my shoulder, trying to explain Linux stuff to her based on my memory of what she should be seeing, and trying to unpack what she sees.
Thanks to “Picture Smart”, this is now a thing of the past.
I was able to send off the screenshots through a JAWS command, get feedback on what the AI interpreted from the image, and was able to clarify what was really on the screen and what was highlighted in the very primitive graphical installer of this 23-year-old operating system. It was sometimes incorrect but, with patience, I got through it with my AI friend.
That was game changing for me. From asking if my camera is turned on with my laptop to asking it to explain UI, it can do a pretty decent job.
Apps with AI Sauce
My second bit of AI-powered tech comes via the iPhone and apps like Seeing AI from Microsoft and, of course, the ChatGPT mobile app.
Microsoft’s Seeing AI
Now, Microsoft’s Seeing AI app isn’t great but it does do some things which can certainly be helpful. Much of it isn’t AI at all I suspect but it has some AI flavor added. The most common use case for this tool is the ability to decode a barcode to figure out what a product is or have it OCR text on the fly to try to read labels of things.
The AI part comes in on picture analysis and the best I can say about that is it can be kind of useful. The information tends to be pretty bland which is why this leans more towards machine learning than true AI.
I think about how much technologies impact my ability to do things, and product identification and label reading was a non-starter for me, so good job Microsoft on this one.
ChatGPT Mobile App
The utility of ChatGPT on the phone is seldom used. The ability to engage voice mode and have a conversation with the AI is mildly entertaining, but the true utility isn’t there for me on the phone.
I use ChatGPT via the desktop mode far more frequently, and from an accessibility perspective I can’t say it brings much more to the table than being a somewhat non-authoritative search engine that can get things just wrong enough to truly drive me nuts. As this matures it may end up being a great leveler, where people can do tasks that they were either not educated for, or simply don’t have the capability to do. By the time the AI can do that though, the question comes to whether we need the humans at all!
Meta Ray-Ban Sunglasses
These sunglasses have somewhat inspired some true excitement in the accessibility world. By using your phone’s cellular connection, these AI-integrated sunglasses can give the user an immediate option to send a picture to Meta’s AI and get information back about that image.
I was truly excited to get my hands on these, and I think maybe my excitement climbed to an unrealistic level. I had such a good experience working with the AI through JAWS Picture Smart that I thought the experience would be the same but more “on the go”.
This expectation has been half met. Let me give you some examples:
- Me: Hey Meta! Look and tell me if you see any potential hazards in my path.
- Meta: There are some potential hazards, I see a road, and in the distance there is a lake. These could both be potential hazards.
That is great, you might think. Yep, the AI described two potential hazards. What it didn’t mention is the flight of stairs that is just ahead of me on the walkway. It also described a pond as a lake, but I can get past that. I can’t get past the fact that the obvious stairs were missed…
- Me: Did you not see the stairs?
- Meta: I’m sorry, now that you mention it, yes there are stairs and these could also present a potential hazard.
Here are a couple of other experiences…
- Me: Hey Meta, are there any propane tanks visible on my back patio?
- Meta: Yes, there is a propane tank beside the grill.
- Me: Really?
- Meta: I’m sorry I am mistaken, there is no propane tank beside the grill, there is one next to the window.
- Me: Really?
- Meta: Sorry, I am mistaken, there is a propane tank beside the grill.
And another favorite!
- Me: Hey Meta, is the garage door open?
- Meta: Yes, the garage door is open.
- Me: Really?
- Meta: I’m sorry, I was mistaken, the garage door is closed.
I share this detail for my extended family of assistive tech users. You cannot trust this tech right now. Trust your orientation and mobility skills, and at the most use this as an augment.
I am sure people will be equally upset when they hope to be able to get some semblance of what our fellow human beings might look like. The AI will not describe people without a good deal of persuasion. In fact, it wouldn’t describe my test subjects at all until I explained to my glasses that I am blind.
Wow, I never thought I’d be writing about talking to my sunglasses. This world is going to get more strange!
Final Thoughts
There is promise here and potential breakthroughs that could make a true difference in the lives of many assistive tech users. As is often heard though, beware of hallucinations from the AI. If you trust what you are being told, you might be lucky to escape with bumps and bruises.
I’ve signed up for Meta Live AI which will be my next interesting AI-powered experiment! Stay tuned.
Well, unless I trust the AI too much and end up in the pond or something!
