Map My Mart

What it does

Map My Mart is a mobile app that uses speech-to-text and text-to-speech for grocery list creation, store navigation and object identification. It enhances the user’s independence and shopping experience through increasing the accessibility of grocery shopping.

Your inspiration

When tasked with creating a product to assist a client with Usher’s syndrome, a rare genetic disorder which causes vision and hearing loss, we were informed through client interviews of the difficulties of grocery shopping. The client recalled experiences of mispurchases and frustrations caused by the lack of visual aid which was highlighted when a team member attempted to navigate a grocery store blindfolded. This experience proved the need for a solution that combined navigation and object recognition into one simple and visually accessible product to address barriers in common spaces affecting tasks which are crucial to one’s independence.

How it works

Map My Mart is a mobile application designed to provide comprehensive assistance when grocery shopping for individuals with visual impairments. Focusing on high-contrast visuals, verbal input and audio output this product allows users to create a grocery list, then activates its indoor navigation system when shopping begins. Powered by indoor positioning technology, Wi-Fi signals and Bluetooth sensors for precise real-time positioning users can input their desired item to receive turn-by-turn audio and visual directions for the most efficient and accessible route. Upon reaching the location, the app's camera feature allows for product identification. The user can take a picture, which is processed by OpenAI’s Vision API, and the item’s name and brand is audibly narrated. Items can then be checked off the grocery list using voice commands and when complete the navigation guides the user directly to checkout, streamlining the entire shopping journey.

Design process

The design journey for Map My Mart began in response to our client’s struggles with grocery shopping due to Usher's Syndrome, sparking initial brainstorms on separate navigation and object recognition solutions. A blindfolded simulation of the client’s experience in a grocery store solidified the need to combine both aspects into a single, integrated product. This led to our initial prototypes, focusing on basic indoor positioning and image recognition. Reviews of the design prompted us to refine the AI's output to strictly the name and brand of products, revamp the user interface to neon and white on black color schemes for visual accessibility, and integrate speech-to-text and text-to-speech functions. The navigation system underwent extensive evolution, moving from a rudimentary GPS overlay to a Situm-based solution, requiring numerous manual calibrations, including adding over 1,600 precise junction points and paths. This iterative cycle of design, fueled by user feedback, resulted in an object identification accuracy of 100% with an average processing time of 24 seconds. The comprehensive Map My Mart prototype we have today accumulated average user scores of 9.6/10 for visual accessibility and consistency and 9.9/10 for ease of use, confirming the successful iterations.

How it is different

Our product has been designed to allow individuals with disabilities to enjoy simple pleasures, such as grocery shopping, with less effort and frustration. The multi-store applicability and feature integration makes this product unique. Users typically require multiple applications for navigation and object recognition features which we have combined into one simple product. Our design focuses on a high contrast, easily interpretable user interface, combining both speech-to-text and text-to-speech controls, and optimizing audio guidance for visual displays to ease the user’s navigation. Our design is intended to add to an individual’s quality of life as seemingly simple tasks, like grocery shopping, aiding visual disabilities along with finding accessible routes for physical disabilities and interpreting product labels for individuals who cannot read the language. Our product is designed to instill greater independence and create a more equitable experience.

Future plans

With more testing and resources Map My Mart improvements would include testing edge cases with weaker Wi-Fi signals to evaluate impacts on object recognition and navigation. The object scanner system would undergo further testing to enhance its recognition and the app would be improved by analyzing accessible pathways, ensuring suitable directions for those using assistive devices. The app would also be extended to more locations, partnering with companies to expand this product and technology across different retail companies. While adding different app languages for navigation and item label translation will broaden the user base and value.

Awards

Our team was selected as one of the top six teams, from a total of 212 teams, to present at the McMaster Engineering First Year Showcase. This opportunity allowed us to present our invention to a panel of judges, including faculty and industry professionals, as a life enhancing product for our client with Usher’s Syndrome.