Job Project
ROle
I came up with this idea during my bachelor thesis research and I designed and programmed every AR and MR prototype. I also drafted the research papers and patent manuscripts with others.
Time
September 2020-November 2021
Category
Human-Computer Interaction (HCI), Augmented Reality (AR), Mixed Reality (MR), Financial Technology, User Research, Digital electronics
Team
Azam Anwar, Ms. Neha, Nikhil Bhatt… +5
How do you think we can access the ATM in a socially distanced way, especially during COVID-19?
How could normal Indians survive without safe access to cash?
The invention integrates AR/MR, IoT, and computer vision to minimize physical contact with ATMs, addressing installation costs and pandemic concerns. Utilizing MR technology, it offers virtual inputs, visual identification, and MR-guided assistance via an MR headset. This enhances traditional ATMs' functionality and efficiency, offering a cost-effective XR-based solution for banking. This innovation paves the way for improved user experiences and operational efficiency in the banking sector.
During my bachelor's thesis, I teamed up with Nikhil Bhatt to explore touchless interactions for ATMs using augmented reality (AR). Proficiency in Unity 3D and online resources aided my AR experience design. Encouraged by my professor, I considered patenting the idea. Our work evolved to mixed reality (MR), including electronics prototype development with expert input. I led AR/MR interface development, collaborating with Nikhil on patent drafting. Using my Android phone creatively for AR due to equipment constraints was challenging. Transitioning to MR enhanced user experience by bridging real and virtual worlds via headsets. Despite lacking a functional MR prototype, I embraced limitations as learning opportunities. This academic journey enriched my understanding of AR/MR in ATM interactions amidst practical challenges.
Innovative steps
Novelty in this invention is the use of AR/MR in financial transactions.
The MR dashboard design for keypads was also unique which helps trigger the virtual buttons without using any high-end mobile camera. This was essential to make it affordable in India.
The MR banking unit is also a novel transactional unit which is mentioned in the description and converts upgrade ATMs so that they can function with AR/MR
Challenges
Accessibility: How to make it accessible for a blind person? This is one of the biggest hurdles I am facing with XR technologies. I am still working on solutions that involve gestures, sound, and wearable sensor technology.
Affordability: So the solution had to be economically viable to be able to implement in India. My potential solution was to work on upgrading current ATMs to be able to work with AR/MR and that would be cheaper than installing new ATMs. Another solution was to develop virtual buttons that can work even with a low-end mobile cameras.
Resources: Since AR/MR is pretty much a new topic in Indian Institutes, I had to rely on other sources to solve technical and. practical problems I faced during the development of my project
Patent details:
Patent name: Implementation of Mixed Reality in Banking Sector for Financial Transaction
Application no. : 202211002426
Publication date: 21/01/2022
Previous Versions
References:
Matthew A. Calman, E. S. R. (2013, May 3). Conducting financial transactions based on identification of individuals in an augmented reality environment. Google Patents. Retrieved from https://patents.google.com/patent/US8438110B2 (US Patent 8438110B2)
Kang, Y.-S., & Chang, Y.-J. (2020). Using an augmented reality game to teach three junior high school students with intellectual disabilities to improve atm use. Journal of Applied Research in Intellectual Disabilities, 33 (3), 409–419.
Angelakis, E., Azhar, E. I., Bibi, F., Yasir, M., Al-Ghamdi, A. K., Ashshi, A. M., . . . Raoult, D. (2014). Paper money and coins as potential vectors of transmissible disease. Future microbiology, 9 (2), 249–261.
Aggarwal, R., & Singhal, A. (2019). Augmented reality and its effect on our life. In 2019 9th international conference on cloud computing, data science & engineering (confluence) (pp. 510–515).
Babaei, H. R., Mohurutshe, P. L., & Lashkari, A. H. (2013). Image-processing with augmented reality (ar). In International conference on graphic and image processing (icgip 2012) (Vol. 8768, p. 87684G).
Carmigniani, J., Furht, B., Anisetti, M., Ceravolo, P., Damiani, E., & Ivkovic, M. (2011). Augmented reality technologies, systems and applications. Multimedia tools and applications, 51 (1), 341–377.
Ganesan, P., & Meena, R. (2020, 10). Customer’s perception on effectiveness of digital banking services during covid-19 lockdown period. South African Journal of Economic and Management Sciences, 23 , 1-4.
Penelle, B., & Debeir, O. (2014). Multi-sensor data fusion for hand tracking using kinect and leap motion. In Proceedings of the 2014 virtual reality international conference (pp. 1–7).
Hasan, H. S., & Kareem, S. A. (2012, November). Human computer interaction for vision based hand gesture recognition: a survey. In 2012 International Conference on Advanced Computer Science Applications and Technologies (ACSAT) (pp. 55-60). IEEE.
Han, F., Hu, J., Yu, X., Feng, Y., & Zhou, J. (2006, January). A novel hybrid crypto-biometric authentication scheme for ATM based banking applications. In International Conference on Biometrics (pp. 675-681). Springer, Berlin, Heidelberg.
Faried, I. (2018). Application Model for Information ATM Location Using Location-Based Augmented Reality.
Chi, H. L., Kang, S. C., & Wang, X. (2013). Research trends and opportunities of augmented reality applications in architecture, engineering, and construction. Automation in construction, 33, 116-122.
Lu, Y., & Smith, S. (2009). GPU-based real-time occlusion in an immersive augmented reality environment. Journal of Computing and Information Science in Engineering, 9(2).
Cruz, E., Orts-Escolano, S., Gomez-Donoso, F., Rizo, C., Rangel, J. C., Mora, H., & Cazorla, M. (2019). An augmented reality application for improving shopping experience in large retail stores. Virtual Reality, 23(3), 281-291.
de Souza Cardoso, L. F., Mariano, F. C. M. Q., & Zorzal, E. R. (2020). A survey of industrial augmented reality. Computers & Industrial Engineering, 139, 106159.
Oosterhoff, B., Palmer, C. A., Wilson, J., & Shook, N. (2020, April 6). Adolescents’ Motivations to Engage in Social Distancing during the COVID-19 Pandemic: Associations with Mental and Social Health. https://doi.org/10.31234/osf.io/jd2kq
Kołodziej, M. (2017). Virtual reality as the indicator of the modern banking. Financial Sciences. Nauki o finansach, (2 (31)).