Abstract: 3D reconstruction from anatomical slices permits anatomists to create three-dimensional depictions of real structures by tracing organs from sequences of cryosections. A wide variety of tools for 3D reconstruction from anatomical slices are becoming available for use in training and study. In this chapter, we present Anatomy Studio, a collaborative Mixed Reality tool for virtual dissection that combines tablets with styli and see-through head-mounted displays to assist anatomists by easing manual tracing and exploring cryosection images. By using mid-air interactions and interactive surfaces, anatomists can easily access any cryosection and edit contours, while following other user’s contributions. A user study including experienced anatomists and medical professionals, conducted in real working sessions, demonstrates that Anatomy Studio is appropriate and useful for 3D reconstruction. Results indicate that Anatomy Studio encourages closely coupled collaborations and group discussion, to achieve deeper insights.

ZORZAL, E. R.; SOUSA, M.; MENDES, D.; FIGUEIREDO, P. S.; RODRIGUES, P.; JORGE, J.; LOPES, D. S. A Tool for Collaborative Anatomical Dissection. Human-Computer Interaction Series. 1ed.: Springer International Publishing, 2021, v., p. 41-58. (http://dx.doi.org/10.1007/978-3-030-61905-3_3)

Abstract: Tools for training and education of dental students can improve their ability to perform technical procedures such as dental implant placement. Shortage of training can negatively affect dental implantologists’ performance during intraoperative procedures, resulting in lack of surgical precision and, consequently, inadequate implant placement, which may lead to unsuccessful implant supported restorations or other complications. We designed and developed IMMPLANT a virtual reality educational tool to assist implant placement learning, which allows users to freely manipulate 3D dental models (e.g., a simulated patient’s mandible and implant) with their dominant hand while operating a touchscreen device to assist 3D manipulation. The proposed virtual reality tool combines an immersive head-mounted display, a small hand tracking device and a smartphone that are all connected to a laptop. The operator’s dominant hand is tracked to quickly and coarsely manipulate either the 3D dental model or the virtual implant, while the non-dominant hand holds a smartphone converted into a controller to assist button activation and a greater input precision for 3D implant positioning and inclination. We evaluated IMMPLANT’s usability and acceptance during training sessions with 16 dental professionals. The conducted user acceptance study revealed that IMMPLANT constitutes a versatile, portable, and complementary tool to assist implant placement learning, as it promotes immersive visualization and spatial manipulation of 3D dental anatomy. IMMPLANT is a promising virtual reality tool to assist student learning and 3D dental visualization for implant placement education. IMMPLANT may also be easily incorporated into training programs for dental students.

ZORZAL, E. R.; PAULO, S. F; RODRIGUES, P.; MENDES, J. J.; LOPES, D. S. An immersive educational tool for dental implant placement: A study on user acceptance. INTERNATIONAL JOURNAL OF MEDICAL INFORMATICS, v. 146, p. 104342, 2021. (http://dx.doi.org/10.1016/j.ijmedinf.2020.104342)

Abstract: Studies indicate that Augmented Reality (AR) is quickly becoming an effective technology for training programs. This systematic literature review (SLR) aims to screen works published on AR for corporate training. We describe AR training applications, discuss current challenges, literature gaps, opportunities, and tendencies of corporate AR solutions. We structured a protocol to define keywords, semantics of research, and databases used as sources of this SLR. From a primary analysis, we considered 1952 articles in the review for qualitative synthesis. We selected 60 among the selected articles for this study. The survey shows a large number of 41.7% of applications focused on automotive and medical training. Additionally, 20% of selected publications use a camera-display with a tablet device, while 40% refer to head-mounted-displays, and many surveyed approaches (45%) adopt marker-based tracking. Results indicate that publications on AR for corporate training increased significantly in recent years. AR has been used in many areas, exhibiting high quality and provides viable approaches to On-The-Job training. Finally, we discuss future research issues related to increase relevance regarding AR for corporate training.

MARTINS, B. R.; JORGE, J.; Zorzal, E. R. Towards augmented Reality for corporate training. INTERACTIVE LEARNING ENVIRONMENTS, v. 29, p. 1-19, 2021. (http://dx.doi.org/10.1080/10494820.2021.1879872)

Abstract: The incidence of diabetes mellitus (DM) is a significant and growing health problem for all countries, regardless of their development level. Providing mobile computing solutions that assist the patients in their daily treatment routine can be an excellent ally. This systematic review aims to assess how mobile apps for DM treatment follow-up can contribute to improving patients’ quality of life. We searched the bases ACM Digital Library, PubMed, Springer Link, IEEE Explore, and Science Direct databases. We obtained 1608 primary studies; 1485 were used in the screening phase and 791 in the full screening phase, resulting in 106 articles for data extraction from 2013 to 2018. We have identified the main functionalities and characteristics necessary for an application for managing DM, such as automation and ease of data collection. We found that apps for type 1 DM help treat patients, facilitate the collection and access to data, and allow better daily monitoring of treatment.

DE ARAÚJO, W. R. V. C.; MARTINS, L. E. G.; ZORZAL, E. R. Mobile apps for the treatment of diabetes patients: a systematic review. RESEARCH ON BIOMEDICAL ENGINEERING, v. 1, p. 1, 2021. (http://dx.doi.org/10.1007/s42600-021-00150-7)

Abstract: This article aims to evaluate the impact of Augmented Reality (AR) applicability and usefulness on real industrial processes by employing a systematic literature review (SLR). The SLR was performed in five digital libraries to identify articles and reviews concerning the AR applicability from 2012 to 2018. A patent search in Google’s patents database was also conducted, for the same period. This paper describes how AR has been applied, which industries are most interested in the technology, how the technology has been developed to meet industry needs, as well as the benefits and challenges of AR. This survey concludes by providing a starting point for companies interested in integrating AR into their processes and proposing future directions for AR developers and researchers.

DE SOUZA CARDOSO, L. F.; MARIANO, F. C. M. Q.; ZORZAL, E. R. A survey of industrial augmented reality. COMPUTERS & INDUSTRIAL ENGINEERING, v. 139, p. 106159, 2020. (http://dx.doi.org/10.1016/j.cie.2019.106159)

Abstract: One of the most promising applications of Optical See-Through Augmented Reality is minimally laparoscopic surgery, which currently suffers from problems such as surgeon discomfort and fatigue caused by looking at a display positioned outside the surgeon’s visual field, made worse by the length of the procedure. This fatigue is especially felt on the surgeon’s neck, as it is strained from adopting unnatural postures in order to visualise the laparoscopic video feed. Throughout this paper, we will present work in Augmented Reality, as well as developments in surgery and Augmented Reality applied to both surgery in general and laparoscopy in particular to address these issues. We applied user and task analysis methods to learn about practices performed in the operating room by observing surgeons in their working environment in order to understand, in detail, how they performed their tasks and achieved their intended goals. Drawing on observations and analysis of video recordings of laparoscopic surgeries, we identified relevant constraints and design requirements. Besides proposals to approach the ergonomic issues, we present a design and implementation of a multimodal interface to enhance the laparoscopic procedure. Our method makes it more comfortable for surgeons by allowing them to keep the laparoscopic video in their viewing area regardless of neck posture. Also, our interface makes it possible to access patient imaging data without interrupting the operation. It also makes it possible to communicate with team members through a pointing reticle. We evaluated how surgeons perceived the implemented prototype, in terms of usefulness and usability, via a think-aloud protocol to conduct qualitative evaluation sessions which we describe in detail in this paper. In addition to checking the advantages of the prototype as compared to traditional laparoscopic settings, we also conducted a System Usability Scale questionnaire for measuring its usability, and a NASA Task Load Index questionnaire to rate perceived workload and to assess the prototype effectiveness. Our results show that surgeons consider that our prototype can improve surgeon-to-surgeon communication using head pose as a means of pointing. Also, surgeons believe that our approach can afford a more comfortable posture throughout the surgery and enhance hand-eye coordination, as physicians no longer need to twist their necks to look at screens placed outside the field of operation.

ZORZAL, E. R.; CAMPOS GOMES, J. M.; SOUSA, M.; BELCHIOR, P.; DA SILVA, P. G.; FIGUEIREDO, N.; LOPES, D. S.; JORGE, J. Laparoscopy with Augmented Reality Adaptations. JOURNAL OF BIOMEDICAL INFORMATICS, v. 107C, p. 103463, 2020. (http://dx.doi.org/10.1016/j.jbi.2020.103463)

Abstract: One major manufacturing challenge of the fourth industrial revolution is the quality of product development information used in the assembly line. One of the most studied information visualization technologies is Augmented Reality (AR), whose deployment via mobile devices is currently being promoted. However, the industrial sector still has doubts regarding the technology’s maturity and the costs incurred by such production processes, restricting its deployment in production environments. Thus, in the present study, we statistically analyze a markerless AR application, using a hand-held mobile device, that addresses the requirements of the structural assemblies used in aeronautical industries comparing them to those of traditional assembly methods. The main technological benefits and limitations of such applications are identified. To perform this analysis, we conducted a field evaluation where qualified professionals marked the positions of small brackets in an aeronautical structure by using AR. After collecting the time and measurement data for each bracket’s position, a statistical analysis was performed to compare these positionings against those achieved using a traditional, metric scale-based method. The results indicate that AR offers a faster solution for highly complex assemblies, despite several limitations regarding the positioning tolerance when using the technology. As a future study, several application modifications are suggested to reduce markup errors and overall usage time.

DE SOUZA CARDOSO, L. F.; MARIANO, F. C. M. Q.; ZORZAL, E. R. Mobile augmented Reality to support fuselage assembly. COMPUTERS & INDUSTRIAL ENGINEERING, v. 148, p. 106712, 2020. (http://dx.doi.org/10.1016/j.cie.2020.106712)

Anstract: 3D reconstruction from anatomical slices allows anatomists to create three dimensional depictions of real structures by tracing organs from sequences of cryosections. However, conventional user interfaces rely on single-user experiences and mouse-based input to create content for education or training purposes. In this work, we present Anatomy Studio, a collaborative Mixed Reality tool for virtual dissection that combines tablets with styli and see-through head-mounted displays to assist anatomists by easing manual tracing and exploring cryosection images. We contribute novel interaction techniques intended to promote spatial understanding and expedite manual segmentation. By using mid-air interactions and interactive surfaces, anatomists can easily access any cryosection and edit contours, while following other user’s contributions. A user study including experienced anatomists and medical professionals, conducted in real working sessions, demonstrates that Anatomy Studio is appropriate and useful for 3D reconstruction. Results indicate that Anatomy Studio encourages closely-coupled collaborations and group discussion, to achieve deeper insights.

ZORZAL, E. R.; SOUSA, M.; MENDES, D.; DOS ANJOS, R. K.; MEDEIROS, D.; PAULO, S. F.; RODRIGUES, P.; MENDES, J. J.; DELMAS, V.; UHL, J-F; MOGORRÓN, J.; JORGE, J.; LOPES, D. S. Anatomy Studio: A tool for virtual dissection through augmented 3D reconstruction. COMPUTERS & GRAPHICS-UK, v. 85, p. 74-84, 2019. (http://dx.doi.org/10.1016/j.cag.2019.09.006)